Meet Emil: an LLM That Lives on My Minecraft Server

Emil started as an admission of defeat. I had spent a long time building my own coding agent — the harness, the tools, and fine-tuned models to drive it — and it works. But I could never get myself to use it every day, because Claude Code is simply miles ahead, and reaching for the worse tool out of principle gets old fast. Competing with frontier labs on their home turf is a boring way to lose.

The part I actually cared about was never the coding assistant, though — it was the agent loop itself: what the model observes, what it’s allowed to do, what it remembers, how it fails after hours of running. Those questions don’t need a coding task. They need an environment that pushes back — real time pressure, real consequences, other players — and one where no frontier product is waiting to make my version pointless. I had one lying around: a Minecraft server.

The result is Emil, an NPC that lives there. Every action comes out of a language model, running on the same Rust agent loop as my coding harness. No scripted policies, no behavior trees. If Emil chops a tree, it’s because the model decided to chop a tree.

If you haven’t worked with LLM agents: an agent is a model in a loop. Each turn, the loop hands the model an observation of the world, the model responds with tool calls — small structured commands like mine_block or craft — the loop executes them and feeds the results back. Everything in this project is engineering on that loop.

That loop is easier to show than to tell. Below is the first bench run Emil ever passed — one unedited take of the easiest scenario on his capability ladder: spawn into a fresh world, find a tree, collect three logs. The current checkpoint still fails it more often than not; this time he did it in 66 seconds. Beside the video runs the actual traffic between my harness and the model at that moment — what the harness sends, what the model thinks and calls, what comes back. Mistakes included.

The long-term goal is an NPC that genuinely lives on the server — its own goals, its own memories, conversations with whoever logs in. The measurable milestone on the way is beating the game: reaching the Ender Dragon forces every survival skill I care about. Emil is not there yet. As I write this he’s still fumbling around the early game — and why that’s hard is exactly what makes this interesting to write about.

The rules

Three constraints I set early and keep enforcing, because each one closes an escape hatch that would make the project meaningless:

• No scripts. The Rust side provides body primitives, never decisions. The moment I hard-code “if night, go home,” I’ve stopped doing research and started writing a very slow behavior tree.
• No wallhacks. Every percept and every tool target passes a line-of-sight check. Emil can’t see diamonds through stone or fight mobs through walls. If a human player couldn’t perceive it, neither can he.
• The base model never gets swapped for a bigger one. Emil has to get better through the harness — perception, memory, tools, training data — not through someone else’s next release.

The stack

• Body: a Rust bot handling pathfinding, block placement, inventory. Pathfinding is body, not brain — the model says go to the cave entrance, the A* search is not its job.
• Brain: a 30B mixture-of-experts model (Nemotron 3 Nano) served locally. Each turn it gets a structured observation — health, inventory, nearby blocks, recent chat — and responds with tool calls: mine_block, craft, goto, chat.
• Memory: a RAG corpus of game knowledge, a self-authored diary, and a cognitive map — a landmark graph with explored-area tracking. A sparse map of places that matter beats photographic recall of coordinates; my dead bot at the bottom of a spawn pit taught me that before the papers did.
• Hardware: one NVIDIA DGX Spark on my desk. 128 GB of unified CPU/GPU memory that training, serving, and two Minecraft servers all fight over.

I don’t run this alone: a coding agent babysits the training runs, watches the logs, and does the grunt work. The design calls and the debugging questions are mine. That division of labor is half of what this project is teaching me.

The flywheel

The part I care most about is the loop that improves Emil:

1. Live episodes. Emil plays continuously; every observation, tool call, result, and chat gets logged.
2. Curation. Episodes are audited before training. One hard rule: world facts never go into weights. A model that memorizes my base’s coordinates isn’t smart, it’s contaminated. Facts belong in memory, behavior in weights.
3. Fine-tuning. LoRA SFT on the curated set, on the same desk machine.
4. The eval gate. A candidate never goes live on vibes. It gets a fixed-seed, fixed-duration run on a disposable server, scored on advancements, survival, and tool-error profile.
5. Serve or reject. Winners get merged, quantized, served. Losers get an investigation — which usually teaches me more than a win would.

That last point is not resume-speak. The gate just rejected my newest checkpoint, and root-causing why uncovered a train/serve mismatch that no loss curve would have shown me.

Why Minecraft

Because it’s an honest world. Real physics, real failure (Emil dies), real time pressure (nights are dangerous), real players saying things the model has to answer now. Every hard problem in agent engineering shows up here in miniature — and I get to watch it happen.