I Rejected a Model at 44 tok/s. The Model Was Doing 108.

Emil, the LLM-driven NPC on my Minecraft server, has run on Nemotron 3 Nano 30B-A3B since the start. It runs on a DGX Spark on my desk — a single box, 128 GB of memory shared between CPU and GPU, and a memory bandwidth ceiling that makes token throughput the number that governs everything else. Emil’s turn latency is how long it stands still while a skeleton shoots at it.

So when I evaluated Qwen3.6-35B-A3B as a replacement and measured 44.1 tok/s against Nemotron’s 63, I wrote it down as too slow and moved on.

That number was correct. It was also a measurement of my own configuration, and it took three separate mistakes to produce it.

The three mistakes

Re-tested properly — temperature 0, 256 tokens, ignore_eos, thinking off, a UUID-prefixed prompt so nothing gets served from cache, token counts read from the API’s own usage field, three warmup requests discarded — the same box gives this:

serve config	tok/s
Qwen3.6-35B-A3B-NVFP4-Fast, MTP=3	110.9
same, under Emil’s own serve script	107.9
same checkpoint, MTP off	71.6
Qwen3.6-35B-A3B-NVFP4 (the artifact I first tested)	44.1
Nemotron 3 Nano 30B-A3B NVFP4	63.0

The wrong checkpoint. I had grabbed a uniform-NVFP4 build. The one that matters is unsloth’s -Fast, which is mixed precision — NVFP4 experts, FP8 attention and lm_head, BF16 embeddings — and ships its multi-token-prediction head inline in the shards. That single substitution is worth 62% before any speculative decoding is turned on.

MTP not actually working. With the right checkpoint the draft head accepts 80–84% of its proposals at 3.4–3.5 tokens per step, against 45% and 2.35 before. Most of that is greedy decoding: drafts get accepted far more often when the target is sampling at temperature 0, which the benchmark is and normal serving isn’t.

Pinning the MoE backend to marlin. Prior lore on this box said marlin was the fast choice, so it was pinned. Left on auto, vLLM picks FLASHINFER_CUTLASS here, which is faster — and more importantly, marlin cannot serve the draft head at all.

Nine reloads of 23 GiB

That last one didn’t fail quietly. The MTP head ships unquantized, because the quantization pass copies those tensors through untouched and only the main model becomes NVFP4. The backend flag is global, so pinning marlin applied it to the draft model too:

ValueError: moe_backend='marlin' is not supported for unquantized MoE.
            Expected one of ['triton','flashinfer_trtllm','flashinfer_cutlass','aiter']

The engine core dies during draft init. The container is started with --restart unless-stopped, so Docker did exactly what I told it to: reloaded 23 GiB of weights and died again. It had gone around nine times before I noticed and cut the loop. A restart policy is for transient failures, and a configuration error is not transient — it will fail identically forever, as politely as you like.

Two more dead ends went into the script’s comments so I don’t repeat them: the pinned CUDA 13 nightly I had been running can’t load a mixed-precision FP8 lm_head at all, and the vLLM version the published recipe pins is built for arm64 only against CUDA 12.9, which isn’t Blackwell-capable and dies in a cutlass call on first compute. The recipe’s exact pin is unreachable on this hardware. Exactly one build — a fresher nightly — both loads the checkpoint and has kernels for this GPU.

Like for like

While re-measuring I ran Nemotron through the identical harness. Its own model card claims about 80 tok/s. Under this methodology it does 63.0, and 63.0 is the number the comparison has to use. I left the claim alone in the script that carries it rather than editing a file the change didn’t otherwise touch — the point isn’t that the card is wrong, it’s that a throughput figure without a methodology attached isn’t comparable to anything.

That’s the shape of the original error too. 44.1 tok/s was a real measurement that honestly answered a question I hadn’t meant to ask.

The constant that changes size

Then the swap broke something that had nothing to do with speed.

One evening the live run started returning 400 on every single turn, for six minutes, in a retry loop that never got smaller:

This model's maximum context length is 32768 tokens. However, you requested
6144 output tokens and your prompt contains at least 26625 input tokens

The interesting part is that the harness has a context-window backstop, a ladder of increasingly aggressive strategies for making history fit, and it believed it had already trimmed to fit. It was working correctly on a number that had quietly changed meaning.

The budget is denominated in estimated tokens, and the estimator is a fixed four-characters-per-token heuristic that never sees a tokenizer. That estimate is not a property of my harness. It’s a property of the pair (harness, base model), and I swapped one half of the pair.

Measured on Emil’s own live content against Qwen3.6:

content	chars per token
tool results	2.64
tool call arguments	2.60
reasoning	3.41
tool specs	3.75
what the estimator assumes	4.00

The estimate is worst exactly where the window is heaviest. The same “18000” that meant about 21k real tokens on Nemotron meant about 26.6k on Qwen3.6, and 26,625 plus 6,144 of output goes straight past a 32,768 ceiling.

I fixed it on the serving side rather than by re-deriving the constant. A per-tokenizer constant is a thing I’d have to get right again for every base model I ever try, and getting it wrong is silent until every turn dies at once. Raising the served window to 65,536 costs almost nothing here, because the KV cache is pinned by bytes rather than by token count: the 8 GiB pin buys around 505,000 tokens, so the old 32,768 ceiling was using 8.7% of it.

There was a second symptom hiding behind the first. Under a too-small estimated window the trim ladder was firing constantly — around 78 messages evicted every 90 seconds to three minutes during active play, which left Emil roughly a two-minute memory horizon. It kept forgetting what it had been doing, and I had been reading that as a model weakness. The budget is 36,000 now, against a 65,536 serve, with about 6k of slack.

The draft head has to survive the pipeline

MTP is 1.55× of the measured serving speed, which makes it load-bearing rather than a nice-to-have — and Emil’s model gets fine-tuned, so every checkpoint goes through LoRA merge and then quantization before it’s served. The Nemotron-era pipeline would have dropped the draft head in both places.

The quantizer now leaves the mtp.* tensors in BF16, because vLLM’s draft path doesn’t run quantized weights. And the merge step verifies tensor parity against the base checkpoint’s index after saving, because a model class that fails to materialize a module loses its tensors behind a transformers warning — not an error. That is now a hard abort. The failure it prevents is a checkpoint that merges fine, quantizes fine, serves fine, and is 35% slower than it should be with nothing anywhere saying why.

There’s also a rule on this box that speculative decoding must stay off, written after n-gram speculation corrupted tool calls. A trained draft head is a different mechanism, so that rule doesn’t automatically transfer — but it doesn’t automatically clear either, so MTP shipped off by default until it had been checked against the thing that actually matters here: tool-call validity under constrained decoding, 10 out of 10.

What I still haven’t measured

None of this says Qwen3.6 plays Minecraft better than Nemotron. It says it generates tokens 1.7× faster like for like, and that the earlier number saying otherwise was mine, not the model’s.

I had a capability A/B planned to answer the real question, and I cancelled it. Nemotron is retired — not on a benchmark, but on months of watching it play, where it was buggy in specific ways and simply less capable than the alternative. Running a fair head-to-head to produce a number I’d already decided not to act on would have been theatre, and the honest version is to say the decision was made on judgement and leave the bench for measuring what the new base can actually do.

Every wrong number in this project has failed the same way. It wasn’t fabricated and it wasn’t noise. It was an accurate measurement of a configuration, mistaken for a measurement of the thing.