Optimistic Tool Calling: Acting Before the Model Stops Talking

My Minecraft NPC Emil thinks at local speed. The model behind him is a 30B mixture-of-experts served off the machine on my desk, and in a world with hostile mobs, every second between observing and acting is a second something can go wrong. A turn that produces a paragraph of reasoning and three tool calls used to mean the body stood perfectly still until the last token landed.

That felt like latency I was leaving on the table. I was pretty sure I wasn’t the first person to want an LLM to act while it’s still talking, so I had my coding agent sweep the literature before we built anything.

What the papers offer

The sweep came back with a family of ideas, each attacking the think-vs-act gap from a different side:

• AsyncLM (arXiv 2412.07017) makes function calls non-blocking: the model keeps generating while calls run, and the executor injects interrupt tokens into the stream when results land. It needed a small call notation and fine-tuning on the protocol — prompting alone wasn’t enough — and bought 1.6–5.4× lower end-to-end latency.
• Speculative Interaction Agents (arXiv 2605.13360) let the model itself speculate: emit a call early, overwrite or cancel it by ID if later input invalidates it. The safety trick is a CPU-style write buffer — read-only calls fire instantly, state-mutating calls wait at a commit point. Wrong speculation costs a wasted read, never a wrong write. Speedups of 1.3–2.2×, at a few points of accuracy.
• PASTE, “Act While Thinking” (arXiv 2603.18897) speculates entirely in the runtime: mine past execution traces for patterns, pre-execute the predicted next call during idle time, serve the result instantly on a match. The model is untouched.
• The Hierarchical Language Agent (AAMAS 2024) splits cognition into a Slow Mind, a Fast Mind, and a reactive executor that keeps acting while the LLMs deliberate — which validated a split Emil already had: pathfinding is body, not brain.
• The Rapid-Reflex Async-Reflect Agent (arXiv 2506.07223) runs reflexes immediately and lets async reflection refine subsequent actions when it lands, and contributes the metric I now want on every eval: the fraction of time an agent spends reasoning versus acting.

The one that looked shiniest to me wasn’t the most sophisticated. Emil’s situation inverts the papers’ setting: his input arrives as a clean snapshot per turn, but his output dribbles out at local-serving speed. The cheap win is stream dispatch — the harness already streams tokens from vLLM, so fire each tool call the moment its closing tag appears in the stream, instead of waiting for the generation to finish. No speculation on the semantics: the model really did decide that action. It just hasn’t stopped talking yet.

The design: queued, then armed

The version we built goes one step further than stream dispatch, because reasoning models leak their intentions early. Emil’s model reasons before it acts, and the reasoning mentions tools long before the action block commits to them. So:

• A recognizer watches the streaming reasoning for tool-shaped intentions — a tool name plus plausible arguments, in prose or half-syntax.
• A recognized call goes into the action queue flagged as reasoning-sourced: validated, name-resolved, prepared — but not executed. Reasoning is noisy; models routinely consider an action and reject it (“I could mine the birch here… but I should finish the shelter first”).
• When the same call shows up in the real action output, the entry is promoted in place — the expensive prep already happened during decoding. If the turn ends without it, the entry is dropped and logged as a mispredict. That’s the papers’ commit-point rule, applied to a Minecraft body: a wrong speculation may never place a block.

Alongside it sits a static repair layer, because a fine-tuned 30B will emit BirchLogs when it means birch_log. Repair is alias-table first, fuzzy-match second — and fuzzy matching on Minecraft identifiers is a trap, because near-miss names are usually different real things: birch_log, birch_wood, and birch_planks all exist. Below a confidence threshold the harness refuses to guess and returns a structured error naming the closest candidates. One wrong guess on a mutating action costs more than one round-trip.

The part I care most about is the telemetry, because it feeds the flywheel. Every repair logs the raw emission, the fix, and the confidence; every speculation logs whether it matched and how much head-start it would have bought. And the dataset builder rewrites repaired calls to their canonical form in the training targets — if the log recorded BirchLogs followed by success, the fine-tune would learn that BirchLogs works, and the crutch would become load-bearing. Written back canonically, the crutch trains itself away. The repair hit rate going down across model generations is the metric that says the flywheel is working.

What the A/B actually said

We ran it honestly: same model, same seed, stream dispatch on versus off, ~11 minutes each. The result is the kind the papers don’t lead with.

The per-round win was small. This model emits all its reasoning first, then its tool calls, then stops almost immediately — so by the time the first call’s tag closes, there’s barely any generation left to overlap with. The gap between a call appearing and the stream ending was about one second. Meanwhile the actual turn latency, twenty to a hundred seconds, lives almost entirely in the pre-call reasoning — which no amount of dispatch cleverness touches.

The speculation telemetry was noisy too — hundreds of recognizer rows, a sub-1% hit rate on the first pass — but it produced the single number that justifies the whole experiment: one reasoning-sourced speculation matched its eventual call with a 74.6-second head start. The intent was sitting in the reasoning stream, fully recognizable, over a minute before the model committed to it.

Where the 74 seconds lives

Stream dispatch stays on — it’s free, it’s proven not to corrupt episodes, and it pays off whenever a round has multiple calls. But the real levers are now obvious and they’re all upstream of dispatch:

1. Train multi-call turns properly, so one generation legitimately carries several actions and dispatch has something to overlap.
2. Cap the thinking budget, because reasoning length is the latency.
3. The trained async protocol — the big one. The speculation logs now measure exactly the gap between “intent visible in reasoning” and “call committed.” That’s the dataset for teaching the model to emit calls at the earliest justifiable moment, the way AsyncLM and SIA did with fine-tuning. Both papers saw accuracy dip when calls move earlier, so this one goes through the eval gate like everything else.

The pattern of this project keeps repeating: the shiny mechanism is cheap to build, the measurement says “not yet,” and the measurement itself turns out to be the asset. Sixty-nine furnace placements taught me that loss curves lie about behavior. The dispatch A/B taught me that latency lies about where it lives. Both times, the fix moved from the harness into the training data — which is exactly where, on a fixed base model, it has to go.