We taught a 12M model to call tools - then a Qwen paper made us rethink the whole thing

TL;DR. Qwen-AgentWorld proposes training an LLM to be the environment - a "Language World Model" that predicts what a tool returns. We borrowed the core idea to improve Fern Bud, our 12-million-parameter tool-calling model, with a hand-built simulator and an automated train→eval→fix loop. The simulator-as-data-factory worked well; our honest negative results were just as useful. This is the full story, warts included.

The paper that started it

Qwen-AgentWorld (Qwen Team, 2026) makes a deceptively simple move. Everyone trains the agent - the thing that decides which tool to call. Qwen-AgentWorld trains the environment instead: a model that, given the state, the history, and the agent's action, predicts the next observation the real system would return - the file contents, the search results, the tool's JSON, the next screen. They call it a Language World Model (LWM).

Why bother? Two payoffs:

1. Decouple - use the LWM as a simulator to train agents with reinforcement learning. Strikingly, agents trained inside the simulator beat agents trained against the real environment, because the simulator is scalable (spin up thousands of scenarios) and controllable (inject errors and edge cases on demand). Real APIs rarely throw the hard cases often enough to learn from.
2. Unify - world-model pretraining as a warm-up makes the model a better agent across the board.

Their 397B model tops AgentWorldBench (58.7), edging out the frontier. But the part that grabbed us wasn't the scale - it was the mechanism: a fake environment that manufactures both training data and an exact reward, offline, with no live API. That is gold for a small-model tool-calling company.

Our world: Fern Bud, 12M parameters

Fernfly's bet has always been the opposite of "one giant model." We run Fern Bud - a ~12M-parameter, weight-shared transformer that converts a sentence into a structured tool call. One tiny specialist per app. Cheap, offline, on-device. The trade is obvious and intentional: not frontier accuracy, but frontier economics.

So we asked: can Qwen-AgentWorld's "decouple" idea make our tiny specialists better - without a live API in the loop?

What we built (the honest version)

We did not train a world model. We're not at frontier scale, and a learned LWM is a research project in itself. Instead we built the cheapest thing that captures the idea: a hand-coded deterministic simulator. For the calendar domain, ~150 lines of Python that, given a tool call, returns what a calendar would say - created, not_found, no free slot, unsupported. Same interface as an LWM (step(action) → observation), different engine.

On top of it we built three things:

• A controllable evaluation - scenarios bucketed by difficulty (in-distribution, messy phrasing, out-of-vocabulary values, multi-constraint, error-inducing, out-of-scope), each grounded in a concrete seeded state.
• An automated loop - generate data → train the 12M → evaluate in the sim → diagnose the failures → fix the recipe → repeat, with a gate that only keeps a round if nothing regressed against a small set of hand-written real queries (the honesty check the loop can't generate for itself).
• A regex "hybrid extractor" - the model routes and writes free text; a per-domain regex fills the structured args (ids, dates, channels, amounts) the tiny model can't copy reliably.

Because our generator picks the gold answer first and then phrases it, the simulator knows the right answer by construction - so it scores the model exactly, for free, with no human labels. That's the decouple idea, scaled down to 12M.

What worked

Calendar went from leaky to deployable. The simulator immediately exposed three real failures in our deployed calendar model: it force-fit out-of-scope requests into a tool 100% of the time, it couldn't extract arguments from multi-constraint sentences, and it fell back to memorized values on unseen inputs. We fixed them in two moves - a retrain (add an out_of_scope action + wide natural value pools) and the hybrid extractor - and watched full-call accuracy climb:

45.6% → 55.6% → 60.0% full-call, and out-of-scope force-fitting 100% → 17%.

Then we automated that exact cycle and let it run unattended on calendar for several gated rounds - it diagnosed its own over-abstention, mined the misfired queries as hard negatives, and self-corrected, while the gate rejected the rounds that regressed. The simulator wasn't just an eval; it was a closed training loop.

What we took from Qwen-AgentWorld - and what we didn't

To be precise, because the difference matters:

We borrowed the plumbing idea, not the learned-LWM-plus-RL core. And our experiments quietly validated their central claim from the other direction: a controllable simulator is a fantastic, cheap source of training signal - even a deterministic one, even for a 12M model.

Takeaways if you're building small tool-callers

1. A simulator beats a live API as a data/eval factory - controllable, offline, exact reward, no keys.
2. Build an honesty gate. A closed loop optimizes its own proxy; a handful of real hand-written queries is what keeps it honest. Ours caught generator-overfit every single time (94% on generated data, 42% on real).
3. Split the labor. Let the tiny model route; let a regex fill structured args. That single move took calendar's hardest fields from broken to deployable.
4. Match the domain to the model. Distinct-verb apps are a great fit for a 12M; CRUD-heavy apps need a bigger router or contrastive data.
5. Diversity over size. For a small model, a broad corpus beats more parameters. We'd been starving ours.
6. Publish the negative results. "The fleet hurt a strong brain" is the most valuable thing we learned this month. It saved us from building an RL pipeline over a chain that wasn't ready.

Small models won't win on agentic-benchmark accuracy - and that was never the bet. They win on cost, privacy, and the edge. Qwen-AgentWorld gave us the tool to make them reliably good at the one thing they're for: turning a sentence into the right call, cheaply, offline, every time.

Next up soon: pretraining a broad Fern Bud from scratch on 250M diverse tokens - and whether breadth fixes the routing cap. Follow along.