The Playground Is Live: A 12M Model vs GPT-4o, Side by Side

Two weeks ago, Gaurav made the case that most AI work isn't reasoning - it's the repeatable 80%, and paying frontier prices for it is a habit, not a requirement. Last week I said I'd stop asserting that and start showing it, in the open.

Today it's live: the Fernfly Playground. No signup, no catch.

TL;DR. Pick a preset intent - "move deal 401 to proposal," "add a kickoff Thursday at 1pm, invite Wei" - and watch a 12M-parameter Fern Bud model and GPT-4o produce the tool call side by side, with latency, cost, and exact-match accuracy shown for each. Two domains (Calendar and CRM), curated tasks, and a comparison we deliberately built to be honest: nothing rescues the small model when it's wrong.

What it does

The Playground runs the same request through two models at once and shows you both results next to each other:

• The structured tool call each model produces - the function name and the arguments it filled.
• Latency - how long each took (including network trip).
• Cost - what each call actually costs.
• Accuracy - whether the call matches the gold answer, scored live.

That's the entire pitch made concrete. On a bounded, repeatable intent→action task, the small specialized model should produce the same structured call as the frontier model. The interesting part is everything in the other three columns.

One asymmetry the cost column makes visible: GPT-4o has to be handed the full tool schema on every call, and it pays for re-reading that spec in input tokens each time. Fern Bud has the tool set baked in, so it doesn't. At one call that's a rounding error; at a million calls a day it's the whole argument.

Presets you can trust, and your own prompts

The Playground has two modes, and the difference between them is the point.

Presets are the measured comparison. Each curated task is drawn from a model's held-out test split (never seen in training), so it has a known-correct answer we can score against in real time, on exact-match, right-tool, and schema-validity. That gold answer is what makes the accuracy numbers below real rather than a vibe check.

Free-text is for your own curiosity. Type any request and both models run it live, side by side. There is no gold answer for an utterance we have never seen, so we report schema-validity instead of a correctness verdict, and we do not store your prompt. It is capped at a few hundred characters: enough for a real intent, small enough that the demo cannot be turned into a free LLM proxy.

Two domains to start: Calendar (book, list, cancel, find a free slot) and CRM (create deals and contacts, move deal stages, log calls, add tasks). Bounded, high-volume, distinct-verb work, exactly the shape the 80% argument is about.

How we kept the comparison honest

It would be easy to build a demo that flatters us. We went the other way on purpose:

• No safety net for the small model. In our production stack, a low-confidence Fern call can fall back to a bigger model. The Playground turns that off. If Fern Bud gets it wrong, you see it get it wrong. No fallback quietly fixing the score.
• GPT-4o gets a fair shot. It receives the same tool schema, inlined, and we scored against its native tool-calling output. Where a domain's published schema was too thin to be fair to a frontier model, we authored a complete one so GPT-4o has real arguments to fill.
• Scoring is strict and legible. Exact means the right tool and the right arguments (order-insensitive); we also show right-tool-only and schema-valid so you can see exactly how a call missed, not just that it did.

One honest footnote on latency: in the Playground we serve Fern Bud from our GPU for concurrency, so the numbers you see are demo-serving numbers. "Runs on a CPU, offline" is a statement about the model's size and economics in production, not a claim about the millisecond count on this page. We'd rather tell you that than let a chart imply something we didn't measure.

The honest 20%

A small model does not win everything, and the Playground will show you where it doesn't. Open-ended reasoning, novel or ambiguous problems, long-context synthesis - that's the genuine 20%, and it still belongs to the frontier models. The claim was never "small beats big." It's "small beats big on the 80%, and you should stop paying the 20% rate for it."

So when Fern Bud matches GPT-4o on a task here, you're watching a real match. And when it misses, that's real too, and I want to know about it.

The numbers

I ran the held-out gold set - the exact tasks the Playground serves - through a live scorer script. No cherry-picking; here's the whole board:

Fern Bud is self-hosted, so its per-call cost is $0; the 23 GPT-4o calls cost about $0.017 total (~$0.0007 each, small until you multiply it by a million calls a day).

The honest read: on held-out tasks, Fern Bud picks the right tool every single time and returns a schema-valid call every single time, at ~25% lower latency and no per-call cost. On getting the arguments exactly right it's split: it beats GPT-4o on Calendar (6/11 vs 4/11) and clearly trails on CRM (2/12), where copying deal names and amounts out of unseen phrasing is exactly where a 12M model still slips. That CRM argument gap is the next thing I'm closing. Per the whole point of this, you can watch it happen in the open.

Go break it (gently)

Run the tasks yourself, compare the columns, and tell me where the small model holds up and where it falls down. This is week two of building Fernfly in the open. The numbers on the page are the real ones, and next up I'll write about what they say once enough of you have put it through its paces.

Open the Playground →