AI Token Cost Reduction: Tips and Tricks

On August 22 we ran the second session in our webinar series. Where Agentic AI essentials was about building an agent, this one is about what happens when the agent you built runs a million times instead of once.

TL;DR. In a typical agent run, roughly 94% of what you pay for is input, and most of that is the same context resent every turn. The user types about 0.1% of the prompt. Nine levers bring it down, four of them need no new infrastructure, and the order you pull them in matters more than which ones you pick. Three live runs against Gemini 2.5 Flash closed the session: capping a tool result took $28,955 down to $843 per million calls, declaring a cache took $3,848 down to $542, and nine words of instruction took $9,033 down to $528. Same questions. Same answers.

The room told us where the money goes

Registration asked three questions, so the session opened with the audience's own answers rather than our assumptions.

What are you spending on LLM APIs, per month?

71% spend under $1,000 a month, which is exactly the Goldilocks range: annoying enough to notice, small enough to keep ignoring until the product scales and the bill scales with it.

The bar worth staring at is the third one. 16% answered "no idea, that's partly why I'm here." That is not a gap in the data. It is the most honest answer on the form, because no dashboard tells you which prompt, which tool schema, or which retry burned the money.

What have you already tried to cut the bill? An even split: 46% nothing yet, 54% something. Prompt caching and routing between models tied at 21%, trimming prompts or schemas 15%, a smaller or self-hosted model 13%, semantic caching 4%.

What is burning the most tokens? Coding agents 33%, a chat assistant or support bot 27%, a tool-calling agent in a long loop 16%, RAG over documents 10%, not in production yet 10%, batch jobs 4%.

Add the first and third rows and half the room is burning it inside a loop. That number set the shape of the rest of the session, because a loop resends its history every turn. The cost of turn ten is not one message. It is ten.

Act one: three meters are always running

The foundations, quickly, because they are the part everyone thinks they can skip.

The model never sees your words. "Turn this into a weekly report." is seven words and eight tokens, and you are billed for the tokens. Common words are cheap and rare ones are not: the and report are one token each, unbundling is three, Grzegorz is five.

The same meaning can cost 5x more depending on how you format it. The identical content is 12 tokens as plain English, 20 as a CSV row, 48 as JSON with keys and quotes, and 110 as a base64 blob. The rule of thumb is about four characters per token, which holds for prose (4.1) and is badly wrong for code and JSON (2.8). Languages differ too. Devanagari runs roughly 2.8x the token count of the same meaning in English.

Two directions, two prices. Input, which is your instructions plus context plus history plus data, runs about $2.50 per million tokens. Output, which is what the model writes back, runs about $10.00. Output is four times the price of input, and that single fact decides which lever you pull first.

Then the framing that carried the whole talk: every token you send pays three bills, not one.

• Money. The per-million rate, on both sides.
• The context window. Instructions, tool definitions, memory, history, retrieved data, and only then your question. Most people picture the window as their question. It is everything except their question.
• Time. Prefill reads everything you sent in one parallel gulp, and that is your time to first token. Decode writes one token at a time, and that is the rest of the wait. Trim the input for a faster first response. Trim the output to finish sooner.

The context meter is the one that fails badly. Overflow is not slow, it is broken. There is no "a bit over budget": under the limit you get a grounded answer, over it the excess is silently dropped and you get a confident, wrong one. Nothing warns you.

Money is the least interesting of the three. Tokens are a design decision about how you architect the system, not a bill that arrives at the end of the month.

Act two: enough machinery to understand caching

Every token looks at every token before it. Ten tokens make 45 comparisons, and the work grows with the square, so doubling the prompt roughly quadruples the reading. This is why a long prompt costs more than a long answer to process, even though the answer is priced higher per token.

The model takes notes while it reads, and those notes are the cache. Send the same beginning and the provider consults its notes instead of re-reading. Cached input is billed at 0.1x, a 90% discount, and it is near-free on a self-hosted vLLM prefix cache. Writing the cache costs about 1.25x, once.

The catch is the whole game: one changed byte at the front throws it all away. So order the prompt by how often each part changes, most stable first:

instructions → tools → stable context → volatile → the question

Put what changes last. If you journal your day and ask "how many calories did I burn", the day's narrative is stable and the number is not. Stuff the changing number in the middle and you pay full price for the whole prompt every time. Put it at the end and you pay 10% of it.

Act three: agents change the maths

A chatbot sends one prompt. An agent sends the same prompt back to itself, over and over, getting fatter each time. That changes the arithmetic from linear to quadratic:

total  ≈  n·B  +  (n²/2)·g

where B is what every call carries (system prompt, tool schemas, the standing context), g is what each turn adds (the reply, the tool result, the next question), and n is the number of turns. The resend overtakes the fixed cost at n > 2B/g. With a 3,000-token base and 1,200 tokens added per turn, that crossover lands on the fifth turn, and most agents run a lot longer than five turns.

Measured, not modelled: one 800-row database answer, 59,307 tokens, carried through five steps of a single run on a production automation platform. 296,535 tokens of the same rows.

Where the tokens actually go, measured across production AI features as ranges rather than averages:

The user typed about 0.1% of that. You did not write most of your prompt. Your system did, which is the good news, because it means you can change it. And the top row is where you start, because it is both the biggest and the most volatile.

Nine levers

The first four cost you nothing but a change to prompts you already send. A few of them are worth expanding.

Lever 1, cap what comes back. Ten of 800 rows plus the column names, or a schema plus a sample plus the aggregates. Simple common sense, and very uncommon. It is the biggest single win available to most teams.

Lever 2, pay once for the part that never changes. Reported cost reductions run 41% to 85% in production. But note carefully: caching saves money, not tokens. If your dashboard counts tokens, caching will look like nothing happened at all.

There is an honest read on caching that the industry mostly skips:

Exact caching is arithmetic. Semantic caching is a bet.

A byte-identical prefix either hits or it does not, the question is decidable, a hit is always the right answer, and it is billed at 0.1x. 90% off, guaranteed.

"Close enough" by embedding distance means picking a similarity threshold. Loose, and you serve a confidently wrong answer. Tight, and you barely hit at all. The claim is 90%. What we see in practice is 20% to 45%. A miss costs you a call. A false hit costs you a customer.

Lever 4, cap the answer, not just the question. Output is 4x input, so this is the cheapest lever you have. No limit set is 1.0x. A max length plus "be concise" is 0.25x.

Lever 6, send the neighbourhood, not the world. Whatever the big object is, send its surroundings: an entire codebase becomes the file and its imports, a whole spreadsheet becomes the sheet and its header row, a thousand-turn thread becomes the last few turns and a summary. Managers editing a deck with an assistant: send the slide you are changing, not the deck.

Lever 7, not every job needs the biggest model. Extract a date, classify, tag this, pick a category: small model. Plan the migration, write the report: big model. And at the end of that road, lever 9's more interesting cousin. The cheapest model is one that only does your job. Routing picks between models you rent. Distillation gives you one you own, and the tool schemas stop travelling with every request because they are baked into the weights. A frontier call carrying a system prompt, 14 tool schemas, history and the question is about 3,400 rented tokens. The distilled equivalent is the 30-token question and one tool call back.

Lever 9, squeeze what's left. Do this one last. It needs a model of its own, and aggressive settings lose detail.

Three live runs

The session closed on real API calls to Gemini 2.5 Flash, priced at frontier rates of $2.50 in and $10.00 out per million tokens, projected to a million calls.

Lab 1, cap what the tool returns. Same question ("which region took the most revenue, and roughly how much?"), same answer (Manchester, about 7238.44). The only difference is how much of the tool result got pasted in.

98% less input, and it came back nearly four times faster.

Lab 2, pay once for the part that never changes. A support-agent brief of about 1,490 identical tokens, three different questions appended at the end. This one is worth reading closely, because it is the demo that did not go the obvious way:

Automatic caching reused nothing on either of the two calls that could have used it. Declaring the cache reused 1,469 of 1,491 tokens. Implicit caching is best-effort and often reuses nothing at all; it also depends on the gap between calls, so back-to-back requests (exactly the regime an agent runs in) are the worst case. Declaring it explicitly is 99% off, every call, with no luck involved.

Lab 3, bound the answer. "Explain prompt caching to a developer who has never used it", with and without nine extra words: "Answer in at most two sentences. No preamble, no list."

95% off the output bill from one instruction, and six times faster.

Across the EverydaySeries platform, with these levers wired in as switches that report their own savings: 97% off an 800-row database answer (59,307 to 1,625 tokens), 99% off a 1 MB API response (267,868 to 1,662), 84% off a 30-slide deck used as context (6,798 to 1,107). Recorded by the levers themselves on real runs, not estimated.

The order matters more than the list

Bound the worst case before you optimise the average. Never build a search index before you have capped results, or you will just retrieve into a prompt that one query can still blow up.

1. Cap what tools return. One function. Removes your tail risk.
2. Cache the prefix and stop repeating. Reorder, then de-duplicate.
3. Cap the answer. A day's work. Output is the expensive side.
4. Route the model. Build the evaluation first.
5. Retrieve, slice, compact. The infrastructure investment.
6. Compress the residual. Last.

And when you measure, track two numbers, not one. Truncation removes tokens. Caching removes cost. A dashboard that only counts tokens will tell you caching did nothing at all. The session's own scoreboard read -40% tokens sent and -65% dollars spent, and neither number alone would have told the story. Keep a holdout run with the guards off, so the total stays honest.

The routing layer, built

Lever 7 is the one we build, so the middle of the session detoured into Fernfly: describe your API, get back a tiny model that calls it. Natural language in, tool calls out, at about 840 ms p50, 85% tool-call accuracy, and a flat per-request rate rather than a per-token one.

The live build was a revenue dashboard. A tools.json of 12 Chart.js functions went in, the platform generated roughly 1,400 training utterances from it, and a trained model drove the chart from plain English: "show this as bars", "make enterprise red", and the good one, "make the small numbers bigger", which came back as a logarithmic y axis. Nobody said the word "logarithmic".

That is the same pipeline as Twelve Functions and a Sentence, and it exists here for a specific reason. A distilled model is what makes lever 9 real: the 14 tool schemas stop being 3,400 tokens you rent on every single request.

The dashboard is open source at antelligent-org/chart-command-bar. Plain HTML, CSS and JavaScript with no build step, and it ships the seed training pairs, so you can read what the model learned from rather than take our word for it.

Five things worth stealing

1. You didn't write most of your prompt. Your system did, so you can change it.
2. Cap what tools return, first. In a loop, one fat result is paid for again at every later step.
3. Put what never changes at the front. Then the provider stops charging you to re-read it.
4. Send the neighbourhood, not the world. The slide you're editing, the file you're changing, the last few turns.
5. Make it a switch with a meter. Every lever reports its own saving; a holdout keeps the total honest.

The one line to leave with: tokens are a design decision, not a bill you receive.

Next

The next session is roughly three weeks out, with the topic still open. If there is something you want covered, or you want to give the talk yourself, tell us. The event calendar is where they get announced.

If your answer to question one was "no idea", start with lever 1 and a meter. You will know more in an afternoon than a month of dashboards will tell you.