The token bill gets all the attention. It is the part of the problem that arrives as an invoice.
Token prices fell about 80% between 2025 and 2026. A production-tier model runs around $3 per million input tokens and $15 per million output. Cheap tiers undercut that by an order of magnitude. Batch halves it again.
So inference is cheap now. Fine. That is roughly the point at which it stopped being worth optimising.
Here is the pipeline I keep running into. Fifty thousand applications a month, each one arriving as a pack of documents that somebody has to turn into a decision. Fifteen model calls per application, because the decision has fifteen questions behind it and somebody wired them up one after another. The whole pack goes into the window every time, because the window is big enough now and nobody wanted to build a retrieval layer.
Add it up and you get $401,261 a month. Tokens are $146,250 of that.
Thirty-six percent. Big enough to notice on an invoice, small enough that fixing it will not save you. That combination is the trap.
Before the money, the shape of the problem.
Every process sits somewhere on a spectrum of how much determinism it needs. Affordability calculations and regulatory disclosures sit at one end, where the same input has to produce the same output every time and a 1% variance is an incident rather than a quality issue. Drafting an underwriter's note sits at the other, where variance is closer to the point.
Nobody argues with that in the abstract. What happens in practice is that a single team builds one pipeline that spans the whole spectrum, because it is one workflow from the customer's side, and the parts that needed to be provable get handled the same way as the parts that needed to be flexible.
Bubble size is roughly transaction volume. The shaded corner is the expensive one: high determinism requirement, weak fit for an LLM-first approach, and enough volume that the mismatch compounds.
The corner bottom right is where budgets go to die. Everything sitting in it needs a deterministic core with the model pushed out to the edges, and most of it gets built the other way round because the demo was easier that way.
Every number on this page comes out of the workbench at the bottom. Change the inputs and the charts move with you. I would rather you argued with my assumptions than took my word for the results.
Everything else follows from one bit of arithmetic, so it goes first.
Fifteen steps, each right 90% of the time, does not give you 90%. It gives you 0.9 to the fifteenth. Twenty-one percent. Four applications in five come out wrong somewhere along the chain.
I have watched capable people miss this in planning meetings. Someone says "each stage is around ninety percent" and the room hears "the thing works about ninety percent of the time," and nobody does the multiplication until pilot results come back looking nothing like the demo.
Below 80% a human review queue stops being a phase you pass through and becomes a permanent cost centre.
Ninety percent per step is not sloppy work. It is where careful teams end up after guardrails, structured outputs, validators between stages, and a golden dataset somebody actually maintains. The last few points are stubborn because the errors left over are the ambiguous ones, and ambiguity is the reason you reached for a model in the first place.
The published numbers say the same thing from a different angle. Models score 80 to 90% on single-turn tasks and fall to something like 18 to 24% on long multi-step work that crosses systems. ReliabilityBench found pass@1 overstates real reliability by 20 to 40%, because it measures whether an agent can get there under good conditions, not whether it does on a Tuesday with a malformed input.
Going from 90% to 95% per step does not cost you 5% more. It costs you the second half of the project. The eval harness, the retry logic and the permanent review queue all come out of that push, and none of the three were in the original estimate.
Set temperature to zero, pin the seed, run the same prompt twice. You still will not get the same two answers.
This one surprised me. Thinking Machines Lab traced it to batch-size dependence in reduction kernels: your prompt gets dispatched inside whatever batch happens to be forming at that instant, the reduction tree changes shape with the batch, and floating point addition is not associative. Temperature zero fixes the sampling step. It has nothing to say about the logits arriving at that step.
It is fixable. Batch-invariant kernels gave bit-identical output across a thousand runs on Qwen3-8B, at about 61.5% of throughput, later trimmed to roughly 34% by the SGLang team with CUDA graphs.
Determinism is purchasable. It is priced in throughput, and nobody puts it on the slide.
The biggest line is cost of poor quality, and I want to spend a minute on it because it is the one people push back on.
Manufacturing has costed this formally for forty years. They even have the vocabulary: a defect that gets past inspection and reaches the customer is an escape, and the rule of thumb is that a defect costs about a dollar to prevent, ten dollars to catch in process, and a hundred once it has left the building. Every quality function you have ever met tracks this. AI programmes somehow do not.
The arithmetic needs four numbers, all of which you either have or can estimate. Fifty thousand applications, times one minus 0.9 to the fifteenth, gives 39,705 that come out wrong somewhere. Detection catches nine in ten, so 3,971 escape. One hour of a $50 reviewer to find each one and unpick it. That is $198,527 a month, and it is half the bill.
Now the objections, because you will get all three.
You cannot know that number. Correct. Neither can you know it is zero, which is precisely what leaving it out of the business case asserts. If you have a better estimate, use it; anyone who has run an operations team will produce one inside a minute and it will be higher than mine.
Our detection is better than 90%. Then move the slider. At 99% detection this line falls to about $20,000 and the argument survives, because tokens and review do not move and the ranking of the three architectures does not change. That objection is worth conceding on the spot.
Fifty dollars is far too low for lending. Yes it is. A wrong affordability calculation is a remediation exercise and possibly a regulatory one, and pricing it at a single reviewer-hour is the friendliest assumption available. I left it there on purpose so that nobody could accuse the model of inflating its own conclusion.
The framing that lands with a CFO is this. You are not adding a new cost to the business case. You are moving an existing one onto the page where the decision gets made. It is already being paid, by the operations team absorbing rework, by the customer who got the wrong limit and rang up about it, by the credit committee spending twenty minutes on a file that should have taken two. The model does not make errors free. It makes them somebody else's line item.
Tokens, $146,250. Fifteen calls at 30,000 tokens apiece is 450,000 tokens an application, 585,000 once you count retries.
Human review, $33,333. A fifth of the output sampled, four minutes each, reviewer loaded at $50 an hour. This one grows with volume and with your error rate at the same time, which is why it never comes down on its own.
Platform and prompt engineering, $15,000. Version bumps mean revalidation. Fixing case A breaks case B. Without a regression suite you are refactoring your most business-critical logic blind.
Eval and observability, $8,000. What it costs to be able to prove anything at all.
We ran electricity and carbon too, because leaving them out felt like picking the convenient slice. It came to $150 and changed the ranking of nothing, so that is the last you will hear about it.
Run the same workload at every chain length against a deterministic engine and the decision resolves earlier than you would guess.
Each extra call adds tokens going in and compounds the failure rate coming out. The deterministic line barely notices.
One call and the model wins comfortably. Two calls and the deterministic engine takes the lead and never gives it back. Fifteen calls and the gap is $299,594 a month.
Two is lower than I expected before I ran it. It is low because each extra call hurts twice over, and because almost nothing else in the model shifts it. Double the token price: barely moves. Double the retry rate: barely moves. The one input that genuinely drags the crossover around is how many minutes a person spends checking each output.
Which tells you what this decision is really about. Not the price of the model. The cost of checking its work.
Halve your token price and you save about $73,000 on a $401,000 problem. Take it. It changes nothing structural.
The question I would ask first is why a model is reading the same 25,000 token pack fifteen separate times to answer fifteen questions about it.
Same model, same price per token. The only thing that changed is what goes into the window, and how often.
Read the pack once, deterministically, and structure what comes out. Then hand the model the four or five fields the judgment actually needs.
Input falls from 25,000 to under 3,000. Output falls from 5,000 to 700, because now it is answering a question rather than restating a document pack. Calls fall from fifteen to three, because twelve of them were doing retrieval, and a parser does retrieval better and for nothing.
$146,250 down to $3,803. Same model, same price per token, 97.4% less. Nobody negotiated anything.
Same applications, same guardrails, same reviewers. The only thing changing is how much of the job the model is handed.
As built, fifteen calls: $401,261 a month, $8.03 an application, 20.6% end to end.
Hybrid, deterministic layer feeding three clean calls: $211,416, $4.23 an application, 72.9% end to end. Three chained steps forgive what fifteen never will.
Deterministic only, engine handling 90% and the rest escalating: $101,667, $2.03 an application.
A quarter of the cost of the thing it replaces, or $3.6 million a year at this volume. One warning, though. The hybrid pays its licence and usage whether it does much work or not, so under about five chained calls the model on its own is cheaper. Hybrid earns its keep on long chains and fat document packs, which is conveniently where the pain already is.
Cost is one axis and the least interesting one. Here are the same three architectures on the axes a review board actually asks about.
Scored out of 100 on my own judgement, not measured. The point is the shape rather than any individual score.
LLM-first wins decisively on two axes: flexibility to handle a case nobody anticipated, and how fast you get something working. Those happen to be the two axes that make a project easy to get funded, which is most of the explanation for how we all ended up here.
It loses on determinism, auditability and cost predictability, which are the three that matter once the thing is carrying real volume. The hybrid gives up some of the speed to get most of the provability back.
Log scale on both axes. Those steps in the deterministic line are the annual usage tiers kicking in.
Turn the volume up and the fixed costs melt away. Three floors appear and none of them is zero. The floor is tokens plus review plus remediation, all three charged per application, none of them touched by scale. Growth solves fixed costs. Fixed costs were never your problem.
The shaded gap is work you paid for and binned. The dashed vertical is the 90% baseline the calculator holds fixed.
Divide by the outputs that survived verification instead of the outputs produced, and each correct answer costs $38.98 rather than $8.03. Everything between those two numbers is work you paid for and threw in the bin.
Cost per token is a vendor's metric. Cost per correct, accepted, unreviewed output is yours.
If I could get teams to report three things next to the API bill, it would be these. Cost per correct output. Autonomy rate, meaning the share that ships without a human touching it, which had better be climbing every quarter or you have built an assistant on an automation's budget. And escape rate times blast radius: how many errors get out, multiplied by what each one costs once it does.
Everything above redraws from the inputs below. The dashed fields are pinned to a baseline so we are all comparing the same thing, with a ×2 button on the ones worth stress-testing. The rest is yours.
I have picked numbers I think are fair. You will disagree with some of them. Good. That is what the boxes are for.
| Cost line | Per month | Share |
|---|
Up to here it has been arithmetic. From here it is opinion, and I would rather label it than smuggle it.
For deterministic work at volume I think a deterministic engine is the cheaper choice and the more responsible one. Cheaper is the easy half and the charts do that work. Responsible is the half worth arguing over. An engine that gives the same answer to the same input every time can be audited. A probabilistic pipeline can be monitored, which is not the same thing, and "the model decided" has never once survived a conversation with a regulator. It also draws a fraction of the power, which will never show up on your P&L and matters anyway.
For most teams I would go hybrid. Deterministic layer reads and structures. Model does the two or three things that genuinely need judgment, on small clean inputs, where it is at its best. The 97% cut in tokens is a side effect of an architecture you should have wanted regardless.
I could be wrong about the ratios. If your per-step accuracy is really 97%, or your documents run 4,000 tokens rather than 25,000, the whole thing shifts and you should shift with it. What I am confident about is the shape. Chains punish you multiplicatively. Review is the largest line nobody budgets for. And the token price gets watched because it is the one number that arrives in an email.
None of this is an argument against language models. It is an argument against asking one to be a search engine, a parser, a rules engine and a judge, fifteen times per application, and then acting surprised at the invoice.
Four questions before you build anything. Can the right answer be derived from rules? Then write the rules; models are for when the rules cannot be written down, not for when writing them is boring. What does one wrong answer cost, and who finds out? How many calls, and how fat is each one, once you multiply context by call count and look at what you are paying to re-read? And if two runs have to agree, you are buying determinism, so price it.
The model proposes. The rules dispose.
We spent two years asking what a token costs. The question was always what a wrong answer costs, and who has to find it.
