An AI feature usually clears its budget review on the strength of a demo: a handful of calls, a clear win, a modest bill. Three months into production, the same feature is serving thousands of users, the finance team is asking why the cloud invoice doubled, and nobody on the engineering side can say precisely why. This is not a pricing problem. It is an architecture problem that happens to show up on an invoice.
We build AI features into products we ship ourselves, including MeetWise AI, which transcribes and summarises meetings in real time, and Money Master AI, which reasons over a user’s transactions. Both involve exactly the pattern that makes LLM costs hard to predict: variable-length input, per-user usage that scales with how much someone likes the product, and a temptation to keep adding context “just in case” it improves the answer. What follows is how we think about keeping that spend under control, not a promise of a specific saving, because the right answer depends on your traffic pattern, your model choice and how tolerant your feature is of latency.
Where the money actually goes
Three things drive spend on a production LLM feature, and they compound.
- Context, not just the user’s message. System prompts, retrieved documents, conversation history and tool schemas are all billed as input tokens on every single call. A verbose system prompt or an unbounded chat history can outweigh the actual question by a wide margin, and it is easy not to notice because it never shows up as a discrete line item.
- Agentic loops. Any feature that lets a model call tools, re-read its own output, or retry on a bad parse multiplies the number of calls per user action. A single “summarise this meeting” request can become five or six model calls once you add retrieval, verification and formatting steps, and each one carries its own context.
- Redundant work. The same or a near-identical prompt is often sent more than once: a user rephrasing a question, a retry after a timeout, or a background job re-processing data that has not actually changed.
None of these are visible from a per-token price list. They only show up once you look at how the feature actually behaves under real usage, which is one reason cost control has to be part of the design, not a post-launch cleanup task.
The levers that actually work
Cache what repeats
The major model providers now offer prompt caching for content that stays constant across calls, such as a system prompt, a long reference document, or a set of tool definitions. If your feature sends the same several-thousand-token preamble on every request and only the last few hundred tokens actually change, caching that preamble is close to a free win: it costs nothing to implement beyond structuring your prompt so the static part comes first and stays byte-for-byte identical between calls. Separately, an application-level cache for genuinely repeated queries, such as identical or near-identical questions against the same document, avoids calling the model at all for a fraction of traffic. That layer needs its own invalidation logic, so it is worth building only where repetition is common enough to justify the complexity.
Match the model to the task
Not every call in a feature needs the largest model you have access to. Classification, extraction and formatting steps are usually well served by a smaller, cheaper model, while the step that genuinely requires reasoning over ambiguous input can use a larger one. Routing by task, decided at design time rather than dynamically guessed at runtime, is simpler to build, easier to test, and avoids the failure mode where a router sends something complex to a model that cannot handle it. Where a feature genuinely needs dynamic routing between models of different capability, treat that router as a piece of production logic with its own tests and monitoring, not a shortcut.
Batch what does not need to be instant
Overnight report generation, bulk re-categorisation of historical data, and any other workload where a user is not staring at a spinner do not need the real-time API. Providers offer batch endpoints, priced lower than synchronous calls, precisely for this kind of work. The trade-off is turnaround time measured in hours rather than seconds, which is irrelevant for a nightly job and unacceptable for a live chat feature. Separating “must respond now” traffic from “can wait” traffic is a design decision worth making explicitly, rather than defaulting everything to the synchronous path because that is what the prototype used.
Trim context deliberately
Longer context is not automatically better context. Unbounded chat history, retrieved documents included wholesale rather than the relevant excerpt, and defensive “include everything in case the model needs it” habits all inflate every subsequent call. Summarising or truncating history after a fixed number of turns, retrieving smaller and more targeted passages, and periodically auditing what is actually in the system prompt are unglamorous but effective. This is also where quality and cost intersect in a useful way: a bloated, poorly organised prompt is often both more expensive and less accurate than a tight one, because the model has more irrelevant material to sift through.
Treat it as a budget line, not an afterthought
The teams that keep AI spend under control generally do one thing the others skip: they instrument it. That means attributing token spend to a feature, and ideally to a customer or plan tier, not just watching a single aggregate number on the provider’s dashboard. It means setting a budget alert before the invoice arrives, not after. And for anything approaching usage-based pricing, it means knowing your cost per user or per action well enough to price the feature sensibly, rather than discovering the unit economics once a few power users have run up disproportionate usage. This is the same discipline most engineering teams already apply to cloud infrastructure spend; LLM APIs simply need it applied a little earlier, because usage can scale unpredictably the moment a feature is genuinely useful.
A checklist before you ship the next AI feature
- Is the static part of every prompt (system prompt, tool schemas, reference material) structured so it can be cached, and does it actually stay identical between calls?
- Does every step in the flow need the largest model available, or would a cheaper model handle classification, extraction or formatting just as well?
- Is there work in this feature that does not need a real-time response and could move to a batch endpoint?
- Is chat history or retrieved context growing unbounded, or is it being trimmed and summarised on a defined schedule?
- Can you attribute token spend to a feature and, ideally, a customer, before launch, not after the first surprising invoice?
- Is there a budget alert configured, and does someone own the response when it fires?
None of this is exotic engineering. It is the same instinct that should apply to any metered dependency: know what drives the cost, decide deliberately where you are willing to spend more for quality or speed, and put monitoring in place before usage grows rather than after. An AI feature that is well-instrumented from day one is much easier to keep, and much easier to price, than one that only gets this attention once the bill has already become a problem.