The uncomfortable moment in most AI product work is not the demo. It is the second invoice. A feature that cost pennies during development starts costing real money once a few thousand people use it daily, and nobody on the team can say precisely which part of the product is responsible. We build AI into our own products — MeetWise AI, Money Master AI and Assistant Mama AI — and the engineering that keeps those features viable has less to do with prompt cleverness than with arithmetic done early.
When a language model is the wrong tool
A language model is a general-purpose reasoner sold by the token. That makes it superb at problems where the input is genuinely open-ended and poor value where the input is not. Before writing a prompt, it is worth asking whether the task has a deterministic answer that a cheaper mechanism already produces.
Categorising a bank transaction is the clearest example. A large share of transactions in any personal finance product are repeats: the same merchant string, the same user, the same category as last month. Sending each one to a model is paying reasoning prices for a lookup. The sensible architecture puts a deterministic layer first — exact merchant matches, user-defined rules, a learned mapping from previous confirmed categorisations — and reserves the model for what actually falls through: unfamiliar merchants, ambiguous strings, cases where the user has corrected the system before and the rule needs revisiting.
Models earn their cost where inputs are messy, varied and unbounded — free-text meeting talk, a photograph of a handwritten note, a question phrased in a way nobody anticipated. Where the input is a structured document from a known source, a parser is faster, cheaper and fails loudly rather than quietly.
A useful test: if you can describe the correct output as a rule in one sentence, write the rule.
Cost per request, and how it escapes
Providers price input and output tokens separately, and output is typically the more expensive of the two. That single fact should shape the design. A feature that reads a long document and returns a short structured answer has very different economics from one that reads a short instruction and writes a long essay, even though both feel like “one AI call” to a product manager.
The costs that surprise teams are almost never the ones on the design document. In our experience the recurring culprits are:
- Context that grows silently. A conversational assistant that appends the full history to every turn has a per-request cost that rises with session length. The tenth message in a conversation can cost several times what the first did, and nobody notices because the average looks fine.
- Retrieval that over-fetches. Pulling twenty candidate chunks when five would answer the question is a permanent tax on every request. Retrieval quality and retrieval cost are the same problem.
- Retries and validation loops. If a call fails schema validation and you retry, that request cost twice. A 10% failure rate is a 10% cost increase before anyone reads a log.
- Chained calls presented as one feature. “Summarise this meeting” may in fact be transcription, then chunk summarisation, then a synthesis pass, then action-item extraction. Four calls, four line items.
The instrumentation that fixes this is unglamorous: log token counts in and out, model used, latency and outcome, tagged by feature and by user, for every call. Without that, cost optimisation is guesswork. With it, the expensive feature identifies itself within a day.
Caching, properly understood
There are two distinct caches and teams conflate them. The first is a result cache: the same question has been asked before, so return the stored answer without calling anything. This works where inputs repeat exactly and answers are stable — a definition, a policy explanation, a classification of an identical merchant string. It requires a cache key built from everything that affects the answer, including the prompt version and the model, or you will serve stale outputs after your next deployment.
The second is provider-side prompt caching, where a stable prefix — system instructions, a schema, reference material shared across requests — is charged at a reduced rate on subsequent calls. This rewards a specific prompt architecture: put everything constant at the front, everything variable at the back. Teams that interleave user data through the prompt for readability give up the discount without realising it.
Matching model size to the task
Using the largest available model everywhere is the equivalent of running every query on the biggest database instance you can rent. It works, and it is not a plan.
Tasks separate reasonably cleanly. Classification into a known set of labels, tidying up a transcript line, deciding whether a passage contains a commitment, routing a question to the right handler — these are well served by small, fast models. Tasks with genuine reasoning depth — reconciling contradictory statements across an hour of discussion, producing financial guidance where the wrong nuance matters, deciding what to escalate to a human — justify the larger model.
The pattern worth adopting is a cascade: a small model attempts the task and reports its confidence or fails validation; only then does the request escalate. The economics work when the small model handles most of the volume, which is usually true because most real inputs are ordinary. The failure mode to watch is a cascade where escalation is frequent — then you are paying for both models and would have been better off with one.
Deciding this requires an evaluation set. A few hundred real inputs with known-good outputs, run against each candidate model, tells you the accuracy difference in the terms that matter for your product rather than in benchmark scores. Building that set is a day of work and it is the highest-leverage day in the project.
Batching, and the latency question nobody asks
Providers commonly offer a batch mode at a discount, in exchange for results arriving within hours rather than seconds. The relevant question is not technical but behavioural: does the user need this answer now, or do they need it before they next look?
A weekly spending review, a monthly summary, a re-categorisation of historical records after a rule change, an overnight enrichment pass — none of these are interactive. A user asking a question in the interface plainly is. Much of what gets built as synchronous is synchronous only because that was the easiest thing to write, and moving it to a queue costs a job runner and buys a materially lower bill.
Proving the feature is worth it
Cost per request is only half the equation. The other half is whether anyone benefits, and this is where most AI roadmaps are weakest, because usage is measured and value is assumed.
Useful signals are behavioural and specific to the feature. Does the user accept the generated output, edit it, or discard it? An edit rate is a quality metric with no annotation cost. Does the feature get used again by the same person a week later, or only once out of curiosity? Does its presence correlate with retention or conversion in a way you could defend to a sceptical finance director? For an action-item extractor, the honest measure is whether the extracted items are subsequently acted on — not how many were produced.
Set a cost ceiling per user per month before building, derived from what you charge them. If the feature cannot fit inside it at plausible usage, the answer is not a better prompt. It is a smaller model, a narrower scope, a cache, a batch job, or not building it.
The questions to answer before you build
- Can a rule, a parser or a lookup produce this answer for most inputs?
- What is the expected cost of one request, split into input and output tokens, at the largest realistic input?
- How many calls does the feature really make, counting retries and chained steps?
- Which parts of the prompt are constant, and are they at the front?
- Does the user need the result within a second, or before they next open the app?
- What observable behaviour would tell us in a month that this feature is earning its place?
If you cannot answer the last one, that is the work to do first.