A payment gateway either authorises a transaction or it doesn’t. A login form either accepts the password or rejects it. Most software you commission behaves this way: an outcome is right or wrong, and a test suite can say which. An AI feature built on a large language model does not behave this way. Ask it the same question twice and you can get two different answers, both defensible, one of them wrong in a way that only becomes obvious three weeks later when a customer complains.
This is the part of commissioning an AI feature that gets skipped most often, because it doesn’t show up in a demo. A vendor can show you a transcription that reads well, a summary that sounds plausible, a categorised expense that looks correct. What a demo cannot show you is how the feature behaves across the hundreds of messy, ambiguous, real inputs it will actually receive once it’s live. That’s a separate piece of engineering work, and if nobody’s asked for it, it probably hasn’t been done.
Why the usual QA process doesn’t cover this
Conventional testing assumes a fixed input produces a fixed, checkable output. You write a test case, assert the expected result, and the test passes or fails forever until the code changes. LLM output breaks that assumption at the root: the same prompt against the same model can produce different phrasing, different structure, occasionally a different conclusion. A test that asserts an exact string match will fail constantly on correct output and pass on subtly wrong output, which makes it worse than no test at all.
The failure mode is also different. A broken API call throws an error you see in your logs within seconds. A meeting summary that quietly drops the one action item your client cared about throws nothing. It looks like a normal response. The system is healthy by every conventional metric while the feature itself is producing output nobody would sign off on if they read it closely. That gap, between “the system is up” and “the output is right,” is what an evaluation process exists to close.
Build the evaluation set from real inputs
The single biggest difference between a team that evaluates AI features properly and one that doesn’t is where the test inputs come from. Synthetic examples, written by the same engineers who built the feature, tend to be clean, well-formed, and unrepresentative. Real customer inputs are messy: a meeting transcript with three people talking over each other, a receipt photographed at an angle, a family calendar entry typed on a phone keyboard with half the words autocorrected wrong. Those are the inputs that expose where a feature actually breaks.
In practice this means collecting a working set of real or realistic inputs before launch, running the feature against every one of them, and keeping the outputs somewhere reviewable, rather than trusting a handful of manual spot-checks in a demo environment. The set doesn’t need to be enormous to be useful. A few dozen inputs that deliberately cover edge cases, ambiguous cases, and the ordinary case will surface more problems than a thousand near-identical happy-path examples.
Match the evaluation method to what’s actually at stake
Not every AI feature needs the same level of scrutiny, and treating them all the same wastes effort in the low-stakes cases and under-invests in the high-stakes ones. A few methods, roughly in order of effort:
Deterministic checks. Did the output parse as valid JSON? Does the categorised amount match the receipt total? Is a required field present? Cheap to run, and they should catch anything that’s checkable this way before a human ever looks at the output.
Reference comparison. Where a correct answer can be written down in advance, score the output against it. This works well for structured extraction and categorisation tasks, less well for open-ended writing.
Model-assisted review. Using a second model to score output against a rubric scales further than manual review and catches more than deterministic checks, but it inherits its own blind spots and needs periodic calibration against human judgement, not blind trust.
Human review. Slower and more expensive, but there’s no substitute for it on the highest-stakes outputs, and it’s the only method that reliably catches the kind of subtle, plausible-sounding error the other three miss.
What counts as acceptable also depends entirely on what the output is used for. A meeting summariser that occasionally rephrases something slightly awkwardly is a minor annoyance. A feature that reads a bank transaction and assigns a spending category can be wrong sometimes without doing real harm, provided the user can correct it easily. A feature that reads a video feed and flags a safety concern, or one that decides whether an expense claim is approved automatically, needs a much higher bar and probably a human in the loop before any consequential action is taken. Ask what happens downstream of a wrong answer before deciding how much evaluation effort it deserves.
Evaluation doesn’t stop at launch
A model provider updates its underlying model. Someone tweaks a prompt to fix one specific complaint and inadvertently changes behaviour elsewhere. The mix of real user input drifts over time as the product gets used in ways nobody anticipated at launch. Every one of these is a reason a feature that worked well in testing can quietly get worse in production, with no error, no alert, and no obvious trigger.
The evaluation set built before launch is the same asset that catches this afterwards. Re-running it against a new model version or a changed prompt, before rolling that change out to everyone, turns “we think this update is fine” into something you can actually check. Without that comparison, every prompt change or model upgrade is a guess made on production traffic, which is a way of finding out about regressions from your customers instead of before them.
What to ask before you commission an AI feature
If you’re evaluating a proposal that includes an AI feature, a short set of questions separates a team that’s thought about this from one that hasn’t:
What does the test set look like, and where do the inputs come from?
How is quality actually measured, beyond someone reading a few outputs and deciding they look fine?
What happens when the underlying model is updated or a prompt is changed? Is anything re-checked before that reaches users?
For the specific feature in question, what does a wrong answer cost, and does the review effort match that cost?
Is there a human review or override step anywhere a wrong answer would otherwise reach a customer directly, a decision, or money?
None of this is exotic. It’s the same discipline as any other engineering QA process, adapted to a kind of software that doesn’t fail the way the rest of your stack fails. The teams that skip it aren’t cutting corners on something invisible; they’re choosing to find out about the problem later, and from the people it affects, rather than earlier, from a test set they built themselves.