Ask three vendors how to customise a large language model for your product and you will likely get three different answers, each presented as the obvious choice. One will push prompt engineering because it is fast to demo. Another will build you a retrieval-augmented generation (RAG) pipeline because it is the current default architecture. A third will offer to fine-tune a model because it sounds like the more serious engineering effort. All three can be correct, depending on what is actually wrong with the model’s output. The mistake is treating them as tiers of sophistication rather than as answers to different problems.
Three different problems, not three levels of effort
Prompt engineering changes what you ask the model and how you ask it: instructions, examples, formatting constraints, system prompts. It costs nothing to try and iterates in minutes. It works well when the model already has the knowledge and capability it needs, and the problem is that your instructions are vague or your examples are missing.
RAG addresses a knowledge gap. The model does not know your company’s pricing policy, your product’s current specification, or what happened in a document uploaded five minutes ago, because that information was never in its training data and never will be. RAG retrieves relevant passages at query time and puts them in the prompt, so the model reasons over facts it was never trained on. It does not change how the model behaves or reasons; it changes what the model has in front of it when it answers.
Fine-tuning addresses a behaviour gap. The model has the underlying knowledge and capability, but it consistently produces the wrong shape of output: the wrong tone, the wrong format, a reasoning style that does not match how your domain experts actually make decisions, or it needs to follow a narrow, repetitive task pattern with more consistency than prompting can reliably deliver. Fine-tuning adjusts the model’s weights so that behaviour becomes the default, rather than something you have to keep re-explaining in the prompt.
The diagnostic question is simple to state and harder to answer honestly: is the model wrong because it does not know something, or because it knows it and still answers badly? Get that wrong and you will spend a project fine-tuning your way around a knowledge gap that no amount of retraining will close, because the facts you need were never in the data to begin with.
What each option actually costs, after launch
The build cost is the visible part of the estimate. The running cost is where budgets get surprised.
Prompt engineering has close to no infrastructure cost, but it is not free to maintain. Prompts drift as model providers update their underlying models, and a prompt tuned carefully against one model version can behave differently after a provider-side upgrade you did not ask for. Someone needs to own the prompt as a piece of code, with version control and regression tests, not treat it as a one-off configuration.
RAG adds a standing pipeline: a vector store or search index, an embedding model, a chunking and ingestion process for every document type you support, and a retrieval step that runs on every request and adds latency. None of that is a one-time build. Source documents change, so the index needs re-ingestion on a schedule or on a trigger. Retrieval quality degrades quietly if nobody monitors what is actually being retrieved versus what should have been, which means evaluation is an ongoing job, not a launch milestone.
Fine-tuning has the highest fixed cost before you see any output: collecting and labelling a training set large and clean enough to shift behaviour reliably, plus the training runs themselves. After launch, the cost is less visible but persistent. A fine-tuned model is tied to the base model version it was trained on. When the provider deprecates that version, or releases a materially better base model, you face a decision between staying on ageing infrastructure or retraining from scratch. That is a recurring cost most estimates leave out entirely.
Questions worth answering before any budget is committed
- Does the required knowledge change weekly, monthly, or effectively never? Fast-changing knowledge points towards RAG; static knowledge that fits comfortably in a prompt may not need retrieval infrastructure at all.
- Can the product tolerate a retrieval step on the critical path? RAG adds latency. If the feature needs a sub-second response, that constraint shapes the architecture before anything else does.
- Is there a labelled test set today? Without one, none of the three approaches can be shown to have improved anything. This is worth building before choosing an approach, not after.
- Who is accountable for the index or the fine-tuned model six months after launch? Both require ongoing ownership; a project plan that ends at go-live is incomplete for either.
- Would better prompting and a larger context window solve this without new infrastructure? Modern context windows are large enough that some problems people reach for RAG to solve can be handled by simply including the relevant document directly.
Where this goes wrong in practice
The most common mistake is reaching for fine-tuning to fix hallucination on facts, when the actual fault is a missing knowledge source. Fine-tuning on a small set of correct answers will not teach a model general facts about your business; it teaches a narrow pattern, and it will still invent details outside that pattern. The second most common mistake is the reverse: building a full RAG pipeline, with all its ongoing maintenance, for a knowledge base small and stable enough to sit directly in a system prompt with no retrieval step at all. The third is skipping evaluation entirely and judging the result by whether a handful of manual tests look plausible, which tells you nothing about how the system behaves on the cases you have not tried.
These approaches are not mutually exclusive. A production system commonly uses careful prompting as the baseline, RAG for information that changes, and fine-tuning only where a narrow, repetitive behaviour needs more consistency than prompting delivers. Choosing one architecture up front and assuming it will cover every part of the product is itself a common source of rework.
A checklist before you specify the approach
- Write down whether the failure is missing knowledge or wrong behaviour, with examples, before proposing a fix.
- Build a test set of realistic queries and correct answers before choosing an architecture, so any approach can be measured against it.
- Try prompting first, on the actual base model you intend to use in production, and record where it fails.
- Only add RAG if the knowledge genuinely will not fit in a prompt or changes too often to hardcode, and budget for ongoing index maintenance, not just the initial build.
- Only add fine-tuning if prompting and RAG have both been tried against the same test set and a specific behavioural gap remains, and plan for what happens when the base model is deprecated.