How to Choose Between Fine-Tuning, RAG, and Prompting

Three techniques, often misapplied

The most common architectural question we get from teams new to production AI: “should we fine-tune the model, use RAG, or just do prompt engineering?” The question is asked as if these are competing approaches to the same problem. They aren’t. Each solves a different problem, and using one to solve another’s problem is a routine source of wasted effort.

The decision is straightforward once you see the shape of the work. Below is the framework, with the failure modes you avoid by picking right.

What each one actually does

Prompting (prompt engineering)

The model is unchanged. You design the input — system prompt, examples, instructions — to get the model to do what you want with each call. State is in the prompt; structure is in the prompt; intent is in the prompt.

Prompting works when:

• The base model has the capability you need.
• The task can be specified clearly in language.
• The variation per call is bounded by what fits in the prompt.

Prompting is the default and should be the starting point for nearly every task. Most production AI in 2026 is dominantly prompt engineering with the other techniques layered on selectively.

RAG (retrieval-augmented generation)

You inject relevant external information into the prompt at runtime. The model is unchanged. The prompt is dynamically constructed to include facts retrieved from a knowledge base.

RAG works when:

• The model needs information it doesn’t have in its training data.
• The information changes over time (so you can’t just include it in a fine-tune).
• The information set is too large to fit in every prompt.

RAG is the right answer when the constraint is “the model needs to know things it doesn’t currently know.” It’s the wrong answer when the constraint is something else.

Fine-tuning

You modify the model itself by training it on examples of the input-output mapping you want. The model learns patterns, preferences, and outputs from the training data.

Fine-tuning works when:

• The task has a consistent shape across many examples.
• You have meaningful training data (at minimum thousands of labeled examples).
• You need the behavior to be reliable across many calls without long prompts.
• The cost of running with shorter prompts justifies the up-front investment.

Fine-tuning is increasingly less central than it was in 2023. Improvements in frontier model capability have closed the gap; many things that required fine-tuning two years ago can now be prompted reliably. Fine-tuning still has a role, but the role is narrower than it once was.

The decision tree

For a typical task, the decision walks like this:

1. Try prompting first. Always. Use a frontier model with a clean system prompt and a few well-chosen examples. Measure quality. If quality is acceptable, stop here.

2. If quality isn’t acceptable because the model doesn’t have the information it needs: add RAG. Build a retrieval layer over the relevant knowledge base. Re-measure quality. If acceptable, stop here.

3. If quality isn’t acceptable because the task has a specific output shape the model isn’t reliably producing: consider structured output mode, or consider few-shot prompting with more examples. If those don’t close the gap, consider fine-tuning.

4. If quality is acceptable but cost is too high because the prompt is very long: consider fine-tuning to reduce prompt length, or consider prompt caching if the prefix is stable.

5. If quality is acceptable but latency is too high because of multi-step reasoning: consider fine-tuning a smaller model that can replace the multi-step process with a single call.

This decision tree picks the right tool 90% of the time. The remaining 10% involves combinations (prompting + RAG + a small fine-tune for a specific capability) but those are refinements, not different starting points.

The mistakes we see most often

Three patterns of misapplication:

Mistake 1: Fine-tuning for information

The team fine-tunes a model on company-specific documents thinking the model will then “know” the information. The result is a model that can be coaxed into producing company-specific content but that hallucinates frequently and can’t be updated when the information changes.

The right approach: RAG. Don’t bake information into the model weights when retrieval can deliver it cleanly.

Mistake 2: Prompting for consistent output shape

The team has a task that requires the exact same output shape every call. They use careful prompting with examples. Most calls produce the right shape; ~5% don’t, and the downstream system breaks.

The right approach: structured output mode (the structured-output post covers this) or, for very specific shapes that structured output can’t enforce, light fine-tuning on the desired output format.

Mistake 3: RAG for everything

The team builds a RAG pipeline as the default architecture. Every prompt gets retrieved context added. This works for some tasks and adds noise for others — tasks where the model didn’t need additional information get worse, not better.

The right approach: route by task type. Tasks that need information get RAG. Tasks that don’t, don’t.

How they compose

The three techniques are not mutually exclusive. Sophisticated systems combine them:

• Prompting for the core instruction and task definition.
• RAG for information the model needs at runtime.
• Fine-tuning for specific behaviors that need to be reliable across many calls.

The discipline is to layer them in the right order. Start with prompting. Add RAG when information access is the constraint. Add fine-tuning when the prompting+RAG pattern works but has cost, latency, or reliability ceilings.

This composition is much more common than pure fine-tuning, pure RAG, or pure prompting in production systems. The frontier-model API space has matured to the point where the right architecture for most production work is “prompt-driven, RAG-augmented, sparingly fine-tuned.”

What’s changed in 2026

A few specific shifts worth knowing:

Long context made RAG less necessary for small corpora. Where the knowledge base fits in context (now ~200k+ tokens for many providers), you can sometimes skip retrieval entirely. The right answer is “load it all” for some workloads.

Prompt caching changed the prompting-vs-fine-tuning math. A long prompt that’s cached costs nearly nothing per call. The cost argument for fine-tuning (shorter prompts = cheaper inference) is weaker than it used to be.

Structured output mode obsoleted some fine-tuning. Tasks that required fine-tuning to produce reliable JSON now run with structured output mode directly. Less fine-tuning is needed for shape-reliability.

Better evaluation tooling makes prompting iterations faster. A well-tooled prompting workflow can match fine-tuning quality on many tasks in a fraction of the time.

The cumulative effect: prompting + structured output + RAG covers more of the production AI surface than it did two years ago. Fine-tuning is still in the toolkit, but the bar to reach for it is higher.

When fine-tuning still wins

Specific scenarios where fine-tuning is the right answer:

• High-volume classification or extraction where unit cost matters and the task is bounded. Fine-tune a small model to do it; ship the small model.
• Specific output formats or styles that prompting cannot reliably produce.
• Domain language where the base model’s vocabulary is off (specific scientific, legal, or technical domains).
• Latency-critical paths where a small fine-tuned model can replace a long prompt + large model.
• Cost optimization at scale where the upfront fine-tuning investment amortizes over millions of calls.

Outside these cases, fine-tuning is usually overkill. The right answer is more often prompting + RAG.

The take

Prompting, RAG, and fine-tuning solve different problems. Prompting is the default. RAG fills information gaps. Fine-tuning fills behavior gaps at scale. The decision tree is mechanical: try prompting, add RAG if information is the constraint, add fine-tuning if scale or shape requires it. Most teams reach for fine-tuning too early and end up paying the operational cost of a trained model when prompt engineering would have done the job. Reverse the default. Reach for the simpler technique first; complicate only when necessary.

---

Technique selection is part of the architecture conversation in every Custom AI Build. Schedule a call if you’re staring at a fine-tuning project and want to validate whether it’s really the right move.