Almost every enterprise LLM conversation eventually arrives at the same question: should we fine-tune a model, or build retrieval-augmented generation (RAG) on top of a general-purpose one? The honest answer is that they solve different problems, and most production systems end up using both.
What RAG Is Actually Good At
RAG grounds a model's responses in retrieved documents at inference time. It is the right default when:
- The underlying knowledge changes frequently (pricing, policies, inventory, documentation)
- You need citations or traceability back to a source document
- You want to add or remove knowledge without retraining anything
- The domain vocabulary is close enough to the base model's training distribution that it doesn't need new "skills," just new facts
What Fine-Tuning Is Actually Good At
Fine-tuning changes how the model behaves, not just what it knows. It earns its cost when:
- You need a consistent output format, tone, or reasoning style that prompting can't reliably enforce
- The task requires a skill the base model is weak at — domain-specific classification, structured extraction from messy documents, a narrow coding convention
- Latency or cost constraints rule out long retrieved contexts at inference time
- You have enough high-quality labeled examples to actually move the needle — sub-hundreds of examples rarely justify the exercise
The Decision Framework We Use
- Start with the failure mode. Is the model wrong because it doesn't know something, or because it doesn't behave the way you need? The former points to RAG, the latter to fine-tuning.
- Estimate data freshness. If the answer changes weekly, fine-tuning will always be stale. RAG stays current by construction.
- Price out the context window. Long retrieved contexts on every request get expensive at scale — sometimes a lightly fine-tuned smaller model plus a thinner retrieval layer is cheaper and faster than a large general-purpose model with a heavy RAG pipeline.
- Check your evaluation data before committing either way. Both approaches are only as good as the offline eval set you're using to compare them.
Why Hybrid Wins More Often Than Either Alone
In most of our enterprise deployments, the winning architecture is a lightly fine-tuned model — tuned for format, tone, and domain-specific instructions — sitting on top of a RAG pipeline for facts that change. Fine-tuning handles how the model responds; retrieval handles what it knows. Neither approach alone tends to survive contact with a real enterprise knowledge base for very long.
Common Mistakes
- Fine-tuning on FAQ pairs instead of just putting the FAQ in a retrieval index — this is almost always the wrong tool for the job
- Skipping a retrieval evaluation harness and judging RAG quality by eyeballing a handful of outputs
- Treating chunking strategy as an afterthought — poor chunking undermines even a well-tuned model far more often than model choice does