Retrieval-augmented generation, or RAG, has become one of the most talked-about ideas in enterprise AI. Most of the explanations written about it are aimed at the people who build it: data scientists, ML engineers, platform teams. Far fewer are written for the person who actually has to decide whether it's worth funding. This is that version.
What RAG actually is
Strip away the acronym and RAG is a fairly simple idea. Instead of asking a large language model to answer a question purely from what it learned during training, you first retrieve the most relevant material from your own documents, systems, or knowledge base, and hand that to the model alongside the question. The model then answers using what it was just given, not just what it happened to absorb from the public internet.
That matters because it changes where the answer comes from. A general-purpose model has no idea what your latest procurement policy says, what changed in your product last quarter, or what your internal handbook actually states about expenses. Retrieval gives it that context on demand, without retraining the model itself.
Why it's not the same as "just use ChatGPT"
A lot of the scepticism around AI in organisations comes from a reasonable place: general chat tools can sound confident while being wrong, and nobody wants that confidence pointed at a customer or a regulator. RAG doesn't remove that risk entirely, but it narrows it considerably, because the model is being asked to work from a defined, current set of source material rather than guessing from memory.
It also solves a problem that fine-tuning a model doesn't: currency. Fine-tuning bakes knowledge into the model at a point in time, and that knowledge goes stale the moment your policy, pricing, or product changes. Retrieval pulls from a live source, so updating the underlying documents updates what the system can answer, without retraining anything.
Where it earns its keep
The organisations getting real value from RAG tend to have one thing in common: a large body of internal knowledge that's hard to search today. A few patterns come up repeatedly:
- Internal knowledge search. Staff spending time hunting through policy libraries, procedural documents, or SharePoint sites for an answer that already exists somewhere in the organisation.
- Customer or citizen support grounded in real documents. Answering questions against actual terms, policies, or product documentation, with the source cited, rather than a generic chatbot response.
- Compliance and regulated-sector queries. Situations where an answer needs to be traceable back to a specific clause or document, not just plausible-sounding.
This is technology that's mature enough to deploy properly today, including fully on-premise where data residency and governance require it. Most organisations that would benefit simply haven't had someone build it for them correctly yet.
Where it doesn't help
RAG is not a fix for messy data. If your source documents are outdated, contradictory, or scattered across five systems with no consistent ownership, retrieval will faithfully surface that mess back to the user, just with more confidence. It's also not a way to eliminate hallucination outright: a model can still misread or over-generalise from what it retrieves, which is why evaluation and human review matter as much as the retrieval pipeline itself.
And it isn't free to maintain. Documents need to be kept current, access permissions need to carry through to what the model can see, and someone needs to own the quality of the answers over time. Treating it as a one-off build rather than an ongoing capability is the most common way these projects underdeliver.
Questions worth asking before you invest
- Is our source material actually retrievable? Structured, reasonably current, and not locked in formats or systems nobody can query.
- Who can see what? Retrieval needs to respect the same access controls your documents already have, not flatten them.
- How will we know if an answer is wrong? What does evaluation and ongoing monitoring look like once this is live, not just at launch.
- Where does the data live? Whether it needs to stay on-premise or within a specific jurisdiction, and what that means for the architecture.
- Who owns this after go-live? Retrieval quality degrades if nobody is responsible for keeping the underlying content current.
The honest starting point
RAG is a genuinely useful pattern, not a trend to be sceptical of by default. But it rewards organisations that scope it carefully far more than it rewards ones that jump straight to a full rollout. A well-bounded pilot against one knowledge base, with clear evaluation criteria and a named owner, tells you more in a few weeks than a large programme tells you in a year.
If you're weighing whether this is worth pursuing and want to think it through with someone who isn't trying to sell you a platform first, that's a conversation we're happy to have.