Ask a general AI assistant about your own returns policy or the payment terms in a supplier contract and it will either say it doesn't know or, worse, make something up that sounds right. Retrieval augmented generation, usually shortened to RAG, is the standard way to fix that. It lets an AI system look things up in your own documents before it answers, and show you where the answer came from.
If you're weighing an internal assistant, a customer chatbot or a tool that reads contracts, you will hear "RAG" in every proposal. Here is what it means in plain terms, what it fixes, and what it doesn't.
The short version
A large language model learns from a huge amount of public text, then stops learning. It knows nothing about your price list, your staff handbook or last week's policy change.
RAG adds a search step. When someone asks a question, the system first finds the most relevant passages in your documents, then hands those passages to the model with an instruction along the lines of: answer using only this material, and say which part you used.
Think of the difference between asking a new hire a question from memory and asking them to answer with the policy binder open in front of them. The second is slower to set up and far more reliable.
The idea isn't new. Researchers at Facebook AI Research, University College London and New York University described the approach in 2020, showing that pairing a language model with a searchable store of documents produced more specific and factual answers than the model alone.
How retrieval augmented generation works, step by step
Collect the sources. You choose which documents count: policies, manuals, contracts, product sheets, past tickets. This choice matters more than any technical setting.
Split and index them. Documents are cut into passages and stored in a search index that can match by meaning, so a question about "time off" finds the section headed "annual leave."
Retrieve. When a question arrives, the system pulls the handful of passages that match it most closely, filtered by what this user is allowed to see.
Generate. The model writes an answer from those passages only.
Cite. The answer links back to the document and section it used, so a person can check it in seconds.
When a document changes, you update the index, not the model. That is a big reason many business AI assistants are built this way: keeping answers current becomes a matter of updating documents.
What a good answer looks like
Say a site manager types: "Can contractors use our forklifts?" A well-built RAG assistant replies with a short, direct answer, quotes the relevant line from the plant safety procedure, and links to section 4.2 of that document. If the procedure doesn't cover contractors, it says so plainly and suggests who to ask.
Compare that with a general AI tool, which might give a confident, generic answer about forklift safety that has nothing to do with your rules. Both may use the same underlying model. The RAG assistant looked in your documents first and showed its work.
Five myths about RAG
Myth: RAG stops AI from making things up
Fact: It reduces the problem. It doesn't remove it. The U.S. National Institute of Standards and Technology lists "confabulation," AI producing confident but false content, as one of the main risks of generative AI in its Generative AI Profile.
A Stanford and Yale study of commercial legal research tools built on RAG found they still produced incorrect or misgrounded answers between 17% and 33% of the time in its tests, despite marketing claims of being hallucination-free. Citations, confidence checks and human review for anything that matters are still needed.
Myth: You need to train your own AI model
Fact: For most businesses, no. RAG works with an existing model and your own documents. Training or fine-tuning a model is expensive, goes stale as your documents change, and makes it hard to show where an answer came from. RAG keeps your knowledge in documents you control.
Myth: Just point it at the shared drive
Fact: The answers are only as good as the documents. If the drive holds three versions of the expense policy, the system may quote the wrong one. Most RAG projects spend real time deciding which documents are authoritative, retiring old ones and naming an owner for each. That work improves life for staff whether or not AI is involved.
Myth: Everyone can safely ask it anything
Fact: Access control has to be built in. If the index contains salary data or board papers, a careless setup can surface them to anyone who asks the right question. The OWASP Gen AI Security Project lists weak access controls on the stores behind RAG among the top security risks for AI applications. The fix is to filter results by the user's own permissions before the model ever sees them.
Myth: More documents always means better answers
Fact: Not always. A focused, well-kept set of sources usually beats a huge pile. Start with one department's documents and one type of question, measure the answers, then widen.
Where RAG fits in a business
RAG is a good fit when the answer already exists in writing and people spend time finding it. Common examples for operations-heavy firms:
Staff help desk: HR policies, IT how-tos and safety procedures answered in a chat window, with the policy linked.
Customer support: product specs, warranty terms and delivery rules answered on your website, with handover to a person for exceptions.
Contracts and compliance: "Which supplier contracts renew in the next 90 days and what are the notice periods?" answered across hundreds of PDFs.
Technical manuals: field and plant staff asking how to reset a machine, with the manual page attached.
It is a poor fit when the answer needs fresh calculation across live data (that is reporting, or a connection to your ERP), or when the honest answer depends on judgment rather than a document.
Questions to ask before you approve a RAG project
Which documents will it answer from, and who keeps them current?
Does every answer show its source?
What does it do when the documents don't contain the answer?
How are permissions enforced, so people only get answers from documents they could already open?
Where do the documents, the index and the chat logs live? Ideally, in your own cloud account.
Is the AI service set so it doesn't train on your data?
How will accuracy be tested before launch, and on what set of real questions?
Frequently asked questions
Is retrieval augmented generation the same as a chatbot?
No. A chatbot is the conversation window. RAG is the method that lets the chatbot answer from your documents. Many chatbots use RAG, and RAG also powers tools with no chat at all, such as a system that drafts replies or summarizes contracts.
Do our documents get sent to the AI provider?
The passages relevant to each question are sent to the model to write the answer. That is why the provider's terms matter: use business services with training switched off, or, for sensitive work, models hosted in your own cloud.
How accurate will it be?
That depends mostly on your documents and how the system is tested. Build a set of real questions with known answers before launch and score the system against it. Anything that will be sent to a customer, or that changes money or a contract, should still be approved by a person.
How long does a first version take?
A narrow first version on one department's documents is a matter of weeks, not months. Most of the time goes into choosing and cleaning the sources and testing the answers.
Where to start with retrieval augmented generation
We build AI agents and chatbots that answer from your own documents, show the source behind every answer and pass anything uncertain to your team. If you want to see what RAG could do for one department, the free 5-day audit ends with a short list of tasks and a fixed price for the first one. Or start with a 30-minute call with a founder.
