Definition
Retrieval-augmented generation, or RAG, is a method for making a generative AI model answer questions using specific, trusted information instead of relying only on what it memorised during training. When a user asks a question, a RAG system first retrieves the most relevant passages from a collection of documents — a policy handbook, a product knowledge base, contracts, support tickets — and then passes those passages to the language model along with the question. The model generates its answer from that supplied context and can cite where each point came from.
The approach was formalised in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Patrick Lewis and colleagues at Facebook AI Research (now Meta), University College London and New York University. It has since become the standard way companies connect AI assistants to their own data. Microsoft Copilot answering from your SharePoint files, a customer-service bot quoting your help centre, and ChatGPT's file-upload and search features all rely on retrieval-based patterns.
Why RAG exists
Large language models have three limits that matter in business:
- They do not know your private data. Your HR policies, pricing and processes were never in their training set.
- Their knowledge has a cut-off. Anything that changed after training is invisible to them.
- They hallucinate. Without sources, they fill gaps with plausible inventions.
Retraining or fine-tuning a model every time a document changes would be slow and expensive. RAG solves the problem at question time: keep the model as it is, and simply hand it the right pages whenever someone asks.
How RAG works, step by step
- Prepare the documents. Files are split into chunks of a few hundred words.
- Create embeddings. Each chunk is turned into an embedding — a list of numbers that captures its meaning — and stored in a vector database or search index.
- Retrieve. When a question arrives, it is also turned into an embedding, and the system finds the chunks whose meaning is closest. Many systems combine this semantic search with classic keyword search.
- Augment. The top chunks are inserted into the prompt with instructions such as "Answer only from the sources below and cite them."
- Generate. The LLM writes the answer, ideally with references to the source documents.
Example
A 2,000-person company receives hundreds of repetitive questions to its HR inbox every month: "How many days of parental leave do I get?", "Can I carry over unused holiday?", "What is the expense limit for client dinners?" The answers are spread across a 90-page handbook, country-specific addenda and a dozen intranet pages.
The HR team builds a RAG assistant on their approved enterprise AI platform. They load the current handbook and policies, and set the instruction: "Answer employee questions using only the documents provided. Quote the policy section. If the documents do not cover the question, say so and suggest contacting HR."
An employee in Germany asks about carrying over holiday. The system retrieves the German addendum and the general leave policy, and the assistant replies with the exact rule, the deadline and a link to section 4.3. When someone asks about a topic not covered, the assistant says it cannot find the answer and routes them to a person. The HR team still reviews a sample of answers each week and updates documents when policies change, which instantly updates the assistant — no retraining needed.
Benefits of RAG
- Grounded answers with citations that users can check.
- Up-to-date knowledge: update the document, and the answers update.
- Lower hallucination rates compared with asking a model from memory.
- Access control: well-built systems only retrieve documents the user is allowed to see.
- Cost: far cheaper than training a custom model.
Limits and pitfalls
RAG is not a guarantee of truth. If retrieval finds the wrong chunk, the model will confidently answer from the wrong source. Outdated, duplicated or contradictory documents produce contradictory answers — "garbage in, garbage out" still applies. Tables, scanned PDFs and images can be hard to chunk well. Permissions must be designed carefully so that the assistant never reveals salary data or confidential files to the wrong person. And the model can still occasionally add details not in the sources, so high-stakes answers need human review.
RAG versus fine-tuning
Fine-tuning changes the model's behaviour or style by training it further on examples. RAG changes what the model knows at the moment of answering. A useful rule: use RAG when the problem is "the model doesn't know our facts"; consider fine-tuning when the problem is "the model doesn't behave or write the way we need". Many production systems use both.
Key takeaways
- RAG retrieves relevant documents and gives them to the model at question time.
- It is the standard way to ground AI assistants in company knowledge.
- Quality depends on clean, current documents and good retrieval.
- Citations plus human review keep RAG answers trustworthy.