How RAG helps AI answer from company documents with citations, access controls and a clear evaluation process.
- RAG does not retrain the model; it retrieves relevant documents and supplies that context when generating an answer.
- Quality depends on documents, chunking, metadata, access controls and a representative evaluation set.
- Every important answer should cite its source and refuse when the evidence is insufficient.
How does RAG work?
RAG stands for Retrieval-Augmented Generation. When a user asks a question, the system does not rely only on knowledge inside the model. It searches the company's knowledge store for relevant passages, supplies those passages with the question and asks the model to answer from the retrieved evidence.
A basic pipeline ingests documents, cleans and chunks them, creates searchable representations, stores an index, retrieves relevant material and generates an answer. Vector databases are common for semantic search, but they are not the whole solution. Structured data such as product codes, effective dates and contract status may require metadata filters or ordinary database queries.
- Ingestion: collect and normalise source documents.
- Retrieval: find material relevant to the question.
- Generation: answer within the retrieved evidence.
RAG does not turn disorganised documents into trusted knowledge. It makes the quality—and the disorder—of the source material more visible.
Which problems suit RAG?
RAG works well when answers live in documents that change: internal policy, product catalogues, operating guidance, project records, contract templates and support knowledge. Compared with putting every document into a prompt, retrieval reduces context, lets teams update knowledge without retraining and creates a path for users to verify the source.
It is not suitable for every problem. Exact calculations, transaction state and data-changing actions should use deterministic tools and APIs. When documents conflict or have no owner, a chatbot can produce a plausible answer from the wrong source. The first problem is then knowledge governance, not model selection.
- Internal policy and process questions.
- Sales access to specifications, case studies and product material.
- Permission-aware knowledge search for employees or customers.
Prepare the data before building the chatbot
Create a source inventory: which source is authoritative, who owns it, how often it changes and which groups may view it. Remove duplicates, expired material and files without a clear version. Attach metadata such as department, product, language, effective date and sensitivity so retrieval can use more than text similarity.
Chunking should follow meaningful structure. Very small chunks lose context, while very large chunks dilute relevance and increase cost. Preserve headings, section numbers and relationships so an answer can point to the right place. For tables, forms and scanned PDFs, inspect extraction quality rather than assuming the system read them correctly.
- A named owner and update rhythm for every important source.
- Metadata for access, version and context filters.
- Manual inspection of representative ingested content.
Evaluate quality beyond a few demo questions
Build an evaluation set from real situations, including easy questions, multi-step questions, questions with no answer and questions the user is not authorised to answer. Record the expected source, required points and refusal conditions. Run the same set whenever the model, embedding, chunking or prompt changes.
Separate retrieval failures from generation failures. If the system does not find the right source, prompt changes rarely solve the root problem. If it retrieves the right source but answers incorrectly, review instructions, context limits and model behaviour. Also track citation validity, source coverage, user feedback and indexing delay after a document changes.
- Retrieval: is the correct source among the top results?
- Groundedness: is the answer supported by those sources?
- Abstention: does the system refuse when evidence is missing?
Security and a four-week pilot
Enforce permissions before retrieval rather than merely telling the model that a document is secret. A user should search only sources they are allowed to view. Logs can retain the question, source IDs and outcome without duplicating all sensitive text. Treat external documents as untrusted input because they can contain instructions intended to manipulate the model.
Over four weeks, select one owned knowledge set, build ingestion and retrieval, create the evaluation set and open the system to a small group. The first goal is not to answer everything. It is to answer one valuable scope correctly, cite the evidence and refuse safely. Add more sources, tools or actions only after that evidence exists.
- Week 1: sources, permissions and reference questions.
- Week 2: ingestion, retrieval and citation interface.
- Weeks 3–4: pilot, failure review, data fixes and a scale decision.
FAQ
Frequently asked questions
Is RAG the same as fine-tuning?
No. RAG supplies relevant documents at answer time; fine-tuning changes model behaviour or output patterns through training examples.
Does RAG require a vector database?
No. Vector search is useful for meaning-based retrieval, but many systems combine it with keyword search, metadata filters and structured queries.
Does RAG eliminate hallucinations?
Not completely. It improves grounding, but still requires evaluation, citations, refusal rules and access controls.
References
Sources used in this guide
We prioritise official guidance and primary technical sources. Visit each source for full context and the latest updates.