RAG (retrieval-augmented generation) makes an AI model answer from your own documents. When someone asks a question, the system first searches your files for the most relevant passages, then gives them to the model with an instruction to answer only from them. Answers are grounded in your policies, prices and manuals, ideally citing the source.
Think of it as an open-book exam. The model is a capable writer, but it hasn't read your handbook. RAG hands it the right pages just before it answers.
Why can't AI just answer from my documents on its own?
General AI models know a lot about the world but nothing about your business unless you give it to them at the moment of the question. Without that context, they guess, and guesses sound confident.
Ask a general chatbot "What's the cancellation fee for a December booking?" and it will produce something plausible that may be completely wrong for you. RAG fixes this by supplying the actual cancellation policy alongside the question, and instructing the model to say "I don't know" if the answer isn't there.
How does RAG work, step by step?
RAG has two halves: preparing your documents once (and whenever they change), and retrieving from them every time someone asks.
Preparation:
- Collect the source material: PDFs, web pages, FAQs, policy documents, product sheets, help articles.
- Clean and split it into chunks, typically a few paragraphs each, keeping headings and context attached.
- Create embeddings for each chunk. An embedding is a list of numbers that represents meaning, so passages about "refunds" and "getting my money back" end up close together.
- Store the chunks and embeddings in a vector database, a search index built for finding similar meanings, along with labels like document name, date and who's allowed to see it.
At question time:
- Convert the question into an embedding the same way.
- Retrieve the handful of chunks closest in meaning, often combined with ordinary keyword search for names, codes and numbers.
- Generate an answer: the model receives the question, the retrieved chunks and instructions ("answer only from these sources; cite them; if unsure, say so and offer a human").
- Show the answer with links or references to the source documents.
Most of the quality comes from steps 2, 6 and 7, not from which model you pick.
How is RAG different from fine-tuning or pasting documents into a prompt?
RAG fetches knowledge on demand; fine-tuning changes how a model behaves; pasting everything into the prompt works only for small, fixed content.
| Approach | How it works | Best for | Weak spot |
|---|---|---|---|
| Paste into the prompt | Put the whole document in every request | A short FAQ or a single policy | Costly and slower as content grows; no access control |
| RAG | Search documents, send only relevant passages | Large or changing knowledge, many documents | Answers are only as good as retrieval and source quality |
| Fine-tuning | Further train a model on examples | Consistent style, format or classification | Poor way to add facts that change; retraining needed |
| RAG + light tuning or detailed prompts | Retrieval for facts, prompts or tuning for tone | Mature customer-facing assistants | More engineering and testing |
For most businesses, RAG plus well-written instructions is the right starting point. Fine-tuning is rarely needed to teach facts, and it's harder to update when prices or policies change.
What can a RAG system do for a business?
Anywhere people repeatedly look things up in documents, RAG can answer faster. Common uses:
- Customer-facing assistants that answer from your FAQs, packages, terms and policies, then capture the lead. See how this fits support teams in AI customer support: what it can handle and what still needs humans.
- Staff knowledge assistants that answer "How do we process a return?" or "What's our travel charge outside the city?" from your SOPs.
- Sales support, pulling the right case study, spec sheet or pricing rule into a proposal.
- Compliance-heavy checks, such as an immigration consultancy checking a client's situation against its own internal checklists. The assistant prepares; a qualified person decides.
- Technical documentation, letting customers or field staff query manuals by describing the problem.
What makes RAG answers accurate or wrong?
Bad sources, poor chunking and weak retrieval cause most wrong answers; the model is usually doing what it was told with the passages it was given.
| Problem | What it looks like | Fix |
|---|---|---|
| Outdated or conflicting documents | Quotes last year's price | Remove old versions; add dates; set one source of truth |
| Chunks cut in the wrong place | Answers miss conditions or exceptions | Split by headings and sections, keep context with each chunk |
| Scanned PDFs and images | Missing or garbled text | Run proper text extraction; check the output |
| Tables and price grids | Numbers attached to the wrong item | Convert tables to structured text or data |
| Retrieval misses exact terms | Can't find product codes or names | Add keyword search alongside meaning-based search |
| Model fills gaps | Plausible but unsupported answers | Strict instructions, citations, "I don't know" path, human handover |
| Wrong person sees wrong data | Staff or customers see restricted content | Filter by permissions before retrieval, not after |
Two practices make the biggest difference:
- Build a test set. Collect 50–100 real questions people have asked, with the correct answers. Run them after every change to documents, prompts or models, and track how many pass.
- Show sources. When users can see where an answer came from, errors get spotted and reported quickly.
Is RAG safe for confidential documents?
It can be, if access control and data handling are designed in from the start. The main questions to settle:
- Permissions: Should every user see every document? If not, tag chunks with access rules and filter before retrieval.
- Provider terms: Use model providers under business API terms, and check their current data-retention and training policies.
- Hosting and region: Decide where the vector database and logs live, especially if customers are covered by India's DPDP Act 2023, Australia's Privacy Act 1988 or the GDPR.
- Prompt injection: A document could contain text that tries to instruct the AI. Treat retrieved content as information, never as commands.
- Logs: Keep question-and-answer logs for quality review, but redact sensitive details and set a retention period.
How much does a RAG system cost?
An AI chatbot or assistant that answers from your own documents typically costs ₹2–8 lakh in India, AUD 15k–60k in Australia, or US$10k–40k globally. These are typical market ranges; the final figure depends on scope.
What pushes it up: lots of messy or scanned documents, per-user permissions, several channels, multiple languages and integrations that let the assistant take actions. Ongoing costs include model usage, vector database hosting, re-indexing when documents change, and maintenance of roughly 15–25% of the build per year. For tier-by-tier detail, see how much a custom AI chatbot costs.
If your documents are few and non-sensitive and only your team needs answers, a tool like a custom GPT with uploaded files may be enough to start. Our comparison of custom GPTs and custom AI apps explains when to move up. For the bigger picture, see our practical guide to AI automation for small and mid-size businesses.
How do you prepare your documents for RAG?
Clean, current, clearly structured documents do more for accuracy than any technical trick. Before a build:
- List every document the assistant should use, with an owner for each.
- Delete or archive outdated versions.
- Add clear headings, and date anything time-sensitive.
- Write down answers that currently live only in people's heads.
- Mark anything confidential or restricted to certain roles.
- Collect the real questions customers and staff ask; they become the test set.
Frequently asked questions
Does RAG train the AI on my data?
No. RAG doesn't change the model; it supplies relevant passages at the moment of each question. Whether a provider may use API data for training depends on its terms, so check your provider's current business data policy.
How often do documents need re-indexing?
Whenever they change. Good systems re-index automatically when a document is added, edited or removed. Stale indexes are a common reason assistants quote old prices or policies.
Can RAG handle spreadsheets and price lists?
Yes, but tables need careful handling so numbers stay attached to the right items. For data that changes often, such as stock or live pricing, it's often better to query your database directly through an integration rather than index a spreadsheet.
Will a RAG chatbot ever make things up?
It can, especially when retrieval finds nothing relevant. Strict instructions to answer only from sources, visible citations, a clear "I don't know" response and regular testing reduce this significantly.
How long does it take to build a RAG assistant?
As a rough estimate, one to three months for a focused assistant, depending mostly on how ready your documents are and how many systems it connects to. Document preparation often takes longer than the build itself.
Put your documents to work
If your team or customers keep asking questions your documents already answer, a RAG assistant can help, provided the sources are right. We'll review your content, tell you what's realistic and quote a fixed price. Learn more on our services page, or start the conversation here.