RAG, without the mystique

RAG gets talked about like an architecture. It’s a sentence:

Before you ask the model a question, search your own documents and paste the best bits into the prompt.

That’s it. That’s the pattern. The “retrieval” half is the search you’ve already built in the language guides. The “augmented generation” half is string concatenation.

The acronym is doing an enormous amount of reputational work for what is, structurally, a very ordinary idea.

Why bother

A language model knows what was in its training data. It does not know:

• Your company’s holiday policy
• What your customer said in a support ticket yesterday
• Anything that happened after its training cutoff
• Anything behind your login

Ask anyway and you get a confident, plausible, invented answer — because the model’s job is to produce likely-sounding text, and it has no mechanism for noticing that it doesn’t know.

RAG fixes this the boring way: put the facts in the prompt, then ask. The model stops recalling and starts reading.

Why not fine-tune instead?

Fine-tuning teaches a model style, format and behaviour — how to sound, what shape to answer in. It’s a poor way to teach it facts: the facts get blurred into the weights, you can’t cite them, you can’t delete one, and you retrain every time something changes.

RAG’s facts live in a database. You can update a document, delete a record on request, show the user where an answer came from, and change nothing about the model. For factual grounding, retrieval wins on nearly every axis that matters operationally.

The whole pipeline

Two halves that run at completely different times. Keeping that distinction clear in your head prevents most RAG design mistakes.

  AHEAD OF TIME (when a document changes)
  ┌──────────────────────────────────────────────────┐
  │  document → chunk → embed → store with metadata  │
  └──────────────────────────────────────────────────┘

  AT QUESTION TIME (per request)
  ┌──────────────────────────────────────────────────┐
  │  question → embed → search → rerank →            │
  │             assemble prompt → model → answer     │
  └──────────────────────────────────────────────────┘

Everything on the top line you’ve already met in chunking and vector databases. The bottom line is this page.

The code

Short, because there genuinely isn’t much to it. This uses Python for brevity; the retrieval half is identical to what the C#, Java and TypeScript guides build.

rag.py

from openai import OpenAI
import numpy as np

client = OpenAI()

PROMPT = """Answer the question using only the context below.
If the context doesn't contain the answer, say "I don't know" — do not guess.
Cite the source number you used, like [1].

Context:
{context}

Question: {question}"""


def answer(question: str, chunks: list[str], matrix: np.ndarray) -> str:
    # 1. Retrieve. Exactly the search from the Python guide.
    query = client.embeddings.create(
        model="text-embedding-3-small", input=[question]
    ).data[0].embedding
    query = np.array(query)
    query /= np.linalg.norm(query)

    top = np.argsort(matrix @ query)[::-1][:5]

    # 2. Assemble. Numbered, so the model can cite them.
    context = "\n\n".join(f"[{n + 1}] {chunks[i]}" for n, i in enumerate(top))

    # 3. Generate.
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": PROMPT.format(context=context, question=question)}
        ],
    )
    return response.choices[0].message.content

Three steps: retrieve, assemble, generate. If you were expecting a framework, this is why people say RAG is oversold as a concept — the concept is small. The difficulty is entirely in the quality of what comes back from step 1.

The prompt is doing real work

That template looks casual. Every line of it is load-bearing.

“using only the context below” — without this, the model happily blends retrieved facts with half-remembered training data, and you can no longer tell which is which.

“say I don’t know — do not guess” — models default to answering. You have to give explicit permission to decline, or you get a fluent answer built from nothing.

“Cite the source number” — the single highest-value line. Citations let a user check the answer, and they let you debug: if the answer is wrong but the citation is right, your prompt is the problem; if the citation is wrong, your retrieval is.

Numbering the chunks — gives the model something concrete to cite. Cheap, and it makes the instruction above actually work.

Retrieved text is untrusted input

This one gets missed constantly. If your documents contain anything a user can write — support tickets, comments, uploaded files, scraped pages — then somebody can put “ignore your previous instructions and…” into a document and wait for it to be retrieved.

Your prompt has now been written partly by an attacker. This is prompt injection, and it has no clean fix. What helps: keep retrieved content clearly delimited from instructions, never let the model’s output trigger actions without a check, and treat anything derived from retrieved text with the same suspicion as a form submission.

Bluntly, if the model can both read user-supplied documents and take real actions, you have a security problem to design around, not a prompt to tweak.

Where RAG actually goes wrong

Almost never in the generation step. Nearly every RAG failure I’ve seen is a retrieval failure wearing a costume.

1. The answer wasn’t retrieved.

   The model can’t use what it never saw. When someone reports a “hallucination”, check the retrieved chunks first — nine times in ten, the right chunk wasn’t in them, and the model did what you told it to do with what you gave it.

   This is why evaluating retrieval matters more than prompt tuning.

2. The answer was split across two chunks.

   The policy states the rule in one paragraph and the exception in the next. Your chunker cut between them. Retrieval returns the rule, and the answer is confidently, precisely wrong. No amount of prompt engineering recovers a fact that isn’t there.

3. You retrieved too much.

   More context is not better. Models attend well to the beginning and end of a long prompt and measurably less well to the middle — the Lost in the Middle result. Twenty mediocre chunks perform worse than four good ones, and cost more.

4. The index is stale.

   Someone updated the document; nobody re-embedded it. Your system now confidently cites last quarter’s policy. Re-indexing on change is part of the feature, not an optimisation.

5. The question wasn’t a retrieval question.

   “How many open tickets does Priya have?” cannot be answered by finding semantically similar text. It’s a COUNT with a WHERE. See alternatives to vector embeddings — some questions need a query, not a search.

Things that reliably help

In rough order of value for effort:

• Rerank the shortlist. Retrieve 20, run them through a cross-encoder, keep the best 4. Usually the largest single quality gain available, and it’s a few lines.
• Go hybrid. Merge keyword and vector results. Rescues every query that hinges on an exact term, code or name.
• Return citations to the user. Improves trust, and turns silent wrongness into something a person can catch.
• Log the retrieved chunks. Not just the answer. Without this you cannot debug a single complaint.
• Fix chunking before touching the prompt. It’s the least glamorous lever and usually the biggest.

Check whether you need RAG at all

If your entire corpus fits in a context window — a product manual, a policy document, one contract — you can skip all of this and paste it in. No vector database, no chunking strategy, no embedding pipeline, and none of the failure modes above.

Modern context windows are large. “Does this even need retrieval?” is a cheap question to ask before building a pipeline, and an expensive one to skip. More on this in alternatives.

What RAG does not do

It doesn’t make the model reason better. It doesn’t fix a model that’s bad at your task. It doesn’t guarantee truthfulness — it makes truthfulness possible by putting the facts within reach, and then the model still has to use them properly.

And it cannot rescue bad retrieval. If the right chunk never surfaces, RAG has given you an expensive, confident wrapper around a search that didn’t work.

Which is the honest summary of this whole page: RAG is only as good as your search. Everything else on this site is about making that search good.

Next

Evaluating retrievalHow to tell whether your search is actually finding the right chunks. The measurement RAG depends on.

Chunking and vector databasesThe ingest half of the pipeline, in detail.

Alternatives to vector embeddingsIncluding when to skip retrieval entirely.

Security and privacyPrompt injection, permissions, and PII in a retrieval system.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020, the paper that named it
• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., why stuffing the prompt backfires
• OpenAI — Retrieval guide and Embeddings guide
• OWASP Top 10 for LLM Applications — prompt injection is LLM01, and it is first for a reason
• Cohere — Rerank overview — the shortlist-reranking step
• LangChain and LlamaIndex — framework implementations, if you’d rather not assemble it yourself