Evaluating retrieval

Several pages on this site tell you to measure retrieval against your own data rather than trusting a number from a tutorial. This is the page that shows you how.

It is genuinely an afternoon’s work, and it changes how you make every subsequent decision. Without it you’re choosing models by vibes, and “the new one feels better” is not something you can defend in a review.

The problem with judging by eye

You type a few queries, the results look sensible, you ship. Two things go wrong with this.

You test the queries you thought of. They’re the ones the system already handles, because you had them in mind while building it. Real users ask differently, and their failures are invisible to you.

You can’t tell whether a change helped. You switch to a bigger model. Results still look fine. Better? Worse? Identical? No idea, and you’ve now committed to double the storage cost on a hunch.

The fix is small: write down the right answers once, then measure against them forever.

Build a golden set

A golden set is a list of real queries paired with the chunk that should come back. Fifty is enough to be useful. A hundred is comfortable. You do not need thousands.

1. Get real queries.

   Search logs are ideal. No logs yet? Ask three colleagues for ten questions each and take the support inbox for the rest. What matters is that you didn’t invent them — your imagination is exactly the bias you’re trying to escape.

2. Include the awkward ones.

   A set of easy queries proves nothing. Deliberately include: questions using different words from the document, questions containing a product code, questions where the answer spans two sections, and questions your corpus genuinely can’t answer. That last group matters — you want to know how the system behaves when there is no right answer.

3. Find the right answer by hand.

   For each query, note which chunk actually answers it. Yes, manually. It’s the boring part and it’s the whole value. Budget an afternoon.

4. Store it as data, not a document.

   golden-set.json

   [
     { "query": "how much holiday do I get",        "expected": ["handbook-4.1"] },
     { "query": "who do I tell if I'm off sick",    "expected": ["handbook-4.2"] },
     { "query": "error ERR-4021",                   "expected": ["errors-4021"] },
     { "query": "can I expense a train to a client","expected": ["handbook-7.3", "handbook-7.4"] },
     { "query": "what is the CEO's home address",   "expected": [] }
   ]

   Note the last row. An empty expected means nothing should score highly, and it’s how you catch a system that confidently returns rubbish rather than admitting defeat.

Keep it in the repo

Commit the golden set next to your code and run it in CI. Retrieval quality then behaves like any other test: a change that drops recall shows up in the pull request rather than in production three weeks later.

The three numbers worth knowing

You don’t need the whole information-retrieval literature. Three metrics cover almost every practical decision.

Recall@k — “did we find it at all?”

Of the queries with a right answer, how often was it somewhere in the top k?

This is the one that matters most for RAG, because the generation step can only use what retrieval hands it. If the right chunk isn’t in the top 5, no prompt will save you.

def recall_at_k(results: list[str], expected: list[str], k: int = 5) -> float:
    """Fraction of expected documents that appear in the top k."""
    if not expected:
        return 1.0                      # nothing to find; not a failure
    found = set(results[:k]) & set(expected)
    return len(found) / len(expected)

MRR — “how far down was it?”

Mean Reciprocal Rank. Scores 1.0 if the right answer was first, 0.5 if second, 0.33 if third. It captures something recall ignores: position matters when a human is reading the list.

def reciprocal_rank(results: list[str], expected: list[str]) -> float:
    """1/rank of the first correct result, or 0 if none appeared."""
    for position, doc_id in enumerate(results, start=1):
        if doc_id in expected:
            return 1 / position
    return 0.0

Use recall@k when a model consumes the results. Use MRR when a person does.

Precision@k — “how much noise came with it?”

What fraction of the top k were actually relevant. Matters when you’re paying per token to put chunks in a prompt, or when noise is crowding out the good chunk.

What about NDCG?

NDCG handles graded relevance — this chunk is perfect, that one is partly useful, this one is unrelated. It’s the right metric when relevance genuinely isn’t binary, and it’s what you’ll see in academic papers.

For a first golden set, skip it. Binary right-or-wrong labels are far quicker to produce and will tell you almost everything you need. Reach for NDCG once binary labels start feeling dishonest.

Running it

evaluate.py

import json, statistics

def evaluate(search, golden_set, k: int = 5):
    """`search` takes a query string and returns a ranked list of chunk ids."""
    recalls, rrs, misses = [], [], []

    for case in golden_set:
        results = search(case["query"])
        recalls.append(recall_at_k(results, case["expected"], k))
        rrs.append(reciprocal_rank(results, case["expected"]))
        if case["expected"] and recall_at_k(results, case["expected"], k) == 0:
            misses.append(case["query"])

    return {
        f"recall@{k}": round(statistics.mean(recalls), 3),
        "mrr": round(statistics.mean(rrs), 3),
        "total_misses": len(misses),
        "missed_queries": misses,
    }


if __name__ == "__main__":
    golden = json.load(open("golden-set.json"))
    print(json.dumps(evaluate(my_search, golden), indent=2))

{
  "recall@5": 0.82,
  "mrr": 0.71,
  "total_misses": 9,
  "missed_queries": [
    "error ERR-4021",
    "who signs off contractor invoices",
    ...
  ]
}

missed_queries is the most valuable field in that output. The averages tell you where you stand; the misses tell you what to do next. Read them. They cluster, and the cluster is usually a fixable class of problem: all the exact-code queries failing means add keyword search, all the multi-section answers failing means fix chunking.

Using the numbers

Establish a baseline before you optimise. Run keyword search through the same harness first. If BM25 gets recall@5 of 0.78 and your embedding pipeline gets 0.81, you now know something genuinely useful, and it may not be the answer you were hoping for.

Change one thing at a time. Chunk size, model, k, hybrid on or off. Re-run. Keep a table. This is dull and it is how you end up able to explain your architecture.

Pick your threshold here, if you must have one. This is where the advice on the cosine similarity page cashes out. Plot the scores of correct results against incorrect ones across your golden set and see whether a cut-off actually separates them. Often it doesn’t, which is itself worth knowing before you hard-code 0.8 and wonder why things vanish.

Two ways to fool yourself

Tuning on your test set. If you tweak parameters until the golden set looks good, you’ve fitted to fifty queries rather than improved retrieval. Hold back a slice you only look at occasionally, or add fresh queries periodically.

Only counting the queries you can answer. If you quietly drop the questions your corpus can’t handle, your numbers improve and your system doesn’t. Keep them in, with empty expected, and check that nothing scores highly.

Evaluating the generated answer

Everything above measures retrieval. If you’re building RAG, you may also want to know whether the final answer was any good — a much harder problem, since there are many correct phrasings.

Two practical approaches:

• Faithfulness checking. Ask a second model whether every claim in the answer is supported by the retrieved context. Catches the specific failure where retrieval worked and generation invented anyway.
• A small human-graded set. Twenty questions, graded by someone who knows the domain, re-run at each release. Unfashionable, slow, and still the most trustworthy signal available.

Fix retrieval first, though. An answer-quality score that’s really measuring a retrieval miss will send you optimising the wrong half of the system.

Next

Going to productionCost, caching, re-embedding migrations and what to monitor.

RAG, without the mystiqueThe pattern these metrics exist to protect.

Chunking and vector databasesWhere most retrieval misses are actually created.

References

• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR Models — Thakur et al., the standard retrieval benchmark suite
• MTEB leaderboard — how published models compare, and which metrics they report
• Mean reciprocal rank and Discounted cumulative gain — formal definitions
• Evaluation measures in information retrieval — the wider family, if you need more than three
• Ragas — a library for faithfulness and answer-quality scoring
• TREC — decades of retrieval evaluation methodology, where most of this originated