What is a good recall at k for RAG?

There is no universal target, because it depends on the domain and how many chunks you send. Aim for the relevant chunk to appear in the top results for the large majority of important questions, and focus on the failures. Track recall over time rather than chasing a single number.

Can I use an LLM to evaluate retrieval?

Yes, as an aid. Language models can judge whether retrieved chunks are relevant to a question at scale. Validate their judgements against human labels on a sample first, watch for systematic bias, and recheck agreement periodically, especially after changing the judge model.

How do I evaluate RAG without labelled data?

Start by creating a small labelled set from real questions; even a few dozen help. Meanwhile, use production signals such as empty or low-score retrievals, rephrased questions, and user feedback, and sample queries for human relevance judgements. Grow the labelled set from the failures you find.

How to Evaluate RAG Retrieval Quality in Production

Evaluate RAG retrieval in production with an offline set of real questions labelled with their relevant chunks, measured by recall at k, precision, and ranking metrics such as MRR; judged relevance on samples of live queries; production signals such as empty retrievals and user feedback; and regression tests whenever chunking, embeddings, or indexes change.

Why evaluate retrieval separately

When a RAG answer is wrong, the cause is either retrieval, which sent the wrong context, or generation, which misused the right context. Measuring retrieval on its own tells you which part to fix. Many answer problems turn out to be retrieval problems.

Build a labelled test set

Collect real questions from logs, support tickets, or users, and for each record which documents or chunks contain the answer. Start with a few dozen and grow to a few hundred, covering common questions, rare ones, and known hard cases. Store the source references rather than chunk IDs where possible, so the set survives re-chunking.

Core offline metrics

Three families of metrics cover most needs. Recall matters most; the others refine the picture.

Recall at k

The share of questions where a relevant chunk appears in the top k results. The most important metric: if the answer is not retrieved, the model cannot use it.

Precision at k

The share of retrieved chunks that are relevant. Low precision wastes context and can distract the model.

MRR and nDCG

Ranking metrics that reward placing relevant chunks near the top, which matters when only a few chunks are sent.

Judging relevance at scale

Labelling every live query by hand is impractical. Sample queries regularly and have people judge whether retrieved chunks are relevant. Language models can assist with relevance judgements, but check their agreement with human labels before trusting them, and recheck periodically.

Production signals

Monitor queries with no results or only low similarity scores, which suggest coverage gaps. Track whether answers cite retrieved chunks, how often users rephrase the same question, explicit feedback such as thumbs up or down, and escalations to humans. Rising rates of any of these point to retrieval issues.

Latency and cost

Retrieval quality includes speed and size. Track retrieval latency, the number and size of chunks sent, and tokens per answer. A change that improves recall slightly but doubles context size may not be worth it.

Regression testing

Run the offline test set automatically whenever you change chunking, embedding models, index parameters, reranking, or filters. Compare metrics with the previous version and block changes that reduce recall on important question groups.

Diagnosing failures

For each failed question, check whether the relevant content exists in the index, whether it was split badly across chunks, whether metadata filters excluded it, whether the query wording differs from document wording, and whether stale content outranked current content. Each cause has a different fix: indexing, chunking, filters, hybrid search, or freshness.

Improving retrieval

Common improvements include structure-aware chunking, hybrid search combining keywords and embeddings, reranking the top candidates with a cross-encoder, query rewriting, metadata filters, and keeping the index in sync with sources. Measure each change against the test set before shipping.

Retrieval quality and cost

Better retrieval does more than improve answers. When the right chunk is found, the model needs less context, and on repeated questions the answer can come straight from retrieval. RetroDeck's stacked retrieval checks private chunks first, then the public network, and calls the LLM only as the fallback, so retrieval quality directly sets how often a model is called at all.

Segmenting results

Overall averages hide problems. Break metrics down by question type, source collection, user group, and language. A change that improves average recall can still hurt one important segment, such as questions about recent releases or a specific product line.

Connecting retrieval to answer quality

Track end-to-end answer accuracy alongside retrieval metrics on the same test set. When retrieval recall is high but answers are wrong, look at generation: prompts, context ordering, or model choice. When both are low, fix retrieval first.

Ownership and cadence

Assign someone to own retrieval quality. Review production signals weekly, rerun the full test set before releases, and refresh the labelled set quarterly with new real questions, so evaluation keeps pace with how users actually ask.

Evaluating access control

Retrieval quality includes returning only what the user may see. Add test cases where users without access ask questions about restricted content, and confirm retrieval returns nothing from those sources. A leak here is a security incident, not a relevance issue.

Freshness checks

Include questions whose answers changed recently and verify that the current version is retrieved, not the old one.

Frequently asked questions

What is a good recall at k for RAG?: There is no universal target, because it depends on the domain and how many chunks you send. Aim for the relevant chunk to appear in the top results for the large majority of important questions, and focus on the failures. Track recall over time rather than chasing a single number.
Can I use an LLM to evaluate retrieval?: Yes, as an aid. Language models can judge whether retrieved chunks are relevant to a question at scale. Validate their judgements against human labels on a sample first, watch for systematic bias, and recheck agreement periodically, especially after changing the judge model.
How do I evaluate RAG without labelled data?: Start by creating a small labelled set from real questions; even a few dozen help. Meanwhile, use production signals such as empty or low-score retrievals, rephrased questions, and user feedback, and sample queries for human relevance judgements. Grow the labelled set from the failures you find.