Tools · RAG · 07

Measure the ranking you retrieve, not only the final answer.

Change k, the relevance threshold and reranking depth on a visible synthetic ranking. Separately, change chunk size and overlap to see how much text you duplicate in the index without pretending that this storage cost predicts semantic retrieval quality.

Precision@krelevant retrieved ÷ k

Recall@krelevant retrieved ÷ total relevant

nDCG@kgraded gain discounted by rank

Ranking evaluation

Evaluation k

Candidates reranked0 keeps the initial ranking; a larger value reranks only that prefix.

Minimum grade counted as relevant

Total relevant items in the qrelsRecall needs a denominator outside the top-k. Use the known relevant count for your query.

Chunking footprint

Corpus · tokens

Chunk size · tokens

Overlap · tokensMust stay below the chunk size.

Copy link Export CSV Reset

Precision@k—purity of the top-k

Recall@k—coverage of known relevant items

MRR@k—rank of the first relevant item

nDCG@k—ordering + graded relevance

Relevant inside k—

Estimated chunks—

Indexed tokens—

Overlap duplication—

Stride—

Metric reading—

Chunking reading—

Deliberate separation: the chunking panel computes index size and textual duplication. It does not turn chunk size or overlap into a supposed relevance gain. Demonstrating a gain requires running the retriever against real qrels.

—results inside k are highlighted

Method

Each metric answers a different question.

Precision@k asks what proportion of the first k results clears your relevance threshold. Recall@k asks what fraction of all known relevant items that top-k managed to recover.

Precision@k = relevant_in_top_k / k · Recall@k = relevant_in_top_k / total_relevant

MRR@k depends only on the rank of the first relevant result. It is useful when finding one good piece of evidence early is enough, but it cannot distinguish an excellent ranking from a mediocre one after that first hit.

MRR@k = 1 / rank_of_first_relevant

nDCG@k supports graded relevance. This lab uses gain 2^rel − 1 with logarithmic rank discount, divided by the ideal DCG for the same judgments.

DCG@k = Σ (2^relᵢ − 1) / log₂(i + 1) · nDCG@k = DCG@k / IDCG@k

Reranking. The lab applies a second score only to the prefix you select. This makes a common pattern explicit: a cheap retriever generates candidates, then a more expensive reranker tries to improve their order without rescuing documents that never entered the candidate set.

Chunking. Stride is chunk_size − overlap. Overlapping windows increase indexed tokens; that redundancy can reduce boundary cuts, but it also increases storage and redundant candidates. The lab computes that footprint exactly for the configured corpus and leaves quality as a separate measurement.

Incomplete judgments. Recall requires knowing how many relevant results exist. If your qrels are partial, the denominator is partial too; do not read a high score as absolute coverage of the knowledge base.

Sources: Karpukhin et al. (DPR, EMNLP 2020), Lewis et al. (RAG, 2020), Thakur et al. (BEIR, 2021), and Järvelin & Kekäläinen (2002).

Connect retrieval, context and evaluation

01Context budgetTurn retrieved chunks into an explicit token budget.Tool 02Model evaluationKeep the metric separate from the decision you are trying to make.Concept 03Engineering notesArchitecture and operation of production AI systems.Engineering