RAG Retriever Eval Calculator: Precision@k, Recall@k, MRR & MAP

A RAG pipeline answers a question in two steps: a retriever picks chunks from your corpus, then a language model writes an answer from them. If the first step fails, nothing the model does can save the answer. These four numbers describe how well the first step works, using a small set of test questions where you already know which chunks are relevant.

• Precision@k = relevant chunks in the top k, divided by k. It is the share of your context window that is useful. A Precision@5 of 0.4 means two of every five chunks you pay tokens for are relevant, and three are noise.
• Recall@k = relevant chunks in the top k, divided by all relevant chunks that exist. It is the share of the evidence the model gets to see. If recall is low, the answer is missing facts no matter how good the model is.
• Hit rate@k = the share of queries with at least one relevant chunk in the top k. It is a forgiving version of recall for questions that need only one supporting chunk.
• MRR (mean reciprocal rank) = the average of 1 divided by the rank of the first relevant chunk, counting 0 when none is returned. Rank 1 scores 1.0, rank 2 scores 0.5, rank 4 scores 0.25. It rewards putting the answer at the very top.
• MAP (mean average precision) = for each query, add up the precision at every rank where a relevant chunk appears and divide by the total number of relevant chunks, then average across queries. It rewards ranking all the relevant chunks early and charges you for any that were never retrieved.

Precision, recall and hit rate are measured at a cutoff k. MRR and MAP are measured over the whole list you paste for each query, so pasting a longer list can raise them.

These are the standard definitions from information retrieval: see Manning, Raghavan and Schütze, Introduction to Information Retrieval (Cambridge University Press, 2008), chapter 8 on evaluation in ranked retrieval, and the reciprocal rank definition from the TREC-8 question answering track (Voorhees, 1999).

Top-k is the number of chunks you paste into the prompt, and it pulls two ways. Raising k can only keep recall the same or raise it, since the top 8 always contains the top 5. But precision usually falls as k grows, because later ranks are less likely to be relevant, and every extra chunk costs input tokens and gives the model more irrelevant text to be distracted by.

The suggestion box finds the smallest k that reaches 95% of the best recall your lists allow, and shows the precision and the average number of noise chunks at that point. Read it as a plateau detector: if recall is 0.90 at k = 4 and 0.92 at k = 10, six extra chunks bought two points of recall.

• Recall still climbing at your k: raise k, or improve the ranking so relevant chunks come sooner.
• Recall flat but precision low: lower k and save tokens. Check what a chunk costs with the LLM context window calculator and price the difference with the AI token cost calculator.
• Recall low at every k: the right chunks are not being retrieved at all. Look at chunking, the embedding model, and query rewriting before tuning k. The embedding similarity playground shows how close a question and a chunk really are, and the AI embedding cost calculator prices a re-embed if you change models.

Several tools on this site use the words precision and recall, but they answer different questions.

• The NDCG calculator scores one ranked list with graded relevance (for example 0 to 3) and discounts lower ranks logarithmically. Use it when some chunks are better answers than others.
• The precision-recall curve calculator takes scored examples from a binary classifier and sweeps a score threshold. There are no ranks per query and no top-k; it is for classifiers, not retrievers.
• The confusion matrix calculator turns true and false positive and negative counts into classification precision and recall. It has no notion of rank position or a cutoff.
• This calculator works per query on rank positions with binary labels and then averages across queries, which is exactly the shape of a RAG eval set.