Measuring context precision and recall (2026)

TL;DR

Measuring context precision and recall requires a labeled target for relevant evidence. Precision tells you whether retrieved chunks are useful and well ranked, while recall tells you whether required evidence was missed. Evaluate at multiple cutoffs, preserve IDs and counts, and connect failures to retrieval, filtering, reranking, or context assembly.

Key Takeaways

• Context precision measures how much retrieved context is useful and, in ranked variants, whether useful chunks appear early.
• Context recall measures how much required evidence was retrieved, so it needs reference contexts, claims, or another coverage target.
• Choose the relevance unit and denominator before calculating a score because chunks, documents, passages, and claims produce different results.
• Use ID-based metrics for deterministic corpus snapshots and semantic or claim-based judges when exact IDs are unavailable.
• Report precision and recall at several k values with counts, slices, and no-answer behavior rather than one global average.
• Diagnose retrieval separately from context assembly because reranking, permissions, deduplication, and truncation can change what the generator sees.
• Version queries, gold evidence, corpus, chunker, embeddings, index, filters, reranker, and evaluator for meaningful regression comparisons.

Measuring context precision and recall tells a RAG team whether retrieval supplies useful evidence without missing what the answer needs. Precision focuses on noise and ranking, while recall focuses on coverage. Both require an explicit definition of relevance, a cutoff such as k, and a reproducible corpus snapshot.

These metrics evaluate evidence selection, not the final prose. A retriever can score well while the generator invents a claim, and a generator can produce a correct answer from model memory after retrieval fails. This guide defines the metrics, implements deterministic ID-based evaluation, shows the current Ragas collections API, and turns score changes into actionable diagnosis.

TL;DR

Always state the unit, cutoff, relevance rule, denominator, and aggregation. "Recall is 0.8" is incomplete without them.

1. Define Measuring Context Precision and Recall

Context precision estimates the usefulness of retrieved context. In a simple set-based form, precision@k is the number of relevant items among the first k results divided by k. A ranked variant rewards systems that place relevant chunks earlier by averaging precision calculated at ranks where a relevant item appears.

The RAG context precision metric must therefore state whether it measures a set, ranking, or final assembled prompt.

Context recall estimates evidence coverage. In an ID-based form, recall@k is the number of known relevant items found in the first k results divided by the total number of known relevant items. In a claim-based form, it is the number of reference-answer claims supported by retrieved context divided by all reference claims.

The RAG context recall metric also needs a named coverage target, since recall has no defensible denominator without one.

The word "context" can mean raw vector results, results after metadata filters, reranked chunks, or the final prompt context after truncation. Name the stage. Retriever recall may be high even when the final generator context loses a required chunk. Measuring both pre-rerank and assembled-context stages helps identify ownership.

Relevance is task-specific. A chunk may be topically related but insufficient to answer the question. For retrieval evaluation, label whether it contributes evidence needed for the approved answer, including conditions, dates, exceptions, or definitions. Similarity alone is not the oracle.

2. Choose the Evaluation Unit and Gold Target

Decide whether an item is a document, section, passage, chunk, table row, image region, or stable source span. Chunk IDs are convenient for deterministic tests, but chunking changes can invalidate gold IDs without changing information coverage. Document-level labels survive rechunking but may over-credit a result whose relevant passage was not retrieved.

A robust dataset can retain both source spans and current chunk mappings. The semantic target is the passage needed to support the answer. A snapshot process maps that span to chunk IDs for fast evaluation. When the chunker changes, remap and review rather than automatically treating every new ID as a retrieval regression.

Define minimal sufficient evidence. Some questions have several interchangeable passages, while others require a set of complementary sources. Store acceptable alternatives and required evidence groups. For example, a policy question might need one chunk for eligibility and one for an exception, but any current regional handbook can satisfy the first group.

No-answer queries need a special target. If no authorized corpus evidence can answer, correct retrieval may return nothing or only low-confidence candidates that the application rejects. Ordinary recall has a zero denominator, so report abstention or empty-retrieval accuracy separately instead of assigning an arbitrary perfect recall.

3. Build a Retriever Evaluation Dataset

Collect representative queries from approved analytics, support patterns, search logs, domain experts, and known incidents. Minimize sensitive text and include stable query IDs. Cover direct fact lookup, paraphrases, acronyms, spelling variation, multiple languages supported by the product, multi-hop questions, filters, temporal questions, ambiguous requests, rare entities, tables, and unanswerable questions.

Retrieval evaluation for RAG should mirror the routes, permissions, source formats, and answerability mix the product actually serves.

For each item, store the query, normalized form if applicable, tenant and permission context, gold evidence spans or claims, acceptable alternatives, required evidence groups, expected no-answer behavior, risk, and reviewer rationale. Do not store only a reference answer because retriever evaluators need to know which evidence supports it.

Use independent reviewers for high-impact items. Ask them to label evidence before seeing a candidate retriever's ranking when possible, reducing bias toward existing output. Adjudicate disagreements about whether a passage is necessary, merely helpful, or irrelevant. These labels define the metric, so annotation quality deserves review.

Keep a frozen regression set and a development set. Tune embeddings, filters, rank fusion, and rerankers on development data, then evaluate once on held-out items. Add minimized production failures to the next version without rewriting history. The golden dataset workflow for LLM evals provides useful governance patterns.

4. Calculate Precision@k and Recall@k Correctly

Assume the gold relevant chunk IDs are A, B, and C. A retriever returns A, X, B, Y, and Z. At k=5, precision is 2/5 and recall is 2/3. At k=1, both precision and ranking look strong because A is first, but recall is only 1/3. The cutoff changes the product conclusion.

When stakeholders say precision at k and recall at k, confirm the evidence unit and whether fewer than k results change the precision denominator.

Evaluate several cutoffs that match system decisions, such as candidate retrieval depth, reranker input, and final context limit. Do not select k after looking for the most flattering result. A retriever may achieve high recall at 100 while the generator can only receive five chunks, which provides little user benefit.

For ranked context precision, calculate precision at each rank containing a relevant result, then average those values under a stated denominator. In the ranking A, X, B, precision at relevant ranks is 1/1 and 2/3. This rewards A appearing before X. Implementations differ in normalization when fewer gold items are retrieved, so document the exact formula or library metric.

Macro averaging gives every query equal weight. Micro averaging pools relevant and retrieved counts, so queries with more gold evidence influence the result more. Report both when they answer useful questions. Always include the number of queries, retrieved items, and gold items.

5. Implement Runnable ID-Based Retrieval Metrics

When the corpus snapshot has stable IDs, deterministic metrics are cheap, fast, and explainable. The following Python script calculates precision@k, recall@k, reciprocal rank, and average precision for one query. It uses only the standard library and validates edge cases explicitly.

from dataclasses import dataclass


@dataclass(frozen=True)
class RetrievalScores:
    precision_at_k: float
    recall_at_k: float | None
    reciprocal_rank: float
    average_precision: float | None


def score_retrieval(
    retrieved_ids: list[str],
    relevant_ids: set[str],
    k: int,
) -> RetrievalScores:
    if k <= 0:
        raise ValueError("k must be positive")

    top_k = retrieved_ids[:k]
    hits = [item_id in relevant_ids for item_id in top_k]
    hit_count = sum(hits)
    precision = hit_count / len(top_k) if top_k else 0.0
    recall = hit_count / len(relevant_ids) if relevant_ids else None

    first_hit_rank = next((rank for rank, hit in enumerate(hits, 1) if hit), None)
    reciprocal_rank = 1.0 / first_hit_rank if first_hit_rank else 0.0

    precision_sum = 0.0
    running_hits = 0
    for rank, hit in enumerate(hits, 1):
        if hit:
            running_hits += 1
            precision_sum += running_hits / rank
    average_precision = (
        precision_sum / len(relevant_ids) if relevant_ids else None
    )

    return RetrievalScores(
        precision_at_k=precision,
        recall_at_k=recall,
        reciprocal_rank=reciprocal_rank,
        average_precision=average_precision,
    )


scores = score_retrieval(
    retrieved_ids=["A", "X", "B", "Y", "Z"],
    relevant_ids={"A", "B", "C"},
    k=5,
)
print(scores)

Run it with python retrieval_metrics.py. This average-precision implementation divides by all gold relevant IDs, so missed evidence reduces the score. Some context-precision definitions normalize only by relevant retrieved items and therefore emphasize ranking more than missing gold items. Neither choice should be hidden behind an unlabeled metric name.

Handle duplicates before scoring. Repeated chunk IDs should not earn repeated hits, and duplicate content under different IDs may also distort results. Keep a separate duplicate-rate diagnostic rather than quietly changing rankings in the evaluator.

6. Use Claim-Based Recall When Gold IDs Are Weak

Stable IDs are not always available, especially across evolving chunkers or external search. Claim-based context recall starts from an approved reference answer or a list of required factual claims. Each claim is checked for support in the retrieved context, and recall is supported reference claims divided by all reference claims.

This approach evaluates information coverage rather than exact source identity. It can credit an alternative approved passage that contains the same evidence. It is also more expensive and subjective because claims must be extracted and entailment judged. Numbers, dates, negation, scope, and partial support require a strict rubric.

Claim recall can miss source requirements. If policy says an answer must use the current employee handbook, a copied blog containing the same sentence should not receive credit. Include allowed source identity, authority, effective date, tenant, and permission in the relevance decision.

Use a hybrid design for important systems: deterministic ID or span labels for a fixed core set, semantic claim evaluation for alternative evidence and changing corpora, and human review for disputed high-severity cases. Track which method produced each score because their uncertainty differs.

7. Run Current Ragas Context Metrics

Ragas provides collections-based ContextPrecision and ContextRecall metrics. ContextPrecision compares retrieved contexts with a reference answer and rewards useful chunks appearing earlier. ContextRecall checks whether claims in the reference can be attributed to retrieved contexts. The current documented pattern uses an evaluator LLM created through llm_factory.

Ragas ContextPrecision and Ragas ContextRecall are model-assisted options, so their evaluator configuration and calibration belong in the result manifest.

import asyncio
import os

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision, ContextRecall


async def main() -> None:
    client = AsyncOpenAI()
    evaluator = llm_factory(os.environ["EVALUATOR_MODEL"], client=client)
    precision_scorer = ContextPrecision(llm=evaluator)
    recall_scorer = ContextRecall(llm=evaluator)

    sample = {
        "user_input": "Who can approve a return after 30 days?",
        "reference": "A store manager can approve an exception after 30 days.",
        "retrieved_contexts": [
            "A store manager may approve return exceptions after 30 days.",
            "Standard returns are accepted at the service desk.",
        ],
    }

    precision = await precision_scorer.ascore(**sample)
    recall = await recall_scorer.ascore(**sample)
    print(f"context_precision={precision.value:.3f}")
    print(f"context_recall={recall.value:.3f}")


if __name__ == "__main__":
    asyncio.run(main())

Install ragas and openai, set OPENAI_API_KEY and an available EVALUATOR_MODEL, then run the script. Pin compatible dependencies in the evaluation project. Ragas also documents synchronous score methods for these collection metrics.

Model-based results need calibration. Compare them with expert relevance labels, include adversarial distractors, and inspect case-level reasons. Do not mix a Ragas score and a deterministic ID score under one series without naming the methodology change.

8. Evaluate Reranking and Final Context Assembly

Record retrieval at multiple stages: initial candidate set, metadata and permission filtering, hybrid fusion, reranking, deduplication, and final context packing. Calculate recall on early candidates to assess search coverage, ranked precision after reranking, and evidence recall in the exact final context supplied to the generator.

This decomposition reveals tradeoffs. If candidate recall is high but final recall is low, the chunk exists but later logic removes it. A reranker may favor a topically similar overview over a narrow exception. A token-budget packer may keep several redundant passages and discard complementary evidence. A permission filter may correctly remove a gold chunk that the dataset incorrectly assumed the user could access.

Evaluate ordering because many generators attend unevenly across long context and product prompts may tell them to prioritize earlier passages. Test whether relevant chunks remain near the top without letting one source monopolize the budget. Add redundancy and source-diversity measures when multi-source synthesis matters.

Final context metrics should operate on the exact text, including truncation. A chunk ID counted as retrieved may be clipped before the relevant sentence. Span-aware evidence mapping or claim-based evaluation catches this failure. Link final context results with answer faithfulness evaluation to distinguish missing evidence from ignored evidence.

9. Slice Results and Diagnose Tradeoffs

Precision and recall oppose each other in many settings. Increasing k generally gives retrieval more opportunities to find gold evidence, improving recall, while adding irrelevant results can lower precision and consume context tokens. The best cutoff depends on reranker quality, context capacity, latency, cost, and how generation handles noise.

Slice metrics by query type, language, entity frequency, source format, temporal sensitivity, tenant, permission rule, question complexity, number of required sources, and answerability. A strong average can hide poor recall for rare product codes or poor precision for tables. Include confidence intervals or repeated samples when model judges introduce noise.

Inspect false positives. Are they topically related but non-answering, outdated, wrong region, wrong tenant, duplicate, low authority, or permission-inappropriate? Inspect false negatives. Did query rewriting remove a key term, embeddings miss an identifier, a filter exclude a source, hybrid weights suppress lexical match, or the corpus lack content?

Use counterfactual tests. Change a date, product code, region, or permission and verify the ranking changes appropriately. Add a distractor with high lexical overlap but wrong scope. Paraphrase the query while preserving intent. These tests expose retrievers that score well on a static set through brittle cues.

10. Set Thresholds and Regression Gates

There is no universal good precision or recall. Derive objectives from product risk and context budget. A compliance answer may require recall of every mandatory condition, while a discovery assistant may value diverse top results. Set critical-case rules separately from aggregate targets.

Use paired comparison. Run the production and candidate retrieval configurations on the same dataset and corpus snapshot. Report per-query changes at each meaningful k, not only new averages. Review lost critical evidence even if the candidate gains more low-risk cases.

For deterministic ID metrics, CI can enforce exact expectations on a small stable set. Broader model-judged evaluation may use calibrated thresholds, repeated scoring, and a review queue. Keep evaluator failures distinct from retrieval failures. A timeout or invalid judge response should not become a zero retrieval score silently.

Version the query set, gold labels, corpus, chunker, embedding model, index settings, filters, query rewriter, fusion weights, reranker, cutoff, packer, evaluator, and rubric. Without this manifest, a trend cannot identify what changed. The LLM evals in CI guide can help structure evidence and approvals.

11. Operationalize Measuring Context Precision and Recall

Build a fast layer and a deep layer. The fast layer runs deterministic known-item and permission tests on each relevant change. The deep layer evaluates a representative dataset across several cutoffs before releases and after material corpus or model updates. Scheduled production sampling can detect query drift when privacy controls permit it.

Create dashboards that pair metric trends with counts and system versions. Show precision@k, recall@k, no-answer behavior, empty results, duplicates, latency, final-context token use, and critical misses. Avoid a single blended score. Make failure drill-down show query, ranking, gold evidence, filters, scores, and final context.

Monitor annotation and evaluator drift. As policies and documents change, gold evidence expires. Set review dates for temporal items and use stable canary queries. When an evaluator or chunker changes, run overlap analysis before declaring improvement or regression.

Close the loop with developers and content owners. Some failures need retrieval tuning, some need metadata repair, and some reveal missing or contradictory source documents. The metric system should route the failure to the owning layer instead of encouraging endless embedding changes.

Interview Questions and Answers

Q: What is the difference between context precision and context recall?

Context precision asks how much retrieved content is relevant and whether useful content is ranked early. Context recall asks how much of the required evidence was retrieved. Precision diagnoses noise and ordering, while recall diagnoses missing coverage.

Q: Why does context recall require a reference?

Recall needs a denominator representing all relevant evidence. That can be gold chunk IDs, required source spans, reference contexts, or claims from an approved answer. Without a target, the evaluator cannot know what retrieval missed.

Q: How do you choose k?

I evaluate cutoffs that correspond to real pipeline stages, such as initial candidate depth, reranker output, and final context capacity. I do not choose k after seeing the best score. Product latency, token budget, and risk determine the operating cutoff.

Q: When would you use ID-based metrics?

I use them for a controlled corpus snapshot with stable document or chunk IDs. They are deterministic, cheap, and explainable. I supplement them with span or claim-based evaluation when chunking changes or multiple sources can provide equivalent evidence.

Q: How do you evaluate unanswerable queries?

Ordinary recall has no meaningful denominator when no evidence should exist. I measure correct empty retrieval, low-confidence rejection, unauthorized-source exclusion, and downstream abstention separately. I never assign a perfect recall by convention without labeling it.

Q: What does high retriever recall but low final-context recall indicate?

The evidence was found early but lost during filters, reranking, deduplication, truncation, or context packing. I compare stage-level rankings and exact assembled text to locate the removal. The retriever itself may not need changing.

Q: How would you gate a retriever change in CI?

I compare candidate and production configurations on the same frozen corpus and query set at defined cutoffs. Deterministic critical cases have hard gates, and broader semantic metrics use calibrated thresholds. Any lost high-severity evidence gets case-level review even if the average improves.

Common Mistakes

• Calling topically similar chunks relevant even when they cannot support the answer.
• Reporting precision or recall without naming k, unit, denominator, and aggregation.
• Using the whole corpus as gold evidence without identifying minimal sufficient passages.
• Evaluating only raw retrieval and ignoring permission filters, reranking, truncation, and packing.
• Treating a changed chunk ID as an information failure after rechunking.
• Assigning arbitrary recall to no-answer cases instead of measuring abstention behavior.
• Optimizing recall with a huge k that the generator can never receive.
• Hiding critical misses inside a stronger global average.
• Mixing deterministic and model-judged metrics in one trend without methodology labels.

Conclusion

Measuring context precision and recall creates a clear view of RAG retrieval quality. Define relevance and the evidence unit, calculate performance at product-relevant cutoffs, inspect both raw and final context, and preserve case-level counts and source identities.

Start with 30 to 50 carefully reviewed queries covering common, difficult, permission-sensitive, and unanswerable behavior. Implement deterministic ID metrics, add claim-based evaluation only where needed, and compare every retrieval change against the same snapshot. The result is a diagnostic system, not just a leaderboard.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Measuring LLM latency and cost in tests (2026)
• Measuring RAG faithfulness and relevancy (2026)
• AI test data generation with Faker and LLMs (2026)
• API error handling and negative testing: A Practical Guide (2026)
• Appium gestures and swipe: A Practical Guide (2026)
• Bug severity and priority examples (2026)