Measure RAG Context Precision with Ragas

TL;DR

Use Ragas LLMContextPrecisionWithReference to score each query's ordered retrieved chunks against a trusted reference. Keep case-level results, pair them with IDBasedContextPrecision, and gate regressions using thresholds calibrated on your own reviewed dataset.

Key Takeaways

• Context precision measures whether relevant chunks rank ahead of irrelevant chunks.
• Use reference-aware scoring for reviewed release suites and response-aware scoring only for exploratory unlabeled data.
• Preserve retrieval order and keep per-case results so low scores remain diagnosable.
• Pair semantic LLM judgment with deterministic ID-based precision checks.
• Calibrate CI thresholds from repeated runs instead of copying a universal benchmark.
• Combine precision with recall, faithfulness, correctness, and citation tests.

To measure RAG context precision with Ragas, score the ordered chunks returned by your retriever against a trusted reference answer, then inspect whether relevant chunks appear before irrelevant ones. A high score means useful evidence is concentrated near the top, where the generator is most likely to use it.

This tutorial builds a repeatable Python evaluation for local and CI runs. For the complete quality model, including recall, faithfulness, correctness, and operational checks, read the RAG application evaluation complete guide.

You will prove that rank order changes the score, evaluate a dataset, add deterministic ID checks, and create a regression gate.

What You Will Build

By the end, you will have:

• A Python project that uses Ragas context precision metrics.
• A controlled test showing why relevant chunks must rank first.
• A JSONL evaluation set with queries, references, and ordered retrieved chunks.
• An LLM-judged batch evaluator for semantic relevance.
• A deterministic ID-based precision check for labeled documents.
• A CI-friendly report and threshold gate.
• A debugging process for low or unstable scores.

The examples use a small employee benefits knowledge base so the relevance decisions are easy to audit. Replace those records with captured output from your own retriever after the harness works.

Prerequisites

Use Python 3.11 or newer and create an isolated environment. The commands below install the current Ragas package line together with the OpenAI integration used by the evaluator. Pin the resolved versions in your lock file after the first successful run so local and CI results use the same dependencies.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install ragas openai
python -m pip freeze > requirements.lock

Set an API key for the evaluator model:

export OPENAI_API_KEY="your-key"

You also need basic Python and JSON knowledge. You do not need a vector database because the tutorial evaluates captured retrieval results. In production, serialize each query and its ordered chunks immediately after retrieval. That separation makes retrieval failures reproducible without repeatedly calling the live index.

Ragas offers several context precision choices:

This tutorial uses the with-reference metric as the main signal because it judges chunks against a trusted expected answer. It also adds ID-based precision as a fast, stable companion metric.

Step 1: Create a Controlled Ranking Experiment

Create check_ranking.py. The first sample puts the relevant policy chunk first. The second puts the same relevant chunk after a distractor. Keeping the evidence constant while changing only order isolates ranking behavior.

import asyncio
import os

from openai import AsyncOpenAI
from ragas import SingleTurnSample
from ragas.llms import llm_factory
from ragas.metrics import LLMContextPrecisionWithReference


async def score(contexts: list[str]) -> float:
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    evaluator_llm = llm_factory("gpt-4o-mini", client=client)
    metric = LLMContextPrecisionWithReference(llm=evaluator_llm)

    sample = SingleTurnSample(
        user_input="How many paid parental leave weeks are available?",
        reference="Eligible employees receive 16 weeks of paid parental leave.",
        retrieved_contexts=contexts,
    )
    result = await metric.single_turn_ascore(sample)
    return float(result)


async def main() -> None:
    relevant = (
        "Parental leave: Eligible employees receive 16 weeks of paid leave "
        "after birth, adoption, or placement."
    )
    distractor = (
        "Vacation: Employees accrue 20 vacation days each calendar year."
    )

    relevant_first = await score([relevant, distractor])
    relevant_second = await score([distractor, relevant])

    print(f"relevant_first={relevant_first:.3f}")
    print(f"relevant_second={relevant_second:.3f}")
    assert relevant_first > relevant_second


if __name__ == "__main__":
    asyncio.run(main())

Run it:

python check_ranking.py

The metric judges the relevance of each chunk and applies an average-precision-style ranking calculation. An irrelevant result below the first relevant result reduces the score. A distractor after the only relevant result may not reduce it in the same way, which is why context precision measures ranking quality rather than simply counting every irrelevant chunk.

Verify: The script prints two values between 0 and 1, and relevant_first is greater than relevant_second. Exact decimals can vary with the evaluator model, so verify the ordering, not a copied numeric value.

Step 2: Build a Small Evaluation Dataset

Create evaluation.jsonl. Each line represents one query and the ordered result list returned by the retriever. Preserve order exactly. If you sort chunks during export, the evaluation no longer tests the production ranking.

{"case_id":"benefits-001","user_input":"How many paid parental leave weeks are available?","reference":"Eligible employees receive 16 weeks of paid parental leave.","retrieved_contexts":["Parental leave provides 16 paid weeks for eligible employees.","Vacation allowance is 20 days per calendar year.","The dental plan covers two preventive visits annually."],"retrieved_context_ids":["leave-16","vacation-20","dental-2"],"reference_context_ids":["leave-16"]}
{"case_id":"benefits-002","user_input":"When does health insurance begin for a new hire?","reference":"Health insurance begins on the first day of the month after the employee starts.","retrieved_contexts":["New-hire health coverage begins on the first day of the month following the start date.","Open enrollment occurs each November.","Employees can change beneficiaries in the HR portal."],"retrieved_context_ids":["health-start","open-enrollment","beneficiary"],"reference_context_ids":["health-start"]}
{"case_id":"benefits-003","user_input":"What is the annual learning budget?","reference":"Each employee receives a 1,500 USD annual learning budget.","retrieved_contexts":["Travel expenses require manager approval.","Each employee has a 1,500 USD learning budget per calendar year.","Course completion certificates belong in the HR portal."],"retrieved_context_ids":["travel","learning-budget","certificates"],"reference_context_ids":["learning-budget"]}

Good evaluation records come from real user intents and reviewed source material. Write the reference answer from the approved policy, not from the current RAG response. Otherwise, a wrong generated answer can redefine relevance and hide retrieval problems.

Include difficult negatives that share vocabulary with the query. For a learning-budget question, an irrelevant expense policy is more valuable than a random recipe. Add queries requiring filters, dates, product versions, or tenant-specific rules. If your corpus changes often, keep stable document IDs and version metadata alongside the text.

Before expanding the set, learn how to build an adversarial RAG evaluation dataset. Adversarial cases expose ranking weaknesses that easy one-fact questions miss.

Verify: Run python -m json.tool evaluation.jsonl against individual lines, or load all lines with the next script. Confirm that every retrieved_contexts list aligns positionally with retrieved_context_ids.

Step 3: Measure RAG Context Precision with Ragas in a Batch

Create evaluate_context_precision.py. This script reads JSONL, scores every sample with the reference-aware metric, writes a detailed report, and prints a summary. It deliberately evaluates samples one at a time so the case ID remains attached to errors and scores.

import asyncio
import json
import os
from pathlib import Path
from statistics import mean
from typing import Any

from openai import AsyncOpenAI
from ragas import SingleTurnSample
from ragas.llms import llm_factory
from ragas.metrics import LLMContextPrecisionWithReference

DATA_PATH = Path("evaluation.jsonl")
REPORT_PATH = Path("context-precision-report.json")


def load_cases(path: Path) -> list[dict[str, Any]]:
    cases = []
    with path.open(encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            if not line.strip():
                continue
            case = json.loads(line)
            required = {"case_id", "user_input", "reference", "retrieved_contexts"}
            missing = required - case.keys()
            if missing:
                raise ValueError(
                    f"Line {line_number} missing fields: {sorted(missing)}"
                )
            if not case["retrieved_contexts"]:
                raise ValueError(f"Line {line_number} has no retrieved contexts")
            cases.append(case)
    return cases


async def main() -> None:
    cases = load_cases(DATA_PATH)
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    evaluator_llm = llm_factory("gpt-4o-mini", client=client)
    metric = LLMContextPrecisionWithReference(llm=evaluator_llm)

    rows = []
    for case in cases:
        sample = SingleTurnSample(
            user_input=case["user_input"],
            reference=case["reference"],
            retrieved_contexts=case["retrieved_contexts"],
        )
        score = float(await metric.single_turn_ascore(sample))
        rows.append({"case_id": case["case_id"], "score": score})
        print(f'{case["case_id"]}: {score:.3f}')

    summary = {
        "metric": "llm_context_precision_with_reference",
        "case_count": len(rows),
        "mean_score": mean(row["score"] for row in rows),
        "cases": rows,
    }
    REPORT_PATH.write_text(
        json.dumps(summary, indent=2) + "\n",
        encoding="utf-8",
    )
    print(f'mean: {summary["mean_score"]:.3f}')


if __name__ == "__main__":
    asyncio.run(main())

Run the batch:

python evaluate_context_precision.py

A single mean is useful for trend reporting but insufficient for diagnosis. Keep the per-case scores and investigate the bottom group. Segment results by language, content type, query class, tenant, or retriever configuration when those dimensions matter.

Do not treat 0.8 as a universal pass mark. Establish a baseline on your reviewed dataset, make one retrieval change, and compare paired cases. Choose the release threshold from your risk tolerance and observed evaluator variance.

Verify: The command prints three case scores, prints a mean between 0 and 1, and creates context-precision-report.json. The third case should be weaker than an equivalent case with the learning-budget chunk at rank one, although judge variation is possible.

Step 4: Compare With and Without a Reference

Reference-aware evaluation is preferred for release gates, but teams often begin before gold answers exist. Ragas can compare retrieved contexts with the generated response through LLMContextPrecisionWithoutReference.

Create compare_metric_modes.py:

import asyncio
import os

from openai import AsyncOpenAI
from ragas import SingleTurnSample
from ragas.llms import llm_factory
from ragas.metrics import (
    LLMContextPrecisionWithReference,
    LLMContextPrecisionWithoutReference,
)


async def main() -> None:
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    evaluator_llm = llm_factory("gpt-4o-mini", client=client)

    contexts = [
        "Parental leave provides 16 paid weeks for eligible employees.",
        "Vacation allowance is 20 days per calendar year.",
    ]
    with_reference = LLMContextPrecisionWithReference(llm=evaluator_llm)
    without_reference = LLMContextPrecisionWithoutReference(llm=evaluator_llm)

    gold_sample = SingleTurnSample(
        user_input="How many paid parental leave weeks are available?",
        reference="Eligible employees receive 16 paid weeks.",
        retrieved_contexts=contexts,
    )
    response_sample = SingleTurnSample(
        user_input="How many paid parental leave weeks are available?",
        response="Eligible employees receive 16 paid weeks.",
        retrieved_contexts=contexts,
    )

    gold_score = await with_reference.single_turn_ascore(gold_sample)
    response_score = await without_reference.single_turn_ascore(response_sample)
    print(f"with_reference={float(gold_score):.3f}")
    print(f"without_reference={float(response_score):.3f}")


if __name__ == "__main__":
    asyncio.run(main())

The without-reference mode answers a different question: are chunks useful for the response that was produced? If the response is wrong, incomplete, or copied from an irrelevant chunk, that score can be misleading. Use it for exploratory monitoring and dataset triage. Promote important cases to reference-aware evaluation after subject-matter review.

Verify: Both values print successfully. Replace the response with a deliberately wrong statement and observe why the without-reference score should not be used as a truthfulness signal.

Step 5: Add Deterministic ID-Based Precision

LLM judgment captures semantic relevance, but it costs time and can vary. If reviewers can label the IDs of relevant source chunks, calculate exact precision with IDBasedContextPrecision.

Create evaluate_id_precision.py:

import asyncio
import json
from pathlib import Path
from statistics import mean

from ragas import SingleTurnSample
from ragas.metrics import IDBasedContextPrecision


async def main() -> None:
    cases = [
        json.loads(line)
        for line in Path("evaluation.jsonl").read_text(encoding="utf-8").splitlines()
        if line.strip()
    ]
    metric = IDBasedContextPrecision()
    rows = []

    for case in cases:
        sample = SingleTurnSample(
            retrieved_context_ids=case["retrieved_context_ids"],
            reference_context_ids=case["reference_context_ids"],
        )
        score = float(await metric.single_turn_ascore(sample))
        rows.append(score)
        print(f'{case["case_id"]}: {score:.3f}')

    print(f"mean_id_precision={mean(rows):.3f}")


if __name__ == "__main__":
    asyncio.run(main())

Run it:

python evaluate_id_precision.py

ID-based precision checks how many returned IDs belong to the relevant ID set. It does not grade semantic text and does not reward ranking the relevant ID earlier. Therefore, keep it beside the ranking-sensitive LLM metric, not in place of it. It is excellent for catching filter failures, wrong namespaces, or a sudden influx of unrelated results.

Document-ID granularity matters. If one document contains ten chunks and only one chunk answers the query, a document-level label may count irrelevant sibling chunks as relevant. Prefer stable chunk IDs when your test needs chunk-level discrimination.

Verify: Each example returns one relevant ID among three retrieved IDs, so the deterministic score should be approximately 0.333. If it is not, check ID spelling and list contents.

Step 6: Diagnose Low Scores With Rank-Level Evidence

A score says that ranking is weak, not why. Add a human-readable rank audit before tuning embeddings. Create inspect_case.py:

import json
import sys
from pathlib import Path


def main(case_id: str) -> None:
    cases = [
        json.loads(line)
        for line in Path("evaluation.jsonl").read_text(encoding="utf-8").splitlines()
        if line.strip()
    ]
    case = next(item for item in cases if item["case_id"] == case_id)
    relevant_ids = set(case["reference_context_ids"])

    print(f'query: {case["user_input"]}')
    print(f'reference: {case["reference"]}')
    for rank, (chunk_id, text) in enumerate(
        zip(case["retrieved_context_ids"], case["retrieved_contexts"]),
        start=1,
    ):
        label = "RELEVANT" if chunk_id in relevant_ids else "not labeled"
        print(f"\nrank={rank} id={chunk_id} label={label}\n{text}")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python inspect_case.py CASE_ID")
    main(sys.argv[1])

Inspect the intentionally weak case:

python inspect_case.py benefits-003

Classify the cause before changing the system:

• Relevant chunk is present but low: tune ranking, reranking, metadata boosts, or query rewriting.
• Relevant chunk is absent: measure recall and inspect indexing, filters, and candidate count.
• Wrong-version chunk ranks first: improve version metadata and effective-date filters.
• Chunks are broad mixtures: revise chunk boundaries and preserve headings.
• Reference is ambiguous: repair the test case before judging the retriever.

Context precision and recall answer complementary questions. Precision asks whether useful items rank ahead of noise. Recall asks whether required evidence was retrieved at all. Use the RAG retrieval recall at k tutorial when relevant evidence is missing rather than merely misplaced.

Verify: The output marks learning-budget as relevant at rank two. You now have an actionable diagnosis: ranking is the issue, not complete retrieval failure.

Step 7: Create a Regression Gate for CI

An LLM metric should not fail a release because one call changed slightly. Use a reviewed baseline, a minimum aggregate, a per-case floor for critical journeys, and a small retry policy at the job level. Keep evaluator model and prompts fixed during comparisons.

Create gate_context_precision.py:

import json
import sys
from pathlib import Path

REPORT = Path("context-precision-report.json")
MINIMUM_MEAN = 0.75
MINIMUM_CASE = 0.45
CRITICAL_CASES = {"benefits-001", "benefits-002"}


def main() -> None:
    report = json.loads(REPORT.read_text(encoding="utf-8"))
    failures = []

    if report["mean_score"] < MINIMUM_MEAN:
        failures.append(
            f'mean {report["mean_score"]:.3f} < {MINIMUM_MEAN:.3f}'
        )

    scores = {row["case_id"]: row["score"] for row in report["cases"]}
    for case_id in CRITICAL_CASES:
        if case_id not in scores:
            failures.append(f"missing critical case {case_id}")
        elif scores[case_id] < MINIMUM_CASE:
            failures.append(
                f'{case_id} {scores[case_id]:.3f} < {MINIMUM_CASE:.3f}'
            )

    if failures:
        print("Context precision gate failed:")
        for failure in failures:
            print(f"- {failure}")
        sys.exit(1)

    print("Context precision gate passed")


if __name__ == "__main__":
    main()

Run the evaluator and gate in sequence:

python evaluate_context_precision.py
python gate_context_precision.py

The thresholds are illustrative, not universal benchmarks. Calibrate them from repeated runs on your own dataset. Store the report as a CI artifact, record the retriever configuration and corpus version, and review score changes case by case.

For pull requests, run deterministic ID checks on every change and reserve the LLM suite for retrieval-related changes or scheduled jobs if cost is a concern. Fail closed when critical cases disappear from the report.

Verify: The gate prints either a clear pass or specific failures and returns a nonzero exit code on failure. Temporarily set MINIMUM_MEAN = 1.01 to prove that CI detects the failing path, then restore the calibrated value.

Step 8: Interpret the Score Without Overclaiming

Context precision ranges from 0 to 1, with higher values indicating that relevant chunks are ranked earlier. It is not the percentage of the answer that is correct. It also does not prove that every needed fact was retrieved, that the model used the evidence, or that citations point to the right source.

Use a layered quality view:

Review both aggregate and slice-level results. A healthy overall mean can conceal failures for a product version, language, customer, or query type. Track a distribution and the worst critical cases, not only the average.

When the answer includes citations, continue with RAG citation correctness examples. A retriever can rank excellent evidence while the generator attaches the wrong citation to a claim.

Verify: For each dashboard or release gate, write one sentence stating what the metric proves and one stating what it does not prove. If the second sentence mentions answer correctness, you are avoiding the most common interpretation error.

Troubleshooting

Problem: OPENAI_API_KEY is missing or authentication fails -> Export the key in the same shell that runs Python. In CI, map a protected secret to the environment variable and never write it into the dataset or report.

Problem: Ragas import names differ after an upgrade -> Recreate the environment from requirements.lock. Check the installed Ragas documentation for your exact version. Do not silently mix a legacy example with the collections-based API in a single harness.

Problem: Scores change between identical runs -> Keep the evaluator model fixed, use a larger reviewed set, repeat borderline cases, and gate on calibrated tolerances. Treat LLM judgment as a measurement with variance.

Problem: Every score is high -> Add hard negatives, stale versions, near-duplicate policies, and intentionally misordered chunks. Verify that references were written independently from generated responses.

Problem: Scores are low even when the right document appears -> Inspect chunk granularity and rank. The answer-bearing sentence may be buried in a broad chunk, or the relevant chunk may rank below distractors.

Problem: The metric call is slow or expensive -> Run deterministic ID precision first, cache immutable evaluation inputs where policy permits, evaluate changed slices, and schedule the full LLM suite. Never cache across a changed evaluator model, prompt, corpus, or retrieval result.

Where To Go Next

You now have a working measurement loop: capture ordered retrieval output, score semantic ranking with Ragas, compare deterministic labels, inspect weak cases, and enforce a calibrated regression gate.

Use the complete RAG evaluation guide to place this signal inside a broader test strategy. Then:

• Measure missing evidence with retrieval recall at k tests.
• Validate claim-to-source alignment with RAG citation correctness examples.
• Expand coverage using an adversarial RAG evaluation dataset.
• Check whether generated claims are grounded with RAG faithfulness testing.

Change one retrieval variable at a time, such as chunk size, embedding model, candidate count, hybrid weighting, or reranker. Compare paired cases, inspect regressions, and keep the configuration only when it improves the slices that matter without breaking critical journeys.

Interview Questions and Answers

Q: What does Ragas context precision measure?

It measures whether relevant retrieved chunks appear ahead of irrelevant chunks for a query. It is a ranking-quality signal. It does not directly measure answer correctness, completeness, or faithfulness.

Q: Why can moving the same relevant chunk change context precision?

The calculation rewards relevant evidence at earlier ranks. An irrelevant chunk before a relevant one reduces precision at the rank where the relevant item appears. This makes the metric useful for testing rerankers and retrieval ordering.

Q: When should you use the with-reference metric?

Use it when a reviewed reference answer exists and you need a release-quality semantic judgment. The reference anchors relevance to approved truth rather than to whatever answer the application generated.

Q: What is the risk of context precision without a reference?

It compares chunks with the generated response. If the response is wrong or incomplete, chunks supporting that response may look useful even though they do not support the correct answer.

Q: How do context precision and context recall differ?

Precision asks whether retrieved results are relevant and well ordered. Recall asks whether the retriever found the evidence required to answer. A system can have high precision but low recall when it returns one excellent chunk while missing another required fact.

Q: Why pair LLM context precision with ID-based precision?

The LLM metric recognizes semantic relevance and rank quality. ID-based precision is deterministic, inexpensive, and good at catching filter or namespace failures. Together they give richer coverage than either alone.

Q: How should context precision be gated in CI?

Use a reviewed dataset, fixed evaluator configuration, an aggregate threshold, and stricter checks for critical cases. Calibrate thresholds from repeated baseline runs and preserve per-case reports for diagnosis.

Best Practices

• Preserve the exact result order returned by production retrieval.
• Write references from authoritative sources, not generated answers.
• Include hard negatives and stale, similar-looking documents.
• Store stable chunk IDs beside text for deterministic checks.
• Pin dependencies and evaluator configuration.
• Compare changes on the same cases and corpus snapshot.
• Track case-level and slice-level scores, not only the mean.
• Use recall, faithfulness, correctness, and citation tests alongside precision.
• Review low scores before tuning embeddings or changing chunk size.
• Recalibrate baselines when the corpus or evaluator intentionally changes.

Conclusion

To measure RAG context precision with Ragas, evaluate ordered retrieved chunks against trusted references and retain the per-case evidence behind the score. The metric is most useful when you prove its ranking behavior with controlled examples, pair it with deterministic labels, and interpret it strictly as retrieval-ranking quality.

Start with the three-case dataset, connect the exporter to your retriever, and add real failure cases from production. Once precision is stable, test recall and citation correctness so a strong ranking score becomes one part of a complete RAG quality gate.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Measuring context precision and recall (2026)
• Building a QA chatbot with RAG (2026)
• Building a QA copilot with RAG (2026)
• Evaluate RAG Citation Correctness with Examples
• Evaluating RAG retrieval precision (2026)
• Accessibility testing with Cypress (2026)