Evaluate RAG Citation Correctness with Examples

TL;DR

Split an answer into atomic claims, map each citation to the exact source passage, and decide whether that passage entails the claim. Report correctness and coverage separately, calibrate any LLM judge on human labels, and gate releases with claim-level regression tests.

Key Takeaways

• Evaluate each atomic claim against its cited passage, not against the full retrieved context.
• Separate citation correctness from citation coverage, retrieval quality, and answer style.
• Use deterministic fixtures first, then add an LLM judge behind a strict structured-output contract.
• Treat source identity, entailment, and citation placement as independently debuggable checks.
• Calibrate thresholds on labeled examples instead of accepting a judge score at face value.
• Store claim-level evidence so every failed regression is explainable.

To evaluate RAG citation correctness examples reliably, test a narrow question: does the cited source actually support the claim beside it? A fluent answer with real-looking references can still cite the wrong passage, overstate the evidence, or leave important claims unsupported.

This tutorial builds a small Python evaluator that checks source identity, claim-to-citation mapping, lexical evidence, and judge-based entailment. It fits into the broader RAG application evaluation complete guide, where citation correctness is one quality signal alongside retrieval, faithfulness, relevance, latency, and cost.

You will start with transparent fixtures, add a structured LLM judge only where semantics require it, calibrate the decision threshold, and turn the result into a repeatable release gate. The examples are intentionally small so you can see exactly why each claim passes or fails.

What You Will Build

By the end, you will have:

• A JSONL fixture format that keeps answers, claims, citations, and passages traceable.
• Deterministic checks for missing sources and weak evidence overlap.
• A structured entailment judge with explicit supported, contradicted, and insufficient labels.
• Separate citation correctness and citation coverage metrics.
• A calibrated threshold and a pytest regression gate.
• Claim-level failure output that a developer can reproduce without rereading an entire conversation.

The final evaluator is deliberately modular. You can replace its local heuristic or judge client without changing the dataset, metric definitions, or CI assertions.

Prerequisites

Use Python 3.11 or newer. Create an isolated environment and install the only required test dependency:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install pytest

The deterministic path runs offline. For a production semantic judge, use the supported SDK for your chosen model provider and pin that dependency in your project. Keep credentials in environment variables, never in fixtures or source control.

You should already have RAG traces containing the generated answer, stable source IDs, and the exact passages given to the model. If you only store document URLs, update instrumentation first. A URL cannot prove which passage supported a claim.

Verify prerequisites: run python --version and pytest --version. Confirm Python reports 3.11 or later and pytest starts without an import error.

Step 1: Define Evaluate RAG Citation Correctness Examples

A useful fixture represents claims explicitly. Do not score an entire multi-sentence answer as one unit because one supported sentence can hide an unsupported one. Create examples.jsonl:

{"case_id":"supported","answer":"The Atlas API limits clients to 100 requests per minute [S1].","claims":[{"claim_id":"c1","text":"The Atlas API limit is 100 requests per minute.","citation_ids":["S1"],"is_verifiable":true}],"sources":[{"source_id":"S1","title":"Atlas API limits","passage":"Each client may send up to 100 requests per minute."}],"expected":{"c1":"supported"}}
{"case_id":"overstated","answer":"Atlas guarantees 99.99% uptime [S1].","claims":[{"claim_id":"c1","text":"Atlas guarantees 99.99% uptime.","citation_ids":["S1"],"is_verifiable":true}],"sources":[{"source_id":"S1","title":"Atlas reliability","passage":"The monthly uptime target is 99.9%."}],"expected":{"c1":"contradicted"}}
{"case_id":"missing-detail","answer":"Atlas stores backups for 90 days [S1].","claims":[{"claim_id":"c1","text":"Atlas stores backups for 90 days.","citation_ids":["S1"],"is_verifiable":true}],"sources":[{"source_id":"S1","title":"Atlas backups","passage":"Backups are encrypted at rest."}],"expected":{"c1":"insufficient"}}

These three labels distinguish positive support, direct conflict, and absence of enough evidence. That distinction matters operationally. A contradiction may indicate stale content or unsafe generation, while insufficient evidence often indicates a retrieval or citation-selection failure.

Keep is_verifiable explicit. Greetings, recommendations, and clearly marked opinions should not reduce coverage merely because they have no citation. For production data, preserve tenant, model, prompt, index version, and trace IDs as optional metadata. Do not put sensitive source text into a shared evaluation repository.

Verify Step 1: run python -m json.tool < <(head -n 1 examples.jsonl) in a shell that supports process substitution. It should print the first record with no parse error. Also manually confirm every cited ID exists in sources.

Step 2: Validate Citation and Source Identity

Before semantic scoring, reject broken references. A citation marker that resolves to no supplied passage cannot be correct. Create citation_eval.py:

from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable

VALID_LABELS = {"supported", "contradicted", "insufficient"}

@dataclass(frozen=True)
class ClaimResult:
    case_id: str
    claim_id: str
    label: str
    confidence: float
    reason: str

def load_cases(path: str) -> list[dict]:
    lines = Path(path).read_text(encoding="utf-8").splitlines()
    return [json.loads(line) for line in lines if line.strip()]

def validate_case(case: dict) -> list[str]:
    errors: list[str] = []
    source_ids = [source["source_id"] for source in case["sources"]]
    if len(source_ids) != len(set(source_ids)):
        errors.append("source IDs must be unique")
    known = set(source_ids)
    for claim in case["claims"]:
        missing = set(claim["citation_ids"]) - known
        if missing:
            errors.append(
                f"{claim['claim_id']} has unknown citations: {sorted(missing)}"
            )
    return errors

def validate_all(cases: Iterable[dict]) -> None:
    failures = {c["case_id"]: validate_case(c) for c in cases}
    failures = {key: value for key, value in failures.items() if value}
    if failures:
        raise ValueError(json.dumps(failures, indent=2))

if __name__ == "__main__":
    cases = load_cases("examples.jsonl")
    validate_all(cases)
    print(f"validated {len(cases)} cases")

This is source identity validation, not entailment. It catches deleted chunks, renumbered references, serialization mistakes, and cases where the UI citation no longer matches the model trace. In a real system, also verify document revision and chunk hash. The same source ID should never silently point to different text across runs.

Verify Step 2: run python citation_eval.py. Expected output is validated 3 cases. Change one fixture citation from S1 to S9; rerun and confirm the validator names the case, claim, and unknown ID. Restore the fixture afterward.

Step 3: Add a Deterministic Evidence Check

A lexical check is not a complete correctness judge, but it is fast, stable, and useful for obvious failures. Add this code below the validator:

import re

STOPWORDS = {
    "a", "an", "and", "are", "as", "at", "be", "by", "for",
    "from", "in", "is", "it", "of", "on", "or", "that", "the",
    "to", "was", "were", "with"
}

def terms(text: str) -> set[str]:
    tokens = re.findall(r"[a-z0-9.]+", text.lower())
    return {token for token in tokens if token not in STOPWORDS}

def overlap_score(claim: str, passages: list[str]) -> float:
    claim_terms = terms(claim)
    if not claim_terms:
        return 1.0
    evidence_terms = terms(" ".join(passages))
    return len(claim_terms & evidence_terms) / len(claim_terms)

def cited_passages(case: dict, claim: dict) -> list[str]:
    wanted = set(claim["citation_ids"])
    return [
        source["passage"]
        for source in case["sources"]
        if source["source_id"] in wanted
    ]

Use overlap only as a triage signal. Synonyms and paraphrases can produce low overlap despite valid support. High overlap can also be wrong: the claim may reverse a negation or change 99.9% to 99.99%. Numbers, negations, comparatives, dates, and modal verbs deserve semantic inspection even when most words match.

A sensible pipeline uses deterministic checks for schema and obvious mismatches, then sends ambiguous claims to a semantic judge. This reduces cost and gives stable explanations for simple defects. Do not publish an overlap threshold as a universal benchmark. Tune it on your content and language.

Verify Step 3: open a Python shell and run from citation_eval import overlap_score, followed by overlap_score('limit is 100 requests per minute', ['up to 100 requests per minute']). The result should be high. Compare it with a passage about encryption and confirm the result drops sharply.

Step 4: Implement a Structured Entailment Judge

Semantic correctness asks whether the cited passages entail the claim. Keep the judge interface provider-neutral so tests run offline and production can call your approved model. Add:

from typing import Protocol

class Judge(Protocol):
    def evaluate(self, claim: str, passages: list[str]) -> dict:
        ...

class FixtureJudge:
    """Deterministic judge for the tutorial fixtures and CI unit tests."""
    def evaluate(self, claim: str, passages: list[str]) -> dict:
        evidence = " ".join(passages).lower()
        normalized_claim = claim.lower()
        if "99.99%" in normalized_claim and "99.9%" in evidence:
            return {
                "label": "contradicted",
                "confidence": 0.99,
                "reason": "The claim and evidence state different uptime values."
            }
        score = overlap_score(claim, passages)
        if score >= 0.5:
            return {
                "label": "supported",
                "confidence": score,
                "reason": "The cited passage contains the material claim details."
            }
        return {
            "label": "insufficient",
            "confidence": 1.0 - score,
            "reason": "The cited passage lacks material claim details."
        }

def check_judge_output(output: dict) -> dict:
    required = {"label", "confidence", "reason"}
    if set(output) != required:
        raise ValueError(f"judge keys must be exactly {sorted(required)}")
    if output["label"] not in VALID_LABELS:
        raise ValueError("invalid judge label")
    confidence = float(output["confidence"])
    if not 0.0 <= confidence <= 1.0:
        raise ValueError("confidence must be between 0 and 1")
    if not output["reason"].strip():
        raise ValueError("judge reason must not be empty")
    return {**output, "confidence": confidence}

For a hosted judge, instruct it to use only the supplied passages, treat partial support as insufficient, preserve numeric precision, and return the exact schema. Set temperature to the provider's deterministic setting when available. Defend against prompt injection in retrieved text by delimiting evidence as data and repeating that source instructions are untrusted. Validate every response before scoring.

Judge confidence is self-reported, not a calibrated probability. Use it for routing only after comparing it with labeled examples. Sample disputed and high-impact claims for human review.

Verify Step 4: call check_judge_output(FixtureJudge().evaluate('Atlas guarantees 99.99% uptime.', ['The monthly uptime target is 99.9%.'])). Confirm the label is contradicted. Delete reason from a sample response and confirm validation raises ValueError.

Step 5: Calculate Citation Correctness and Coverage Separately

Correctness measures whether attached citations support their claims. Coverage measures whether verifiable claims have citations. Combining them hides important behavior. A system that cites only one easy claim can achieve perfect correctness with poor coverage. Add:

def evaluate_case(case: dict, judge: Judge) -> list[ClaimResult]:
    results: list[ClaimResult] = []
    for claim in case["claims"]:
        if not claim["is_verifiable"] or not claim["citation_ids"]:
            continue
        output = check_judge_output(
            judge.evaluate(claim["text"], cited_passages(case, claim))
        )
        results.append(ClaimResult(
            case_id=case["case_id"],
            claim_id=claim["claim_id"],
            label=output["label"],
            confidence=output["confidence"],
            reason=output["reason"],
        ))
    return results

def metrics(cases: list[dict], results: list[ClaimResult]) -> dict[str, float]:
    cited_verifiable = len(results)
    supported = sum(result.label == "supported" for result in results)
    verifiable = sum(
        claim["is_verifiable"]
        for case in cases
        for claim in case["claims"]
    )
    with_citation = sum(
        claim["is_verifiable"] and bool(claim["citation_ids"])
        for case in cases
        for claim in case["claims"]
    )
    return {
        "citation_correctness": supported / cited_verifiable if cited_verifiable else 0.0,
        "citation_coverage": with_citation / verifiable if verifiable else 1.0,
    }

Use macro averaging when every query should count equally, even if one answer contains many more claims. Use micro averaging, as above, when every claim should count equally. Report both if answer lengths vary greatly. Always publish the denominator and label distribution next to the ratio. Ten supported claims out of ten are less convincing than ten thousand out of ten thousand.

Verify Step 5: evaluate all three fixtures with FixtureJudge. The illustrative correctness is one supported claim out of three cited claims, while coverage is three out of three. Do not adopt that value as a quality target; it merely proves the examples exercise all labels.

Step 6: Calibrate the Decision Threshold

A semantic judge may return a support score instead of a label. Select a threshold with a human-labeled calibration set that represents your domains, languages, answer lengths, and risk levels. Never tune on the final regression set.

def confusion(rows: list[dict], threshold: float) -> dict[str, int]:
    counts = {"tp": 0, "fp": 0, "tn": 0, "fn": 0}
    for row in rows:
        predicted = row["support_score"] >= threshold
        actual = row["human_label"] == "supported"
        key = ("t" if predicted == actual else "f") + ("p" if predicted else "n")
        counts[key] += 1
    return counts

def precision(counts: dict[str, int]) -> float:
    denominator = counts["tp"] + counts["fp"]
    return counts["tp"] / denominator if denominator else 0.0

def recall(counts: dict[str, int]) -> float:
    denominator = counts["tp"] + counts["fn"]
    return counts["tp"] / denominator if denominator else 0.0

calibration = [
    {"support_score": 0.96, "human_label": "supported"},
    {"support_score": 0.78, "human_label": "supported"},
    {"support_score": 0.71, "human_label": "insufficient"},
    {"support_score": 0.22, "human_label": "contradicted"},
]
for threshold in (0.70, 0.80, 0.90):
    result = confusion(calibration, threshold)
    print(threshold, result, precision(result), recall(result))

For compliance or medical content, false support may cost more than false rejection, so favor precision and human escalation. For low-risk discovery, you may tolerate more false positives to retain recall. Document that policy instead of calling one threshold objectively best. Recalibrate when you change the judge model, prompt, chunking, citation format, or content domain.

Track inter-annotator agreement and adjudicate ambiguous examples. Human labels are not automatically correct, especially when passages require domain knowledge. The annotation guide should define whether common knowledge, inference across two citations, and conflicting sources count as support.

Verify Step 6: run the snippet and inspect how precision and recall change. Confirm that raising the threshold cannot turn a previously negative prediction positive. Add boundary cases exactly equal to the threshold so the >= policy is tested.

Step 7: Turn Examples into Regression Tests

Create test_citation_eval.py so labeled cases fail loudly when behavior changes:

import pytest

from citation_eval import FixtureJudge, evaluate_case, load_cases, validate_all

CASES = load_cases("examples.jsonl")

@pytest.mark.parametrize("case", CASES, ids=lambda case: case["case_id"])
def test_claim_labels(case: dict) -> None:
    validate_all([case])
    results = evaluate_case(case, FixtureJudge())
    actual = {result.claim_id: result.label for result in results}
    assert actual == case["expected"]

def test_all_required_labels_are_present() -> None:
    labels = {label for case in CASES for label in case["expected"].values()}
    assert labels == {"supported", "contradicted", "insufficient"}

Run deterministic tests on every pull request. Run hosted-judge evaluations on a controlled schedule or candidate release if they are slower, costly, or nondeterministic. Store the model identifier, prompt version, raw structured response, latency, and evaluator code revision with each result. Redact sensitive passages before sending them to any external judge.

A release gate should combine a minimum sample size, correctness floor, coverage floor, and no-regression rule for critical slices. Slice by source type, language, tenant configuration, answer length, and claim type. An aggregate can remain stable while numeric claims deteriorate. Avoid failing a build on one unreviewed model disagreement; route uncertain or changed labels through adjudication.

Verify Step 7: run pytest -q. Expected output is four passing tests: three parameterized cases and the label coverage test. Change the expected label for one case, verify the assertion shows the case ID and difference, then restore it.

Step 8: Diagnose Failures Across the RAG Pipeline

Citation failures are symptoms, not always generation defects. Use the label and trace to identify the responsible stage.

Start with retrieval. Use the test RAG retrieval recall at k tutorial to determine whether the supporting document entered the candidate set. Then inspect noise with measure RAG context precision with Ragas. If retrieval is healthy, compare the exact context seen by the model with the citation rendered to the user. The wider RAG hallucination testing guide helps you test unsupported answers that may have no citation marker at all.

Build failure clusters rather than fixing isolated prompts. Numeric mismatches may require a specialized rule. Multi-hop claims may require two passages and a judge that permits composition. Conflicting sources require a documented authority or recency policy. Add every confirmed production defect to the regression dataset after removing sensitive information.

Verify Step 8: take one failed trace and record its query, retrieved IDs, supplied chunk hashes, answer, citations, and claim labels. Another engineer should be able to identify the failing stage from that bundle without accessing live logs.

Troubleshooting

Problem: the judge marks paraphrases insufficient -> Add representative paraphrases to calibration data and use a semantic judge after deterministic triage. Do not lower the threshold until you confirm false negatives with human review.

Problem: the judge accepts a changed number -> Add exact numeric comparison before the semantic call. Normalize units carefully, but preserve precision, ranges, dates, and qualifiers such as at least.

Problem: correctness is high but users still distrust answers -> Measure coverage and citation placement. A few correct citations do not compensate for important uncited claims or a marker attached to the wrong sentence.

Problem: scores change between runs -> Pin the judge model and prompt, request structured output, reduce sampling randomness, and repeat a stability subset. Treat a provider model update as an evaluator change that requires recalibration.

Problem: the citation opens the right document but wrong text -> Validate chunk hashes and revisions, not only URLs. Store the exact cited passage with the generation trace.

Problem: long passages make nearly every claim look supported -> Evaluate the smallest passage actually exposed through the citation. Large context windows increase accidental overlap and make results less diagnostic.

Common Mistakes and Best Practices

Do not label citations by reading the answer alone. The evaluator must see the exact evidence. Do not treat retrieval relevance as citation correctness: a topically relevant passage may not entail the specific claim. Do not allow the judge to use its own knowledge, browse, or repair missing evidence.

Prefer atomic claims, immutable source identifiers, and explicit label definitions. Test negation, numbers, dates, source conflicts, partial support, multi-citation claims, and uncited factual claims. Keep a balanced challenge set even if production traffic contains mostly easy supported examples. An evaluator that always says supported can appear accurate on a skewed sample.

Review judge errors, not only system errors. If humans repeatedly disagree with the evaluator, improve its rubric or replace it. Version datasets and prompts together, and show claim text, evidence, label, confidence, and reason in reports. Explainability turns a red metric into an actionable defect. Use the structured LLM output testing tutorial to harden the judge response contract separately from its semantic accuracy.

Where To Go Next

Place citation correctness inside a complete evaluation plan using the RAG application evaluation complete guide. Pair it with retrieval recall and context precision so you can separate missing evidence from bad citation selection.

Next, expand beyond three fixtures. The adversarial RAG evaluation dataset tutorial shows how to add conflicting sources, prompt injection, stale documents, near-duplicate passages, and answerable versus unanswerable queries. Keep a small smoke set for pull requests and a broader stratified set for release evaluation.

Finally, connect failures to ownership. Retrieval teams need recall and filter traces. Generation teams need the exact prompt and context. Frontend teams need source-to-marker mapping tests. Shared claim-level artifacts prevent each team from optimizing a different interpretation of citation quality.

Interview Questions and Answers

Q: What is RAG citation correctness?

It is the degree to which a cited passage supports the specific claim associated with it. It does not ask whether the source is merely relevant or whether the answer is generally plausible.

Q: How is correctness different from citation coverage?

Correctness considers cited claims and asks whether their evidence supports them. Coverage considers all verifiable claims and asks whether they received citations. Report both because either metric can look good while the other is poor.

Q: Why evaluate atomic claims instead of complete answers?

An answer can mix supported, contradicted, and unsupported statements. Claim-level evaluation preserves those differences and creates failure output that developers can diagnose.

Q: When should you use an LLM judge?

Use one when paraphrase and semantic inference exceed deterministic rules. Validate structured output, constrain the judge to supplied evidence, calibrate it against human labels, and retain human review for risky or uncertain cases.

Q: What belongs in a citation evaluation dataset?

Include the query, answer, atomic claims, citation IDs, exact passages, expected labels, and trace metadata. Cover support, contradiction, insufficiency, missing citations, numeric changes, conflicts, and multi-source reasoning.

Q: How do you prevent evaluator drift?

Pin the judge model and prompt, version the dataset, retain a stable calibration set, and rerun agreement checks after any evaluator change. Store raw judge outputs so score changes remain auditable.

Conclusion: Evaluate RAG Citation Correctness Examples

To evaluate RAG citation correctness examples well, make the unit of analysis an atomic claim paired with its exact cited passage. Validate reference identity first, judge entailment second, and report correctness separately from coverage and retrieval metrics.

Start with the three transparent fixtures, run them as regression tests, then add anonymized production failures and adversarial cases. Calibrate on human labels before setting a release threshold. That workflow produces a citation score engineers can explain, reproduce, and improve.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Boundary Value Analysis With Examples (2026)
• Build an Adversarial RAG Evaluation Dataset
• Building a QA chatbot with RAG (2026)
• Building a QA copilot with RAG (2026)
• Cypress cy.wait with alias: Examples
• Equivalence partitioning with examples (2026)