Evaluate RAG Citations with LLM Judges (2026)

TL;DR

Evaluate RAG citations with LLM judges by validating citation IDs in code, decomposing the answer into atomic claims, and asking a structured judge whether each cited passage entails its claim. Measure correctness and completeness separately, calibrate against human labels, and fail CI only on meaningful regressions or critical unsupported claims.

Key Takeaways

• Split citation quality into validity, entailment, completeness, and source quality instead of asking for one vague score.
• Use deterministic code for citation parsing and reference validity before spending tokens on semantic judgment.
• Force the judge to return a typed schema so malformed prose cannot silently enter evaluation results.
• Score atomic claims individually and require evidence spans to make failures reviewable.
• Calibrate thresholds against human labels and track disagreement by failure category.
• Gate CI on aggregate regressions and critical unsupported claims, not a brittle perfect-score rule.

To evaluate RAG citations with LLM judges reliably, do not ask a model whether the citations are simply good. Build a claim-level pipeline: validate citation markers deterministically, ask a judge whether each cited source supports the exact claim, measure whether important claims have citations, and calibrate the resulting thresholds against human review.

This tutorial builds that pipeline in Python. You will finish with structured JSON results, unit tests, an aggregate report, and a CI gate that exposes the reason for every failure. If you need the broader evaluation landscape first, read the complete RAG application evaluation guide.

TL;DR

The central design choice is separation. Citation correctness asks whether a citation supports the nearby claim. Citation completeness asks whether claims that need evidence received citations. A response can score perfectly on correctness by citing only one supported sentence while leaving five uncited claims, so never collapse both dimensions prematurely.

What You Will Build

You will create a small evaluator that:

• Accepts a question, answer, and retrieved sources with stable IDs.
• Rejects missing or unknown citation markers without an API call.
• Sends atomic claims and cited passages to an LLM judge.
• Produces typed claim verdicts with evidence quotes and short reasons.
• Calculates citation correctness, weighted completeness, and an overall score.
• Runs a labeled test set and exits nonzero when a release policy fails.

The example deliberately avoids a RAG framework. Plain Python keeps the scoring contract visible and lets you embed it behind LangChain, LlamaIndex, a custom retriever, or an API test later. For dataset design, the adversarial RAG evaluation dataset tutorial is a useful companion.

Prerequisites

Use Python 3.12.11, OpenAI Python 2.46.0, Pydantic 2.11.7, and pytest 8.4.1. The evaluator uses the Responses API and Pydantic structured parsing. Create an isolated project:

mkdir rag-citation-eval
cd rag-citation-eval
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install openai==2.46.0 pydantic==2.11.7 pytest==8.4.1
export OPENAI_API_KEY=your_key_here

Choose an available model that supports structured outputs and pin its snapshot in production. The examples read JUDGE_MODEL so your team controls that choice without editing source. Never place an API key in the dataset, test log, or repository.

Verification: Run python -c "import openai, pydantic, pytest; print(openai.__version__, pydantic.__version__, pytest.__version__)". Confirm the three installed versions, then run python -c "from openai import OpenAI; OpenAI(); print('client ready')".

Step 1: Define the Citation Evaluation Contract

Start with explicit input and output types. A source needs an immutable ID, its exact retrieved text, and metadata used for source-quality policy. A verdict belongs to one atomic claim, not an entire paragraph.

Create citation_eval.py:

from __future__ import annotations

import os
import re
from collections import defaultdict
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel, Field


class Source(BaseModel):
    id: str
    title: str
    text: str
    url: str
    authority: Literal["primary", "secondary", "unknown"] = "unknown"


class Claim(BaseModel):
    id: str
    text: str
    citation_ids: list[str] = Field(default_factory=list)
    importance: int = Field(ge=1, le=3)


class ClaimVerdict(BaseModel):
    claim_id: str
    verdict: Literal["supported", "partial", "unsupported"]
    evidence_quote: str
    reason: str


class JudgeResult(BaseModel):
    verdicts: list[ClaimVerdict]


class EvaluationReport(BaseModel):
    correctness: float
    completeness: float
    overall: float
    invalid_citation_ids: list[str]
    verdicts: list[ClaimVerdict]

The three-level verdict prevents a common distortion. Partial means the passage supports part of a compound claim or supports a weaker statement. It does not mean the judge feels uncertain. Uncertainty should appear in the reason and trigger human review if needed. Importance weights reserve stronger penalties for claims that change the answer, such as a safety limit or eligibility rule.

Verification: Run python -c "from citation_eval import Source, Claim; print(Claim(id='c1', text='A', importance=3))". Pydantic should print a claim with an empty citation_ids list. Changing importance to 4 should raise a validation error.

Step 2: Parse Markers and Fail Fast

Use a citation format that machines can parse. This tutorial expects [S1], [S2], and similar markers in the answer. It does not infer citations from raw URLs because punctuation, redirects, and duplicate URLs complicate attribution.

Append this code:

CITATION_PATTERN = re.compile(r"\[([A-Za-z][A-Za-z0-9_-]*)\]")


def citation_ids(text: str) -> list[str]:
    return list(dict.fromkeys(CITATION_PATTERN.findall(text)))


def invalid_citations(answer: str, sources: list[Source]) -> list[str]:
    known = {source.id for source in sources}
    return sorted(set(citation_ids(answer)) - known)


def citation_validity(answer: str, sources: list[Source]) -> float:
    used = citation_ids(answer)
    if not used:
        return 0.0
    invalid = set(invalid_citations(answer, sources))
    return round((len(used) - len(invalid)) / len(used), 4)

This layer catches hallucinated IDs, a failure an LLM judge should never decide semantically. It also preserves first-seen order for diagnostics while using a set for validation. Duplicate uses of one source do not inflate validity.

Do not treat a valid ID as a correct citation. [S1] can point to a real passage that contradicts the answer. Validity is necessary plumbing, while entailment is the semantic test covered later. For focused examples of that distinction, see RAG citation correctness examples.

Verification: Run python -c "from citation_eval import *; s=[Source(id='S1',title='T',text='x',url='https://e.test')]; print(citation_validity('Fact [S1], other [S9].',s))". The output must be 0.5, and invalid_citations must return ['S9'].

Step 3: Create Atomic Claims

A judge cannot grade citation placement fairly if it receives one long paragraph as one claim. Split the answer during generation when possible. Your RAG service should return claim objects alongside display text, because post hoc sentence splitting mishandles abbreviations, tables, and sentences containing several assertions.

For this runnable evaluator, store a small golden case in cases.py:

from citation_eval import Claim, Source

QUESTION = "What is the support window and refund rule for Acme Pro?"

ANSWER = (
    "Acme Pro includes email support for 12 months [S1]. "
    "Refunds are available within 30 days when usage is below 100 requests [S2]. "
    "Enterprise customers receive telephone support."
)

SOURCES = [
    Source(
        id="S1",
        title="Acme Pro Support Policy",
        text="Each Acme Pro license includes email support for twelve months from purchase.",
        url="https://docs.acme.test/support",
        authority="primary",
    ),
    Source(
        id="S2",
        title="Acme Refund Policy",
        text="Customers may request a refund within 30 days if the account used fewer than 100 API requests.",
        url="https://docs.acme.test/refunds",
        authority="primary",
    ),
]

CLAIMS = [
    Claim(id="c1", text="Acme Pro includes email support for 12 months.", citation_ids=["S1"], importance=2),
    Claim(id="c2", text="Refunds are available within 30 days when usage is below 100 requests.", citation_ids=["S2"], importance=3),
    Claim(id="c3", text="Enterprise customers receive telephone support.", citation_ids=[], importance=2),
]

Claims should be independently verifiable. Split “the plan costs $20 and includes unlimited projects” into two claims because one citation may support only one half. Keep purely connective language out of the set. Mark recommendations and opinions separately if your product permits them, since citation completeness should target factual assertions rather than every sentence.

Verification: Run python -c "from cases import CLAIMS; print(sum(c.importance for c in CLAIMS), len(CLAIMS))". Expect 7 3. Inspect the list and confirm each factual proposition can receive its own verdict.

Step 4: Build a Strict LLM Judge

Now ask the judge only the question it is suited to answer: whether supplied evidence entails each claim. Include no outside knowledge. Require a verbatim evidence span for supported and partial verdicts, then verify that span locally.

Append to citation_eval.py:

def judge_claims(
    question: str,
    claims: list[Claim],
    sources: list[Source],
    client: OpenAI | None = None,
) -> JudgeResult:
    client = client or OpenAI()
    source_map = {source.id: source for source in sources}
    packets = []
    for claim in claims:
        cited = [source_map[sid] for sid in claim.citation_ids if sid in source_map]
        packets.append({
            "claim_id": claim.id,
            "claim": claim.text,
            "sources": [source.model_dump() for source in cited],
        })

    system = """You evaluate citation entailment. Use only the supplied source text.
SUPPORTED: the evidence directly establishes every material part of the claim.
PARTIAL: the evidence establishes some material parts or a weaker version.
UNSUPPORTED: the evidence is absent, contradictory, or merely topically related.
For supported or partial, copy the shortest exact evidence quote.
For unsupported, evidence_quote must be empty. Return one verdict per claim_id."""

    response = client.responses.parse(
        model=os.environ["JUDGE_MODEL"],
        input=[
            {"role": "system", "content": system},
            {"role": "user", "content": str({"question": question, "items": packets})},
        ],
        text_format=JudgeResult,
        temperature=0,
    )
    result = response.output_parsed
    if result is None:
        raise RuntimeError("Judge returned no parsed output")
    return result

A source packet contains only passages cited for that claim. This prevents the judge from rescuing a wrong citation with a different retrieved document. The user question provides context for ambiguous pronouns, but the rubric makes source text the sole evidence. Structured parsing rejects missing fields and unexpected verdict labels.

The exact evidence quote is an audit hook, not decoration. After parsing, normalize whitespace and ensure the quote occurs in one cited passage. If it does not, change the verdict to unsupported or route it to review. That check catches judges that paraphrase despite instructions.

Verification: Set JUDGE_MODEL to your pinned structured-output model and run python -c "from citation_eval import judge_claims; from cases import *; print(judge_claims(QUESTION, CLAIMS, SOURCES).model_dump_json(indent=2))". Expect c1 and c2 to be supported, while c3 is unsupported because it has no cited evidence.

Step 5: Calculate Correctness and Completeness

Turn verdicts into transparent metrics. Use a partial credit value of 0.5 for entailment. Correctness considers only cited claims. Completeness measures the importance weight of supported cited claims against all factual claims. These definitions answer different release questions.

Append the scoring functions:

VERDICT_VALUE = {"supported": 1.0, "partial": 0.5, "unsupported": 0.0}


def score_report(
    answer: str, claims: list[Claim], sources: list[Source], result: JudgeResult
) -> EvaluationReport:
    verdict_by_id = {v.claim_id: v for v in result.verdicts}
    cited_claims = [c for c in claims if c.citation_ids]

    correctness = (
        sum(VERDICT_VALUE[verdict_by_id[c.id].verdict] for c in cited_claims)
        / len(cited_claims)
        if cited_claims else 0.0
    )
    total_weight = sum(c.importance for c in claims)
    supported_weight = sum(
        c.importance * VERDICT_VALUE[verdict_by_id[c.id].verdict]
        for c in claims
        if c.citation_ids
    )
    completeness = supported_weight / total_weight if total_weight else 1.0
    validity = citation_validity(answer, sources)
    overall = 0.45 * correctness + 0.40 * completeness + 0.15 * validity

    return EvaluationReport(
        correctness=round(correctness, 4),
        completeness=round(completeness, 4),
        overall=round(overall, 4),
        invalid_citation_ids=invalid_citations(answer, sources),
        verdicts=result.verdicts,
    )

For the sample, correctness should be 1.0 because both cited claims are supported. Completeness is 5/7, about 0.7143, because the uncited telephone-support claim has weight 2. Overall is an illustrative product policy, not a universal benchmark. Adjust weights only after examining user risk and human labels.

If you also evaluate factual faithfulness beyond explicit citations, keep that as a separate metric. The guide to measuring answer faithfulness explains why groundedness and attribution overlap but are not identical.

Verification: Save a JudgeResult with two supported verdicts and one unsupported verdict, call score_report, and assert correctness equals 1.0, completeness equals 0.7143, and overall equals 0.8857. If an unknown marker is added, validity must fall and invalid_citation_ids must expose it.

Step 6: Add Deterministic and Mocked Tests

Unit tests should never spend tokens. Mock the semantic result, then test parsing, formulas, boundary behavior, and policy logic. Reserve live judge calls for a scheduled or explicitly enabled suite.

Create test_citation_eval.py:

from citation_eval import (
    ClaimVerdict, JudgeResult, citation_ids, citation_validity,
    invalid_citations, score_report,
)
from cases import ANSWER, CLAIMS, SOURCES


def sample_result() -> JudgeResult:
    return JudgeResult(verdicts=[
        ClaimVerdict(claim_id="c1", verdict="supported",
                     evidence_quote="email support for twelve months", reason="Exact policy"),
        ClaimVerdict(claim_id="c2", verdict="supported",
                     evidence_quote="within 30 days", reason="Both conditions match"),
        ClaimVerdict(claim_id="c3", verdict="unsupported",
                     evidence_quote="", reason="No citation"),
    ])


def test_marker_parsing_preserves_order_and_deduplicates():
    assert citation_ids("[S2] then [S1] and [S2]") == ["S2", "S1"]


def test_unknown_ids_are_reported():
    assert invalid_citations("Claim [S1] [S404]", SOURCES) == ["S404"]
    assert citation_validity("Claim [S1] [S404]", SOURCES) == 0.5


def test_scores_correctness_separately_from_completeness():
    report = score_report(ANSWER, CLAIMS, SOURCES, sample_result())
    assert report.correctness == 1.0
    assert report.completeness == 0.7143
    assert report.overall == 0.8857


def test_no_citations_gets_zero_correctness():
    claims = [c.model_copy(update={"citation_ids": []}) for c in CLAIMS]
    report = score_report("No markers.", claims, SOURCES, sample_result())
    assert report.correctness == 0.0
    assert report.completeness == 0.0

This suite protects semantics encoded in arithmetic. Add cases for duplicate markers, an empty answer, all-partial verdicts, unknown claim IDs, repeated sources, and zero factual claims. Also validate that the judge returns exactly the requested claim IDs once each. A typed schema proves shape, not correspondence.

Verification: Run pytest -q. Expect 4 passed. No API key is required because these tests never call judge_claims.

Step 7: Calibrate the Judge Against Humans

Before using a threshold, label a representative sample manually. Two reviewers should independently assign supported, partial, or unsupported to claim-source pairs and resolve disagreements. Include easy positives, subtle partial support, contradicted numbers, stale policy versions, and a source that supports the claim but was not cited.

Create a CSV with case_id, claim_id, human_label, judge_label, importance, and failure_type. Calculate exact agreement and a confusion matrix. Do not celebrate a high overall agreement if the judge misses the rare class that matters most. In a medical assistant, falsely calling unsupported dosage advice supported is more serious than marking harmless background detail unsupported.

Use at least dozens of claim pairs for an initial smoke calibration and expand toward hundreds across real domains before enforcing a production gate. Those are workflow sizes, not promised statistical guarantees. Slice results by document type, answer length, citation count, language, and claim importance. Recalibrate after changing the prompt, model snapshot, chunking strategy, or source serialization.

The LLM judge and human label calibration guide covers disagreement analysis in depth. Keep human labels immutable and version the rubric. If reviewers cannot apply a category consistently, improve the category definition before tuning the judge.

Verification: Manually inspect every false-supported result. Confirm the report distinguishes partial -> supported from unsupported -> supported, because the latter is normally the riskier error. Record the prompt version and model snapshot with each run so future comparisons remain interpretable.

Step 8: Add a CI Regression Gate

A CI gate should compare a candidate against a frozen baseline, not demand perfection from a nondeterministic judge. Cache judge outputs for unchanged case, prompt, model, and source hashes. Run repeated trials on a small stability slice before trusting a one-off movement near the threshold.

Create gate.py:

from dataclasses import dataclass


@dataclass(frozen=True)
class GateResult:
    passed: bool
    reasons: list[str]


def release_gate(
    baseline_overall: float,
    candidate_overall: float,
    invalid_ids: int,
    critical_unsupported: int,
) -> GateResult:
    reasons: list[str] = []
    if candidate_overall < baseline_overall - 0.03:
        reasons.append("overall score regressed by more than 0.03")
    if invalid_ids > 0:
        reasons.append("answer contains unresolved citation IDs")
    if critical_unsupported > 0:
        reasons.append("critical claims lack supporting evidence")
    return GateResult(passed=not reasons, reasons=reasons)


if __name__ == "__main__":
    result = release_gate(0.86, 0.84, 0, 0)
    print({"passed": result.passed, "reasons": result.reasons})
    raise SystemExit(0 if result.passed else 1)

The illustrative 0.03 tolerance is useful only as an example. Derive your value from observed judge variance, human-reviewed risk, and the smallest change your product considers meaningful. Block any unresolved ID because that is deterministic. Block unsupported critical claims even when averages look healthy, since aggregation can hide a severe failure. For broader pipeline patterns, follow building LLM evaluations in CI with Promptfoo.

Verification: Run python gate.py; the sample passes because the 0.02 drop stays inside tolerance. Change the candidate to 0.82 and confirm the process exits with code 1 and names the regression. Set invalid_ids=1 and verify it fails regardless of the aggregate score.

How to Evaluate RAG Citations with LLM Judges Without Fooling Yourself

Judge-based evaluation is scalable, but it is not ground truth. Treat these controls as part of the test system:

• Pin the model snapshot and prompt version. A silent judge change invalidates trend comparisons.
• Randomize answer order when comparing two systems, and run the comparison in both positions to detect positional bias.
• Blind the judge to model names, vendor names, and baseline labels.
• Keep source text verbatim. Summarizing evidence before judging adds another uncontrolled model decision.
• Test adversarial cases where lexical overlap is high but meaning contradicts the claim, especially negation and numeric limits.
• Route low-confidence or high-impact disagreements to a person instead of inventing extra decimal precision.
• Store the full evaluation packet, parsed verdict, latency, token usage, model snapshot, rubric version, and dataset revision.

A single scalar is convenient for dashboards but insufficient for diagnosis. Preserve each verdict so a retrieval engineer can distinguish a missing passage from an answer generator that cited the wrong passage. Retrieval quality can be measured independently with RAG retrieval recall at k.

Troubleshooting

Problem: The judge marks a topically related passage as supported. -> Tighten the rubric around direct entailment. Add counterexamples where the same entities appear but the date, quantity, condition, or direction differs. Require the evidence quote and verify it exists in the cited passage.

Problem: Structured parsing returns no object. -> Confirm the selected model supports structured outputs, the installed SDK matches the pinned version, and every schema field is supported. Log the response ID and refusal state without logging sensitive source content. Retry transient failures with bounded exponential backoff, but do not retry a stable schema error indefinitely.

Problem: Correctness is high while users still see unsupported statements. -> Inspect completeness. Your evaluator may score only claims carrying markers, which rewards selective citation. Add all externally verifiable claims to the denominator and weight consequential claims more heavily.

Problem: Scores change between identical runs. -> Use a pinned snapshot, temperature zero, cached inputs, and repeated trials on a stability set. Temperature zero reduces sampling variability but does not guarantee bit-for-bit determinism. Set a regression margin wider than measured judge noise.

Problem: The judge supports a claim using the wrong document. -> Send only sources explicitly cited for that claim. Do not include the entire retrieval context, because the judge may locate support elsewhere and conceal an attribution defect.

Problem: Evaluation cost grows unexpectedly. -> Run syntax and ID checks first, truncate passages at ingestion with a documented policy, cache by content hash, and judge only changed cases in pull requests. Keep the complete suite scheduled for nightly or release-candidate runs.

Interview Questions and Answers

The interview-ready answers are captured in the structured section below. Be ready to explain why validity, correctness, and completeness need separate denominators; how you detect judge bias; and why critical failures can override an aggregate score. A strong answer connects the metric to a concrete product risk instead of listing evaluation buzzwords.

Common Mistakes

• Asking one model for a score from 1 to 10 without category definitions or evidence.
• Giving the judge all retrieved passages and accidentally allowing it to repair a wrong citation.
• Counting citation markers instead of checking the claims they supposedly support.
• Ignoring uncited factual claims, which inflates correctness while completeness collapses.
• Changing the judge model and application model in the same experiment.
• Using synthetic easy positives only, with no contradictions, partial support, stale sources, or numeric traps.
• Failing a build on a tiny score movement that falls inside ordinary judge variance.
• Logging private retrieved text to a third-party evaluation service without a data-handling review.

Where To Go Next

Start by running the sample against ten human-labeled answers from your own application. Expand the set with retrieval failures, contradictory evidence, uncited major claims, and malformed markers. Once claim-level agreement is acceptable, connect the evaluator to a scheduled build and review every critical false-supported verdict.

Continue with the RAG pipeline hallucination testing tutorial, then add RAG context precision with Ragas. Use the /dashboard resume analyzer to map this evaluation work to evidence in your QA portfolio, or practice explaining the design in an /practice AI testing interview.

Conclusion

To evaluate RAG citations with LLM judges in production, combine deterministic reference checks with narrowly scoped semantic judgments. Grade atomic claims, preserve evidence quotes, report correctness apart from completeness, and make critical unsupported claims visible rather than hiding them inside an average.

The judge becomes trustworthy through calibration and observability, not through a more authoritative prompt tone. Freeze a representative dataset, compare results with human labels, measure variance, and adopt release thresholds that reflect the actual consequence of a citation failure.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Evaluate RAG Citation Correctness with Examples
• Building a QA chatbot with RAG (2026)
• Building a QA copilot with RAG (2026)
• Evaluating an LLM app with DeepEval metrics (2026)
• Measuring answer faithfulness in RAG (2026)
• RAG Application Evaluation Complete Guide (2026)