Testing a RAG pipeline for hallucinations (2026)

TL;DR

Testing a RAG pipeline for hallucinations requires layer-by-layer evidence. Measure whether retrieval found the required authorized sources, whether each answer claim is entailed by the supplied context, whether citations point to the right passages, and whether the system abstains when evidence is absent or conflicting.

Key Takeaways

• Define hallucination as unsupported or contradicted claims relative to approved evidence, then test that invariant claim by claim.
• Evaluate retrieval separately from generation so a failed answer can be traced to missing context, noisy context, or model behavior.
• Use answerable, unanswerable, ambiguous, temporal, conflicting, permission-sensitive, and adversarial questions in the evaluation set.
• Check citation entailment and source identity, not merely whether an answer contains citation-shaped text.
• Test abstention quality because a grounded system must decline when authorized context cannot support an answer.
• Combine deterministic checks, human review, and calibrated model graders instead of trusting one aggregate score.
• Version questions, corpus, chunks, embeddings, retrieval settings, prompts, models, and graders for reproducible regression results.

Testing a RAG pipeline for hallucinations means verifying that answer claims are supported by the authorized evidence the application actually retrieved. The test should also explain where a failure began. A wrong answer may come from poor document ingestion, missing retrieval, an irrelevant reranker, context truncation, generation that ignores evidence, or a citation renderer that points to the wrong source.

A single answer accuracy score hides those distinctions. A practical QA strategy records the retrieval set, assembled context, answer, atomic claims, citations, identity, and versions. It uses answerable and unanswerable questions, deterministic and semantic graders, human calibration, and regression gates that reflect business risk.

TL;DR

Evaluate these layers independently, then add end-to-end tasks. That combination gives both diagnosis and user-level confidence.

1. Define Hallucination When Testing a RAG Pipeline for Hallucinations

For RAG testing, a useful operational definition is: a material answer claim is hallucinated when it is not supported by, or is contradicted by, the approved evidence available to the answer under the tested identity and time. This definition is narrower and more testable than the answer feels wrong. It also recognizes that a fact may be generally true yet unacceptable if the product promises answers only from a controlled corpus.

Break answers into atomic claims. The Pro plan includes SSO and costs $40 contains at least two claims. One source might support SSO while no supplied source supports the price. Grading the sentence as a whole can miss the unsupported detail. Label claims as entailed, contradicted, not in context, nonfactual, or requiring expert judgment.

Define materiality by product risk. A wrong document title is different from a wrong medication instruction, eligibility rule, or contract term. High-risk domains may require every factual claim to have direct approved support and a human-reviewed evaluation set. Lower-risk summaries may allow harmless connective language.

Ground truth is time- and identity-bound. A policy valid last quarter may be wrong today. A document visible to an administrator may be forbidden to a customer. Store effective dates, document versions, tenant, role, locale, and expected sources with every case. For the wider quality model, see how to test generative AI applications.

2. Map the Pipeline and Capture Trace Evidence

Draw the actual flow: source acquisition, parsing, normalization, chunking, metadata enrichment, embedding, indexing, query rewriting, filters, hybrid search, reranking, context packing, prompt construction, model generation, citation mapping, rendering, feedback, and logs. Each transformation can introduce or conceal a grounding defect.

Instrument the pipeline so a failed evaluation stores at least:

• Test case and authenticated principal.
• Original question and rewritten or expanded queries.
• Retrieved chunk IDs, scores, ranks, metadata, ACL decisions, and document versions.
• Reranked order and dropped chunks.
• Exact context sent to the model, subject to secure retention.
• Prompt and model identifiers, parameters, and response ID where available.
• Raw answer, parsed citations, refusal state, latency, and token or cost metadata.
• Grader versions, claim labels, and human overrides.

Do not log protected context into a system with broader access than the source. Use stable IDs and restricted artifact storage. Redact secrets and personal data from evaluation views. Observability must preserve the authorization boundary.

Trace evidence turns the bot hallucinated into a diagnosis. If the gold chunk was never retrieved, tune ingestion, filters, query transformation, or retrieval. If it ranked first but was truncated, fix packing. If it reached the prompt and the answer contradicted it, investigate prompt, model, or conflicting context. If the text is grounded but the displayed citation is wrong, fix mapping or rendering.

3. Build a RAG Hallucination Evaluation Dataset

Start with real task patterns, not synthetic trivia alone. Ask support, operations, legal, product, and QA owners which questions matter, which sources are authoritative, and which mistakes are costly. Sanitize production queries before reuse. For each case, store the question, user context, expected answer properties, required source IDs, forbidden source IDs, reference claims, acceptable abstention, risk tier, and notes.

Create balanced case families:

Include paraphrases without letting them dominate the corpus. Split evaluation by underlying information need, not random question rows, so near-duplicates do not leak across development and holdout sets. Keep a locked challenge set for release decisions and a living set for newly discovered failures.

4. Test Retrieval Quality Before Answer Quality

Retrieval cannot guarantee truth, but generation cannot cite evidence it never receives. For cases with labeled required chunks or documents, measure whether at least one required item appears in the top k, the rank of the first required item, and how much retrieved context is relevant. Use document-level labels when chunk boundaries change frequently, then add chunk-level labels for diagnostic suites.

Recall at k answers whether required evidence was retrieved. Precision at k estimates how much retrieved material is relevant. Mean reciprocal rank or normalized ranking measures can help compare ordering, but no metric replaces risk inspection. A retriever can achieve good average recall while missing every rare high-risk policy question. Report slices by product, language, tenant, document type, query length, and risk tier.

Test metadata and authorization filters as hard invariants. A semantically perfect result from another tenant is a failure even if the generator refuses to quote it. Verify filtering before or during retrieval, not only after generation. Seed forbidden canary documents that are highly similar to the question and confirm they never enter the authorized retrieval set.

Exercise ingestion defects: scanned PDFs, tables, headers, footnotes, merged columns, deleted documents, duplicate versions, broken encoding, and partial updates. Compare indexed content and metadata with the source. The RAG test case design guide provides additional boundary cases for chunking and retrieval.

5. Test Context Assembly and Conflict Handling

The retrieval list is not necessarily the model context. Reranking, token limits, deduplication, prompt templates, and source formatting can drop or distort evidence. Capture the exact packed context and verify the required passage survived. Test questions whose answer appears near the end of a long chunk or document, in a table cell, or across two chunks.

Create controlled conflicts. Place an old policy and a newer approved policy in the corpus with clear effective dates and authority metadata. The expected behavior may be to choose the current authoritative source, state the conflict, or ask for clarification. Encode that rule in application logic or prompt context, then test it. Do not assume embedding similarity understands source authority.

Vary context ordering. If the answer changes when the same approved chunks are reordered, the system may be overly sensitive to position. Add irrelevant but topically similar chunks to measure distraction. Add duplicates to see whether repeated text overwhelms a more authoritative source. Test context truncation at realistic maximum query and history sizes.

Source delimiters and metadata must be unambiguous. Assign stable document and chunk IDs outside the content, and keep untrusted document text separated from application instructions. If citations are generated from model-authored numbers without server mapping, the model can cite nonexistent sources. Prefer a controlled mapping from allowed IDs to rendered links.

6. Measure Faithfulness at the Claim Level

Faithfulness asks whether the answer follows from the supplied context, not whether the answer resembles a reference string. Exact-match metrics are useful for IDs, dates, enumerations, or short canonical values, but they punish valid paraphrases and may reward a copied sentence with the wrong meaning. Claim-level entailment is better for explanatory answers.

Use a pipeline: segment the answer into atomic material claims, retrieve the cited or relevant context for each claim, and label support. Deterministic checks can handle numbers, names, source IDs, and required phrases. Human reviewers handle nuanced policy or high-risk claims. A model grader can scale review after calibration, but it must see only the evidence and rubric, return structured reasons, and be monitored for bias and drift.

Report at least:

• Supported material claims divided by all material claims.
• Contradicted material claim count.
• Unsupported material claim count.
• Fully faithful answers, where every material claim is supported.
• Severe hallucination rate by risk tier.
• Nonfactual or conversational text excluded from scoring.

Do not average away a critical contradiction. A response with nine correct claims and one wrong eligibility rule can be unusable. Establish hard gates for severe claims and broader thresholds for lower-risk explanatory text. Review the LLM evaluation metrics guide for tradeoffs among deterministic, semantic, and human grading.

7. Verify Citations and Attribution

Citation presence is not citation correctness. For every citation, test validity, entailment, completeness, and placement. Validity means the source ID exists and the user may access it. Entailment means the cited passage supports the nearby claim. Completeness means material factual claims have citations when the product promises them. Placement means a reader can tell which claim the citation supports.

Build cases where a source mentions the same entity but not the claimed fact. This catches topical citations that look convincing without providing evidence. Include multiple documents with similar titles, old and new versions, and a secondary source that conflicts with the authoritative one. Verify links resolve to the correct document version and, where supported, the relevant passage.

If the generator returns source IDs, constrain them to the provided candidate list and validate server-side. Never render arbitrary model-generated URLs as trusted citations. Apply normal URL and output-safety controls. If the answer cites source S3 but S3 was not supplied, fail or remove the citation and flag the response.

Test partial support. A sentence may contain two claims with one citation supporting only the first. Claim segmentation should expose that mismatch. For synthesized answers, allow multiple citations per claim and record which source supports which component. Human reviewers should be able to open the exact evidence without reconstructing the prompt.

8. Test Abstention, Ambiguity, and Out-of-Scope Questions

A grounded RAG system must know when authorized evidence is insufficient. Build unanswerable cases whose topics are plausible but absent, cases where the corpus contains only part of the requested answer, and cases where another tenant holds the relevant document. The correct behavior is a clear limitation, not an answer from model memory.

Define acceptable abstention. It may say that the available sources do not contain the answer, ask for a specific missing detail, offer related supported information, or direct the user to an owner. It should not falsely claim that no such information exists anywhere. Test for leaked hints from forbidden sources inside the refusal.

Measure correct abstention and over-refusal separately. A system that declines every question has excellent hallucination avoidance and no utility. Include clearly answerable neighbors for every unanswerable case. Vary typos, paraphrases, history, and locale so abstention does not depend on exact wording.

Ambiguous questions should trigger clarification when different interpretations lead to materially different answers. If the application chooses a default interpretation, it should state that assumption and ground the answer. Test whether conversation history resolves the ambiguity correctly and whether stale history causes the wrong entity or time period to be used.

9. A Runnable Claim and Citation Evaluator

The following standalone Python program evaluates a simplified response artifact. It uses only the standard library, so save it as evaluate_rag.py and run python evaluate_rag.py. Production systems need semantic entailment and human review, but deterministic source and claim mappings remain valuable regression checks.

from dataclasses import dataclass
from typing import FrozenSet

@dataclass(frozen=True)
class Claim:
    text: str
    cited_sources: FrozenSet[str]
    supporting_sources: FrozenSet[str]
    material: bool = True

@dataclass(frozen=True)
class RagCase:
    allowed_sources: FrozenSet[str]
    claims: tuple[Claim, ...]
    should_abstain: bool
    did_abstain: bool

def evaluate(case: RagCase) -> dict[str, float | int | bool]:
    material = [claim for claim in case.claims if claim.material]
    supported = 0
    invalid_citations = 0
    uncited = 0

    for claim in material:
        invalid_citations += len(claim.cited_sources - case.allowed_sources)
        if not claim.cited_sources:
            uncited += 1
        valid_citations = claim.cited_sources & case.allowed_sources
        if valid_citations & claim.supporting_sources:
            supported += 1

    total = len(material)
    return {
        "material_claims": total,
        "supported_claims": supported,
        "faithfulness": supported / total if total else 1.0,
        "invalid_citations": invalid_citations,
        "uncited_claims": uncited,
        "abstention_correct": case.should_abstain == case.did_abstain,
        "fully_grounded": supported == total and invalid_citations == 0,
    }

case = RagCase(
    allowed_sources=frozenset({"policy-v3", "faq-12"}),
    claims=(
        Claim(
            text="Refund requests are accepted within 30 days.",
            cited_sources=frozenset({"policy-v3"}),
            supporting_sources=frozenset({"policy-v3"}),
        ),
        Claim(
            text="A manager always approves the request.",
            cited_sources=frozenset({"faq-12"}),
            supporting_sources=frozenset(),
        ),
    ),
    should_abstain=False,
    did_abstain=False,
)

result = evaluate(case)
print(result)
assert result["faithfulness"] == 0.5
assert result["fully_grounded"] is False

In a real harness, claim labels come from reviewed fixtures or a calibrated grader, not from the answer generator. Store grader reasons and sample disagreements. Run deterministic validations first so an expensive judge is not used to discover a nonexistent source ID.

10. Testing a RAG Pipeline for Hallucinations in CI

Create several test tiers. A fast commit suite checks ingestion contracts, ACL filters, stable retrieval cases, source-ID validity, JSON shape, and deterministic answers. A scheduled suite runs broader paraphrases, multiple trials, semantic faithfulness grading, adversarial documents, and performance checks. A release suite includes locked high-risk cases and human review of changed outcomes.

Version every moving part: corpus snapshot, parser, chunker, metadata schema, embedding model, index, query rewrite, retrieval parameters, reranker, context template, model, generation settings, citation mapper, grader, and dataset. Save a configuration fingerprint with results. Without it, a score change cannot be reproduced or attributed.

Compare candidates against an approved baseline by case and slice, not only by average. Show newly failed questions, newly retrieved forbidden sources, changed citations, abstention shifts, and latency or cost regressions. Use canary deployment and production sampling after release, with privacy-aware review and a rollback trigger.

Treat user feedback as a lead, not a label. A thumbs-down may reflect tone or missing product functionality rather than hallucination. Route samples through claim and evidence review before adding them to the regression set. Periodically deduplicate and rebalance the set so a large easy category does not dominate release metrics.

11. Review Failures With Domain Experts

Automated scores become useful only when experts review the failures they summarize. Hold a regular error-analysis session with QA, retrieval engineers, domain owners, security, and product. Sample severe failures, changed outcomes, grader disagreements, correct abstentions, and over-refusals. For each item, identify the first failed layer and decide whether the dataset, oracle, corpus, policy, or application needs correction.

Maintain adjudication guidance for recurring disputes. It should define source authority, treatment of partially supported claims, acceptable connective language, citation completeness, temporal precedence, and when a clarifying question is required. Record the decision and evidence, then relabel related cases consistently. Do not quietly change gold labels to make a release score improve.

Use disagreement as a diagnostic signal. If qualified reviewers often disagree, the product rule or evidence may be ambiguous. If a model grader disagrees mainly on tables, negation, or long contexts, create a dedicated slice and adjust the evaluation design. Periodically re-review older cases against current policy while retaining historical versions. Expert adjudication turns metrics into accountable release evidence and keeps the suite aligned with the decisions real users depend on.

Interview Questions and Answers

Q: How do you define hallucination in a RAG system?

I define it at the claim level: a material claim is unsupported by or contradicted by the authorized evidence supplied for that answer. The definition is bound to source version, user identity, and time. General model knowledge is not accepted when the product promises corpus-grounded answers.

Q: How do you know whether retrieval or generation caused a wrong answer?

I capture required sources, retrieved chunks, reranked order, packed context, answer claims, and citations. If evidence was missing, retrieval or ingestion failed. If evidence reached context but the answer ignored or contradicted it, generation failed.

Q: Which metrics would you use for RAG evaluation?

I use retrieval recall and rank, context relevance, claim faithfulness, contradiction counts, citation validity and entailment, correct abstention, and over-refusal. I slice by risk and domain and keep hard gates for severe failures rather than relying on one composite score.

Q: Why is citation presence insufficient?

A citation can exist, resolve, and still fail to support the claim. I verify source authorization, source identity, passage entailment, completeness for material claims, and clear placement. Model-generated URLs or IDs are validated against the supplied source set.

Q: How do you test unanswerable questions?

I include plausible questions with no approved evidence, partial evidence, and evidence available only to another identity. The system should state the limitation or ask for clarification without using model memory or leaking forbidden hints. Answerable neighbor cases detect over-refusal.

Q: Can an LLM judge another LLM's faithfulness?

Yes, as one grader after calibration, not as unquestioned ground truth. I provide only the rubric, claims, and evidence, require structured reasons, compare with human labels, monitor disagreements, and combine it with deterministic checks.

Q: What must be versioned in a RAG regression suite?

I version the dataset, corpus, parser, chunks, metadata, embeddings, index, retrieval and reranking settings, prompt, model, citation mapping, and graders. A configuration fingerprint is stored with every run.

Common Mistakes

• Scoring only the final answer and losing the layer where the failure started.
• Defining hallucination as any wording difference from one reference answer.
• Grading whole sentences that contain several independently supported claims.
• Measuring average retrieval recall without inspecting high-risk and long-tail slices.
• Allowing unauthorized but relevant chunks into context and relying on the model not to quote them.
• Checking that citations exist without checking entailment, authorization, version, and placement.
• Omitting unanswerable and ambiguous questions from the evaluation set.
• Using the answer-generating model as the only uncalibrated judge of its own output.
• Logging raw protected context into broadly accessible test dashboards.
• Randomly splitting paraphrases of the same information need across development and holdout sets.
• Comparing aggregate scores across different corpus snapshots without configuration fingerprints.
• Optimizing abstention until the system refuses useful answerable questions.

Conclusion

Testing a RAG pipeline for hallucinations is evidence engineering. Label required and forbidden sources, instrument every transformation, evaluate retrieval and context assembly, break answers into atomic claims, verify citation entailment, and demand appropriate abstention when evidence is missing.

Begin with twenty high-value information needs, including answerable, unanswerable, temporal, conflicting, and permission-sensitive cases. Capture full traces, label each failure by layer, and make the resulting corpus and configuration fingerprint part of every meaningful RAG release.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Penetration testing basics: A Complete Guide for QA (2026)
• Testing a RAG chatbot end to end (2026)
• A/B test validation: A Complete Guide for QA (2026)
• API contract testing with Pact: A Practical Guide (2026)
• API security testing basics: A Practical Guide (2026)
• Building a QA chatbot with RAG (2026)