Build an Adversarial RAG Evaluation Dataset

TL;DR

Build a versioned JSONL dataset from a fixed corpus, label cases by attack type, attach expected evidence and behavior, validate every record, and score each RAG layer separately.

Key Takeaways

• Define attack families from real RAG risks before generating questions.
• Attach stable evidence IDs and explicit expected behavior to every case.
• Score retrieval, answers, abstention, and citations separately.
• Validate schema, references, duplicates, coverage, and answer leakage.
• Use deterministic generation and versioned corpora in CI.
• Preserve difficult failures as regression tests.

A friendly golden set is not enough to expose production failures. To build adversarial RAG evaluation dataset coverage, create controlled examples that pressure retrieval, context selection, grounding, refusal, and citation support while remaining realistic and answerable from a defined corpus.

This tutorial gives you a runnable Python pipeline. It complements the RAG application evaluation complete guide, which connects retrieval and generation metrics into an end-to-end release strategy.

You will create a source corpus, generate deterministic adversarial cases, validate every record, run a local baseline, and add CI checks. The output is a versionable JSONL asset, not a one-time prompt experiment.

What You Will Build

You will build:

• A corpus with stable document and passage identifiers.
• A typed case schema containing questions, attacks, evidence, and expected behavior.
• Seven cases covering lexical mismatch, entity collision, temporal conflict, false premises, unanswerable requests, distractors, and citation traps.
• A validator for duplicate IDs, unknown evidence, missing coverage, and answer leakage.
• A deterministic baseline retriever and layer-specific evaluator.
• A CI gate you can connect to your real RAG endpoint.

Review quality matters equally. Determinism matters. The same corpus and generator revision should produce the same file so every failure can be reproduced and reviewed.

Prerequisites

Use Python 3.12 for the commands in this tutorial. Python 3.11 or newer is compatible with the shown type syntax. Create an isolated project and install pinned major-version ranges:

mkdir adversarial-rag-eval
cd adversarial-rag-eval
python -m venv .venv
source .venv/bin/activate
python -m pip install "pydantic>=2.7,<3" "scikit-learn>=1.5,<2" "pytest>=8,<10"
mkdir -p data tests
python -m pip freeze > requirements.lock

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1. Confirm imports with python -c "import pydantic, sklearn; print('ready')". No API key or hosted model is required.

Step 1: Define the Corpus and Threat Model

Save this compact corpus as data/corpus.json:

[
  {
    "passage_id": "policy-2025#refund",
    "title": "Refund Policy, effective 2025-01-01",
    "text": "Annual Pro plans purchased directly may be refunded within 30 calendar days. Monthly plans are non-refundable. Marketplace purchases follow the marketplace policy."
  },
  {
    "passage_id": "policy-2024#refund",
    "title": "Archived Refund Policy, effective 2024-01-01",
    "text": "Annual Pro plans purchased directly may be refunded within 14 calendar days. This policy was replaced on 2025-01-01."
  },
  {
    "passage_id": "security#retention",
    "title": "Security and Retention",
    "text": "Audit logs are retained for 400 days on Enterprise plans and 90 days on Pro plans. Free plans do not include audit log access."
  },
  {
    "passage_id": "regions#hosting",
    "title": "Regional Hosting",
    "text": "Customer data can be hosted in the United States or European Union. Australia is available only through a private preview agreement."
  }
]

Define attacks before writing questions:

An attack label must name a risk, not merely unusual wording. Include ordinary controls later so you can tell whether a change harms normal queries.

Verification: Run python -m json.tool data/corpus.json. It should exit successfully. Confirm passage_id values are unique and time-sensitive passages expose their dates.

Step 2: Create a Strict Schema

Create schema.py:

from enum import StrEnum
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator

class AttackType(StrEnum):
    LEXICAL_MISMATCH = "lexical_mismatch"
    ENTITY_COLLISION = "entity_collision"
    TEMPORAL_CONFLICT = "temporal_conflict"
    FALSE_PREMISE = "false_premise"
    UNANSWERABLE = "unanswerable"
    DISTRACTOR = "distractor"
    CITATION_TRAP = "citation_trap"

class Expected(BaseModel):
    model_config = ConfigDict(extra="forbid")
    mode: Literal["answer", "abstain", "correct_premise"]
    contains: list[str] = Field(default_factory=list)
    excludes: list[str] = Field(default_factory=list)
    evidence_ids: list[str] = Field(default_factory=list)

    @model_validator(mode="after")
    def valid_oracle(self):
        if self.mode == "answer" and not self.evidence_ids:
            raise ValueError("answer cases require evidence")
        if self.mode == "abstain" and self.contains:
            raise ValueError("abstain cases cannot require answer text")
        return self

class EvalCase(BaseModel):
    model_config = ConfigDict(extra="forbid")
    case_id: str = Field(pattern=r"^[a-z0-9][a-z0-9-]+
quot;)
    question: str = Field(min_length=10, max_length=500)
    attack_type: AttackType
    difficulty: Literal["medium", "hard"]
    expected: Expected
    notes: str = Field(min_length=5)
    tags: list[str] = Field(min_length=1)

Evidence IDs form the retrieval oracle. Text constraints form a lightweight answer oracle. Do not demand one exact response because valid wording varies. For complex answers, replace strings with reviewer-approved propositions or a rubric.

Verification: Import the model with python -c "from schema import EvalCase; print('schema ready')". Add an unknown field to a sample and confirm Pydantic rejects it.

Step 3: Build Adversarial RAG Evaluation Dataset Cases

Create generate.py:

from pathlib import Path
from schema import EvalCase

CASES = [
 {"case_id":"refund-lexical-001","question":"What is the cooling-off period for a yearly Pro subscription bought from the vendor?","attack_type":"lexical_mismatch","difficulty":"medium","expected":{"mode":"answer","contains":["30","calendar days"],"excludes":["14"],"evidence_ids":["policy-2025#refund"]},"notes":"Uses synonyms for annual and directly.","tags":["refund"]},
 {"case_id":"retention-entity-001","question":"A Pro customer asks about Enterprise audit history. What is the Enterprise limit?","attack_type":"entity_collision","difficulty":"hard","expected":{"mode":"answer","contains":["400 days"],"excludes":["90 days"],"evidence_ids":["security#retention"]},"notes":"Competing plan is near target.","tags":["retention"]},
 {"case_id":"refund-temporal-001","question":"Under the policy effective in February 2025, what is the annual Pro refund window?","attack_type":"temporal_conflict","difficulty":"hard","expected":{"mode":"answer","contains":["30 calendar days"],"excludes":["14"],"evidence_ids":["policy-2025#refund"]},"notes":"Requires current policy.","tags":["temporal"]},
 {"case_id":"hosting-premise-001","question":"Since Australia hosting is generally available, how does any customer enable it?","attack_type":"false_premise","difficulty":"hard","expected":{"mode":"correct_premise","contains":["private preview"],"excludes":["generally available"],"evidence_ids":["regions#hosting"]},"notes":"Correct asserted availability.","tags":["premise"]},
 {"case_id":"security-unanswerable-001","question":"Which encryption algorithm protects audit logs at rest?","attack_type":"unanswerable","difficulty":"medium","expected":{"mode":"abstain","contains":[],"excludes":["AES-256","AES"],"evidence_ids":[]},"notes":"Encryption is absent.","tags":["abstention"]},
 {"case_id":"refund-distractor-001","question":"Are monthly Pro plans refundable within 30 days?","attack_type":"distractor","difficulty":"hard","expected":{"mode":"answer","contains":["non-refundable"],"excludes":["monthly plans may be refunded"],"evidence_ids":["policy-2025#refund"]},"notes":"Annual number distracts.","tags":["negation"]},
 {"case_id":"hosting-citation-001","question":"List generally available hosting regions and cite the supporting passage.","attack_type":"citation_trap","difficulty":"hard","expected":{"mode":"answer","contains":["United States","European Union"],"excludes":["Australia is generally available"],"evidence_ids":["regions#hosting"]},"notes":"Private preview is excluded.","tags":["citation"]}
]

validated = [EvalCase.model_validate(item) for item in CASES]
path = Path("data/adversarial_cases.jsonl")
path.write_text("".join(c.model_dump_json() + "\n" for c in validated), encoding="utf-8")
print(f"wrote {len(validated)} cases")

These are explicit transformations, so reviewers can explain each case. Model-assisted generation can propose more variants later, but it must not establish its own ground truth.

Verification: Run python generate.py, then wc -l data/adversarial_cases.jsonl. Both should report seven. Run the generator twice and compare checksums. They should match.

Step 4: Validate Integrity and Leakage

Create validate.py:

import json
from collections import Counter
from pathlib import Path
from schema import AttackType, EvalCase

corpus = json.loads(Path("data/corpus.json").read_text())
known = {p["passage_id"] for p in corpus}
cases = [EvalCase.model_validate_json(line) for line in
         Path("data/adversarial_cases.jsonl").read_text().splitlines() if line]

ids = [c.case_id for c in cases]
assert len(ids) == len(set(ids)), "duplicate case IDs"
assert {c.attack_type for c in cases} == set(AttackType), "missing attack type"
assert {"answer", "abstain", "correct_premise"} <= {c.expected.mode for c in cases}

for case in cases:
    assert set(case.expected.evidence_ids) <= known, case.case_id
    question = case.question.casefold()
    for phrase in case.expected.contains:
        if len(phrase.split()) >= 4:
            assert phrase.casefold() not in question, f"leak: {case.case_id}"

print(f"valid: {len(cases)} cases")

Schema validity alone cannot establish quality. Review whether a user could naturally ask the question, whether the evidence uniquely supports the oracle, and whether the expected refusal is justified. Reject cases that are hard only because their grammar is broken.

Verification: Run python validate.py and expect valid: 7 cases. Change one evidence ID to missing#passage and confirm the validator fails.

Step 5: Run a Reproducible Retriever

Create baseline.py:

import json
from pathlib import Path
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from schema import EvalCase

corpus = json.loads(Path("data/corpus.json").read_text())
cases = [EvalCase.model_validate_json(x) for x in
         Path("data/adversarial_cases.jsonl").read_text().splitlines() if x]
texts = [f'{p["title"]} {p["text"]}' for p in corpus]
vectorizer = TfidfVectorizer(ngram_range=(1, 2))
matrix = vectorizer.fit_transform(texts)
rows = []

for case in cases:
    scores = cosine_similarity(vectorizer.transform([case.question]), matrix)[0]
    ranking = scores.argsort()[::-1][:2]
    retrieved = [corpus[i]["passage_id"] for i in ranking]
    expected = set(case.expected.evidence_ids)
    rows.append({"case_id":case.case_id,
                 "attack_type":case.attack_type.value,
                 "retrieved":retrieved,
                 "evidence_hit_at_2":not expected or bool(expected & set(retrieved))})

Path("data/baseline.json").write_text(json.dumps(rows, indent=2) + "\n")
print(f"evaluated {len(rows)} cases")

This local baseline proves the evaluation path, not production quality. A retrieval miss is useful evidence. Do not weaken the case to improve a score. For ranked retrieval analysis, test RAG retrieval recall at K. Also measure RAG context precision with RAGAS to penalize distracting context.

Verification: Run python baseline.py and python -m json.tool data/baseline.json. Expect seven rows with two passage IDs each.

Step 6: Score Each RAG Layer Separately

Save your system output as data/responses.json. Each object needs case_id, answer, citation_ids, and abstained. Create evaluate.py:

import json
from pathlib import Path
from schema import EvalCase

cases = {c.case_id:c for line in Path("data/adversarial_cases.jsonl").read_text().splitlines()
         if line for c in [EvalCase.model_validate_json(line)]}
responses = json.loads(Path("data/responses.json").read_text())
rows = []

for response in responses:
    case = cases[response["case_id"]]
    answer = " ".join(response["answer"].casefold().split())
    contains = all(x.casefold() in answer for x in case.expected.contains)
    excludes = all(x.casefold() not in answer for x in case.expected.excludes)
    abstention = case.expected.mode != "abstain" or response["abstained"] is True
    citations = (case.expected.mode == "abstain" or
                 set(case.expected.evidence_ids) <= set(response["citation_ids"]))
    rows.append({"case_id":case.case_id,"answer_pass":contains and excludes,
                 "abstention_pass":abstention,"citation_pass":citations,
                 "overall_pass":contains and excludes and abstention and citations})

Path("data/evaluation.json").write_text(json.dumps(rows, indent=2) + "\n")
print(f"passed {sum(r['overall_pass'] for r in rows)}/{len(rows)}")

Use answer constraints only as a first line check. Semantic rubrics and human review remain valuable, especially for premise correction. Citation identifiers must point to passages that support the claims, not merely appear beside them. See RAG citation correctness evaluation examples for claim-level checks.

Verification: Give refund-lexical-001 an answer containing 30 calendar days and the policy-2025#refund citation. It passes. Remove the citation and confirm answer_pass remains true while citation_pass becomes false.

Step 7: Add a CI Gate

Create tests/test_dataset.py:

import json
from pathlib import Path
from schema import AttackType, EvalCase

def cases():
    return [EvalCase.model_validate_json(x) for x in
            Path("data/adversarial_cases.jsonl").read_text().splitlines() if x]

def test_unique_ids():
    values = cases()
    assert len(values) == len({x.case_id for x in values})

def test_attack_coverage():
    assert {x.attack_type for x in cases()} == set(AttackType)

def test_evidence_exists():
    corpus = json.loads(Path("data/corpus.json").read_text())
    known = {x["passage_id"] for x in corpus}
    assert {e for c in cases() for e in c.expected.evidence_ids} <= known

Run the fixed sequence:

python generate.py
python validate.py
python baseline.py
pytest -q

Verification: Pytest reports three passed tests. In CI, call the staging RAG endpoint separately, store its raw responses, run evaluate.py, and compare critical slices against an approved baseline. Never regenerate release questions with a stochastic model in the same job that evaluates them.

Choose thresholds from measured risk, not a universal benchmark. You might require zero regression for unanswerable and temporal cases while allowing a reviewed change elsewhere. Store corpus checksum, dataset version, index revision, model identifier, prompt version, and raw output.

Step 8: Build Adversarial RAG Evaluation Dataset Coverage

Seven cases demonstrate the workflow but do not prove readiness. Expand each family with different entities, evidence positions, writing styles, and difficulty. Split development and holdout data by source scenario, not random question, or near-duplicate paraphrases can leak across sets.

Review every case:

• Can expected behavior be derived only from listed evidence?
• Does one primary attack label describe the risk?
• Is the question realistic rather than artificially confusing?
• Does the oracle permit valid paraphrases?
• Does an abstention case truly lack evidence?
• Are current and archived facts distinguishable?
• Is sensitive information excluded?
• Can a failed run be reproduced?

Report performance by attack and difficulty. Averages can rise while a critical slice falls. Preserve raw retrieval rankings, context, answers, citations, latency, and configuration.

Verification: Run shasum -a 256 data/corpus.json data/adversarial_cases.jsonl and record both values in a manifest. A reviewer using a clean checkout must reproduce them.

Step 9: Add Mutation Tests and Human Review

A validator can prove structural consistency, but it cannot prove that a case detects the failure it claims to detect. Mutation testing supplies that missing signal. Deliberately create a broken response for each oracle dimension, then confirm the evaluator fails it for the intended reason.

Create tests/test_evaluator_mutations.py:

from dataclasses import dataclass

@dataclass
class Oracle:
    contains: list[str]
    excludes: list[str]
    evidence_ids: list[str]
    abstain: bool = False

def score(answer: str, citations: list[str], abstained: bool, oracle: Oracle):
    normalized = " ".join(answer.casefold().split())
    return {
        "contains": all(x.casefold() in normalized for x in oracle.contains),
        "excludes": all(x.casefold() not in normalized for x in oracle.excludes),
        "citations": set(oracle.evidence_ids) <= set(citations),
        "abstention": not oracle.abstain or abstained,
    }

def test_wrong_temporal_value_is_caught():
    oracle = Oracle(["30 calendar days"], ["14"], ["policy-2025#refund"])
    result = score("The window is 14 days.", ["policy-2024#refund"], False, oracle)
    assert result == {
        "contains": False,
        "excludes": False,
        "citations": False,
        "abstention": True,
    }

def test_unsupported_answer_is_caught():
    oracle = Oracle([], ["AES"], [], abstain=True)
    result = score("Audit logs use AES-256.", [], False, oracle)
    assert result["excludes"] is False
    assert result["abstention"] is False

Add one mutation for each attack family. Swap current and archived evidence, substitute Pro for Enterprise, promote private preview to general availability, add a fabricated algorithm, remove a required citation, and invert a negation. If a mutation still passes, strengthen the oracle or evaluator before trusting the dataset.

Mutation tests also reveal coupling. For example, a citation mutation should fail the citation dimension without necessarily failing the answer dimension. If every mutation changes only one opaque overall score, the report will not help an engineer locate the defect.

Human review remains mandatory for semantic validity. Give reviewers the question, source passages, expected behavior, attack definition, and generator history. Hide the candidate system answer during the first review so it does not anchor the judgment. Ask one reviewer to solve the case from the corpus and another to challenge ambiguity for high-risk slices.

Use a simple review status such as proposed, evidence_checked, adversarial_checked, approved, and retired. Record reviewer identity and rationale without putting personal data into the public dataset.

Verification: Run pytest -q tests/test_evaluator_mutations.py. Expect two passed tests. Temporarily change the wrong temporal answer to 30 calendar days and confirm the contains assertion changes, proving the mutation is observable.

Design Balanced Splits and Release Reports

Do not randomly split paraphrases of the same source fact. If a refund passage produces ten questions, keep that family in one split. Otherwise the system can be tuned on nine near-duplicates and appear to generalize on the tenth.

Create a manifest beside the JSONL file:

{
  "dataset_version": "1.0.0",
  "corpus_sha256": "replace-with-generated-checksum",
  "generator_revision": "replace-with-git-commit",
  "split_policy": "group_by_source_scenario",
  "attack_types": [
    "lexical_mismatch",
    "entity_collision",
    "temporal_conflict",
    "false_premise",
    "unanswerable",
    "distractor",
    "citation_trap"
  ],
  "review_status": "approved"
}

Use semantic versioning as a team convention. Increment the patch when correcting metadata without changing case meaning, the minor version when adding backward-compatible cases, and the major version when changing schemas or oracle semantics. The convention matters less than recording exactly what changed.

A clear, reproducible release report should show counts and pass rates by attack type, difficulty, behavior mode, and source domain. Include both numerator and denominator. A perfect score on one unanswerable case is weaker evidence than a slightly lower score across a broad reviewed set.

Compare candidate and baseline on the same immutable cases. Report new failures, fixed failures, and unchanged failures. Attach retrieved passage IDs and citations for diagnosis, but sanitize content before moving production-derived artifacts into logs.

Keep development, regression, and hidden release sets separate. Engineers need immediate feedback from development cases. Regression cases preserve known incidents. A limited group should control hidden cases so repeated prompt tuning does not overfit the release gate.

Verification: Validate the manifest with python -m json.tool data/manifest.json. Recompute the corpus checksum and confirm it matches. Generate a report twice from the same stored responses and verify byte-for-byte identical output.

Troubleshooting

Weak generated questions -> Start from a written threat model and explicit transformations. Require a reviewer to explain the isolated failure.

Archived passages outrank current policy -> Add effective dates to text and metadata, then use date-aware filtering or reranking. Keep the failure as a regression case.

Answer leakage is flagged -> Rewrite the question instead of disabling the check. Document narrowly scoped exceptions for necessary entity names.

Unanswerable cases receive confident answers -> Add evidence-sufficiency logic and explicit abstention instructions. Score refusal separately.

Citations pass unsupported claims -> Evaluate sentence-level propositions. A citation being present does not prove entailment.

CI fluctuates -> Freeze dataset, index, prompt, model configuration, and temperature where supported. Retry infrastructure errors, not semantic failures.

Where To Go Next

Return to the complete RAG application evaluation guide for full release planning. Then:

• Test retrieval recall at K with the evidence IDs you created.
• Measure context precision with RAGAS to find noisy retrieved chunks.
• Evaluate RAG citation correctness with examples at claim level.

Add 10 to 20 reviewed cases per critical attack family as an early working target, then grow from sanitized production failures. Keep a hidden holdout for release decisions. Use the AI test review checklist when approving generated cases, and apply the AI agent testing complete guide when the RAG workflow includes tool-using agents.

Interview Questions and Answers

Q: What distinguishes an adversarial RAG dataset from a golden set?

A golden set represents ordinary expected behavior. An adversarial set applies controlled pressure such as conflicting dates, similar entities, false premises, or absent evidence. Both require realistic questions and defensible oracles.

Q: Why score retrieval and generation separately?

A wrong answer may come from missing evidence, noisy evidence, or failure to use correct evidence. Separate metrics identify the responsible layer and prevent one aggregate score from hiding the cause.

Q: How do you define an unanswerable oracle?

Set the expected mode to abstain, require no evidence, and list claims that must not appear. Human review must confirm the corpus lacks the requested fact.

Q: How do you prevent evaluation leakage?

Keep source-level holdouts, exclude test answers from tuning prompts, detect expected phrases in questions, and review paraphrase families. Version and restrict hidden release data.

Q: When should an LLM generate cases?

Use it to propose variants after establishing the schema, threat model, and seed examples. Validate proposals against evidence and require human review.

Q: What belongs in a CI gate?

Check schema, IDs, evidence references, and attack coverage first. Then enforce non-regression for critical retrieval, abstention, answer, and citation slices with reproducible artifacts.

Common Mistakes

• Creating difficult wording without mapping it to a real risk.
• Using exact string equality as the only answer metric.
• Treating high lexical overlap as proof of relevance.
• Combining retrieval, grounding, citations, and refusal into one score.
• Accepting synthetic cases without evidence review.
• Deleting hard failures instead of preserving regressions.
• Publishing results without corpus and configuration versions.

Conclusion

To build adversarial RAG evaluation dataset coverage that improves a real system, begin with explicit threats, stable evidence IDs, and distinct answer, correction, and abstention behaviors. Generate controlled variants, validate them, and score every RAG layer independently.

Run the frozen dataset in CI, inspect results by attack type, and expand it from observed failures. That turns adversarial evaluation into a durable quality gate.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• RAG Application Evaluation Complete Guide (2026)
• Evaluate RAG Citation Correctness with Examples
• Measuring answer faithfulness in RAG (2026)
• Measuring RAG faithfulness and relevancy (2026)
• Build Page Component Models with TypeScript Generics
• Building a QA chatbot with RAG (2026)