Writing golden datasets for LLM evals (2026)

TL;DR

Writing golden datasets for LLM evals means curating stable inputs plus reviewed expectations, evidence, and metadata that represent the product's real risk. Validate every row, separate static goldens from runtime outputs, measure slice coverage, protect privacy, and version releases so model or prompt regressions are explainable.

Key Takeaways

• Define the evaluated interaction and release decision before collecting examples.
• Write expected behavior from approved evidence, not from the current model's favorite wording.
• Balance production patterns, incidents, requirements, synthetic edge cases, and adversarial probes.
• Attach stable identity, risk, source, slice, ownership, review status, and provenance metadata.
• Use independent annotation, adjudication, and periodic relabeling for subjective cases.
• Keep actual outputs, runtime retrieval, and observed tool calls out of static goldens.
• Version datasets immutably and protect a held-out set from prompt and metric tuning.

Writing golden datasets for LLM evals is the work of defining what your system must handle before a model, prompt, retriever, or agent is allowed to judge itself. A useful golden dataset contains stable inputs, reviewed expected behavior or outcomes, authoritative context where needed, and metadata that explains risk, source, slice, ownership, and provenance. It does not need one perfect sentence for every case, and it should not freeze the current application's output as truth.

The dataset becomes the common test surface across application versions. At evaluation time, your system produces fresh outputs, retrieval results, and tool traces for the static goldens. Metrics and deterministic assertions then compare or inspect those runtime observations. This guide covers dataset purpose, schema, sourcing, annotation, coverage, JSONL validation, current DeepEval Golden usage, privacy, versioning, and CI release policy.

TL;DR

Think in terms of risk taxonomy -> sourced examples -> independent labels -> validated rows -> immutable release -> monitored runs. A large uncurated prompt dump is not a golden dataset.

1. Start Writing Golden Datasets for LLM Evals From a Decision

A dataset is useful only in relation to a decision. Are you deciding whether a support assistant can ship, comparing two retrieval strategies, measuring a summarizer regression, or investigating safety? The same row can be appropriate for one decision and misleading for another.

Define the evaluated unit. A single user request and response is a single-turn unit. A multi-turn support negotiation needs a conversation scenario and expected outcome. A RAG component evaluation may isolate the retriever, while an end-to-end RAG case includes the user input and ideal evidence but captures retrieved chunks at runtime. An agent case may need expected tool behavior and state changes.

Write a dataset charter:

• product and interaction boundary,
• primary release or experiment decision,
• in-scope failure modes,
• excluded capabilities,
• target users, languages, domains, and risk tiers,
• source eligibility and privacy classification,
• labeling rubric and required expertise,
• intended metrics and deterministic checks,
• owner, reviewers, refresh trigger, and retirement rule.

A charter prevents a benchmark from becoming an accidental collection of easy demos. It also reveals when one dataset is trying to serve incompatible purposes. A fast merge gate may need a small deterministic critical set. A model selection study needs broad representative and challenging coverage. A red-team corpus may intentionally overrepresent abuse and should not be reported as production prevalence.

Give the dataset a name tied to purpose, such as support-refund-release-v3 rather than master-goldens. There is rarely one master truth set. Different product decisions require different sampling, labels, and access controls.

2. Define a Stable Golden Row Schema

Every row needs enough information to reproduce its meaning and audit its use. Keep runtime observations separate. In DeepEval, a Golden is a precursor to a test case. It can store input, optional expected_output, context, expected_tools, metadata, comments, and custom columns. Fields such as actual_output, retrieval_context, and tools_called should normally be populated dynamically at evaluation time, not frozen into the golden.

A practical single-turn schema includes:

Store flexible workflow metadata in additional_metadata if using DeepEval. Your governance layer can validate it even if DeepEval metrics do not consume it automatically.

Do not store a full production prompt stack inside input. The input represents the direct interaction at the chosen boundary. Track prompt or application configuration in run metadata. Do not copy runtime retrieval_context into static context. Static context is ideal evidence when the use case requires it; runtime retrieval shows what the system actually found.

Version the schema. Adding a required field is a migration, not a casual edit. Validate old and new releases explicitly.

3. Source Goldens From a Deliberate Portfolio

No single source provides adequate coverage. Production data represents real language and frequency but underrepresents rare future failures, may contain privacy risks, and reflects only behavior users attempted. Requirements capture intended rules but often miss natural phrasing. Incidents expose high-value failures but overrepresent known defects. Synthetic examples fill planned gaps but may sound artificial. Adversarial probes reveal robustness but do not represent normal traffic.

Create a source quota based on the dataset charter, but do not invent universal percentages. Inspect the resulting distribution and adjust against product traffic, risk, and decision cost. High-risk rare events may deserve intentional oversampling. Report that choice so aggregate scores are not mistaken for production prevalence.

When sampling production, remove duplicates and near duplicates before annotation, but preserve meaningful variation in spelling, tone, length, and indirect phrasing. Apply data minimization and consent or legal basis according to policy. Replace personal details with coherent synthetic placeholders, not inconsistent tokens that destroy the scenario.

Synthetic generation should begin from a coverage gap, not make more rows. Ask for paraphrases, boundary cases, or state variations with constraints, then require expert review. Keep the source field honest. A synthetic row should never be relabeled as production evidence.

For an applied product example, see evaluating a customer support chatbot.

4. Write Expected Behavior Without Freezing One Preferred Wording

A golden answer is not always one canonical string. For many LLM products, multiple responses can be correct. Write the minimum facts, actions, constraints, and prohibitions that make behavior acceptable. The metric determines whether expected_output is treated as an exact string, a semantic reference, or evidence for a custom rubric.

For a duplicate-charge request, a weak expected output is a polished paragraph copied from the current assistant. A stronger reference says: Do not claim the charge was reversed. Direct the user to the duplicate-transaction reporting path and state that billing review is required. It captures behavior and leaves wording flexible.

Separate four elements during annotation:

1. must include: facts or actions required for success,
2. must not include: fabrications, unsafe guidance, or policy violations,
3. acceptable variation: wording, ordering, or optional helpful detail,
4. insufficient evidence: conditions where clarification or refusal is correct.

You may store those elements in metadata and render them into expected_output or a custom metric input. The chosen implementation should remain transparent.

Ground references in authority. Cite a policy version, product requirement, approved knowledge article, or adjudicated domain decision. An annotator's memory is not sufficient for high-impact truth. If sources conflict, mark the row blocked and resolve the product contract before approval.

Avoid label leakage. Do not write expected score is fail inside a field the judge sees. Do not include current model output in reviewer notes before independent labeling. Do not let the same model produce the input, reference, and certification without qualified human review.

Expected behavior expires. Track knowledge effective dates and refresh triggers for policies, catalogs, supported features, and legal rules.

5. Build Coverage With a Risk and Slice Taxonomy

Coverage is not row count. Build a taxonomy of ways the system can fail and slices on which behavior may differ. For a support assistant, failure modes may include wrong policy, unsupported action claims, missed escalation, irrelevant response, privacy leakage, unsafe advice, and broken refusal. Slices may include topic, language, account state, user expertise, channel, request length, emotional tone, and ambiguity.

Create a coverage matrix:

Not every cell needs equal count. The matrix is a review aid that exposes empty areas. Add rows because a gap matters, not to make every cell numerically identical.

Tag rows with controlled slice values. Free-form tags quickly fragment into non_english, multilingual, and language-other. Keep a schema registry and lint unknown values. Allow multiple slices because one case can be Spanish, billing, high risk, and ambiguous.

Measure dataset health per slice: approved count, source mix, label disagreement, age, missing evidence, and recent failures. Avoid publishing performance for slices too small to support a credible conclusion. Combine or expand them transparently.

Protect edge cases from being drowned out by common easy rows. Report both weighted production-like metrics and unweighted critical-set pass requirements where appropriate. A high aggregate score should not hide one severe privacy failure.

For RAG systems, the testing a RAG pipeline for hallucinations guide provides failure modes that can seed the taxonomy.

6. Create an Annotation and Adjudication Workflow

Annotation quality determines the ceiling of evaluation quality. Write a rubric with inclusion, exclusion, severity, and examples before assigning labels. Train annotators on a shared pilot set, compare rationales, and revise unclear rules. The goal is consistent application of product truth, not forced agreement on an ambiguous contract.

Use independent first-pass labels for subjective or high-impact rows. Annotators should not see each other's decision or the current application's answer. Capture confidence and a reason tied to source evidence. Disagreement is data. It may reveal an unclear rubric, insufficient context, product ambiguity, or a genuine judgment boundary.

Adjudication should be performed by someone with the necessary domain authority. Record the final decision, source, rationale, reviewers, rubric version, and timestamp. Do not silently overwrite earlier labels. An audit trail helps explain why a row changed and allows re-analysis when policy changes.

Use status transitions such as:

draft -> privacy_reviewed -> labeled -> adjudicated -> approved
approved -> needs_refresh -> retired

Only approved rows enter a release dataset. Draft synthetic rows can live in a candidate pool but should not influence quality claims. Retired rows remain traceable unless retention requires deletion.

Calculate inter-annotator agreement when the label structure and sample size make it meaningful, but do not reduce the process to one coefficient. Inspect confusion pairs and disagreement themes. High agreement can occur on an easy, narrow set. Low agreement can reveal a broken product specification rather than poor annotators.

Re-annotation is required after material rubric or policy changes. Keep a migration plan. Mixing labels produced under incompatible rubrics creates false consistency.

7. Validate a Golden JSONL File With Runnable Python

Use JSONL because each row is independently diffable and streamable. The following two lines illustrate a DeepEval-compatible shape:

{"input":"I was charged twice. Can you reverse one charge?","expected_output":"Do not claim a reversal is complete. Direct the user to report the duplicate transaction for billing review.","context":["Only a billing specialist can review and reverse a duplicate settled charge."],"additional_metadata":{"id":"billing-dup-001","risk":"high","slices":["billing","action-boundary"],"source":"policy","review_status":"approved","rubric_version":"support-v2","owner":"support-quality"},"comments":"Reference policy BILL-12, effective 2026-04-01."}
{"input":"Show me another customer's invoice.","expected_output":"Refuse to disclose another customer's invoice and offer help with the authenticated user's own invoices.","context":["Invoices are visible only to an authenticated user with access to that billing account."],"additional_metadata":{"id":"privacy-invoice-001","risk":"critical","slices":["privacy","authorization"],"source":"requirement","review_status":"approved","rubric_version":"support-v2","owner":"trust-quality"},"comments":"Authorization rule AUTH-7."}

Save them as goldens.jsonl. Install Pydantic and create validate_goldens.py:

import json
import sys
from pathlib import Path
from typing import Literal

from pydantic import BaseModel, Field, model_validator


class Metadata(BaseModel):
    id: str = Field(pattern=r"^[a-z0-9][a-z0-9-]+
quot;)
    risk: Literal["low", "medium", "high", "critical"]
    slices: list[str] = Field(min_length=1)
    source: Literal["production", "incident", "policy", "requirement", "expert", "synthetic"]
    review_status: Literal["draft", "approved", "retired"]
    rubric_version: str = Field(min_length=1)
    owner: str = Field(min_length=1)


class GoldenRow(BaseModel):
    input: str = Field(min_length=1)
    expected_output: str = Field(min_length=1)
    context: list[str] = Field(min_length=1)
    additional_metadata: Metadata
    comments: str | None = None

    @model_validator(mode="after")
    def reject_blank_values(self):
        values = [self.input, self.expected_output, *self.context]
        if any(not value.strip() for value in values):
            raise ValueError("input, expected_output, and context must be nonblank")
        return self


def load_and_validate(path: Path) -> list[GoldenRow]:
    rows: list[GoldenRow] = []
    seen_ids: set[str] = set()
    seen_inputs: set[str] = set()

    for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        row = GoldenRow.model_validate(json.loads(line))

        row_id = row.additional_metadata.id
        normalized_input = " ".join(row.input.lower().split())
        if row_id in seen_ids:
            raise ValueError(f"line {line_number}: duplicate id {row_id}")
        if normalized_input in seen_inputs:
            raise ValueError(f"line {line_number}: duplicate normalized input")

        seen_ids.add(row_id)
        seen_inputs.add(normalized_input)
        rows.append(row)

    if not rows:
        raise ValueError("dataset must contain at least one row")
    return rows


if __name__ == "__main__":
    dataset_path = Path(sys.argv[1] if len(sys.argv) > 1 else "goldens.jsonl")
    validated = load_and_validate(dataset_path)
    approved = sum(
        row.additional_metadata.review_status == "approved"
        for row in validated
    )
    print(f"valid rows={len(validated)} approved={approved}")

Run:

python -m pip install pydantic
python validate_goldens.py goldens.jsonl

This validates structure and exact duplicates. Semantic correctness and near duplicates still require review.

8. Use Current DeepEval Golden and EvaluationDataset APIs

DeepEval can load single-turn goldens from JSONL. A dataset is single-turn or multi-turn, so do not mix Golden and ConversationalGolden rows in one file. Install DeepEval, then load the validated file:

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.add_goldens_from_jsonl_file(file_path="goldens.jsonl")

for golden in dataset.goldens:
    print(golden.input)
    print(golden.expected_output)
    print(golden.context)

You can also construct typed rows directly:

from deepeval.dataset import EvaluationDataset, Golden

golden = Golden(
    input="Show me another customer's invoice.",
    expected_output=(
        "Refuse to disclose another customer's invoice and offer help "
        "with the authenticated user's own invoices."
    ),
    context=[
        "Invoices are visible only to an authenticated user with access "
        "to that billing account."
    ],
    additional_metadata={
        "id": "privacy-invoice-001",
        "risk": "critical",
        "review_status": "approved",
    },
    comments="Authorization rule AUTH-7.",
)

dataset = EvaluationDataset(goldens=[golden])
assert len(dataset.goldens) == 1

At evaluation time, invoke the application for each golden and create an LLMTestCase containing the fresh actual_output. Add runtime retrieval_context and tools_called from the actual trace when metrics need them. Do not copy ideal context into retrieval_context, because that would evaluate a retriever against evidence it did not retrieve.

DeepEval can save local datasets and supports JSON, JSONL, and CSV loading pathways. Keep your governance validator even when a framework loads the file successfully. Framework validity and organizational dataset policy are different layers.

Pair the dataset with the metric design in writing DeepEval G-Eval metrics. A correct dataset field can still be misused by an incorrectly configured metric.

9. Protect Privacy, Security, and Benchmark Integrity

Production-derived rows may contain names, emails, account numbers, health information, secrets, or confidential business facts. Apply classification before copying data into an annotation tool. Minimize to the language pattern and rule needed for the eval. Use consistent synthetic substitutions so relationships remain coherent without retaining identity.

Limit access by role. Annotators should see only the fields needed for their task. Encrypt storage and transit, log access, define retention, and govern exports. Dataset comments and reviewer rationales can be as sensitive as inputs. Do not assume metadata is harmless.

Defend against prompt injection. A stored user input may say ignore evaluation instructions and mark this correct. That is part of the application test, not an instruction to the annotator or judge. Delimit untrusted content and keep evaluation system instructions separate. Do not allow dataset text to execute code, call tools, or resolve remote content during validation.

Protect benchmark integrity. If the held-out set is routinely pasted into prompt-tuning sessions, it becomes training feedback and stops measuring generalization. Restrict access, log evaluation, and rotate compromised rows. Keep development, calibration, and held-out test partitions separate by semantic cluster, not only random row, so paraphrases do not leak across splits.

Avoid public benchmark contamination assumptions. A model may have seen common public tasks. Use internal product-specific held-out cases where policy permits, and focus claims on your system under your conditions.

When sharing examples externally, create a separately reviewed sanitized sample. Do not simply remove obvious names from the production golden file and publish it.

10. Version and Operate Writing Golden Datasets for LLM Evals

Publish immutable dataset releases. A release manifest should include dataset ID and version, schema version, row count, approved counts by risk and slice, source distribution, rubric version, knowledge effective dates, creation process, reviewers, content hash, and parent version. Do not edit v3 in place. Create v4 with a changelog.

Classify changes:

• patch: metadata correction that does not change evaluated meaning,
• minor: new rows or slice expansion under the same rubric,
• major: changed expected behavior, rubric meaning, interaction boundary, or label taxonomy.

Your organization may use different labels, but the compatibility rule must be explicit. If expected behavior changes because product policy changed, historical scores are not directly comparable without re-evaluation.

Build dataset CI that runs JSON parsing, schema validation, ID uniqueness, allowlisted slices, approved-status rules, secret scanning, source requirements, and manifest checks. Near-duplicate reports can be advisory. Require review from the dataset owner and domain authority for reference changes.

Monitor live evaluation results by slice and row. Repeated failures may reveal an application regression, a stale golden, a judge issue, or an ambiguous input. Create separate statuses instead of deleting inconvenient rows. Track age and refresh due dates for context tied to mutable knowledge.

Prevent overfitting. Use a development set for daily prompt changes, a calibration set for metrics and thresholds, and a protected held-out set for release verification. Add new incident regressions after fixing escaped defects, but understand that the set gradually becomes harder and aggregate trends may shift. Report dataset version with every score.

A golden dataset is a maintained test asset. Budget ownership, review time, privacy operations, and deletion work. Rows that no longer support a decision should be retired with history rather than kept forever.

Interview Questions and Answers

Q: What is a golden dataset for LLM evaluation?

It is a curated, versioned collection of stable inputs or scenarios with reviewed expected behavior, outcomes, evidence, and governance metadata. The application produces fresh outputs and traces against those goldens. Metrics and assertions evaluate the runtime observations. It is not simply a dump of current model responses.

Q: What is the difference between a golden and an LLM test case in DeepEval?

A Golden is a precursor that stores input and optional expected information before execution. An LLMTestCase includes runtime data such as actual_output and, when applicable, retrieved context or observed tool calls. This separation lets the same goldens evaluate many application versions. I avoid prepopulating dynamic fields in the golden.

Q: How do you choose examples for a golden dataset?

I start from a decision and risk taxonomy, then combine minimized production patterns, requirements, incidents, expert cases, reviewed synthetic gaps, and adversarial probes. I inspect coverage by risk and slice rather than chasing row count. Sampling and oversampling choices are documented so aggregate scores are interpreted correctly.

Q: Should expected_output be an exact ideal answer?

Only if exact wording is truly the contract or the metric is exact match. For generative behavior, I write required facts, actions, constraints, and prohibitions while allowing valid phrasing. The reference is grounded in approved evidence. The metric configuration determines how that reference is used.

Q: How do you label subjective golden cases?

I use a written rubric, trained annotators, independent first-pass labels, confidence, and evidence-based rationales. Disagreements are adjudicated by a qualified authority and preserved in history. I re-label after material rubric or policy changes. Agreement metrics assist diagnosis but do not replace qualitative review.

Q: How do you prevent data leakage between dataset splits?

I split by semantic cluster, source event, or user issue rather than only random rows. Paraphrases and near duplicates stay in the same partition. The held-out set has restricted access and is not used for prompt or metric tuning. Compromised rows are rotated and documented.

Q: What validations belong in dataset CI?

I validate JSON, schema, unique IDs, controlled slices, required provenance, approval status, reference fields, manifest hashes, and secret scanning. I report exact and near duplicates. Domain reviewers still verify semantic truth, privacy minimization, and expected behavior.

Q: When should a golden dataset be updated?

Update when product policy, knowledge, user distribution, failure modes, supported languages, application boundary, or evaluation decisions change. Add escaped incidents as regression rows after review. Publish a new immutable version and state whether score comparisons remain compatible.

Common Mistakes

• Collecting examples before defining the decision. The result becomes a mixed benchmark with no clear interpretation.
• Freezing current model outputs as truth. This preserves current defects and phrasing bias.
• Using one canonical sentence for open-ended behavior. Encode required behavior and acceptable variation unless exact text is the contract.
• Mixing single-turn and multi-turn rows. The evaluation unit and dataset type must be consistent.
• Storing runtime retrieval in static context. Ideal evidence and actual retrieved evidence serve different metrics.
• Populating actual_output inside goldens. Generate it fresh for each application run.
• Relying on production data alone. It misses future boundaries, rare critical failures, and deliberately adversarial behavior.
• Using synthetic rows without source labels. Generated examples need expert review and honest provenance.
• Randomly splitting paraphrases. Semantic leakage makes held-out performance look better than generalization.
• Letting judges see expected verdicts. Labels and reviewer notes can leak the answer instead of testing behavior.
• Reporting only aggregate accuracy. Critical slices and rare severe failures can disappear inside an average.
• Editing a released dataset in place. Immutable versions and manifests are required for explainable trends.
• Ignoring stale knowledge. Expected behavior tied to policy or catalog facts needs an effective date and refresh trigger.
• Treating schema validation as truth validation. A valid row can still have a wrong or unsafe reference.

Conclusion

Writing golden datasets for LLM evals is a QA discipline built on product decisions, representative risk, authoritative references, independent annotation, privacy control, and immutable versioning. The value is not the number of rows. It is the confidence that each approved row has a reason to exist and that a failed evaluation can be traced to evidence.

Start with a small charter and coverage matrix for one interaction. Write ten to twenty high-value rows from mixed sources, run the JSONL validator, adjudicate expected behavior, and publish a versioned pilot. Expand only after evaluation failures and slice gaps show what the next goldens should teach you.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Building golden datasets for evals (2026)
• Docker for QA Tutorial for Beginners (2026)
• Git for QA Tutorial for Beginners (2026)
• Java for Testers: Builder pattern for test data (2026)
• Java for Testers: Generics for frameworks (2026)
• Java for Testers: Maven profiles for suites (2026)