Set LLM Evaluation Regression Thresholds

TL;DR

Set LLM evaluation regression thresholds by defining the cost of each failure, freezing a representative paired dataset, and combining absolute quality floors with maximum allowed drops, confidence bounds, and critical-slice gates. Automate the policy as code, preserve the evidence, and route borderline or unstable changes to review instead of forcing every result into pass or fail.

Key Takeaways

• Translate product harm into separate release gates for critical errors, quality, stability, and operational health.
• Compare candidates and baselines on the same frozen examples instead of relying on unrelated aggregate scores.
• Use paired bootstrap confidence bounds so sampling noise does not become a false regression alert.
• Apply absolute floors, maximum allowed drops, and high-risk slice gates together.
• Predeclare thresholds before evaluating a candidate and keep threshold tuning separate from final validation.
• Emit a machine-readable report and make CI fail with a nonzero exit code when any required gate fails.

To set LLM evaluation regression thresholds, define acceptable change in terms of user harm, compare the candidate with the current baseline on identical examples, and fail only when a predeclared gate is breached. A useful policy combines an absolute quality floor, a maximum regression allowance, uncertainty around the paired difference, and zero-tolerance or count-based rules for critical failures.

This tutorial implements that policy as part of the complete LLM evaluation pipeline guide. You will build a dependency-free Python gate that reads cached evaluation results, calculates paired bootstrap bounds, checks overall and slice-level rules, writes JSON evidence, and returns the correct CI exit code.

The illustrative data evaluates whether support answers pass a groundedness rubric. Replace its metrics and limits with values derived from your product risk, human-reviewed baseline, traffic mix, and review capacity. There is no universal score or percentage that makes every LLM application safe to release.

What You Will Build

You will build a small project that can:

• Validate a versioned JSONL result schema before calculating metrics.
• Compare baseline and candidate outputs on the same example IDs.
• Calculate overall pass rate, critical-failure count, and per-slice rates.
• Estimate a paired bootstrap confidence interval for the candidate-minus-baseline change.
• Apply absolute, relative, slice, stability, and infrastructure gates.
• Write regression-report.json and exit with status 1 when CI must block.

The gate will distinguish pass, review, and fail. Pass means all mandatory evidence is acceptable. Review means uncertainty or a warning deserves human judgment. Fail means at least one blocking policy was violated. This separation prevents a noisy estimate from receiving the same treatment as a confirmed critical defect.

Prerequisites

Use Python 3.11 or newer. The implementation uses only the standard library, so it works without a package install. You need Git and a CI runner capable of executing Python.

mkdir llm-regression-gate
cd llm-regression-gate
python3 -m venv .venv
source .venv/bin/activate
python --version

Prepare cached, human-auditable evaluation results for one released baseline and one candidate. Each must cover the same IDs and include a binary passed decision from a frozen evaluator. For a real system, first calibrate the LLM judge against human labels, and record the judge model, rubric, prompt, and threshold alongside every run.

You should also know which slices represent elevated harm, such as safety, refunds, protected-language behavior, or tool execution. Obtain approval for the release policy from product, QA, and risk owners before the candidate results are visible.

Verification: Run python -c "import sys; assert sys.version_info >= (3, 11); print('ready')". Expect ready. Confirm the baseline configuration and result artifacts are immutable and traceable to a released version.

Step 1: Set LLM Evaluation Regression Thresholds From Risk

Start with consequences, not a favorite metric. List failures, the people affected, detectability after release, reversibility, and review cost. Then map each risk to a measurable gate. If you are still designing the surrounding test strategy, use the AI testing strategy guide to connect these release rules to functional, safety, and production checks.

Create policy.json. These numbers are illustrative and intentionally easy to replace.

{
  "policy_version": "support-release-v1",
  "minimum_examples": 100,
  "minimum_overall_pass_rate": 0.80,
  "maximum_overall_drop": 0.02,
  "confidence_level": 0.95,
  "bootstrap_samples": 10000,
  "maximum_critical_failures": 0,
  "maximum_error_rate": 0.01,
  "maximum_flip_rate": 0.03,
  "critical_slices": {
    "safety": {"minimum_examples": 20, "minimum_pass_rate": 0.95, "maximum_drop": 0.01},
    "refunds": {"minimum_examples": 20, "minimum_pass_rate": 0.85, "maximum_drop": 0.03}
  }
}

The overall rule has two parts. A candidate must score at least 0.80 and must not fall more than 0.02 below the baseline. If the baseline is 0.79, the absolute floor still blocks 0.79. If the baseline is 0.94, a candidate at 0.90 clears the floor but fails the allowed-drop rule.

Do not derive a threshold by repeatedly testing candidates until one passes. Estimate an initial policy from prior releases, human-reviewed errors, user impact, and operational capacity. Backtest it on historical runs, revise it on development data, then freeze it before the next release decision.

Verification: Ask each policy owner to explain what defect every field prevents. Confirm every numeric limit has an owner, rationale, unit, effective date, and review trigger. Run python -m json.tool policy.json and expect formatted JSON without an error.

Step 2: Freeze a Paired Regression Dataset

Use the same examples for baseline and candidate. Pairing removes variation caused by evaluating different samples and lets you measure which individual cases changed. The set should represent current traffic while deliberately covering rare, high-risk slices.

Create baseline.jsonl:

{"id":"case-001","slice":"refunds","passed":true,"critical_failure":false,"eval_error":false,"flip_rate":0.0}
{"id":"case-002","slice":"safety","passed":true,"critical_failure":false,"eval_error":false,"flip_rate":0.0}
{"id":"case-003","slice":"general","passed":false,"critical_failure":false,"eval_error":false,"flip_rate":0.0}

Create candidate.jsonl with the same IDs and slices:

{"id":"case-001","slice":"refunds","passed":false,"critical_failure":false,"eval_error":false,"flip_rate":0.0}
{"id":"case-002","slice":"safety","passed":true,"critical_failure":false,"eval_error":false,"flip_rate":0.0}
{"id":"case-003","slice":"general","passed":true,"critical_failure":false,"eval_error":false,"flip_rate":0.0}

Three rows are only a schema demonstration. The policy requires at least 100 examples and each critical slice requires 20, so this fixture must fail coverage checks. Never weaken the policy merely to make a tiny fixture green. Build a production set large enough to support the claim you intend to make.

Keep prompts, references, grader outputs, rationales, and traces in an access-controlled artifact store. This compact file contains only gate inputs. Deduplicate semantically similar cases, group related conversation turns, and version the dataset. Refresh it deliberately when production behavior drifts, but do not change it during a candidate decision.

Verification: Check that both files contain identical ID sets, unique IDs, matching slice assignments, and valid booleans. Confirm the dataset version and content hash are recorded in the evaluation run. Review whether rare but severe failures have explicit coverage rather than relying only on random traffic sampling.

Step 3: Validate Inputs and Pair Results

Create regression_gate.py with imports, loaders, schema checks, and pairing. Reject malformed or mismatched evidence instead of treating it as a quality result.

import argparse
import hashlib
import json
import random
import sys
from collections import defaultdict
from pathlib import Path

REQUIRED = {
    "id": str, "slice": str, "passed": bool,
    "critical_failure": bool, "eval_error": bool,
    "flip_rate": (int, float),
}

def load_jsonl(path):
    rows = []
    for line_number, line in enumerate(Path(path).read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        row = json.loads(line)
        for key, expected in REQUIRED.items():
            if key not in row or not isinstance(row[key], expected):
                raise ValueError(f"{path}:{line_number}: invalid {key}")
        if not 0 <= row["flip_rate"] <= 1:
            raise ValueError(f"{path}:{line_number}: flip_rate outside [0, 1]")
        rows.append(row)
    if len({row["id"] for row in rows}) != len(rows):
        raise ValueError(f"{path}: duplicate IDs")
    return {row["id"]: row for row in rows}

def pair_rows(baseline, candidate):
    if baseline.keys() != candidate.keys():
        missing = sorted(baseline.keys() - candidate.keys())
        extra = sorted(candidate.keys() - baseline.keys())
        raise ValueError(f"ID mismatch; missing={missing[:5]}, extra={extra[:5]}")
    pairs = []
    for item_id in sorted(baseline):
        old, new = baseline[item_id], candidate[item_id]
        if old["slice"] != new["slice"]:
            raise ValueError(f"Slice changed for {item_id}")
        pairs.append((old, new))
    return pairs

def file_sha256(path):
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()

Fail closed on missing rows, duplicate IDs, unknown values, changed slices, and invalid probabilities. An evaluation API timeout belongs in eval_error; it must not silently become passed: false or disappear. Otherwise quality and infrastructure health become impossible to distinguish.

Verification: Temporarily duplicate an ID and run the gate after Step 6. Expect a clear validation exception and a nonzero exit. Restore the file, then change one candidate slice and confirm the mismatch is rejected before metrics are calculated.

Step 4: Calculate Paired Metrics and Confidence Bounds

Append metric functions to regression_gate.py. The paired difference for an item is candidate pass minus baseline pass, producing -1, 0, or 1. Bootstrap those paired differences to estimate uncertainty without breaking the pairing.

def rate(values):
    return sum(values) / len(values) if values else 0.0

def percentile(sorted_values, probability):
    index = (len(sorted_values) - 1) * probability
    lower = int(index)
    upper = min(lower + 1, len(sorted_values) - 1)
    fraction = index - lower
    return sorted_values[lower] * (1 - fraction) + sorted_values[upper] * fraction

def paired_bootstrap_interval(pairs, samples, confidence, seed=20260715):
    differences = [int(new["passed"]) - int(old["passed"]) for old, new in pairs]
    rng = random.Random(seed)
    estimates = []
    for _ in range(samples):
        estimates.append(sum(rng.choice(differences) for _ in differences) / len(differences))
    estimates.sort()
    alpha = 1 - confidence
    return [percentile(estimates, alpha / 2), percentile(estimates, 1 - alpha / 2)]

def summarize(pairs):
    baseline_rate = rate([old["passed"] for old, _ in pairs])
    candidate_rate = rate([new["passed"] for _, new in pairs])
    return {
        "examples": len(pairs),
        "baseline_pass_rate": baseline_rate,
        "candidate_pass_rate": candidate_rate,
        "difference": candidate_rate - baseline_rate,
        "critical_failures": sum(new["critical_failure"] for _, new in pairs),
        "error_rate": rate([new["eval_error"] for _, new in pairs]),
        "maximum_flip_rate": max((new["flip_rate"] for _, new in pairs), default=0.0),
    }

def group_by_slice(pairs):
    groups = defaultdict(list)
    for old, new in pairs:
        groups[new["slice"]].append((old, new))
    return groups

A 95 percent interval is not a magical truth boundary. It describes sampling uncertainty under this resampling procedure and dataset. It does not cover judge miscalibration, label error, dataset shift, prompt leakage, or provider changes. Keep those risks visible through human audits and provenance checks.

Paired bootstrap analysis is especially useful when the same examples are evaluated by both systems. For prompt candidates, the paired prompt evaluation tutorial explains why per-example wins and losses are more informative than two unrelated averages.

Verification: On the three-row example, baseline and candidate pass rates should both be 0.666666..., with a difference of zero. Change only case-003 candidate to false and confirm the difference becomes approximately -0.333333. Restore it afterward.

Step 5: Set LLM Evaluation Regression Thresholds for Overall and Slice Results

Append a helper that records every gate. A blocking rule creates a fail; an uncertainty rule can create review. Keep results explicit so a developer can see what happened without reading CI code.

def check(name, actual, operator, limit, severity="fail"):
    operations = {
        ">=": actual >= limit,
        "<=": actual <= limit,
    }
    passed = operations[operator]
    return {
        "name": name, "actual": actual, "operator": operator,
        "limit": limit, "passed": passed, "severity": severity,
    }

def evaluate_policy(pairs, policy):
    summary = summarize(pairs)
    interval = paired_bootstrap_interval(
        pairs, policy["bootstrap_samples"], policy["confidence_level"]
    )
    gates = [
        check("minimum examples", summary["examples"], ">=", policy["minimum_examples"]),
        check("overall pass floor", summary["candidate_pass_rate"], ">=", policy["minimum_overall_pass_rate"]),
        check("maximum overall drop", summary["difference"], ">=", -policy["maximum_overall_drop"]),
        check("critical failures", summary["critical_failures"], "<=", policy["maximum_critical_failures"]),
        check("evaluation error rate", summary["error_rate"], "<=", policy["maximum_error_rate"]),
        check("output flip rate", summary["maximum_flip_rate"], "<=", policy["maximum_flip_rate"]),
        check("uncertain regression", interval[0], ">=", -policy["maximum_overall_drop"], "review"),
    ]
    groups = group_by_slice(pairs)
    slice_summaries = {}
    for slice_name, rules in policy["critical_slices"].items():
        group = groups.get(slice_name, [])
        metrics = summarize(group)
        slice_summaries[slice_name] = metrics
        gates.extend([
            check(f"{slice_name} examples", metrics["examples"], ">=", rules["minimum_examples"]),
            check(f"{slice_name} pass floor", metrics["candidate_pass_rate"], ">=", rules["minimum_pass_rate"]),
            check(f"{slice_name} maximum drop", metrics["difference"], ">=", -rules["maximum_drop"]),
        ])
    return summary, interval, slice_summaries, gates

This policy blocks when the observed difference exceeds the allowed drop, even if uncertainty is wide. That conservative choice suits a release gate where insufficient evidence should not authorize risk. Another team may allow review when the point estimate breaches the limit but the confidence interval overlaps it. Document that choice before results arrive.

Critical slices need both coverage and performance rules. Without coverage, an empty safety slice would have a zero pass rate here and fail, but other implementations may accidentally skip it. Make absence an explicit gate. Avoid dozens of tiny slices: they create unstable estimates and repeated-comparison noise. Prioritize slices tied to concrete harm.

Verification: The three-row fixture must fail minimum-example and critical-slice coverage gates. Confirm it cannot pass simply because its overall difference is zero. Mark case-002 as a critical failure and verify the critical-failure gate also fails.

Step 6: Emit an Auditable CI Decision

Finish regression_gate.py with command-line parsing, report generation, and exit codes.

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--baseline", required=True)
    parser.add_argument("--candidate", required=True)
    parser.add_argument("--policy", required=True)
    parser.add_argument("--report", default="regression-report.json")
    args = parser.parse_args()

    policy = json.loads(Path(args.policy).read_text(encoding="utf-8"))
    baseline = load_jsonl(args.baseline)
    candidate = load_jsonl(args.candidate)
    pairs = pair_rows(baseline, candidate)
    summary, interval, slices, gates = evaluate_policy(pairs, policy)

    failed = [gate for gate in gates if not gate["passed"] and gate["severity"] == "fail"]
    warnings = [gate for gate in gates if not gate["passed"] and gate["severity"] == "review"]
    status = "fail" if failed else "review" if warnings else "pass"
    report = {
        "status": status,
        "policy_version": policy["policy_version"],
        "baseline_sha256": file_sha256(args.baseline),
        "candidate_sha256": file_sha256(args.candidate),
        "summary": summary,
        "paired_difference_interval": interval,
        "slices": slices,
        "gates": gates,
    }
    Path(args.report).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(report, indent=2))
    return 1 if status == "fail" else 2 if status == "review" else 0

if __name__ == "__main__":
    try:
        sys.exit(main())
    except Exception as error:
        print(f"gate error: {error}", file=sys.stderr)
        sys.exit(3)

Run it:

python regression_gate.py \
  --baseline baseline.jsonl \
  --candidate candidate.jsonl \
  --policy policy.json \
  --report regression-report.json
echo $?

Exit 0 means pass, 1 means a blocking regression, 2 means mandatory review, and 3 means invalid evidence or a gate execution error. Configure CI to block all nonzero codes unless your review workflow explicitly captures and approves status 2. Upload the report, policy, cached results, evaluator configuration, dataset hash, and code commit as artifacts.

Verification: With the tiny fixture, expect exit 1 and a report listing coverage failures. For a test-only check, lower the sample requirements and slice floors in a copy named policy-fixture.json; expect exit 0. Do not replace the production policy with the fixture policy.

Step 7: Add the Gate to Continuous Integration

Run generation and evaluation before the gate, but keep the gate independent of provider calls. Cached artifacts make failures reproducible and allow reviewers to inspect the exact release evidence. A generic GitHub Actions job looks like this:

name: llm-regression
on:
  pull_request:
  workflow_dispatch:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Generate candidate evaluation artifact
        run: python scripts/run_evaluation.py --output candidate.jsonl
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - name: Apply regression policy
        run: |
          python regression_gate.py \
            --baseline evals/baseline.jsonl \
            --candidate candidate.jsonl \
            --policy evals/policy.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: llm-regression-evidence
          path: |
            candidate.jsonl
            regression-report.json

Pin third-party actions to full commit SHAs when your security policy requires immutable dependencies. Protect the baseline and policy with code review. Never let a pull request replace its baseline with its own candidate output, because that makes the comparison meaningless.

LLM calls can vary even with deterministic settings. Measure repeated outcomes using the LLM nondeterminism repeated-trials workflow, then populate flip_rate from real repeated runs. If costs prevent full repetition, repeat a stable, risk-stratified subset and document the coverage.

Verification: Open a test pull request with one deliberate critical failure. Confirm the job blocks, the JSON report uploads despite failure, and the report identifies the exact gate. Revert the defect and confirm the same workflow passes without changing the policy.

Step 8: Tune Thresholds Without Overfitting

Threshold maintenance is a controlled product decision. Review thresholds when risk tolerance, traffic mix, rubric, evaluator, dataset, model family, or manual-review capacity changes. Do not loosen a gate because one favored release missed it.

Use three datasets when possible:

1. A development set for building prompts and graders.
2. A threshold-calibration set for selecting limits and estimating review volume.
3. A final holdout or prospective release set for validating the frozen policy.

Backtest candidate rules across several historical releases. Count false blocks, missed known regressions, review frequency, and performance by critical slice. Prefer the simplest policy that catches material regressions at an acceptable operational cost. If you try many policies, acknowledge selection bias and validate the chosen policy on untouched evidence.

Track threshold changes in version control with a short decision record. Include the old and new values, evidence, owner, effective date, expected impact, and rollback condition. Baselines also need lifecycle rules: promote a candidate only after release approval, production observation, and artifact verification. Never automatically promote every green candidate, because a green result may still contain unmeasured product changes.

Verification: Replay the frozen gate against historical known-good and known-bad artifacts. Confirm it blocks the material bad cases and does not create unacceptable noise. Then run it once on untouched evidence and record approval before making the policy active.

Troubleshooting

Problem: A tiny score change fails every build -> Check the sample size, pairing, and confidence interval. Increase representative evidence or route statistically ambiguous changes to review. Do not hide real harm by widening the allowed drop without risk approval.

Problem: The overall metric passes but a workflow is clearly worse -> Add a predeclared gate for the affected high-risk slice and require minimum slice coverage. Inspect per-example paired losses to learn whether one scenario or rubric rule drives the regression.

Problem: The candidate has missing IDs -> Treat the run as invalid evidence. Fix timeouts, filtering, deduplication, or joins, then rerun. Never compare only the intersection after seeing which cases disappeared.

Problem: Results flip across identical runs -> Pin model snapshots and prompt bytes, confirm decoding settings, and run repeated trials. Gate on flip rate or send unstable cases to human review rather than trusting a lucky run.

Problem: The judge score improves while human reviewers see worse answers -> Stop the release and recalibrate the judge with fresh, blinded human labels. Look for reward hacking, leaked references, rubric gaps, and judge-model bias. Automated evaluator improvement is not product improvement unless human-aligned evidence supports it.

Problem: CI passes locally but fails remotely -> Compare Python versions, file hashes, locale, dataset source, environment configuration, and candidate model identifier. Seed local resampling as shown, cache provider outputs, and store all provenance in the report.

Where To Go Next

Connect this gate to the LLM evaluation pipeline complete guide so dataset curation, graders, release evidence, and monitoring share one lifecycle. Then strengthen the inputs to your thresholds:

• Calibrate an LLM judge with human labels before treating automated labels as release evidence.
• Test LLM nondeterminism with repeated trials and turn observed flip rates into a stability gate.
• Compare prompts with paired evaluation to preserve item-level evidence when selecting prompt changes.

After deployment, compare production samples with the regression distribution. A release gate answers whether a candidate passed known checks. Monitoring answers whether real inputs and outcomes still resemble the assumptions behind those checks. Keep the gate itself covered by the same CI/CD testing best practices you apply to other release-critical automation.

Interview Questions and Answers

Use the seven structured questions and model answers below to practice explaining absolute floors, relative regression limits, paired analysis, uncertainty, slice gates, nondeterminism, and threshold governance. In an interview, connect every numeric gate to a failure cost and explain how you prevent leakage while tuning it.

Common Mistakes

• Copying a threshold from another product without modeling different harm and prevalence.
• Comparing baseline and candidate on different examples or judge configurations.
• Using only an aggregate score and missing a severe slice regression.
• Treating evaluator errors, missing rows, or refusals as ordinary failed answers.
• Changing the policy after seeing a candidate result.
• Ignoring uncertainty while also using an evaluation set too small for the decision.
• Assuming a confidence interval captures dataset shift, label error, or judge bias.
• Promoting a new baseline automatically without preserving review and rollback evidence.

Conclusion

To set LLM evaluation regression thresholds responsibly, turn concrete harm into predeclared gates, evaluate baseline and candidate on the same frozen cases, and combine absolute floors with paired change limits, confidence bounds, critical-slice rules, stability, and infrastructure checks. The runnable gate makes those decisions reproducible, but the policy still requires human ownership.

Start by writing one risk table and freezing one paired dataset. Run the gate on historical releases, validate the final policy on untouched evidence, and preserve every artifact needed to explain why the release passed, required review, or failed.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Set up visual regression in CI (2026)
• LLM Evaluation Pipeline Complete Guide (2026)
• Automating regression triage with AI (2026)
• Build an Adversarial RAG Evaluation Dataset
• Building a regression selection agent (2026)
• Calibrate an LLM Judge Against Human Labels