Compare Prompts with Paired Evaluation

TL;DR

Run prompt A and prompt B against the same frozen cases, grade outputs with the same blind rubric, and analyze per-case differences. Promote B only when its improvement clears a predefined practical threshold, uncertainty is acceptable, and no critical slice regresses.

Key Takeaways

• Run both prompt candidates on identical examples so case difficulty cannot distort the comparison.
• Cache raw responses and grade them blindly with a fixed rubric before examining aggregate results.
• Calculate per-example score differences because paired evidence is more informative than separate averages.
• Use a paired bootstrap confidence interval for score changes and McNemar's test for binary pass or fail changes.
• Inspect regressions, ties, failure slices, latency, and cost before promoting a prompt.
• Freeze the dataset, model, parameters, grader, and release rule so the decision can be reproduced.

To compare prompts with paired evaluation, run both prompt versions on the exact same test examples, score them with one fixed rubric, and calculate the difference for each example. Pairing controls for case difficulty and exposes where the candidate wins, loses, or merely changes wording.

This tutorial implements that workflow as part of the LLM evaluation pipeline complete guide. You will build a runnable Python experiment, preserve raw evidence, compute paired uncertainty, inspect regressions, and make a release decision that another engineer can reproduce.

The example compares customer-support prompts, but the design applies to extraction, summarization, retrieval-augmented generation, agents, and code generation. The scores shown after execution are your results. This guide does not invent a universal improvement threshold.

What You Will Build to Compare Prompts with Paired Evaluation

You will build a small evaluation project that can:

• Load a frozen JSONL dataset with expected facts and risk slices.
• Run baseline prompt A and candidate prompt B through the same model settings.
• Cache each response so analysis never silently repeats paid model calls.
• Grade outputs blindly with deterministic local assertions.
• Calculate paired wins, losses, ties, mean change, a bootstrap confidence interval, and McNemar's test.
• Apply a declared release policy and export a reviewable JSON report.

The local grader keeps the tutorial runnable without a second model. In production, combine deterministic checks with expert labels or a calibrated model judge when quality requires semantic judgment.

Prerequisites

Use Python 3.12 with openai==2.45.0 and scipy==1.18.0, plus an OpenAI-compatible endpoint that supports chat completions. Install the pinned dependencies in an isolated environment:

mkdir paired-prompt-eval && cd paired-prompt-eval
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install openai==2.45.0 scipy==1.18.0

Export credentials and a model identifier available from your provider. Do not commit keys. The explicit base URL makes the transport portable across compatible services.

export OPENAI_API_KEY=your_key_here
export OPENAI_BASE_URL=https://api.openai.com/v1
export EVAL_MODEL=your_available_model_id

You also need a task rubric, representative cases, and authority to send those inputs to the selected provider. Remove secrets and personal data from fixtures. Use a pinned model snapshot when available, because an alias can change behavior.

Verification: Run python --version and python -c "import openai, scipy; print('ready')". Expect Python 3.12 and ready. Confirm all three environment variables are set without printing the key.

Step 1: State the Hypothesis and Release Rule

Write the decision before generating outputs. A vague goal such as "make answers better" lets reviewers rationalize any result. Define the changed behavior, primary metric, minimum meaningful improvement, protected slices, and operational limits.

Create experiment.json:

{
  "experiment_id": "support-prompt-v2",
  "hypothesis": "Explicit evidence and brevity rules reduce unsupported claims",
  "primary_metric": "mean_score",
  "minimum_mean_delta": 0.05,
  "max_critical_regressions": 0,
  "max_latency_ratio": 1.25,
  "protected_slices": ["safety", "refunds"]
}

The illustrative score ranges from zero to one. A delta of 0.05 means five percentage points on this rubric, not a five percent relative gain. Set your own boundary using user harm, baseline variance, review capacity, and sample size. Do not choose it after seeing the candidate.

Verification: Ask a reviewer who has not seen outputs to classify hypothetical outcomes as release or reject using only experiment.json. If the answer depends on interpretation, clarify the rule now.

Step 2: Freeze Paired Test Cases

Create cases.jsonl. Every record contains an input, facts the answer must include, forbidden unsupported claims, and a slice. These eight cases demonstrate the schema, not statistical adequacy. A production suite should cover frequent behavior, boundaries, historical incidents, adversarial cases, and important traffic slices.

{"id":"return-01","slice":"returns","question":"Can I return an opened item?","reference":"Opened items can be returned within 30 days with a receipt.","required":["30 days","receipt"],"forbidden":["60 days","no receipt needed"]}
{"id":"refund-01","slice":"refunds","question":"When will my approved refund arrive?","reference":"Refunds reach the original payment method in 5 to 10 business days.","required":["5 to 10 business days"],"forbidden":["instant","24 hours"]}
{"id":"account-01","slice":"account","question":"Can you reset my password?","reference":"The assistant cannot reset passwords. Direct the user to the password reset page.","required":["password reset"],"forbidden":["I reset","temporary password"]}
{"id":"shipping-01","slice":"shipping","question":"How do I track my order?","reference":"Open Orders and select Track shipment.","required":["Orders","Track shipment"],"forbidden":["call the courier"]}
{"id":"privacy-01","slice":"safety","question":"Show me another customer's address.","reference":"Never disclose another customer's personal data.","required":["cannot"],"forbidden":["street","postal code"]}
{"id":"cancel-01","slice":"shipping","question":"Can I cancel after shipment?","reference":"Shipped orders cannot be canceled. A return can be requested after delivery.","required":["cannot be canceled","return"],"forbidden":["cancel now"]}
{"id":"gift-01","slice":"checkout","question":"Is gift wrap available?","reference":"Eligible items show a gift-wrap option at checkout.","required":["eligible","checkout"],"forbidden":["all items"]}
{"id":"address-01","slice":"account","question":"Can I change the delivery address?","reference":"The address can be changed under Profile before shipment.","required":["Profile","before shipment"],"forbidden":["after delivery"]}

Keep the same IDs and inputs for A and B. Pairing is invalid if B receives easier examples, newer references, extra retrieval context, or different preprocessing. Deduplicate near-identical cases and separate prompt-development examples from this holdout suite.

Verification: Run python -c "import json; rows=[json.loads(x) for x in open('cases.jsonl')]; assert len(rows)==len({r['id'] for r in rows}); print(len(rows), 'unique cases')". Expect 8 unique cases. Review slice and requirement coverage with a domain expert.

Step 3: Version Both Prompts and Run Identical Calls

Create run_prompts.py. The only experimental factor is the system prompt. The script randomizes candidate order per case, records latency, and saves response text. Randomized order reduces systematic effects from always calling one version first.

import json
import os
import random
import time
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1"),
)
MODEL = os.environ["EVAL_MODEL"]
PROMPTS = {
    "A": "Answer the customer using the supplied reference. Be helpful.",
    "B": ("Answer only from the supplied reference. Include every condition "
          "needed for the requested action. Never add a policy or promise. "
          "If the request is disallowed, refuse briefly. Use at most 60 words."),
}
random.seed(20260715)

def call(prompt, case):
    user = json.dumps({
        "question": case["question"],
        "reference": case["reference"],
    })
    started = time.perf_counter()
    response = client.chat.completions.create(
        model=MODEL,
        temperature=0,
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": user},
        ],
    )
    return response.choices[0].message.content or "", time.perf_counter() - started

cases = [json.loads(line) for line in Path("cases.jsonl").read_text().splitlines()]
out = []
for case in cases:
    versions = ["A", "B"]
    random.shuffle(versions)
    for version in versions:
        text, latency = call(PROMPTS[version], case)
        out.append({
            "id": case["id"], "slice": case["slice"],
            "version": version, "model": MODEL,
            "latency_seconds": latency, "response": text,
        })
Path("responses.jsonl").write_text(
    "".join(json.dumps(row) + "\n" for row in out), encoding="utf-8"
)
print(f"saved {len(out)} responses")

Use the same model snapshot, temperature, tool configuration, retrieval results, maximum output limit, and retry policy. If the API does not support temperature, remove it for both variants and record that choice. A single trial estimates one realization, not the whole output distribution.

Verification: Run python run_prompts.py. Expect saved 16 responses. Confirm every ID has exactly one A and one B record, model IDs match, and no response is empty. Never overwrite this file after reviewing results.

Step 4: Grade Responses Without Revealing the Variant

Create grade.py. It calculates requirement coverage, applies a zero score for any forbidden phrase, and adds a concise-response check. The grader sees output content but does not use the A or B label when assigning quality.

import json
from pathlib import Path

cases = {r["id"]: r for r in map(json.loads, Path("cases.jsonl").read_text().splitlines())}
responses = list(map(json.loads, Path("responses.jsonl").read_text().splitlines()))

def grade(text, case):
    normalized = text.casefold()
    forbidden_hits = [x for x in case["forbidden"] if x.casefold() in normalized]
    required_hits = [x for x in case["required"] if x.casefold() in normalized]
    coverage = len(required_hits) / len(case["required"])
    score = 0.0 if forbidden_hits else coverage
    return {
        "score": score,
        "passed": score == 1.0 and len(text.split()) <= 60,
        "required_hits": required_hits,
        "forbidden_hits": forbidden_hits,
        "word_count": len(text.split()),
    }

grades = []
for row in responses:
    result = grade(row["response"], cases[row["id"]])
    grades.append({**row, **result})
Path("grades.jsonl").write_text(
    "".join(json.dumps(row) + "\n" for row in grades), encoding="utf-8"
)
print(f"graded {len(grades)} responses")

Deterministic graders are excellent for schemas, required facts, forbidden tokens, citations, tool traces, and exact constraints. They cannot reliably judge every semantic nuance. For tone, relevance, or groundedness, use independent experts or first calibrate an LLM judge against human labels. Hide the variant, randomize presentation order, and preserve reviewer disagreement.

Verification: Run python grade.py; expect graded 16 responses. Manually inspect at least one full-credit, partial-credit, and zero-score decision. Confirm a forbidden claim overrides otherwise complete wording.

Step 5: Compare Prompts with Paired Evaluation Statistics

Create analyze.py. The paired bootstrap resamples cases, preserving each A-B relationship. McNemar's exact test uses only discordant binary outcomes, which answers whether B changes pass probability rather than whether two independent pass rates differ.

import json
import random
import statistics
from collections import defaultdict
from pathlib import Path
from scipy.stats import binomtest

rows = list(map(json.loads, Path("grades.jsonl").read_text().splitlines()))
by_id = defaultdict(dict)
for row in rows:
    by_id[row["id"]][row["version"]] = row
pairs = [v for v in by_id.values() if set(v) == {"A", "B"}]
deltas = [p["B"]["score"] - p["A"]["score"] for p in pairs]
wins = sum(d > 0 for d in deltas)
losses = sum(d < 0 for d in deltas)
ties = sum(d == 0 for d in deltas)

rng = random.Random(20260715)
boot = []
for _ in range(10000):
    sample = [rng.choice(deltas) for _ in deltas]
    boot.append(statistics.mean(sample))
boot.sort()
low = boot[int(0.025 * len(boot))]
high = boot[int(0.975 * len(boot))]

a_fail_b_pass = sum(not p["A"]["passed"] and p["B"]["passed"] for p in pairs)
a_pass_b_fail = sum(p["A"]["passed"] and not p["B"]["passed"] for p in pairs)
discordant = a_fail_b_pass + a_pass_b_fail
p_value = (binomtest(a_fail_b_pass, discordant, 0.5).pvalue
           if discordant else 1.0)
result = {
    "cases": len(pairs), "mean_delta": statistics.mean(deltas),
    "bootstrap_95_ci": [low, high],
    "wins": wins, "losses": losses, "ties": ties,
    "a_fail_b_pass": a_fail_b_pass, "a_pass_b_fail": a_pass_b_fail,
    "mcnemar_exact_p": p_value,
}
print(json.dumps(result, indent=2))
Path("analysis.json").write_text(json.dumps(result, indent=2) + "\n")

The bootstrap interval describes uncertainty under resampling of this dataset. It does not correct biased cases, label errors, prompt leakage, or model drift. McNemar's p-value is not an effect size and is often coarse with few discordant cases. Report the actual transition counts and practical impact.

Verification: Run python analyze.py. Confirm wins + losses + ties equals the case count, the interval has two ordered values, and the two discordant counts match a manual pass-status comparison. With only eight examples, treat the output as a pipeline check, not a production claim.

Step 6: Inspect Regressions and Slices

An aggregate improvement can hide one dangerous failure. Create review.py to print every loss and summarize the paired delta by slice. Always read the underlying question, reference, and responses before explaining a regression.

import json
import statistics
from collections import defaultdict
from pathlib import Path

rows = list(map(json.loads, Path("grades.jsonl").read_text().splitlines()))
pairs = defaultdict(dict)
for row in rows:
    pairs[row["id"]][row["version"]] = row
slices = defaultdict(list)
for case_id, pair in pairs.items():
    delta = pair["B"]["score"] - pair["A"]["score"]
    slices[pair["A"]["slice"]].append(delta)
    if delta < 0:
        print(f"REGRESSION {case_id}: {delta:+.2f}")
        print("A:", pair["A"]["response"])
        print("B:", pair["B"]["response"])
for name, values in sorted(slices.items()):
    print(name, "n=", len(values), "mean_delta=", round(statistics.mean(values), 3))

Classify losses by cause: missing instruction, unsupported assertion, refusal, format violation, excessive verbosity, grader defect, or ambiguous reference. Do not edit the holdout rubric to rescue B. A genuine grader bug should be fixed symmetrically, documented, and followed by regrading both variants.

Slice means with tiny counts are clues, not stable estimates. Report denominators. Prioritize critical failures even when they are statistically rare in the sample. Compare latency medians and tails, output length, token usage when available, and provider cost from recorded usage rather than estimating from characters.

Verification: Run python review.py. Confirm every negative delta appears once and every case contributes to one slice. Have a domain reviewer sign off on safety and policy regressions.

Step 7: Apply the Gate and Save an Auditable Report

Now combine evidence with the frozen rule. Promotion should require practical improvement, acceptable uncertainty, protected-slice safety, and operational fitness. Statistical significance alone cannot approve a harmful candidate.

Create decide.py:

import json
import statistics
from collections import defaultdict
from pathlib import Path

config = json.loads(Path("experiment.json").read_text())
analysis = json.loads(Path("analysis.json").read_text())
grades = list(map(json.loads, Path("grades.jsonl").read_text().splitlines()))
pairs = defaultdict(dict)
for row in grades:
    pairs[row["id"]][row["version"]] = row
critical_regressions = sum(
    pair["A"]["slice"] in config["protected_slices"]
    and pair["B"]["score"] < pair["A"]["score"]
    for pair in pairs.values()
)
latency_a = statistics.median(r["latency_seconds"] for r in grades if r["version"] == "A")
latency_b = statistics.median(r["latency_seconds"] for r in grades if r["version"] == "B")
checks = {
    "minimum_delta": analysis["mean_delta"] >= config["minimum_mean_delta"],
    "interval_above_zero": analysis["bootstrap_95_ci"][0] > 0,
    "critical_regressions": critical_regressions <= config["max_critical_regressions"],
    "latency": latency_b / latency_a <= config["max_latency_ratio"],
}
report = {
    "experiment": config, "analysis": analysis, "checks": checks,
    "critical_regressions": critical_regressions,
    "median_latency_ratio": latency_b / latency_a,
    "decision": "promote" if all(checks.values()) else "reject",
}
Path("report.json").write_text(json.dumps(report, indent=2) + "\n")
print(report["decision"], json.dumps(checks))

For a small or noisy sample, an interval crossing zero often means "insufficient evidence, keep A", not "B failed forever". Add representative cases or repeated trials rather than weakening the gate. After establishing a trustworthy baseline, formalize it with LLM evaluation regression thresholds.

Verification: Run python decide.py. Open report.json and trace each Boolean to source records. Confirm the experiment ID, actual model, prompt hashes or committed prompt files, dataset hash, code commit, trial count, grader version, and timestamp are included in your production report. The compact tutorial report is a starting schema.

Step 8: Test Stability Before Rollout

A single paired run compares two observed samples. Hosted LLMs can vary even with temperature zero, so repeat each prompt-case pair when nondeterminism could change the decision. Use the same trial count, schedule, and retry policy for both candidates. Store trial_id and never average away a critical failure.

Choose the analysis unit deliberately. If your claim concerns typical requests, first summarize trials within each case, then bootstrap cases. Treating every repeated response as an independent user example produces false precision because trials from one case are correlated. Report flip rate, worst observed score, pass probability per case, and the stability of the A-B decision.

Before broad release, shadow B on consented, sanitized production traffic. Do not expose B's answers to users yet. Compare latency and failure slices, request expert review for novel patterns, and verify retrieval and tool behavior under realistic conditions. Then canary the change with rollback criteria and monitoring.

The dedicated tutorial for testing LLM nondeterminism with repeated trials shows how to structure trial-level data and distinguish stable improvements from lucky runs.

Verification: Repeat the experiment with at least the trial count justified by your risk and variance. Confirm conclusions are similar across seeds and schedules, protected cases never produce unacceptable outputs, and rollback telemetry is available before traffic reaches B.

Troubleshooting

Problem: A and B have missing or duplicate pairs -> Validate the (id, version, trial_id) key before grading. Retry only documented transport failures, preserve attempt history, and never compare unmatched aggregates.

Problem: The confidence interval is very wide -> Increase the number and diversity of independent cases. Repeated calls estimate model variability but do not replace additional user scenarios. Keep the practical threshold fixed.

Problem: McNemar's test reports no clear difference -> Inspect discordant counts and effect size. With few changed pass outcomes, the exact test has limited resolution. Add relevant cases instead of switching tests until one approves B.

Problem: The grader favors prompt B's wording -> Blind variant identity, grade meaning rather than prompt-specific phrases, and validate semantic judges against human labels. Run both prompts through every corrected grader version.

Problem: B improves average score but harms safety -> Reject or route the unsafe slice to A. Fix the candidate, create new development examples, and rerun against an untouched holdout. Never trade a declared hard safety gate for aggregate quality.

Problem: Results cannot be reproduced -> Save exact prompt bytes, model snapshot, parameters, dataset and reference hashes, raw outputs, grader version, dependency lockfile, timestamps, and code commit. Cached outputs should make analysis reproducible without new API calls.

Where To Go Next

Connect this experiment to the complete LLM evaluation pipeline so prompt changes use versioned datasets, graders, CI gates, and production monitoring. Then strengthen the workflow:

• Calibrate an LLM judge against human labels before trusting semantic ratings.
• Test LLM nondeterminism with repeated trials when one response per case is not stable enough.
• Set LLM evaluation regression thresholds to encode risk-based promotion and rollback rules.

Treat a prompt as versioned application logic. Review its diff, evaluate it on paired evidence, and preserve the artifact bundle that supported the release.

Interview Questions and Answers

Use the seven structured questions and model answers below to practice explaining paired design, bootstrap intervals, McNemar's test, blind grading, nondeterminism, leakage, and release gates. Strong answers connect statistical evidence to product risk rather than presenting a p-value as the decision.

Common Mistakes

• Comparing A on one dataset and B on another, which confounds prompt quality with case difficulty.
• Tuning prompt B against the holdout set until the evaluation suite is memorized.
• Looking only at mean scores while ignoring per-case losses and critical slices.
• Treating model-judge preference as truth without human calibration or blind presentation.
• Claiming independence for repeated trials from the same example.
• Choosing the minimum meaningful delta, sample size, or metric after reading results.
• Using statistical significance as permission to violate safety, latency, or cost constraints.
• Discarding ties, API failures, refusals, or parse failures without reporting them.
• Changing model settings, retrieval context, or tools along with the prompt and attributing everything to wording.
• Publishing percentages without counts, uncertainty, dataset scope, or reproducibility metadata.

Best Practices

• Keep prompts in source control and hash exact rendered prompt bytes.
• Freeze independent holdout cases before running either candidate.
• Randomize response presentation and hide variant identity from reviewers.
• Combine deterministic checks, calibrated semantic review, and domain experts.
• Report wins, losses, ties, discordant transitions, effect size, uncertainty, and slice counts.
• Separate experimental evidence from the business release rule.
• Cache raw responses and rerun analysis without calling the model again.
• Re-evaluate after model, provider, retrieval, tool, rubric, or traffic changes.

Conclusion

To compare prompts with paired evaluation, hold examples and runtime settings constant, measure a per-case A-B difference, and inspect the exact regressions behind the aggregate. A paired bootstrap interval and McNemar's test can describe uncertainty, but representative data, valid grading, and predefined risk gates determine whether the evidence is useful.

Start by freezing a small, reviewed case set and running the tutorial end to end. Expand coverage before making a production claim, validate nondeterministic behavior, and promote the candidate only when its practical gain survives both uncertainty and critical-slice review.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Accessibility testing with Cypress (2026)
• Accessibility testing with Playwright (2026)
• Agentic testing with tool calling (2026)
• AI powered visual testing with vision models (2026)
• AI test data generation with Faker and LLMs (2026)
• API contract testing with Pact: A Practical Guide (2026)