Test LLM Nondeterminism with Repeated Trials

TL;DR

Run identical LLM cases across repeated trials, score every response with a versioned evaluator, and retain the raw evidence. Gate releases with risk-based pass rates and sampling uncertainty, not a single successful output.

Key Takeaways

• Run the same controlled case many times instead of trusting one response.
• Preserve raw outputs, seeds, model metadata, scores, and failure reasons.
• Measure correctness and output consistency as separate dimensions.
• Use confidence intervals to show uncertainty around observed pass rates.
• Set risk-based thresholds before viewing candidate results.
• Keep paired conditions stable when comparing prompts or models.
• Upload trial artifacts even when a CI evaluation fails.

Testing LLM output once tells you whether one response passed. It does not tell you whether the behavior is reliable. To test LLM nondeterminism with repeated trials, run the same controlled case many times, record every result, score each result with deterministic checks, and evaluate the pass-rate distribution instead of trusting a single response.

This tutorial builds that repeatable test harness in Python. It fits into the broader LLM evaluation pipeline complete guide, but stays focused on one job: exposing response variation and turning it into regression evidence.

You will use a local fake model first, so every command is runnable without a key or network access. Then you can connect the same harness to your model provider while preserving prompts, trial counts, raw evidence, and acceptance rules.

What You Will Build

By the end, you will have a small command-line evaluation project that:

• runs identical test cases across repeated independent trials;
• stores the seed, latency, raw response, parsed response, and score for every attempt;
• calculates pass rate, exact-match concentration, score mean, and score spread;
• compares observed pass rates with an explicit reliability threshold;
• creates machine-readable JSON artifacts for CI and later investigation;
• reproduces an illustrative failure with a deterministic fake LLM.

The target is not to eliminate randomness. The target is to measure whether variation remains inside a product-specific tolerance.

Prerequisites

Use Python 3.11 or newer. The tutorial uses only the standard library, so you do not need a package manager dependency or an API account.

Create an empty directory and confirm Python:

mkdir llm-repeated-trials
cd llm-repeated-trials
python3 --version

Expected output starts with Python 3.11, 3.12, or 3.13. On Windows, use py -3 in place of python3 if that is how Python is installed.

You also need a terminal and a text editor. Keep temperature, model identifier, system prompt, user prompt, decoding options, and evaluator version under source control. A repeated test is interpretable only when those inputs are known.

Step 1: Define Stable Evaluation Cases

Start with narrow cases whose required behavior can be checked without another LLM. Create cases.json:

[
  {
    "id": "refund-json",
    "prompt": "Return JSON with keys intent and urgency. Message: I need my duplicate charge refunded today.",
    "required_intent": "refund",
    "allowed_urgency": ["high"],
    "min_pass_rate": 0.90
  },
  {
    "id": "password-json",
    "prompt": "Return JSON with keys intent and urgency. Message: I forgot my password and cannot sign in.",
    "required_intent": "password_reset",
    "allowed_urgency": ["low", "medium"],
    "min_pass_rate": 0.85
  }
]

Each case has a stable identifier, an observable contract, and an acceptance threshold. The threshold belongs to the case because the business risk differs. A refund routing error may be more costly than a harmless wording change.

Do not ask for exact prose when multiple phrasings are acceptable. Constrain the output to JSON and assert semantic fields. Exact match is still useful as a variation signal, but it should not automatically become the product verdict.

Verify Step 1: Run this standard-library validation:

python3 -m json.tool cases.json

The command should print two formatted objects and exit with code 0. A syntax error identifies the line and column to fix.

Step 2: Create a Reproducible Model Adapter

Create model_adapter.py. The fake adapter deliberately varies formatting and sometimes returns a wrong label. A seed makes each trial reproducible while different seeds create a distribution.

import json
import random
import time
from dataclasses import dataclass


@dataclass(frozen=True)
class ModelResult:
    text: str
    latency_ms: float
    model: str
    seed: int


def generate(prompt: str, seed: int) -> ModelResult:
    started = time.perf_counter()
    rng = random.Random(seed)

    if "duplicate charge" in prompt:
        intent = "refund" if rng.random() >= 0.12 else "billing_question"
        urgency = "high"
    elif "forgot my password" in prompt:
        intent = "password_reset" if rng.random() >= 0.08 else "account_help"
        urgency = rng.choice(["low", "medium"])
    else:
        intent = "unknown"
        urgency = "low"

    payload = {"intent": intent, "urgency": urgency}
    if rng.choice([True, False]):
        text = json.dumps(payload, sort_keys=True)
    else:
        text = json.dumps(payload, indent=2)

    latency_ms = (time.perf_counter() - started) * 1000
    return ModelResult(
        text=text,
        latency_ms=latency_ms,
        model="deterministic-fake-llm-v1",
        seed=seed,
    )

In production, keep this function signature and replace the body with the official SDK call for your provider. Pass a seed only if the API supports it, and record the returned seed or system fingerprint when available. A seed can improve reproducibility, but it does not guarantee identical output across model revisions, infrastructure changes, or providers.

The adapter returns text rather than parsed JSON. That preserves malformed responses as evidence instead of losing them inside a convenience parser.

Verify Step 2: Run a small probe:

python3 - <<'PY'
from model_adapter import generate
print(generate("duplicate charge", seed=7))
print(generate("duplicate charge", seed=7))
PY

The two ModelResult values should have identical text, model, and seed. Latency can differ slightly.

Step 3: Score Every Trial Deterministically

Create evaluator.py:

import json
from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class Evaluation:
    passed: bool
    score: float
    reason: str
    parsed: dict[str, Any] | None


def evaluate(text: str, case: dict[str, Any]) -> Evaluation:
    try:
        parsed = json.loads(text)
    except json.JSONDecodeError as error:
        return Evaluation(
            passed=False,
            score=0.0,
            reason=f"invalid_json:{error.msg}",
            parsed=None,
        )

    if not isinstance(parsed, dict):
        return Evaluation(False, 0.0, "root_not_object", None)

    intent_ok = parsed.get("intent") == case["required_intent"]
    urgency_ok = parsed.get("urgency") in case["allowed_urgency"]
    score = (float(intent_ok) + float(urgency_ok)) / 2.0

    failures = []
    if not intent_ok:
        failures.append("wrong_intent")
    if not urgency_ok:
        failures.append("wrong_urgency")

    return Evaluation(
        passed=intent_ok and urgency_ok,
        score=score,
        reason="pass" if not failures else ",".join(failures),
        parsed=parsed,
    )

This evaluator separates validity from quality. Invalid JSON receives zero. A valid response can receive partial credit while still failing the strict contract. The reason label makes aggregate failures diagnosable.

Deterministic assertions are ideal for schemas, forbidden content, exact facts, and required fields. For subjective qualities, introduce a human rubric or calibrated judge. The tutorial on calibrating an LLM judge against human labels explains how to validate that scorer before trusting it.

Verify Step 3: Exercise a pass and a failure:

python3 - <<'PY'
from evaluator import evaluate
case = {
    "required_intent": "refund",
    "allowed_urgency": ["high"]
}
print(evaluate('{"intent":"refund","urgency":"high"}', case))
print(evaluate('{"intent":"other","urgency":"high"}', case))
PY

The first result should show passed=True and score=1.0. The second should show passed=False, score=0.5, and wrong_intent.

Step 4: Test LLM Nondeterminism with Repeated Trials

Create run_trials.py:

import argparse
import json
from datetime import datetime, timezone
from pathlib import Path
from statistics import fmean, pstdev
from typing import Any

from evaluator import evaluate
from model_adapter import generate


def run_case(case: dict[str, Any], trials: int, base_seed: int) -> dict[str, Any]:
    attempts = []
    for index in range(trials):
        seed = base_seed + index
        result = generate(case["prompt"], seed)
        evaluation = evaluate(result.text, case)
        attempts.append({
            "trial": index + 1,
            "seed": seed,
            "model": result.model,
            "latency_ms": round(result.latency_ms, 3),
            "response": result.text,
            "parsed": evaluation.parsed,
            "passed": evaluation.passed,
            "score": evaluation.score,
            "reason": evaluation.reason,
        })

    scores = [attempt["score"] for attempt in attempts]
    passed_count = sum(attempt["passed"] for attempt in attempts)
    canonical_outputs = [
        json.dumps(attempt["parsed"], sort_keys=True, separators=(",", ":"))
        if attempt["parsed"] is not None else attempt["response"]
        for attempt in attempts
    ]
    most_common_count = max(
        canonical_outputs.count(value) for value in set(canonical_outputs)
    )

    return {
        "case_id": case["id"],
        "trials": trials,
        "passed": passed_count,
        "pass_rate": passed_count / trials,
        "min_pass_rate": case["min_pass_rate"],
        "meets_threshold": passed_count / trials >= case["min_pass_rate"],
        "mean_score": fmean(scores),
        "score_stddev": pstdev(scores),
        "dominant_output_rate": most_common_count / trials,
        "attempts": attempts,
    }


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cases", default="cases.json")
    parser.add_argument("--trials", type=int, default=30)
    parser.add_argument("--base-seed", type=int, default=1000)
    parser.add_argument("--out", default="artifacts/repeated-trials.json")
    args = parser.parse_args()

    if args.trials < 2:
        parser.error("--trials must be at least 2")

    cases = json.loads(Path(args.cases).read_text(encoding="utf-8"))
    results = [run_case(case, args.trials, args.base_seed) for case in cases]
    report = {
        "created_at": datetime.now(timezone.utc).isoformat(),
        "configuration": {
            "trials": args.trials,
            "base_seed": args.base_seed,
            "cases_file": args.cases,
        },
        "results": results,
    }

    output = Path(args.out)
    output.parent.mkdir(parents=True, exist_ok=True)
    output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")

    for result in results:
        print(
            f"{result['case_id']}: "
            f"{result['passed']}/{result['trials']} passed, "
            f"rate={result['pass_rate']:.1%}, "
            f"threshold={result['min_pass_rate']:.1%}"
        )

    return 0 if all(item["meets_threshold"] for item in results) else 1


if __name__ == "__main__":
    raise SystemExit(main())

Every trial gets a unique deterministic seed. Using the same base seed later replays the same fake-model sample. With a remote model, the result may still change, which is precisely why you retain raw attempts and provider metadata.

The dominant output rate canonicalizes valid JSON, so whitespace differences do not count as semantic variants. A low rate signals diversity, not necessarily failure.

Verify Step 4: Run 30 trials and allow the expected nonzero exit while learning:

python3 run_trials.py --trials 30 --base-seed 1000 || true
python3 -m json.tool artifacts/repeated-trials.json > /dev/null

You should see one summary line per case. The artifact validation should produce no output and exit successfully.

Step 5: Interpret Pass Rates and Variation

A repeated-trial report contains several different signals. Do not collapse them into one vague "stability score."

Suppose 27 of 30 refund trials pass. The observed pass rate is 90 percent, but that does not prove the model's true future pass probability is exactly 90 percent. Thirty trials provide an estimate with substantial uncertainty. Increase the sample when decisions are costly or observed performance sits near the boundary.

Also inspect failures. Three independent safety violations are more important than three harmless formatting deviations. Aggregate metrics help you locate risk, while raw records tell you what the risk actually is.

Verify Step 5: Print the computed metrics:

python3 - <<'PY'
import json
report = json.load(open("artifacts/repeated-trials.json", encoding="utf-8"))
for row in report["results"]:
    print(row["case_id"], {
        "pass_rate": row["pass_rate"],
        "mean_score": row["mean_score"],
        "score_stddev": row["score_stddev"],
        "dominant_output_rate": row["dominant_output_rate"],
    })
PY

Expect two dictionaries with values between 0 and 1. Standard deviation is zero or positive.

Step 6: Add a Confidence Interval

A threshold decision is stronger when it acknowledges sampling uncertainty. Add confidence.py with a Wilson score interval, which behaves better than a simple normal approximation for modest samples and rates near zero or one:

from math import sqrt


def wilson_interval(successes: int, trials: int, z: float = 1.96) -> tuple[float, float]:
    if trials <= 0:
        raise ValueError("trials must be positive")
    proportion = successes / trials
    denominator = 1 + (z * z / trials)
    center = (proportion + z * z / (2 * trials)) / denominator
    margin = (
        z
        * sqrt(
            proportion * (1 - proportion) / trials
            + z * z / (4 * trials * trials)
        )
        / denominator
    )
    return max(0.0, center - margin), min(1.0, center + margin)

Then add this import near the top of run_trials.py:

from confidence import wilson_interval

Inside run_case, immediately after calculating passed_count, add:

    confidence_low, confidence_high = wilson_interval(passed_count, trials)

Add these fields to the returned dictionary:

        "confidence_95": {
            "low": confidence_low,
            "high": confidence_high,
        },

The interval does not account for biased cases, changing model versions, correlated failures, or evaluator mistakes. It only expresses binomial sampling uncertainty under the chosen setup. Treat it as one decision input.

A conservative release gate can require the lower bound to exceed a minimum. That usually requires more trials than checking the observed rate alone. Define that policy before seeing the candidate's results.

Verify Step 6: Rerun and inspect the interval:

python3 run_trials.py --trials 100 --base-seed 1000 || true
python3 - <<'PY'
import json
report = json.load(open("artifacts/repeated-trials.json", encoding="utf-8"))
for row in report["results"]:
    low = row["confidence_95"]["low"]
    rate = row["pass_rate"]
    high = row["confidence_95"]["high"]
    assert 0 <= low <= rate <= high <= 1
    print(row["case_id"], f"{low:.1%} to {high:.1%}")
PY

Every assertion should pass. Increasing trials generally narrows the interval when the underlying rate is similar.

Step 7: Turn the Harness into a CI Regression Gate

A CI job should fail on meaningful reliability regression, not on any text difference. Keep the script's existing exit behavior for the observed threshold, or make the lower confidence bound the gate for high-risk cases.

Create .github/workflows/llm-repeated-trials.yml:

name: LLM repeated trials

on:
  pull_request:
  workflow_dispatch:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Run repeated evaluation
        run: python3 run_trials.py --trials 100 --base-seed 1000
      - name: Upload evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: repeated-trial-report
          path: artifacts/repeated-trials.json

Pin the model version for candidate and baseline runs. If the provider silently updates an alias, record the resolved model identifier or fingerprint when available. Cache neither responses nor evaluator results unless cache keys include every relevant input.

For noisy production models, a candidate-versus-baseline comparison on identical cases and trial seeds is often more informative than an absolute rate alone. Use paired prompt evaluation when comparing two prompts, and define release rules with LLM evaluation regression thresholds.

Verify Step 7: Parse the YAML if your repository already uses a YAML checker, then run the exact CI command locally. For this dependency-free example, the decisive verification is:

python3 run_trials.py --trials 100 --base-seed 1000
echo $?

Exit code 0 means every case met its configured observed pass-rate threshold. Exit code 1 means at least one case failed, and the artifact remains available for diagnosis. Do not append || true in CI.

Test LLM Nondeterminism with Repeated Trials at Scale

Scale in layers. First, expand case coverage across intents, languages, adversarial inputs, long contexts, and safety conditions. Next, use enough trials per risk tier. Finally, schedule a larger nightly evaluation while keeping pull-request checks small enough to provide useful feedback.

Do not multiply every case by hundreds of trials without a budget model. A practical portfolio can use more repetitions for unstable or high-risk cases, fewer for deterministic low-risk checks, and periodic deep runs for the full suite. Record token usage and cost beside latency when using a paid API.

Control concurrency. Parallel requests reduce wall time but can trigger rate limits or introduce load-related effects. Record retry counts and distinguish provider errors from quality failures. Apply the same observable failure rules used in API rate limiting testing. A retry that replaces a failed request can make availability look better than users experience, so preserve both attempts.

Freeze evaluator versions. If a rubric or judge changes, the historical series is no longer directly comparable unless you rescore stored outputs. Keeping raw responses lets you do that.

Finally, segment results. A global 95 percent pass rate can hide a severe failure in one language or intent. Report by case, risk tier, locale, prompt version, and model version before calculating a suite-wide summary. Use an AI test review checklist to make coverage, evidence, and approval criteria explicit during review.

Troubleshooting

The same seed still returns different responses -> Treat the seed as metadata, not a guarantee. Confirm the exact model version, system prompt, tool definitions, sampling parameters, SDK version, and provider fingerprint. Store the response rather than assuming replay will regenerate it.

All trials are identical -> Check whether caching is enabled or temperature is zero. Identical output can be valid, but run diverse cases and confirm requests reach the model. The fake adapter intentionally varies seeds, so identical fake results may simply mean the case has few valid outputs.

Pass rate changes wildly between runs -> Use the same trial seeds for comparison, increase the sample, and inspect confidence intervals. Confirm that cases, evaluators, and model identifiers did not change.

JSON parsing fails even though the answer looks correct -> Preserve the raw response and use provider-supported structured output in the real adapter. Do not strip arbitrary text with a greedy regular expression because it can turn invalid behavior into an apparent pass.

CI fails only because of rate limits -> Classify transport, timeout, and rate-limit failures separately. Reduce concurrency, add bounded backoff, and keep an availability metric. Do not silently exclude failed requests from the denominator.

A judge score disagrees with reviewers -> Pause the gate and recalibrate the judge on representative human labels. Version the judge prompt and model, measure agreement by slice, and retain deterministic checks for objective requirements.

Common Mistakes

• Running one example and calling the prompt stable.
• Changing prompts, seeds, and model versions in the same comparison.
• Using exact text equality when the contract permits semantic variation.
• Reporting only an average and hiding individual severe failures.
• Choosing a pass threshold after looking at candidate results.
• Treating temperature as the only source of nondeterminism.
• Omitting malformed outputs, timeouts, or retries from the denominator.
• Letting an unvalidated LLM judge become the sole release authority.
• Failing to store raw responses and evaluation reasons.
• Comparing aggregate rates while ignoring weak slices.

A good harness makes failures inspectable. It should be possible to answer which input failed, on which trial, under which configuration, with what raw output, and why the evaluator rejected it.

Where To Go Next

Place repeated-trial testing inside the complete LLM evaluation pipeline, where datasets, evaluators, reports, and release gates are versioned together.

Then deepen the parts that repeated sampling depends on:

• Calibrate an LLM judge with human labels before scoring subjective qualities.
• Compare prompts with paired evaluation so both candidates face matched cases and trial conditions.
• Set LLM evaluation regression thresholds before CI decides whether a change ships.

Your next practical move is to replace generate with one real provider adapter, add five representative cases, and run the same fixed seed schedule against your current prompt and one candidate.

Interview Questions and Answers

Q: Why are repeated trials necessary for LLM testing?

A single output is one sample from a variable system. Repeated trials estimate failure frequency, reveal output modes, and expose whether a passing example is typical. They also produce raw evidence for comparing versions.

Q: Does setting temperature to zero make an LLM deterministic?

No. It reduces sampling variation, but infrastructure, numerical computation, model routing, provider updates, tool results, and ties during decoding can still change output. Test the deployed configuration rather than assuming a parameter guarantees determinism.

Q: What should remain fixed across trials?

Keep the model version, prompts, tools, evaluator, decoding settings, input case, and environment fixed. Vary only the trial identity or seed. Record every setting needed to interpret or reproduce the run.

Q: How many trials should a team run?

There is no universal number. Choose it from risk, expected failure rate, decision boundary, cost, and desired uncertainty. Use confidence intervals and increase trials when a result is close to the release threshold.

Q: What is the difference between consistency and correctness?

Consistency asks whether outputs stay similar across trials. Correctness asks whether each output satisfies requirements. A model can be consistently wrong or variably correct, so report both dimensions.

Q: How should timeouts and provider errors be counted?

Preserve and classify them. For user-facing reliability, they usually belong in an end-to-end failure denominator. Also report model-quality pass rate separately so availability problems do not obscure content defects.

Q: Why use paired trials when comparing prompts?

Paired trials reduce noise by exposing both prompts to matched cases and conditions. The comparison focuses on within-case changes instead of differences caused by unrelated samples. Keep raw pairs for review.

Conclusion

To test LLM nondeterminism with repeated trials, freeze the evaluation setup, sample each case repeatedly, preserve every response, apply versioned checks, and interpret pass rates with their uncertainty. Variation is not automatically a bug, but unmeasured variation is an unmanaged product risk.

Start with the dependency-free harness in this tutorial. Replace only the adapter, keep the artifact shape and verification checks, and establish thresholds before the next prompt or model change reaches CI.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Selenium Test Web Components Slots Tutorial: Test Web Component Slots with Selenium
• AI test data generation with Faker and LLMs (2026)
• Automating Jira to test case with n8n (2026)
• Building a test case generator with LangChain (2026)
• Connecting Claude to your test suite with MCP (2026)
• Evaluating an LLM app with DeepEval metrics (2026)