Calibrate an LLM Judge Against Human Labels

TL;DR

Create an adjudicated human-labeled dataset, split it into calibration and holdout sets, run the LLM judge deterministically, and compare predictions with human labels using a confusion matrix, per-class metrics, raw agreement, and Cohen's kappa. Tune only on calibration data, validate once on the untouched holdout set, and define release gates by risk rather than copying a universal threshold.

Key Takeaways

• Freeze a representative, adjudicated human-labeled set before tuning the judge.
• Measure confusion matrices, class-level metrics, raw agreement, and Cohen's kappa together.
• Keep calibration examples separate from the final holdout evaluation set.
• Tune prompts and score thresholds on calibration data, then make one final holdout measurement.
• Inspect disagreements by slice because a good aggregate score can hide serious subgroup failures.
• Version the judge model, rubric, prompt, threshold, dataset, and code as one evaluation artifact.

To calibrate llm judge human labels, treat expert human decisions as the reference, measure where the automated judge disagrees, and tune the rubric or decision threshold on a separate calibration split. Do not accept a judge because a few examples look convincing. Require reproducible evidence on representative data.

This tutorial builds one complete workflow inside the broader LLM evaluation pipeline complete guide. You will create labeled fixtures, call an OpenAI-compatible judge endpoint, calculate agreement and class-level errors, tune a threshold without leaking holdout answers, and save an auditable report.

The example task labels support answers as pass or fail. The same method works for relevance, safety, groundedness, tone, or ordinal ratings, but the rubric and statistics must match the decision.

What You Will Build to calibrate llm judge human labels

You will build a Python calibration project that can:

• Load human-labeled examples with useful slices such as product area and difficulty.
• Ask a deterministic LLM judge for a label, confidence score, and short rationale.
• Calculate a confusion matrix, precision, recall, F1, raw agreement, and Cohen's kappa.
• Select a confidence threshold on calibration data using an explicit cost function.
• Validate the frozen judge configuration on an untouched holdout set.
• Write a JSON report that identifies the exact model, prompt, dataset, and results.

Prerequisites

Use Python 3.11 or newer. Create an isolated environment and install releases without pinning this article to future patch versions:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install openai pandas scikit-learn python-dotenv

Set an endpoint and credentials. The OpenAI Python client works with OpenAI and many OpenAI-compatible providers. Keep secrets out of source control.

export OPENAI_API_KEY=your_key_here
export OPENAI_BASE_URL=https://api.openai.com/v1
export JUDGE_MODEL=your_available_model_id

You also need two domain experts or one expert plus an adjudicator. The reference labels must represent an agreed rubric, not one person's unreviewed intuition. Have at least enough examples to cover each important class and risk slice. There is no universal minimum because the required sample depends on prevalence, desired uncertainty, and failure cost.

Verification: Run python --version and python -c "import openai, pandas, sklearn; print('ready')". Expect Python 3.11 or newer and the word ready.

Step 1: Define the Human Rubric Before Calling a Model

Write the labeling policy first. A good rubric defines the unit being judged, valid labels, positive and negative conditions, precedence rules, and what to do when evidence is missing. Here is the rubric used by the tutorial:

Task: Judge whether the assistant answer is acceptable customer support.

PASS when all conditions are true:
1. It directly addresses the user's request.
2. Its factual claims are supported by the supplied reference.
3. It does not invent policies, prices, or actions.
4. It is understandable and safe to send.

FAIL when any condition is true:
1. A material claim contradicts or exceeds the reference.
2. The requested action is ignored.
3. The answer promises an action the system cannot perform.
4. The answer exposes sensitive information or gives unsafe advice.

Precedence: Any safety or unsupported-material-claim failure overrides style quality.
If evidence is insufficient, return FAIL.

Verification: Give five examples to two reviewers without sharing their answers. Confirm that every disagreement can be resolved by pointing to, or improving, a written rubric rule. Save the agreed text as rubric-v1.txt.

Step 2: Create Representative Calibration and Holdout Data

Create labels.csv with this schema. The small sample demonstrates file format only and cannot establish a reliable production judge.

id,question,answer,reference,human_label,slice
1,Can I return an opened item?,Opened items can be returned within 30 days.,Opened items may be returned within 30 days with receipt.,pass,returns
2,When will my refund arrive?,Your refund is instant.,Refunds normally reach the original payment method in 5 to 10 business days.,fail,refunds
3,Can you reset my password?,I reset it for you.,The assistant cannot reset passwords and should direct users to the reset flow.,fail,account
4,How do I track my order?,Open Orders and select Track shipment.,Customers can track shipped orders from Orders using Track shipment.,pass,shipping
5,Do you store card numbers?,We publish all card numbers for auditing.,Card data must never be exposed; payments are handled by the payment processor.,fail,safety
6,Can I change my address?,Update the address under Profile before shipment.,An address can be changed under Profile only before the order ships.,pass,account
7,Can I cancel a shipped order?,Yes, cancel it from Orders.,Shipped orders cannot be canceled; the customer can request a return after delivery.,fail,shipping
8,Is gift wrap available?,Gift wrap is available at checkout.,Gift wrap is available for eligible products and appears at checkout.,pass,checkout

Split by stable IDs, not after reading judge outputs. Group near-duplicates and conversations from the same source so they cannot cross splits. Use roughly 60 to 80 percent for calibration and the rest for holdout, adjusted for dataset size. Never put holdout examples into the prompt.

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv("labels.csv")
calibration, holdout = train_test_split(
    df, test_size=0.25, random_state=42,
    stratify=df["human_label"]
)
calibration.to_csv("calibration.csv", index=False)
holdout.to_csv("holdout.csv", index=False)
print(calibration["human_label"].value_counts())
print(holdout["human_label"].value_counts())

Verification: Confirm IDs are unique, neither split is empty, both contain each label, and set(calibration.id).isdisjoint(set(holdout.id)) is true. For real conversational data, add a group ID and verify no group crosses the boundary.

Step 3: Implement a Structured LLM Judge

Create judge.py. The Responses API call requests JSON through a strict schema, uses a temperature of zero when supported by the selected model, and validates the returned values again in application code.

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1"),
)
MODEL = os.environ["JUDGE_MODEL"]
RUBRIC_VERSION = "support-pass-fail-v1"

SYSTEM = """You are a strict QA evaluator. Judge only from the supplied
question, answer, reference, and rubric. Do not use outside knowledge.
PASS requires relevance, support for every material claim, no impossible
promise, and safe wording. Any material contradiction, unsupported claim,
ignored request, impossible action, or unsafe content is FAIL. When evidence
is insufficient, choose FAIL. Return only the requested JSON."""

SCHEMA = {
    "type": "object",
    "properties": {
        "label": {"type": "string", "enum": ["pass", "fail"]},
        "pass_probability": {"type": "number", "minimum": 0, "maximum": 1},
        "reason": {"type": "string"}
    },
    "required": ["label", "pass_probability", "reason"],
    "additionalProperties": False
}

def judge(question: str, answer: str, reference: str) -> dict:
    payload = {"question": question, "answer": answer, "reference": reference}
    response = client.responses.create(
        model=MODEL,
        temperature=0,
        input=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(payload)}
        ],
        text={
            "format": {
                "type": "json_schema",
                "name": "judge_result",
                "strict": True,
                "schema": SCHEMA
            }
        }
    )
    result = json.loads(response.output_text)
    if result["label"] not in {"pass", "fail"}:
        raise ValueError("Unexpected label")
    if not 0 <= result["pass_probability"] <= 1:
        raise ValueError("Probability outside [0, 1]")
    return result

A probability supplied by a language model is a decision score, not automatically a statistically calibrated probability. You will use it to rank or threshold cases and later verify its behavior. Keep the rationale short because it is diagnostic evidence, not a hidden source of truth.

Some compatible providers implement chat completions but not Responses or strict schemas. In that case, adapt only the transport layer to the provider's documented structured-output API. Preserve the same input fields, schema validation, and stored metadata.

Verification: Call judge() on one obvious pass and one obvious fail. Confirm both results parse as JSON, contain exactly three fields, and keep pass_probability between zero and one.

Step 4: Run the Judge and Cache Every Result

Create run_judge.py. Caching prevents accidental extra cost and makes analysis reproducible. The script writes one JSON Lines record per example so a partial run remains recoverable.

import json
import time
import pandas as pd
from judge import judge, MODEL, RUBRIC_VERSION

INPUT = "calibration.csv"
OUTPUT = "calibration_predictions.jsonl"

df = pd.read_csv(INPUT)
with open(OUTPUT, "w", encoding="utf-8") as out:
    for row in df.itertuples(index=False):
        for attempt in range(3):
            try:
                result = judge(row.question, row.answer, row.reference)
                break
            except Exception:
                if attempt == 2:
                    raise
                time.sleep(2 ** attempt)
        record = {
            "id": int(row.id),
            "human_label": row.human_label,
            "slice": row.slice,
            "judge_label": result["label"],
            "pass_probability": result["pass_probability"],
            "reason": result["reason"],
            "model": MODEL,
            "rubric_version": RUBRIC_VERSION
        }
        out.write(json.dumps(record, ensure_ascii=False) + "\n")
        out.flush()
print(f"Wrote {len(df)} predictions to {OUTPUT}")

In production, add request IDs, prompt hashes, latency, token usage when the provider returns it, retry details, and an error status. Do not silently convert API failures into fail; infrastructure errors are not quality judgments. Avoid parallel requests until you understand provider rate limits.

Even at temperature zero, hosted model behavior can change and some operations remain nondeterministic. Before trusting a single prediction per item, use repeated trials to test LLM nondeterminism on a representative subset.

Verification: Compare the CSV row count with the JSONL line count. Check that IDs are unique and every record has the expected model and rubric version. Re-run only in a separate output file, then compare it to detect unstable labels.

Step 5: Measure Agreement and Class-Level Errors

Create analyze.py. Accuracy alone can be misleading when one label dominates. Cohen's kappa adjusts observed agreement by the agreement expected from the two label distributions, while class metrics reveal which mistakes drive the result.

import pandas as pd
from sklearn.metrics import (
    accuracy_score, classification_report, cohen_kappa_score,
    confusion_matrix
)

df = pd.read_json("calibration_predictions.jsonl", lines=True)
y_true = df["human_label"]
y_pred = df["judge_label"]
labels = ["fail", "pass"]

print("raw_agreement", round(accuracy_score(y_true, y_pred), 4))
print("cohen_kappa", round(cohen_kappa_score(y_true, y_pred), 4))
print(pd.DataFrame(
    confusion_matrix(y_true, y_pred, labels=labels),
    index=[f"human_{x}" for x in labels],
    columns=[f"judge_{x}" for x in labels]
))
print(classification_report(y_true, y_pred, labels=labels, zero_division=0))

for slice_name, group in df.groupby("slice"):
    agreement = accuracy_score(group["human_label"], group["judge_label"])
    print(slice_name, len(group), round(agreement, 4))

Read the confusion matrix in business terms. A human fail predicted as pass lets a defective answer escape. A human pass predicted as fail creates a false alarm or blocks a good response. Those errors rarely have equal cost.

Verification: Manually recompute matching labels for ten rows and compare with raw agreement. Confirm the confusion-matrix total equals the number of records and inspect every disagreement's input, rationale, and adjudication reason.

Step 6: Analyze Disagreements and Improve the Prompt

Export disagreements instead of immediately adding examples to the prompt. First classify each mismatch as rubric ambiguity, bad reference, human-label error, judge reasoning error, score-threshold error, or unsupported input format.

import pandas as pd

labels = pd.read_csv("calibration.csv")
preds = pd.read_json("calibration_predictions.jsonl", lines=True)
review = labels.merge(
    preds[["id", "judge_label", "pass_probability", "reason"]],
    on="id"
)
disagreements = review[review["human_label"] != review["judge_label"]]
disagreements.to_csv("disagreements.csv", index=False)
print(disagreements[["id", "slice", "human_label", "judge_label", "reason"]])

Fix the source of the problem. Clarify the rubric when reviewers disagree about policy. Correct the dataset when the reference or human label is wrong. Improve prompt instructions when the judge repeatedly ignores precedence. Add a few diverse examples only when a decision boundary is hard to express as a rule.

Change one major variable per experiment. Save each candidate prompt, run all candidates over the same calibration IDs, and compare per-item outcomes. The paired prompt evaluation tutorial shows how paired analysis isolates changes better than comparing unrelated aggregate runs. Never select a prompt by reading holdout failures.

Verification: Every disagreement row should have a reviewed root-cause category and disposition. After a prompt revision, confirm that targeted errors improve without increasing safety-critical false passes or damaging another slice.

Step 7: calibrate llm judge human labels with a decision threshold

The judge's categorical label may not reflect your error costs. Use pass_probability as a score and choose a threshold on calibration data. The example below heavily penalizes a false pass, where the human said fail but the judge allowed the answer.

import pandas as pd
from sklearn.metrics import confusion_matrix

df = pd.read_json("calibration_predictions.jsonl", lines=True)
y_true = (df["human_label"] == "pass").astype(int)

best = None
for threshold_int in range(5, 96, 5):
    threshold = threshold_int / 100
    y_pred = (df["pass_probability"] >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    cost = 10 * fp + 1 * fn
    candidate = {
        "threshold": threshold, "cost": int(cost),
        "false_passes": int(fp), "false_fails": int(fn),
        "true_fails": int(tn), "true_passes": int(tp)
    }
    if best is None or (candidate["cost"], -threshold) < (best["cost"], -best["threshold"]):
        best = candidate
print(best)

Set the cost ratio with product, safety, and operations owners. The value 10 is illustrative, not a recommendation. You can instead require a minimum fail recall, cap false-pass rate, introduce a human-review band, or optimize a domain-specific expected cost.

A useful three-way policy is: automatically fail below a low score, send ambiguous middle scores to human review, and automatically pass only above a high score. That reduces risky automation at the price of review volume. Measure coverage as well as accuracy so the judge cannot appear strong merely by deferring most cases.

Threshold search does not turn self-reported confidence into calibrated probability. If you need probability semantics, inspect reliability bins and consider a documented calibration model fitted only on calibration data. For most QA gates, a validated decision threshold and review band are easier to explain.

Verification: Print all threshold candidates and confirm the selected result follows the declared cost rule. Review false passes individually. Freeze the threshold, prompt, rubric, and model before opening holdout predictions.

Step 8: Validate Once on the Untouched Holdout Set

Change the runner input to holdout.csv, write holdout_predictions.jsonl, and do not revise the judge based on those results. Apply the already selected threshold, then generate a machine-readable report.

import hashlib
import json
from pathlib import Path
import pandas as pd
from sklearn.metrics import accuracy_score, cohen_kappa_score, confusion_matrix

THRESHOLD = 0.80  # Replace with the frozen calibration result.
df = pd.read_json("holdout_predictions.jsonl", lines=True)
y_true = (df["human_label"] == "pass").astype(int)
y_pred = (df["pass_probability"] >= THRESHOLD).astype(int)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()

def sha256(path: str) -> str:
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()

report = {
    "dataset_sha256": sha256("holdout.csv"),
    "model": str(df["model"].iloc[0]),
    "rubric_version": str(df["rubric_version"].iloc[0]),
    "threshold": THRESHOLD,
    "n": len(df),
    "raw_agreement": accuracy_score(y_true, y_pred),
    "cohen_kappa": cohen_kappa_score(y_true, y_pred),
    "confusion": {"tn": int(tn), "fp": int(fp), "fn": int(fn), "tp": int(tp)}
}
Path("holdout_report.json").write_text(
    json.dumps(report, indent=2) + "\n", encoding="utf-8"
)
print(json.dumps(report, indent=2))

Define acceptance rules before the run. Include the minimum sample and slice coverage, a maximum count or rate for critical false passes, class-specific metric floors, uncertainty requirements, and required stability across repeated trials. The right gate follows harm and operational cost, not a copied benchmark. Use LLM evaluation regression thresholds to turn the validated baseline into a maintainable CI policy.

If holdout performance misses the gate, reject the configuration. Return to development with a newly designated calibration set, then obtain a fresh holdout for the next unbiased final claim. Repeatedly tuning against the same holdout converts it into training data.

Verification: Confirm the report hash matches the exact reviewed holdout file. Ensure the confusion total equals n, the threshold matches the frozen decision, and no holdout item appeared in prompts or calibration analyses. Have a reviewer sign off on critical disagreements.

Step 9: Operationalize Drift Checks and Auditability

Package the judge as a versioned evaluation component. Store the model identifier, provider, system prompt hash, rubric version, few-shot example IDs, threshold, code commit, dataset hash, timestamp, and result metrics. A model name alone is insufficient because providers can update aliases and behavior. Pin a snapshot identifier when the provider offers one.

Run a compact canary set on every judge configuration change. Run the broader regression set on application releases and a schedule appropriate to traffic and risk. Sample current production cases for periodic human relabeling because user requests, references, and answer styles drift. Compare new human-judge disagreements with the original slices.

Separate two questions:

1. Did the application response quality change?
2. Did the automated judge's relationship to humans change?

Monitor coverage, class prevalence, false-pass rate on audited samples, agreement, kappa, slice metrics, missing-data rate, refusal rate, latency, cost, and label stability. Alert on data-quality changes as well as metric changes. An empty reference or a new response format can invalidate results without producing an API error.

Verification: Reconstruct a past decision from stored artifacts in a clean environment. You should be able to identify the exact inputs and configuration, reproduce analysis from cached outputs, and explain why the release passed or failed.

Troubleshooting

Problem: Human agreement is low -> Stop model tuning. Clarify overlapping rules, add precedence and counterexamples, train reviewers, then independently relabel a pilot set. Report pre-adjudication agreement so consensus does not hide rubric weakness.

Problem: The judge returns prose or malformed JSON -> Use the provider's documented strict structured-output feature, validate the response, retry transport failures, and record permanent parse failures separately. Do not extract labels with a loose substring search.

Problem: Accuracy is high but failures escape -> Inspect the confusion matrix and fail recall. The pass class may dominate. Increase the false-pass cost, raise the pass threshold, or introduce human review while maintaining representative prevalence in the evaluation set.

Problem: Calibration improves but holdout performance drops -> Suspect prompt overfitting, duplicate leakage, or a nonrepresentative split. Remove overly specific few-shot examples, group related records before splitting, and collect a fresh holdout after revising the process.

Problem: Results change between identical runs -> Confirm exact model snapshot and prompt bytes, cache inputs, and check provider settings. Measure repeated-trial label flip rate. Use majority voting only after validating that its extra cost and latency improve human agreement.

Problem: One slice has too few examples -> Do not publish a confident slice percentage from a tiny denominator. Collect targeted adjudicated cases, report counts with every metric, and keep high-risk sparse slices under manual review.

Where To Go Next

Place this judge calibration workflow inside the complete LLM evaluation pipeline, where datasets, graders, regression gates, and production monitoring connect. Then deepen the pieces that most affect reliability:

• Use repeated-trial nondeterminism testing before assuming one judge call represents stable behavior.
• Use paired evaluation to compare prompt candidates without losing per-example evidence.
• Use regression thresholds for LLM evaluation to convert holdout evidence into release policy.

Interview Questions and Answers

Use the seven interview questions and model answers in the structured interviewQnA section below. They cover calibration meaning, metric selection, leakage prevention, reviewer disagreement, threshold selection, class imbalance, and production monitoring. Practice answering each with the rubric, split strategy, and risk-based release policy from this tutorial.

Common Mistakes

• Treating one reviewer as ground truth without measuring human consistency.
• Tuning prompts on the same records used for the final performance claim.
• Reporting accuracy without class prevalence, confusion counts, or critical slices.
• Assuming self-reported confidence is a calibrated probability.
• Silently treating API and parse errors as quality failures.
• Adding many handpicked few-shot examples until the calibration set is memorized.
• Copying a universal metric threshold without modeling business harm.
• Changing model aliases without rerunning human-alignment checks.

Conclusion

To calibrate llm judge human labels responsibly, begin with a decision rubric humans can apply, preserve an untouched holdout set, and evaluate exact errors rather than a single attractive score. Tune prompts and thresholds only on calibration data, then freeze the complete judge configuration before final validation.

Your next practical step is to adjudicate a representative pilot set and run the scripts end to end. Once the holdout gate passes, keep sampling fresh human labels so calibration becomes an ongoing quality control, not a one-time certification.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Evaluating an LLM app with DeepEval metrics (2026)
• Load testing an LLM API (2026)
• Red teaming an LLM chatbot (2026)
• Testing an LLM agent memory (2026)
• Testing an LLM router (2026)
• Testing structured output from an LLM (2026)