Writing DeepEval G-Eval metrics (2026)

TL;DR

Writing DeepEval G-Eval metrics starts with a narrow quality claim, explicit evaluation steps, and only the relevant SingleTurnParams fields. Add observable rubric anchors, calibrate the threshold against human labels, and run the metric through assert_test or evaluate without treating one stochastic score as ground truth.

Key Takeaways

• Write one metric for one decision-relevant quality dimension.
• Use current DeepEval SingleTurnParams values and include only fields named in the evaluation steps.
• Explore with criteria, then lock explicit evaluation_steps for a controlled metric.
• Anchor non-overlapping 0-10 Rubric bands with observable outcomes.
• Calibrate thresholds against independently labeled representative and adversarial examples.
• Keep deterministic checks beside G-Eval for schemas, permissions, tools, numbers, and exact rules.
• Version the metric, judge configuration, prompt, dataset, application, and threshold as one evaluation contract.

Writing DeepEval G-Eval metrics is the practical way to grade a product-specific quality that a generic metric does not capture, such as escalation quality, evidence-aware refusal, policy-compliant tone, or troubleshooting completeness. A useful G-Eval metric is a versioned measurement instrument. It names one construct, supplies the exact evidence the judge may use, defines ordered evaluation steps, anchors score bands, and demonstrates agreement with qualified human decisions.

This guide uses the current DeepEval single-turn API, including GEval, LLMTestCase, SingleTurnParams, Rubric, assert_test, and evaluate. The code is runnable once DeepEval and a supported judge provider are configured. The deeper focus is metric engineering: deciding what to measure, preventing leakage, calibrating a release threshold, diagnosing disagreement, and keeping a non-deterministic judge from becoming an unexplained quality oracle.

TL;DR

Use G-Eval for subjective, task-specific judgment. Use deterministic assertions for exact properties, and use DeepEval's system-specific metrics when their documented construct already matches the failure you need to detect.

1. Writing DeepEval G-Eval Metrics as Measurement Instruments

G-Eval uses an LLM as a judge to assess supplied test-case fields against a custom definition. DeepEval can generate evaluation steps from criteria, or it can follow explicit evaluation_steps. The current API requires a metric name, either criteria or steps, and a list of SingleTurnParams values that tells the judge which LLMTestCase fields to examine.

That flexibility is powerful and dangerous. A metric called Quality may produce a polished reason and a number, but nobody can tell which product decision the number supports. Start with a construct statement: Escalation quality is the degree to which a support answer recognizes an unsupported account-specific request, avoids pretending to act, and gives the user a concrete safe handoff. The statement identifies observable behavior and excludes unrelated writing style.

Treat the metric like test code plus a test oracle. Document:

• the failure mode it detects,
• the interaction boundary,
• required fields,
• excluded qualities,
• positive and negative anchors,
• intended decision,
• owners and reviewers,
• known limitations,
• calibration dataset and date,
• judge configuration and threshold.

G-Eval results are normalized into the 0 to 1 range in DeepEval, and success is based on the configured threshold. That does not make the score an objective probability. It is a judge output under a specific prompt, model, evidence set, and implementation.

If you are new to the full framework, read evaluating an LLM app with DeepEval metrics. A custom metric should complement a deliberate metric set, not replace system-level evaluation design.

2. Decide Whether G-Eval Is the Correct Metric Type

Reach for G-Eval when the quality is subjective, use-case-specific, and expressible through constrained judgment. Examples include whether a refund response is empathetic without making promises, whether a summary prioritizes decision-critical facts, or whether an agent explanation is actionable for a novice.

Do not use an LLM judge for exact JSON validity, a required key, a numeric limit, a tool name, a permission boundary, latency, token cost, or exact text prohibition. Code can evaluate those properties cheaply and reproducibly. Do not recreate a system-specific metric without a reason. DeepEval has dedicated metrics for areas such as RAG, agents, safety, and other interactions. Their documented data dependencies may align better with the construct.

Ask three questions. First, could two trained reviewers disagree reasonably? If no, encode the rule directly. Second, can the judge receive enough evidence to decide without private hidden knowledge? If no, redesign the test case. Third, will the score change a product or release decision? If no, the metric may be decorative.

A G-Eval metric should not compensate for a missing deterministic contract. For a text-to-SQL product, parse SQL safety rules with code and use G-Eval only for a truly semantic quality that parsing cannot establish.

3. Map the Evaluated Interaction to LLMTestCase Fields

LLMTestCase models one atomic single-turn interaction. The current fields include input and actual_output plus optional fields such as expected_output, context, retrieval_context, tool information, cost, and completion time. The metric's evaluation_params must list only the fields actually referenced by its criteria or steps.

Use the fields semantically:

Suppose the metric asks whether an actual support response correctly resolves the user's issue compared with a curated expected response. Its steps mention input, actual_output, and expected_output, so those three SingleTurnParams values are appropriate. Adding retrieval context simply because the application is RAG-based would leak irrelevant evidence into this metric and may change what the judge rewards.

Conversely, a custom grounded-refusal metric that tells the judge to verify claims against retrieved evidence must include SingleTurnParams.RETRIEVAL_CONTEXT and populate retrieval_context at runtime. Missing that evidence makes the task impossible. The judge may still return a score, which is why field mapping must be reviewed before execution.

Keep input atomic. A complete conversation belongs in a conversational test case and compatible multi-turn metric, not a JSON dump inside input. The interaction boundary determines what a result means.

4. Write Criteria First, Then Lock Explicit Evaluation Steps

DeepEval accepts either criteria or evaluation_steps, never both. Criteria is useful during exploration because DeepEval can derive steps. Explicit steps are better for a controlled metric because they prevent the step-generation phase from changing on every evaluation.

A weak criterion says: Evaluate whether the response is good. A better criterion says: Evaluate whether the actual support response resolves the user's stated problem accurately and completely compared with the expected response, without inventing actions or outcomes. It names evidence, desired behavior, and a serious penalty.

Convert the construct into ordered operations:

1. read the input and identify the user's explicit problem,
2. extract required facts and actions from the expected output,
3. compare the actual output for contradictions and unsupported claims,
4. identify omitted steps that prevent resolution,
5. distinguish harmless phrasing differences from behavioral differences,
6. assign a score using the rubric and explain the decisive evidence.

Each step should be observable and should mention DeepEval field names consistently. Avoid asking for hidden facts. Avoid combining tone, factuality, privacy, and completeness unless the product decision truly treats them as one inseparable construct. A composite score makes failure diagnosis and ownership difficult.

Include penalty priority. A contradiction or fabricated completed action may deserve a much lower band than a minor omission. State that in a step and in the rubric. Do not use subjective intensifiers such as very good without anchors.

Test the steps manually. Give two human reviewers several cases and ask them to follow the instructions. If they interpret a phrase differently, the model will too. Rewrite the instrument before increasing the judge model or adding repeated calls.

5. Add Non-Overlapping Rubric Anchors

DeepEval's Rubric accepts score_range values from 0 through 10, inclusive. Ranges must not overlap. The rubric constrains judge scores to meaningful bands and describes the expected outcome for each band. DeepEval then exposes the metric result on its normalized scale.

A practical four-band rubric for resolution quality is:

• 0 to 2: contradicts the required behavior, fabricates completion, or creates material risk,
• 3 to 5: partially relevant but misses a required action or contains an unsupported claim,
• 6 to 8: resolves the core problem correctly with only a non-critical omission,
• 9 to 10: complete, accurate, actionable, and free of unsupported claims.

Notice that each band contains observable consequences. Poor and excellent alone do not tell a judge where partial completeness belongs. Cover the whole 0 to 10 range and avoid gaps unless you have verified how the implementation handles them. Use a single-score band when a particular outcome deserves an exact anchor.

Rubrics reduce middle clustering, but they do not eliminate judge variance or bias. They also do not substitute for calibration. A threshold of 0.7 means the normalized score must meet that cut, but the business meaning comes from examples around the boundary. A case at 0.69 and a case at 0.71 may be practically indistinguishable.

Write boundary examples outside the production prompt first. Include an obvious fail, obvious pass, and several hard partial cases. Have reviewers justify which band each belongs to. Revise band wording until disagreement centers on true product ambiguity rather than metric ambiguity.

If pass is binary and perfection is required, DeepEval supports strict_mode, which forces a binary result and threshold of 1. Use that only when the construct genuinely requires perfection. Most nuanced quality dimensions need calibrated bands.

6. Runnable Example for Writing DeepEval G-Eval Metrics

Install DeepEval and pytest in an isolated environment:

python -m pip install -U deepeval pytest

Configure credentials for a judge provider supported by your DeepEval setup. Keep credentials in environment or secret management and do not hardcode them. This example intentionally does not pin a judge model string, so the environment's approved DeepEval configuration controls that operational dependency.

Create test_support_quality.py:

import pytest

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.metrics.g_eval import Rubric
from deepeval.test_case import LLMTestCase, SingleTurnParams


def resolution_quality() -> GEval:
    return GEval(
        name="Support resolution quality",
        evaluation_steps=[
            "Read 'input' and identify the user's explicit support problem.",
            "Extract the facts and required actions from 'expected output'.",
            "Check 'actual output' for contradictions or unsupported completed actions.",
            "Identify missing actions that would prevent the user from resolving the problem.",
            "Do not penalize wording differences that preserve the required behavior.",
            "Assign a score using the rubric and explain the decisive evidence.",
        ],
        evaluation_params=[
            SingleTurnParams.INPUT,
            SingleTurnParams.ACTUAL_OUTPUT,
            SingleTurnParams.EXPECTED_OUTPUT,
        ],
        rubric=[
            Rubric(
                score_range=(0, 2),
                expected_outcome=(
                    "Contradicts required behavior, fabricates completion, "
                    "or creates material user risk."
                ),
            ),
            Rubric(
                score_range=(3, 5),
                expected_outcome=(
                    "Partly relevant but misses a required action or "
                    "contains an unsupported claim."
                ),
            ),
            Rubric(
                score_range=(6, 8),
                expected_outcome=(
                    "Resolves the core problem correctly with only "
                    "a non-critical omission."
                ),
            ),
            Rubric(
                score_range=(9, 10),
                expected_outcome=(
                    "Complete, accurate, actionable, and free of "
                    "unsupported claims."
                ),
            ),
        ],
        threshold=0.7,
    )


CASES = [
    LLMTestCase(
        input="I was charged twice. Can you reverse one charge?",
        actual_output=(
            "I cannot reverse a charge here. Please open Billing, select the "
            "duplicate transaction, and choose Report duplicate. A billing "
            "specialist will review it."
        ),
        expected_output=(
            "Do not claim the reversal is complete. Direct the user to report "
            "the duplicate transaction in Billing for specialist review."
        ),
    ),
    LLMTestCase(
        input="I was charged twice. Can you reverse one charge?",
        actual_output="Done, I reversed the second charge.",
        expected_output=(
            "Do not claim the reversal is complete. Direct the user to report "
            "the duplicate transaction in Billing for specialist review."
        ),
    ),
]


@pytest.mark.parametrize("test_case", CASES, ids=["safe-handoff", "fabricated-action"])
def test_resolution_quality(test_case: LLMTestCase) -> None:
    assert_test(test_case=test_case, metrics=[resolution_quality()])

Run it with DeepEval's test command:

deepeval test run test_support_quality.py

The second case is deliberately adversarial and should challenge the threshold. Do not assume expected pass or fail until calibration demonstrates judge behavior.

7. Calibrate Thresholds Against Human Labels

Calibration asks whether metric decisions align closely enough with qualified reviewers for the intended use. Build a set that represents production topics, lengths, user styles, edge conditions, and known failure severity. Include hard cases near the expected decision boundary and counterexamples designed to expose shortcuts.

Create labels independently. At least two reviewers should apply a written rubric without seeing the G-Eval score. Resolve disagreements through adjudication and record whether the cause was product ambiguity, labeling error, or rubric ambiguity. Do not turn the judge output into the human label.

For each candidate threshold, calculate a confusion matrix:

The names assume acceptable is the positive class, so document your convention. A release gate may prioritize reducing false passes for safety-critical failures, while an exploratory dashboard may tolerate more false positives to surface candidates. Choose based on decision cost, not maximum accuracy alone.

Segment results. Overall agreement can hide poor performance on non-English inputs, refusals, long responses, rare domains, or a protected user group. Review false passes first because they show defects the gate would allow. Review false fails because noisy gates train teams to ignore results.

Do not calibrate and report on the same small set without acknowledging overfitting. Keep a held-out verification set or use repeated evaluation across controlled splits when the dataset is large enough. Re-evaluate after changing the judge, steps, rubric, threshold, prompt template, or data distribution.

For dataset construction, see writing golden datasets for LLM evals. The threshold belongs to a metric version and a dataset version together.

8. Reduce Variance, Leakage, and Judge Bias

G-Eval is non-deterministic. Explicit steps and rubrics improve control but do not create a deterministic oracle. Pin the judge configuration where the provider allows it, record the model identifier, and avoid silent fallback. Run a stability study on identical cases to see how often decisions cross the threshold.

Avoid reference leakage. If expected_output includes language that directly states this case should fail, the judge may follow the label rather than evaluate behavior. References should describe correct behavior, not evaluation verdicts. Remove case IDs, reviewer notes, and metadata that reveal the expected score unless the metric requires them.

Watch verbosity and position bias. A longer answer may appear more complete even when it invents details. A judge may favor text phrased like the reference. Include paraphrases, concise correct answers, verbose incorrect answers, reordered facts, and irrelevant polished content in calibration. Counterfactual pairs are valuable: change one decisive behavior while holding style constant.

Do not use the same judge to create expected outputs, write the metric, and certify the metric without independent human checks. Shared model tendencies can inflate apparent agreement. Human labels remain imperfect, but independent review breaks the simplest self-confirming loop.

Repeated judging can support analysis. Use multiple runs or multiple judges only with a predefined aggregation and cost policy. Majority vote can hide systematic bias and is not automatically more correct. Store individual decisions and reasons so disagreement remains visible.

Keep reasons as diagnostics, not proof. The explanation may be plausible after the score rather than a faithful record of internal causation. Use it to route investigation and inspect the actual fields yourself.

9. Combine G-Eval With Deterministic and System Metrics

One custom metric should not carry the whole release policy. Build a compact scorecard that maps each important failure to the cheapest valid check.

For a support RAG assistant:

• schema validation confirms the response envelope,
• permission assertions confirm no unauthorized tool action,
• a faithfulness metric evaluates claims against retrieved evidence,
• an answer relevancy metric checks whether the response addresses the request,
• a custom G-Eval metric grades safe escalation quality,
• latency and cost assertions enforce operational budgets.

Do not average unrelated metrics into one magic score without a decision model. A high tone score must not compensate for an unauthorized refund action. Use veto rules for critical deterministic failures and separate thresholds for independent constructs. Report the failed dimension and evidence.

Limit metric count. Every LLM-based judge adds latency, cost, variance, and another failure mode. DeepEval recommends a tight end-to-end set, and practical suites usually benefit from two or three system metrics plus one or two custom metrics rather than every available metric.

For RAG-specific grounding, use the RAG faithfulness and relevancy measurement guide. Do not write a generic RAG quality G-Eval metric that combines retrieval coverage, claim grounding, answer focus, and style. Component-level metrics make diagnosis possible.

Keep the product output generation and judge failure paths separate. A provider timeout is not a product-quality fail. Mark evaluation infrastructure errors distinctly and apply a documented fail-open or fail-closed policy based on risk. A silent pass on judge error destroys the gate.

10. Operate Writing DeepEval G-Eval Metrics in CI

In CI, use assert_test for pytest-style cases and deepeval test run for DeepEval's test execution features. Use evaluate for scripts, notebooks, or batch evaluation where a pytest assertion is not the desired interface. Both need controlled inputs, provider configuration, timeouts, and error handling.

Split suites by purpose. A small merge gate contains high-value, calibrated cases and critical deterministic checks. A larger scheduled suite explores topic and slice regressions. An experimental suite evaluates new metric versions without blocking releases. Promote a metric only after calibration and stability evidence.

Version a metric contract:

metric_id: support-resolution-quality
metric_version: 3
deepeval_api: SingleTurnParams
judge_policy: approved-default-2026-07
steps_hash: recorded-by-pipeline
rubric_version: 2
threshold: 0.70
dataset_version: support-golden-11
application_version: recorded-by-pipeline
owner: support-quality

The judge_policy should resolve to a controlled provider and model configuration in your environment. Do not place API keys in the manifest. Record prompt template changes if you customize DeepEval's template.

Set cost and concurrency budgets. Cache behavior can improve speed but should not hide changes to the application or metric inputs. Preserve test run metadata and reasons subject to privacy policy. Alert on evaluation infrastructure errors separately from quality failures.

When a case flips, inspect the application output, test-case fields, metric version, judge version, and repeated-run behavior. Do not immediately move the threshold. Threshold changes are policy changes that require recalibration and review. Retire metrics that no longer influence decisions or cannot maintain credible agreement.

Interview Questions and Answers

Q: What is G-Eval in DeepEval?

G-Eval is an LLM-as-a-judge approach for custom, task-specific evaluation. In DeepEval, I define a name, either criteria or explicit evaluation steps, and the relevant SingleTurnParams fields. The metric returns a normalized score and reason, with success based on a threshold. I treat it as a calibrated measurement instrument rather than objective ground truth.

Q: Should you use criteria or evaluation_steps?

I use criteria while exploring the construct and inspecting the steps DeepEval derives. For a controlled metric, I lock explicit evaluation steps so the step-generation phase does not vary. DeepEval accepts one or the other, not both. Any change to the steps creates a new metric version that needs calibration.

Q: Why should evaluation_params include only referenced fields?

Extra fields expose irrelevant evidence and can change what the judge rewards. Missing fields make the requested judgment impossible. I map each evaluation step to the exact LLMTestCase field it names and include only those SingleTurnParams values. This also makes leakage review straightforward.

Q: How do you choose a G-Eval threshold?

I compare metric decisions with independently labeled representative and adversarial examples. I inspect false passes, false fails, subgroup performance, and repeated-run flips at candidate thresholds. The selected cut reflects the cost of each error for the intended gate. I never copy a default and call it calibrated.

Q: What makes a good Rubric?

A good rubric uses non-overlapping 0 to 10 score bands with observable outcomes and severity. It covers contradiction, omissions, acceptable variation, and complete behavior for the construct. Boundary examples help reviewers and judges distinguish adjacent bands. Adjectives without behavior are weak anchors.

Q: How do you handle G-Eval non-determinism?

I pin and record judge configuration, provide explicit steps and anchors, and measure repeated-run stability. I analyze threshold crossings rather than assuming temperature or one run solves variance. Critical exact rules remain deterministic gates. Reasons and individual run results remain available for diagnosis.

Q: When should you avoid G-Eval?

I avoid it for exact schemas, numbers, permissions, tool names, latency, and rules that code can evaluate directly. I also prefer a documented system metric when it matches the construct. G-Eval is appropriate when nuanced product-specific judgment adds real decision value.

Q: How would you deploy a G-Eval metric in CI?

I start in a non-blocking experimental suite, calibrate it, and promote a small stable set to the merge gate. I separate judge infrastructure errors from product-quality failures, control secrets and budgets, and version the complete metric contract. Threshold or judge changes require revalidation.

Common Mistakes

• Naming the metric Quality. A broad name usually hides multiple constructs and produces unactionable failures.
• Passing criteria and evaluation_steps together. Current DeepEval expects one approach. Use steps for a locked instrument.
• Using the old field enum from memory. Current examples use SingleTurnParams for single-turn G-Eval configuration.
• Adding every LLMTestCase field. Irrelevant fields create leakage and distract the judge.
• Writing steps that require unavailable facts. The judge cannot verify a policy or retrieval source it never receives.
• Using overlapping rubric ranges. Rubric bands must be non-overlapping within 0 to 10.
• Copying a threshold from sample code. A tutorial threshold is not release policy for your product.
• Calibrating on easy cases only. Obvious passes and fails hide boundary disagreement and shortcut behavior.
• Letting verbosity substitute for correctness. Include concise correct and polished incorrect counterexamples.
• Averaging away critical failures. A strong style score cannot compensate for unauthorized action or fabrication.
• Treating judge errors as product failures. Provider timeouts and configuration errors need a separate status.
• Changing a threshold after one noisy run. Investigate provenance and stability, then recalibrate through governance.
• Hardcoding provider secrets in tests. Use approved secret injection and keep run artifacts sanitized.

Conclusion

Writing DeepEval G-Eval metrics is successful when a custom score has a precise construct, correct evidence mapping, explicit steps, anchored bands, and demonstrated agreement with human decisions. The Python class is the easy part. The durable value comes from calibration, counterexamples, versioning, and a release policy that never asks a subjective judge to enforce exact rules.

Choose one product-specific failure that generic metrics do not capture. Write its construct statement, label a small set independently, implement the current SingleTurnParams example, and keep the metric non-blocking until false passes, false fails, and repeated-run stability are understood.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Writing custom DeepEval metrics (2026)
• Evaluating an LLM app with DeepEval metrics (2026)
• Prompt patterns for bug report writing (2026)
• Writing a bug report that gets fixed (2026)
• Writing a defect report (2026)
• Writing a Dockerfile for test automation (2026)