Writing custom DeepEval metrics (2026)

TL;DR

Writing custom DeepEval metrics means turning one product quality rule into a documented scoring function that follows DeepEval's metric lifecycle. Implement and unit-test the metric first, calibrate it against human labels, then use it in `assert_test` with versioned thresholds and datasets.

Key Takeaways

• Create a custom metric only when its score maps to a clear product decision and existing metrics do not express the rule.
• Subclass BaseMetric for single-turn cases and implement measure, a_measure, is_successful, and a readable metric name.
• Set score, success, reason when useful, and error consistently on every execution path.
• Prefer deterministic code for schemas, citations, exact policies, calculations, and other machine-checkable requirements.
• Calibrate thresholds against independently labeled examples and inspect disagreement slices rather than choosing convenient constants.
• Unit-test the metric as production code, including empty, malformed, boundary, Unicode, async, and exception cases.
• Version metric code, configuration, datasets, judge settings, and score semantics together.

Writing custom DeepEval metrics is appropriate when your application has a quality rule that built-in metrics, G-Eval, or DAG metrics do not express precisely enough. A good custom metric converts one documented requirement into a reproducible score, reason, and pass decision. It is not a convenient wrapper around an arbitrary number.

This 2026 guide uses DeepEval's current custom metric contract: inherit BaseMetric, implement synchronous and asynchronous measurement, set score and success, expose is_successful, and name the metric. You will build a runnable deterministic citation-coverage metric, test its failure paths, calibrate its threshold, integrate it with pytest, and decide when judge-based or composite designs are more appropriate.

TL;DR

The metric should fail clearly when required test-case data is absent. A score of zero must mean measured poor quality, not parser failure, missing labels, network error, or programmer exception.

1. When Writing Custom DeepEval Metrics Is Justified

Writing custom DeepEval metrics is justified when a product requirement has stable semantics, the necessary evidence exists in an LLMTestCase, and the resulting score supports a decision. Examples include required citation coverage, exact policy-reference use, JSON field completeness, product-specific refusal rules, controlled terminology, escalation triggers, domain calculations, or a calibrated combination used by an established release policy.

Do not write a metric merely because the framework allows it. First ask whether a deterministic assertion, existing metric, GEval, DAGMetric, or a human review is clearer. A custom class creates maintenance responsibility: score semantics, test data requirements, threshold, async behavior, exceptions, versioning, and documentation all belong to your team.

Define the metric card before code:

• quality property and product risk,
• evaluated unit,
• required LLMTestCase fields,
• scoring range and direction,
• interpretation of 0, intermediate values, and 1,
• pass threshold and who owns it,
• reason format,
• invalid-input and error behavior,
• known blind spots,
• calibration dataset and refresh trigger.

A citation-coverage metric can prove that required evidence identifiers appear in the answer. It cannot prove that the cited source supports the adjacent claim. That limitation belongs in the metric card and usually requires a separate entailment or human check.

If you are new to the framework, evaluating an LLM app with DeepEval metrics covers test-case construction and built-in metric selection before customization.

2. Choose Deterministic, Judge-Based, DAG, or Composite Scoring

The scoring mechanism should match the property. Deterministic code is ideal for exact, machine-verifiable requirements. A judge model is appropriate for semantic qualities that cannot be reduced safely to string rules. A DAG makes explicit decisions easier to inspect. A composite combines signals only when the aggregation itself has product meaning.

Prefer separate component metrics while exploring. A response with relevancy 1.0 and faithfulness 0.0 should not become an unexplained 0.5 that narrowly passes. If the release policy says both must pass, use a conjunction of pass states or a minimum score, not an average. If policy weights dimensions, document and calibrate the weights with stakeholders.

Do not call a judge for requirements code can prove. Regex can detect a citation token, JSON Schema can validate structure, and a domain function can recalculate tax. Conversely, regex cannot prove that a paragraph faithfully represents a policy.

The DeepEval vs Ragas comparison can help teams choose a framework boundary, but metric design principles remain the same: define evidence, score meaning, and release decision before implementation.

3. Understand the BaseMetric Lifecycle

For a single-turn custom metric, inherit BaseMetric and accept an LLMTestCase in measure() and a_measure(). The measurement methods must set self.score and self.success. A metric can also set self.reason and should set self.error when execution fails, then re-raise so infrastructure errors are not mislabeled as quality failures.

measure() is the synchronous path. a_measure() is the asynchronous path used when DeepEval runs metrics concurrently. Both must implement the same scoring semantics. If the work is trivial CPU logic, a_measure() can call measure(). If it performs network I/O, implement a real async client or offload blocking work carefully. Calling blocking network code directly in an async method defeats concurrency.

is_successful() translates score and error state into the metric's status. The common higher-is-better rule is score >= threshold. Some built-in metrics use lower-is-better semantics, so document direction and do not copy a threshold rule blindly. Expose a readable __name__ property for reports.

Initialize all state in __init__ and reset run-specific state at the beginning of measurement. This prevents stale score, reason, or error values if a metric object is reused. Prefer creating a new metric instance per test unless the framework's execution model and your class are explicitly safe for shared concurrency.

For multi-turn evaluation, use the conversational base and ConversationalTestCase, not a single-turn metric with the conversation flattened into one string. The test-case type tells DeepEval how to route the metric and which evidence is available.

4. Implement a Runnable Deterministic BaseMetric

The example metric checks whether an answer contains every required source identifier. The test case stores required IDs as a JSON array in expected_output, while actual_output contains citations such as [POLICY-1]. In a production project, you could use a dedicated field in your dataset adapter, but the metric still needs a clear serialization contract.

# citation_metric.py
import json
import re
from typing import Any

from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

CITATION = re.compile(r"\[([A-Z][A-Z0-9_-]*)\]")

class CitationCoverageMetric(BaseMetric):
    def __init__(self, threshold: float = 1.0, async_mode: bool = True):
        if not 0 <= threshold <= 1:
            raise ValueError("threshold must be between 0 and 1")
        self.threshold = threshold
        self.async_mode = async_mode
        self.score: float | None = None
        self.success: bool = False
        self.reason: str | None = None
        self.error: str | None = None

    def measure(self, test_case: LLMTestCase) -> float:
        self.score = None
        self.success = False
        self.reason = None
        self.error = None
        try:
            required: Any = json.loads(test_case.expected_output or "null")
            if not isinstance(required, list) or not required:
                raise ValueError("expected_output must be a nonempty JSON array")
            if not all(isinstance(item, str) and item for item in required):
                raise ValueError("every required citation ID must be a string")

            required_ids = set(required)
            observed_ids = set(CITATION.findall(test_case.actual_output or ""))
            covered = required_ids & observed_ids
            missing = sorted(required_ids - observed_ids)

            self.score = len(covered) / len(required_ids)
            self.success = self.score >= self.threshold
            self.reason = (
                f"Covered {len(covered)} of {len(required_ids)} required citations. "
                f"Missing: {missing or 'none'}."
            )
            return self.score
        except Exception as exc:
            self.error = str(exc)
            self.success = False
            raise

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        if self.error is not None or self.score is None:
            self.success = False
        else:
            self.success = self.score >= self.threshold
        return self.success

    @property
    def __name__(self) -> str:
        return "Citation Coverage"

This metric scores coverage, not citation correctness, placement, uniqueness, or source validity. Keep those as separate checks so the reason remains diagnostic.

5. Unit Test the Metric Before Evaluating the App

A custom metric is production test code and needs its own unit tests. Test the scoring function without calling the LLM application or DeepEval runner. Cover full, partial, and zero coverage; duplicates; extra citations; case sensitivity; malformed labels; empty expected data; threshold boundaries; repeated metric use; and the async path.

# test_citation_metric_unit.py
import asyncio
import pytest
from deepeval.test_case import LLMTestCase
from citation_metric import CitationCoverageMetric

def case(actual: str, required: str) -> LLMTestCase:
    return LLMTestCase(
        input="Summarize the approved refund policy.",
        actual_output=actual,
        expected_output=required,
    )

def test_full_coverage_passes() -> None:
    metric = CitationCoverageMetric(threshold=1.0)
    score = metric.measure(case("Refunds need approval [POLICY-1] [POLICY-2].",
                                '["POLICY-1", "POLICY-2"]'))
    assert score == 1.0
    assert metric.is_successful()

def test_partial_coverage_has_diagnostic_reason() -> None:
    metric = CitationCoverageMetric(threshold=1.0)
    score = metric.measure(case("See [POLICY-1].",
                                '["POLICY-1", "POLICY-2"]'))
    assert score == 0.5
    assert not metric.is_successful()
    assert "POLICY-2" in (metric.reason or "")

def test_invalid_expected_output_is_an_error_not_zero_quality() -> None:
    metric = CitationCoverageMetric()
    with pytest.raises(ValueError):
        metric.measure(case("See [POLICY-1].", "not-json"))
    assert metric.error is not None
    assert not metric.is_successful()

def test_async_and_sync_scores_match() -> None:
    metric = CitationCoverageMetric()
    test_case = case("See [POLICY-1].", '["POLICY-1"]')
    assert asyncio.run(metric.a_measure(test_case)) == 1.0

Use exact floating-point assertions when the fraction is exactly representable, or pytest.approx for general calculations. Add property tests for invariants such as adding a required valid citation cannot lower coverage, duplicate citations cannot raise it, and extra unrelated citations cannot cover a missing required ID.

A unit failure here is a metric defect, not an application regression. Keep these reports separate.

6. Integrate the Metric With DeepEval and Pytest

After unit testing, use the metric through DeepEval's normal test path. assert_test accepts an LLMTestCase and a list of metrics. This runnable example passes because both required source IDs appear.

# test_policy_answer.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from citation_metric import CitationCoverageMetric

def test_policy_answer_cites_all_required_sources() -> None:
    test_case = LLMTestCase(
        input="When can a customer receive a refund?",
        actual_output=(
            "A refund is available within the approved eligibility window "
            "[POLICY-ELIGIBILITY]. A manager must approve exceptions "
            "[POLICY-APPROVAL]."
        ),
        expected_output=(
            '["POLICY-ELIGIBILITY", "POLICY-APPROVAL"]'
        ),
    )
    metric = CitationCoverageMetric(threshold=1.0)
    assert_test(test_case, [metric], run_async=False)

Run it with DeepEval's test command according to your installed version and CI setup. Keep normal pytest markers for deterministic versus provider-backed suites. This metric does not need a judge API, so it belongs in a fast merge lane.

Do not reuse one mutable metric instance across parameterized tests running concurrently. Construct it inside the test or fixture with the correct scope. If the metric loads a large immutable resource, separate that resource from per-run score state.

When several metrics evaluate one case, report each score and reason. Avoid wrapping them into one composite until the decision rule is established. A hard citation requirement can gate independently, while a judge-based tone metric might use an aggregate candidate-versus-baseline policy.

Capture metric name, code version, threshold, test-case ID, dataset version, score, reason, success, and error. Without these, a CI failure cannot be reproduced after the metric evolves.

7. Calibrate Scores and Thresholds Against Human Labels

A default threshold is not a product policy. Calibrate the metric on independently reviewed examples that represent real tasks and risks. For deterministic citation coverage, stakeholders might require 100 percent on high-risk policy answers but permit a lower value for exploratory research summaries. That is a product decision, not a framework constant.

Create labels before tuning. Reviewers should classify pass or fail and explain required evidence without seeing the metric score when practical. Run the metric, compare disagreements, and inspect slices. A high overall agreement can hide severe misses on long answers, multilingual output, unusual source IDs, or answers with citations detached from claims.

For continuous scores, examine candidate thresholds across false-pass and false-fail costs. Prefer a simple policy that stakeholders understand. If the metric's score is not monotonic with human quality, changing the threshold will not fix the design. Revisit evidence extraction or split the property into multiple metrics.

Use a held-out set after calibration. Otherwise prompt, regex, weights, or judge instructions can overfit the same examples. Version the labeled dataset and define refresh triggers such as new product domains, source-ID formats, language support, or production incidents.

Track disagreement as a first-class artifact. A false fail may reveal a legitimate alternative citation format. A false pass may reveal that merely mentioning an ID is insufficient. Update the metric card and increment its version when score meaning changes. Do not silently modify a metric while retaining historical trend lines as if they were comparable.

The golden dataset guide for LLM evals provides a fuller approach to sourcing, labeling, provenance, and held-out evaluation.

8. Add Judge Calls or Composite Logic Carefully

A custom judge metric needs the same lifecycle plus model-call engineering. Define the judge prompt, structured response schema, model, timeout, retry policy, temperature or reasoning controls offered by the provider, and invalid-response handling. Preserve judge errors as errors. Do not convert them to score zero, because that confuses infrastructure availability with application quality.

Prefer built-in GEval when natural-language criteria are enough. Write a custom judge class only when you need special evidence assembly, model routing, scoring math, or privacy controls that existing metrics cannot support. Unit-test prompt construction and response parsing with doubles, then run a provider contract suite.

Composite metrics are similarly risky. If combining citation coverage C and faithfulness F, possible rules include:

• both must pass,
• score is min(C, F),
• weighted average with a hard floor,
• fail citation first, then evaluate faithfulness,
• report both without a composite score.

These rules express different policy. An average lets one dimension compensate for another. A minimum emphasizes the weakest. A gate sequence can reduce judge cost by skipping doomed cases. Choose through stakeholder impact and calibration, not mathematical elegance.

If child metrics make network calls, implement true async execution and preserve each child's reason and error. A composite reason should show component values and the aggregation rule. Avoid mutating child thresholds behind the caller's back.

The metric should remain understandable to the engineer who receives a 2 a.m. release failure. A compact, decision-aligned design is usually stronger than a complicated score with impressive decimals.

9. Engineer Errors, Performance, Privacy, and Versioning

Define three distinct outcomes: measured fail, evaluation error, and skipped or not applicable. A measured fail has valid evidence and a score below policy. An evaluation error means the metric could not compute, perhaps because input was malformed, a judge timed out, or a dependency failed. Skipped means the metric does not apply to that test-case type. Collapsing them into zero corrupts quality reporting.

Reset state before every measurement. Validate required fields with useful messages. Bound input sizes. Redact secrets from reasons and logs. If actual output contains customer data, avoid echoing full text into reason. Store case IDs and short evidence summaries.

For performance, profile dataset size, metric complexity, judge concurrency, retry behavior, and caching. Cache only when the key includes every semantic input: metric code or prompt version, model and settings, test-case evidence, and relevant configuration. A stale judge score can hide a regression. Deterministic metrics are cheap enough that caching may add unnecessary complexity.

Version all factors that affect meaning:

For sensitive workloads, document where judge data goes and how it is retained. A custom metric is part of the data flow and must follow the same privacy and access controls as the application evaluation pipeline.

10. Scale Writing Custom DeepEval Metrics in CI

Writing custom DeepEval metrics for CI requires ownership and change control. Run metric unit tests first. Run deterministic application evaluations on every relevant change. Run judge-based metrics on a smaller merge set, scheduled benchmark, or release lane according to cost and risk. Failures should link directly to case evidence, score, reason, threshold, metric version, and owner.

A practical quality gate separates:

1. metric implementation tests,
2. dataset schema and label checks,
3. application execution,
4. deterministic metric evaluation,
5. judge-based evaluation,
6. candidate-versus-baseline analysis,
7. human review for disagreements,
8. production monitoring and dataset refresh.

Fail immediately on metric exceptions in required lanes. Do not report the application as low quality when the evaluator is broken. For probabilistic metrics, define retry and aggregation policy in advance. Re-running until a score passes is not testing.

Report slices by task, risk, language, output length, source count, model version, and known failure mode. A global score can improve while critical citation coverage declines. Use hard gates for exact requirements and calibrated distributional gates for subjective quality.

Add a metric review checklist to pull requests: purpose, required fields, score math, error behavior, unit tests, async behavior, threshold evidence, dataset, privacy, performance, reason quality, and version change. Retire metrics that no longer influence a decision. More metrics are not automatically more confidence.

For broader production controls, guardrails testing for production LLMs shows how deterministic safety rules and model-based evaluations can coexist without confusing their authority.

Interview Questions and Answers

Q: When would you create a custom DeepEval metric?

I create one when a product-specific quality rule has defined evidence and score semantics that existing metrics cannot express clearly. I first check whether a deterministic assertion, GEval, DAG, or human review is simpler. The metric must support a real test or release decision.

Q: What methods must a custom single-turn metric implement?

It inherits BaseMetric and implements measure() and a_measure() with the same scoring semantics. It sets score and success, implements is_successful(), and exposes a readable name. Reason and error handling should also be consistent.

Q: How do you distinguish a metric error from a quality failure?

A quality failure has valid input and a computed score below threshold. A metric error means scoring could not complete because of malformed evidence, code failure, timeout, or dependency error. I set the error and fail the evaluation infrastructure path rather than returning zero.

Q: How would you test a custom metric?

I unit-test score boundaries, malformed and missing data, duplicates, Unicode, state reset, threshold equality, reason text, exceptions, and sync versus async equivalence. I add property invariants when possible, then test integration with assert_test.

Q: How do you choose a threshold?

I compare scores with independent human labels on representative examples and examine false-pass and false-fail costs by risk slice. I protect a held-out set and document the policy owner. A convenient default is not evidence.

Q: When should a_measure() call measure() directly?

That is acceptable for quick deterministic work that does not block on network or heavy I/O. For judge calls or other asynchronous dependencies, I use a real async path or a safe offloading strategy. Blocking I/O inside the event loop defeats concurrency.

Q: What is risky about composite metrics?

Aggregation can hide a critical component failure and weaken diagnosis. An average, minimum, conjunction, and gated sequence express different policies. I keep components separate until stakeholders define and calibrate a meaningful combined decision.

Q: How do you version custom metric results?

I store metric code version, configuration, threshold, dataset and labels, parser, judge prompt and model when used, plus the application evidence adapter version. Historical scores are comparable only when their semantics and population remain compatible.

Common Mistakes

• Creating a metric without a decision: A score is useful only when someone knows what action it supports.
• Using model judgment for exact rules: Prefer deterministic validation for machine-checkable requirements.
• Returning zero on exceptions: Separate evaluator failure from measured poor quality.
• Forgetting run-state reset: Reused instances can leak old error, score, or reason.
• Implementing fake async with blocking I/O: Use a real async client for network work.
• Choosing thresholds by intuition: Calibrate against independently labeled cases.
• Averaging incompatible dimensions: A strong score can conceal a critical zero.
• Testing the app but not the metric: The evaluator needs its own unit and contract tests.
• Writing vague reasons: Report evidence, missing items, component scores, and policy clearly.
• Changing metric semantics without versioning: Trend lines become misleading.

Conclusion

Writing custom DeepEval metrics is metric engineering, not just subclassing. Start with a precise quality property, choose the simplest valid scoring mechanism, implement the full lifecycle, test the evaluator independently, and calibrate thresholds against reviewed evidence.

Build one narrow deterministic metric first, such as citation coverage or required-field completeness. Once its score, errors, and decision policy are trustworthy, integrate it into DeepEval and CI. That discipline produces metrics engineers can debug and stakeholders can use.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Writing DeepEval G-Eval metrics (2026)
• Evaluating an LLM app with DeepEval metrics (2026)
• How to Use Cypress custom commands (2026)
• Prompt patterns for bug report writing (2026)
• Writing a bug report that gets fixed (2026)
• Writing a defect report (2026)