Evaluating an LLM app with DeepEval metrics (2026)

TL;DR

Model one real LLM interaction as an `LLMTestCase`, choose only metrics whose required fields match that interaction, and run them with `assert_test` through `deepeval test run`. Calibrate thresholds and custom G-Eval rubrics against human labels before making them release gates.

Key Takeaways

• Define the evaluated interaction and failure model before importing a metric.
• Populate each LLMTestCase field from real application traces, not from a combined prompt dump.
• Use AnswerRelevancyMetric for on-topic responses and FaithfulnessMetric for claims grounded in retrieval_context.
• Use GEval for a precise product rubric, then calibrate it against human labels.
• Treat thresholds as release policy derived from evidence, not universal quality constants.
• Version datasets, prompts, application models, judge models, retrieval snapshots, and metric configuration together.
• Keep deterministic assertions beside model-based metrics for schemas, tools, permissions, and exact business rules.

Evaluating an LLM app with DeepEval metrics is not a matter of adding every available score to pytest. First define the interaction you are grading, the evidence captured from the application, and the failures that matter. Then select a small metric set whose required LLMTestCase fields match that evidence.

This guide builds a practical DeepEval suite for a retrieval-augmented application, adds a custom G-Eval rubric, and explains calibration, CI, cost, variance, and failure diagnosis. The examples follow the current DeepEval interfaces and can be adapted to chatbots, summarizers, copilots, and agents. The emphasis is test engineering: every score must have a clear data dependency, owner, and debugging path. That discipline matters because a judge can produce a precise-looking number even when a case is missing the evidence needed to assess the product. The sections below keep dataset validation, application execution, semantic measurement, and release policy visible as separate concerns.

TL;DR

Use two or three system metrics plus one product-specific metric. Add deterministic assertions for exact requirements. More metrics can mean more cost, slower feedback, correlated signals, and harder triage without improving the release decision.

1. What Evaluating an LLM App with DeepEval Metrics Involves

DeepEval represents a single atomic interaction with LLMTestCase. At minimum, it carries the user input and application actual_output. Depending on the metric, it can also contain expected_output, curated context, runtime retrieval_context, tools_called, expected tools, token cost, and completion time. The first design decision is where that interaction begins and ends.

For a RAG pipeline, a useful end-to-end interaction includes the user question, retrieved chunks, and generated answer. That scope supports answer relevancy and faithfulness. For an agent, the interaction may include selected tools and final output. For a retriever component, generation metrics are the wrong choice because the output under test is a ranked context set.

Write a failure model at the same scope. A RAG answer can be irrelevant, unsupported, incomplete, unsafe, too verbose, or based on poor retrieval. A metric is useful only if a score movement maps to an actionable defect. If a low value could mean five unrelated things, add more targeted evidence or reduce the interaction scope.

DeepEval is an evaluation framework, not the test oracle itself. Its built-in metrics encode particular definitions, and model-based metrics inherit judge limitations. Your team remains responsible for product rubrics, goldens, thresholds, sampling, and the decision that a failure blocks delivery.

2. Build a Versioned Evaluation Dataset

Start from real user tasks and known risks. Collect frequent intents, complex reasoning paths, policy boundaries, ambiguous queries, adversarial inputs, empty retrieval, conflicting sources, and cases that previously failed. Remove personal or confidential data while preserving the structure that made each interaction difficult.

A golden record should include stable identifiers and enough state to reproduce the application call:

{
  "id": "refund-policy-eligible-01",
  "input": "Can I return shoes delivered 12 days ago?",
  "expected_facts": ["returns are accepted within 30 days"],
  "source_ids": ["returns-policy-v7"],
  "risk": "medium",
  "tags": ["returns", "policy", "rag"]
}

Generate actual_output by calling the application during the test or attach a previously captured output when evaluating a frozen candidate. Capture the exact retrieval_context returned at runtime. Do not put the system prompt or serialized message list into input; DeepEval expects the user input at the interaction boundary. Track prompt configuration separately as a version or hyperparameter.

Split datasets by purpose. A small deterministic smoke set can run on every pull request. A balanced regression set can gate a candidate. A larger discovery set can run nightly. Keep an untouched holdout for major tuning decisions so repeated prompt changes do not overfit every visible case. The LLM application testing strategy explains how these layers fit a broader QA plan.

3. Map Failure Modes to DeepEval Metrics

Choose metrics from the defect backward. If users complain that answers wander, answer relevancy is appropriate. If a RAG system adds facts absent from retrieved chunks, use faithfulness. If the product must state a decision, reason, and next step in a specific order, create a G-Eval rubric or deterministic schema check.

Pay attention to field semantics. retrieval_context is what the retriever actually supplied to the generator. context is curated ground truth. expected_output is an ideal output when your metric needs a reference. Interchanging these fields changes what is being measured. A faithfulness failure points primarily at generated claims relative to the retrieved evidence, while poor retrieval needs contextual retrieval metrics or separate ranking evaluation.

Also check score direction. Most quality metrics pass when the score is at or above threshold. Metrics such as toxicity and hallucination express an undesirable proportion, so lower values are better and the threshold is a maximum. Read the metric contract instead of applying one comparison rule to every score.

Avoid redundant score collections. Relevancy, completeness, and a broad custom correctness rubric may overlap heavily. Begin with a compact set, label real defects, and ask whether each metric finds a distinct, actionable class. A metric that never changes a decision or diagnosis is instrumentation cost without quality value.

4. Run DeepEval Metrics with Pytest

Create a virtual environment, then install the framework and pytest. DeepEval model-based metrics require access to a configured judge model. Its default provider uses an OpenAI API key, or you can configure another supported provider or a custom model.

python -m pip install -U deepeval pytest
export OPENAI_API_KEY='your-test-project-key'

The following test_llm_eval.py is runnable and uses a frozen application output so the example isolates evaluation. Replace the fixture values with a call to your application when you are ready to run end to end.

from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

answer_relevancy = AnswerRelevancyMetric(
    threshold=0.7,
    include_reason=True,
)
faithfulness = FaithfulnessMetric(
    threshold=0.8,
    include_reason=True,
)
resolution_quality = GEval(
    name='Resolution Quality',
    criteria=(
        'Determine whether the actual output answers the user request, '
        'states only facts supported by the retrieval context, and gives '
        'a clear next step when one is required.'
    ),
    evaluation_params=[
        SingleTurnParams.INPUT,
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.RETRIEVAL_CONTEXT,
    ],
    threshold=0.7,
)

def test_refund_answer() -> None:
    case = LLMTestCase(
        input='Can I return shoes delivered 12 days ago?',
        actual_output=(
            'Yes. The policy allows returns within 30 days of delivery. '
            'Open your order and choose Start a return.'
        ),
        retrieval_context=[
            'Standard items may be returned within 30 days after delivery.',
            'Customers start an eligible return from the order details page.',
        ],
    )
    assert_test(
        test_case=case,
        metrics=[answer_relevancy, faithfulness, resolution_quality],
    )

Run the suite with:

deepeval test run test_llm_eval.py

The numeric thresholds are illustrative starting points, not recommendations for every product. Calibrate them before using the test as a release gate. Keep credentials in CI secrets, restrict judge access, and avoid sending sensitive evaluation content to an external provider without an approved data policy.

5. Interpret Answer Relevancy and Faithfulness Correctly

AnswerRelevancyMetric asks whether the actual_output addresses the input. It does not prove that the answer is factually correct. A confident but invented answer can be highly relevant. Use it to catch tangents, excessive unrelated content, misunderstood intent, and responses that avoid the question.

FaithfulnessMetric evaluates claims in actual_output against retrieval_context. It is valuable for RAG generators because it tests grounding in the evidence that actually reached the model. A real-world fact absent from those chunks can still be marked unfaithful. That is expected: the metric is checking support by retrieved evidence, not encyclopedic truth.

Interpret the combination:

Do not stop at the quadrant. Inspect metric reasons, atomic claims, and the retrieved chunks. A low faithfulness result can expose an unsupported claim, but retrieval may still be the upstream contributor if the necessary source never arrived. Add contextual precision, recall, or relevancy only when you have the reference data needed to diagnose retrieval. For browser-facing answers, combine semantic evaluation with stable UI assertions such as those in the Playwright role locator guide.

6. Create a Product-Specific G-Eval Rubric

Generic metrics rarely cover the full acceptance criteria. GEval lets you define a named criterion or explicit evaluation steps and select the SingleTurnParams the judge may use. This is where product language must become observable behavior.

Weak criterion: The answer should be good. Strong criterion: The answer identifies the eligibility decision, gives the policy basis using retrieved context, provides the next action, and does not claim that an action occurred unless a tool result confirms it. The strong version separates dimensions that reviewers can label.

For consequential workflows, use explicit steps:

from deepeval.metrics import GEval
from deepeval.test_case import SingleTurnParams

policy_compliance = GEval(
    name='Policy Compliance',
    evaluation_steps=[
        'Identify each policy claim in the actual output.',
        'Check every claim against the retrieval context.',
        'Check that the answer does not promise an exception.',
        'Check that the next step matches the stated eligibility decision.',
        'Assign a high score only when all material checks pass.',
    ],
    evaluation_params=[
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.RETRIEVAL_CONTEXT,
    ],
    threshold=0.8,
)

Keep one construct per rubric. A single judge combining correctness, tone, safety, brevity, and formatting produces a score that is difficult to act on. Put exact formatting in code. Put privacy and unauthorized actions in hard gates. Use G-Eval where meaning and context require judgment.

Version the name, criterion, steps, judge model, and threshold. Changing any of them changes the measurement instrument, so compare old and new instruments on the same calibration set before migrating trends.

7. Calibrate Judges and Thresholds

Threshold selection is an empirical classification task. Ask domain reviewers to label a representative set as release-acceptable or unacceptable and record defect types. Run the metric, inspect false passes and false failures, and choose a threshold that reflects the cost of those errors. A safety-adjacent rubric usually prioritizes catching unacceptable outputs, while a style metric may tolerate more borderline variation.

Measure judge-human agreement by important slice, not only overall. A judge may perform well on English fact questions and poorly on terse multilingual support responses. Include boundary cases and difficult negatives in calibration. Review confusion examples together and refine ambiguous rubric language before blaming the model.

Model-based evaluation can vary. On a repeatability subset, run the same cases more than once under controlled configuration and track label flips near the threshold. Options include widening the review band, requiring human review for borderline results, using more explicit steps, or replacing a fuzzy rubric with deterministic logic. Repeated runs should be a measured policy, not a way to reroll failures until they pass.

When the judge model changes, treat it like a test framework migration. Re-score the calibration set, compare classifications and reasons, review changed cases, and establish a new baseline. Never splice scores from different judges into one trend line without marking the boundary.

8. Put DeepEval in CI Without Creating Noise

Layer the pipeline. Fast deterministic contract tests run on every change. A small DeepEval smoke set runs when prompts, retrieval, model configuration, or LLM-facing code changes. Broader evaluation runs before release or on a schedule. This controls cost and feedback time while preserving coverage.

Pin Python dependencies with the repository's normal lock mechanism. Record the application revision, dataset revision, prompt version, generator model identifier, judge configuration, retrieval index snapshot, and metric settings in the run. Cache evaluation results only when the complete input and configuration identity match. A stale cached judgment can hide a real model or rubric change.

Handle provider failures separately from quality failures. Rate limits, timeouts, authentication errors, and service outages should report an infrastructure status, not a zero quality score. Define bounded retry behavior for transient errors and fail the pipeline as indeterminate if the evaluation did not complete. Do not silently skip cases.

Control concurrency according to provider limits and budget. A large case-by-metric matrix can generate many judge calls. Start small, measure duration and cost, and parallelize within safe quotas. Use deepeval test run output for debugging and export stable summaries for the CI system. The quality gate should list failing scenario IDs and metric names, not only a red job.

9. Diagnose Metric Failures Systematically

A failed eval starts an investigation. First confirm the test input and expected evidence are correct. Then inspect actual_output, retrieval_context, tool traces, and metric reason. Classify the failure as application, retrieval, dataset, evaluator, environment, or policy. That taxonomy prevents every red score from becoming a prompt edit.

For low relevancy, check intent routing, ambiguous input, refusal policy, and irrelevant verbosity. For low faithfulness, enumerate claims and locate support in the retrieved chunks. If required evidence is missing, reproduce the retrieval query and ranking. For G-Eval, compare the judge reason with rubric steps and human labels. A reason that discusses an unstated preference indicates rubric or judge drift.

Preserve passing outputs too. A prompt change can improve one case by making every response longer, which later damages latency or user experience. Compare paired outputs and supporting metrics. Test the smallest suspected component before rerunning the full matrix.

Turn confirmed production and evaluation defects into new goldens. Minimize each case while preserving the failure mechanism, attach a source or policy version, and mark the owning failure class. Over time, this converts subjective model review into a durable regression asset.

10. Scale Evaluating an LLM App with DeepEval Metrics

As the suite grows, create a metric registry that documents purpose, required fields, direction, threshold owner, calibration set, provider, expected cost class, and blocking policy. Reject cases missing required evidence before invoking a judge. A data validation failure is clearer and cheaper than a misleading score.

Separate end-to-end and component evaluation. End-to-end tests answer whether users receive acceptable results. Component evals locate retrieval, generation, agent planning, or tool selection defects. Use trace identifiers to connect them, but avoid comparing unlike scores. A generator can be faithful to poor retrieval while the end-to-end answer remains wrong.

Build dashboards around decisions and defects. Show pass rate with raw counts, critical failures, slice regressions, judge-human disagreement, infrastructure errors, latency, and evaluation spend. Provide drill-down to the exact case, output, evidence, and reason. A score without artifacts is not a test report.

Finally, establish ownership. Product and domain experts own acceptance meaning, QA owns coverage and measurement discipline, engineering owns component fixes, and risk stakeholders own critical policy. DeepEval supplies useful execution and metrics, but a sustainable evaluation program is a cross-functional quality system.

Interview Questions and Answers

Q: What is an LLMTestCase in DeepEval?

It represents one atomic LLM application interaction. input and actual_output are fundamental, while fields such as expected_output, context, retrieval_context, and tool data support particular metrics. The chosen boundary should match the component or end-to-end behavior being evaluated.

Q: What is the difference between context and retrieval_context?

context is curated reference material supplied as ground truth. retrieval_context is the material the application retriever actually returned at runtime. Faithfulness uses retrieval context to assess whether generated claims are grounded in what the model received.

Q: Why can answer relevancy pass when an answer is wrong?

Relevancy measures whether the response addresses the input, not whether every claim is true. A fabricated but directly responsive answer can score well. Pair relevancy with grounding, reference-based, or deterministic correctness checks.

Q: When would you use G-Eval?

Use G-Eval when the product has a semantic acceptance criterion not covered adequately by a built-in metric. Define observable criteria or evaluation steps, pass only relevant test-case fields, and calibrate results against human labels.

Q: How do you set a DeepEval threshold?

Label a representative calibration set, run the metric, study false passes and false failures, and choose a threshold based on risk. Revalidate it whenever the dataset, rubric, judge, or product policy materially changes.

Q: How do you reduce flaky LLM evaluation?

Use precise rubrics, stable inputs, controlled judge configuration, deterministic checks where possible, and a measured repeatability set. Route borderline or unstable cases to review rather than rerunning until a desired score appears.

Q: What belongs in a CI report for DeepEval?

Include failing case IDs, metric names, scores, thresholds, reasons, infrastructure errors, and links to inputs, outputs, retrieval, and version metadata. Separate evaluation execution failure from application quality failure.

Common Mistakes

• Adding many metrics before defining the product failure model.
• Passing a complete prompt transcript into input instead of the user input at the evaluated boundary.
• Using context and retrieval_context interchangeably.
• Assuming every metric passes when its score is above the threshold.
• Copying illustrative thresholds into a release gate without calibration.
• Asking G-Eval to judge exact schema, tool parameter, or permission rules.
• Changing the judge model and comparing its raw scores to the old trend as if the instrument were unchanged.
• Treating provider timeouts as low application quality.
• Evaluating only averages and ignoring critical cases or weak slices.
• Saving scores without the output, evidence, metric reason, and version identity.

The cure is measurement discipline: explicit scope, correct fields, limited metrics, calibrated decisions, and inspectable evidence.

Conclusion

Evaluating an LLM app with DeepEval metrics works when metrics express a known failure model. Build versioned cases, capture the actual interaction evidence, combine answer relevancy and faithfulness with one focused product rubric, and preserve deterministic gates for exact behavior.

Run the example against a small human-labeled set before connecting it to a release rule. The first valuable milestone is not a dashboard full of scores. It is one reliable failure signal that leads the team to the right defect and the right fix.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Evaluating an AI code reviewer (2026)
• Evaluating an AI test-case generator (2026)
• Load testing an LLM API (2026)
• Red teaming an LLM chatbot (2026)
• Testing an AI agent with LangSmith (2026)
• Testing an LLM agent memory (2026)