Measure AI Agent Task Completion Rate

TL;DR

Measure completion by executing independent checks against the task's required final state. Divide strictly completed trials by all valid trials, then report the sample size, confidence interval, partial-credit score, and failure-stage distribution beside the rate.

Key Takeaways

• Define completion as verified final state, not a confident final message.
• Use deterministic checks for artifacts, side effects, constraints, and tool evidence.
• Keep strict completion rate separate from partial-credit quality scores.
• Run repeated trials because the same agent can produce different outcomes.
• Label failure stages so the rate leads to an actionable engineering fix.
• Report sample size and a confidence interval with every completion rate.
• Gate releases on representative task suites, not one happy-path prompt.

To measure AI agent task completion rate, define an independently verifiable success contract for each task, run representative tasks repeatedly, and divide fully successful trials by all valid trials. Do not treat a polished final answer, a stopped run, or a claimed success as proof of completion.

This tutorial builds a small evaluation harness that checks artifacts and side effects, records failure stages, calculates strict and weighted metrics, and enforces a CI release gate. Use it alongside the AI Agent Testing Complete Guide for 2026 when you need the broader strategy for planning, tool use, memory, safety, and production monitoring.

The examples use Python 3.12 and pytest, but the measurement design works with any agent framework. Replace the sample adapter with your own agent call while keeping the evaluator independent from the system under test.

What You Will Build

By the end, you will have a local evaluation project that can:

• Describe task completion with machine-checkable criteria.
• Run a deterministic sample agent through success and failure cases.
• Calculate strict task completion rate and a weighted quality score.
• Estimate a Wilson confidence interval without third-party statistics packages.
• Classify failures as planning, execution, verification, or infrastructure issues.
• Fail a CI job when reliability falls below an explicit threshold.

The final report separates three questions that teams often mix together: Did the agent finish? How much of the task did it satisfy? Where did an unsuccessful run break? Keeping those answers separate prevents an attractive average score from hiding incomplete tasks.

Prerequisites

Use Python 3.12 or newer and pytest 8 or newer. The code uses standard-library dataclasses, pathlib, statistics, and JSON support. Create an isolated project with these commands:

mkdir agent-completion-eval
cd agent-completion-eval
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install 'pytest>=8,<9'
mkdir -p tests eval_workspace

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1. Confirm the environment before adding code:

python --version
python -m pytest --version

You should see Python 3.12 or newer and pytest 8.x. Pin exact versions in your own lock file after validating them in your environment. The bounded range above avoids claiming that an unreleased version is required.

You do not need an API key for the tutorial because the sample agent is deterministic. That makes the metric logic easy to test. In production, inject credentials through your CI secret store and never write prompts, tokens, or customer data into the result file.

Step 1: Define AI Agent Completion Criteria

Start with the expected final state. A task is complete only when all required criteria pass. Each criterion gets a stable ID, a weight for partial-credit diagnosis, and a function that inspects evidence outside the agent's narration.

Create evaluation.py:

from __future__ import annotations

import json
import math
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Sequence

@dataclass(frozen=True)
class Criterion:
    id: str
    weight: float
    check: Callable[[Path], bool]

@dataclass(frozen=True)
class TaskCase:
    id: str
    prompt: str
    criteria: Sequence[Criterion]

@dataclass(frozen=True)
class TrialResult:
    task_id: str
    trial: int
    completed: bool
    score: float
    failed_criteria: tuple[str, ...]
    failure_stage: str | None
    latency_ms: int

def file_exists(name: str) -> Callable[[Path], bool]:
    return lambda workspace: (workspace / name).is_file()

def json_field_equals(
    name: str, field: str, expected: object
) -> Callable[[Path], bool]:
    def check(workspace: Path) -> bool:
        try:
            value = json.loads((workspace / name).read_text(encoding="utf-8"))
        except (OSError, json.JSONDecodeError):
            return False
        return value.get(field) == expected
    return check

This contract checks observable output. It does not ask the agent whether it succeeded. In a browser agent, a criterion might query the test database for an order ID. In a coding agent, it might run tests and inspect the patch. In a support agent, it might verify both the ticket state and the approved response policy.

Use binary checks for must-have requirements. Weights help diagnose near misses, but a high weighted score must not convert an incomplete task into a completed one.

Verify Step 1: Run python -m py_compile evaluation.py. The command should exit with no output and status code 0. If it reports a syntax error, confirm that the entire block was copied and that Python 3.12 is active.

Step 2: Create Representative Tasks and an Agent Adapter

Add a task that asks the agent to create a JSON order artifact. Then provide a deterministic adapter with three modes. Your production adapter can call an SDK, an HTTP endpoint, or a queued workflow, but it should return control only after the run reaches a terminal state.

Append to evaluation.py:

ORDER_TASK = TaskCase(
    id="create-order",
    prompt="Create order.json with status confirmed and total 49.99",
    criteria=(
        Criterion("artifact_exists", 0.20, file_exists("order.json")),
        Criterion(
            "status_confirmed", 0.40,
            json_field_equals("order.json", "status", "confirmed")
        ),
        Criterion(
            "total_correct", 0.40,
            json_field_equals("order.json", "total", 49.99)
        ),
    ),
)

def run_sample_agent(prompt: str, workspace: Path, mode: str) -> None:
    del prompt
    if mode == "success":
        payload = {"status": "confirmed", "total": 49.99}
        (workspace / "order.json").write_text(
            json.dumps(payload), encoding="utf-8"
        )
    elif mode == "partial":
        payload = {"status": "draft", "total": 49.99}
        (workspace / "order.json").write_text(
            json.dumps(payload), encoding="utf-8"
        )
    elif mode == "tool_error":
        raise ConnectionError("simulated tool outage")
    else:
        raise ValueError(f"unknown mode: {mode}")

Representative coverage matters more than a large collection of trivial prompts. Include normal work, ambiguous inputs, dependency failures, policy constraints, and boundary values. Freeze task wording during a release comparison. If prompts change between builds, the rates no longer describe the same benchmark.

Keep the adapter thin. It should capture agent and tool traces, but the evaluator should not depend on the model's self-reported success field. If the agent can influence both the work and the judge, it can accidentally reward its own misunderstanding.

Verify Step 2: Run python -c 'from evaluation import ORDER_TASK; print(len(ORDER_TASK.criteria))'. The expected output is 3. This proves the module loads and all three completion checks are registered.

Step 3: Execute and Independently Judge Every Trial

Now run the adapter in a clean workspace, evaluate every criterion, and preserve an explicit failure stage. Infrastructure failures stay in the data instead of silently disappearing from the denominator.

Append this function:

def evaluate_trial(
    task: TaskCase, trial: int, workspace: Path, mode: str
) -> TrialResult:
    workspace.mkdir(parents=True, exist_ok=True)
    for item in workspace.iterdir():
        if item.is_file():
            item.unlink()

    started = time.perf_counter()
    failure_stage: str | None = None
    try:
        run_sample_agent(task.prompt, workspace, mode)
    except (ConnectionError, TimeoutError):
        failure_stage = "infrastructure"

    passed: list[Criterion] = []
    failed: list[str] = []
    for criterion in task.criteria:
        try:
            ok = criterion.check(workspace)
        except Exception:
            ok = False
        if ok:
            passed.append(criterion)
        else:
            failed.append(criterion.id)

    total_weight = sum(item.weight for item in task.criteria)
    score = sum(item.weight for item in passed) / total_weight
    completed = not failed and failure_stage is None
    if not completed and failure_stage is None:
        failure_stage = "verification"

    latency_ms = round((time.perf_counter() - started) * 1000)
    return TrialResult(
        task_id=task.id,
        trial=trial,
        completed=completed,
        score=score,
        failed_criteria=tuple(failed),
        failure_stage=failure_stage,
        latency_ms=latency_ms,
    )

Catching Exception around each criterion is deliberate because a broken judge must produce visible failure evidence. In a real harness, also log the exception type. Do not catch BaseException, which would swallow interrupts and process-exit signals.

A fresh workspace prevents one trial from inheriting another trial's successful artifact. For remote side effects, generate a unique correlation ID per trial and clean up through a supported API. Never point an evaluator at production records.

Verify Step 3: Run this command:

python -c 'from pathlib import Path; from evaluation import *; print(evaluate_trial(ORDER_TASK, 1, Path("eval_workspace"), "partial"))'

The result should contain completed=False, score=0.6, failed_criteria=('status_confirmed',), and failure_stage='verification'. Latency varies by machine.

Step 4: Measure AI Agent Task Completion Rate

Strict task completion rate is the count of fully completed trials divided by the count of valid trials. Report it with sample size and a confidence interval. A result of 9 out of 10 and 900 out of 1,000 can both display 90 percent, but they provide very different certainty.

Append the aggregation code:

@dataclass(frozen=True)
class Summary:
    trials: int
    completed: int
    completion_rate: float
    mean_score: float
    ci95_low: float
    ci95_high: float

def wilson_interval(successes: int, trials: int) -> tuple[float, float]:
    if trials <= 0:
        raise ValueError("trials must be positive")
    z = 1.959963984540054
    proportion = successes / trials
    denominator = 1 + (z * z / trials)
    center = (proportion + z * z / (2 * trials)) / denominator
    margin = (
        z
        * math.sqrt(
            proportion * (1 - proportion) / trials
            + z * z / (4 * trials * trials)
        )
        / denominator
    )
    return max(0.0, center - margin), min(1.0, center + margin)

def summarize(results: Sequence[TrialResult]) -> Summary:
    if not results:
        raise ValueError("at least one result is required")
    completed = sum(result.completed for result in results)
    low, high = wilson_interval(completed, len(results))
    return Summary(
        trials=len(results),
        completed=completed,
        completion_rate=completed / len(results),
        mean_score=sum(result.score for result in results) / len(results),
        ci95_low=low,
        ci95_high=high,
    )

The Wilson interval behaves better than the simple plus-or-minus formula near zero or one and with smaller samples. Treat 95 percent as the chosen reporting convention, not a magical guarantee. Trials should be independent enough for the interval to be informative. Shared outages, cached responses, and repeated seeds can create correlated results.

Do not average criterion scores and call the result completion rate. A task that satisfies two of three required outcomes is incomplete even when its score is high.

Verify Step 4: Execute python -c 'from evaluation import wilson_interval; print(wilson_interval(9, 10))'. You should get values close to (0.596, 0.982). Minor final-digit differences are normal.

Step 5: Repeat Runs and Segment the Results

One run per prompt cannot characterize a stochastic agent. Repeat trials across controlled seeds or fresh sessions, then segment results by task family, risk, model configuration, and failure stage. Do not pool unlike tasks into a number that hides a critical workflow.

Create run_evaluation.py:

from dataclasses import asdict
import json
from pathlib import Path

from evaluation import ORDER_TASK, evaluate_trial, summarize

modes = [
    "success", "success", "partial", "success", "tool_error",
    "success", "partial", "success", "success", "success",
]
results = [
    evaluate_trial(
        ORDER_TASK,
        trial=index,
        workspace=Path("eval_workspace") / str(index),
        mode=mode,
    )
    for index, mode in enumerate(modes, start=1)
]
summary = summarize(results)
report = {
    "summary": asdict(summary),
    "results": [asdict(result) for result in results],
}
Path("report.json").write_text(
    json.dumps(report, indent=2), encoding="utf-8"
)
print(
    f"completed={summary.completed}/{summary.trials} "
    f"rate={summary.completion_rate:.1%} "
    f"mean_score={summary.mean_score:.1%}"
)

These modes are illustrative fixtures, not benchmark claims. Seven trials pass, two create partial output, and one simulates an outage. Because the outage was observed after the evaluation began, this simple policy counts it as an unsuccessful trial. Your organization may separately mark confirmed evaluator outages as invalid, but define that rule before seeing results and publish both invalid and valid counts.

For production, run enough repetitions to narrow the interval to a useful decision range. There is no universal sample size. High-risk money movement needs stronger evidence than a low-risk drafting assistant. Stop based on a predeclared sample plan, not when the graph looks favorable.

Verify Step 5: Run python run_evaluation.py. The console should print completed=7/10 rate=70.0% mean_score=82.0%, and report.json should contain ten detailed results. The confidence interval will be wide because ten trials provide limited evidence.

Step 6: Diagnose Why Completion Failed

A rate is a signal, not a root cause. Add stage labels using trace evidence from the agent runtime. Use a small, mutually understandable taxonomy:

• planning: the trace omitted a necessary subgoal or selected an invalid sequence.
• execution: the plan was reasonable, but a tool call, argument, or action was wrong.
• verification: the agent stopped without checking the final state, or the independent checks failed.
• policy: the agent correctly refused a disallowed task. Report this separately from capability failure.
• infrastructure: an external dependency, evaluator, or runtime failed.

The sample harness assigns verification when output checks fail and infrastructure for simulated dependency errors. In a real system, store sanitized tool-call status, terminal reason, retry count, and criterion evidence. Then map the earliest causal failure, not every downstream symptom.

Planning failures deserve their own targeted suite. Follow the step-by-step AI agent planning failure tutorial to test decomposition, ordering, and replanning. If traces show repeated actions or exhausted budgets, apply the AI agent loop termination testing guide instead of merely increasing timeouts.

Review samples from both successful and unsuccessful buckets. False positives are especially dangerous because they inflate the headline metric. Periodically have two human reviewers independently label a stratified sample, resolve disagreements, and update checks only through versioned changes. Keep prior reports tied to the evaluator version that produced them.

Verify Step 6: Open report.json and search for failure_stage. You should find two verification entries, one infrastructure entry, and seven null values. Confirm that the counts add up to all ten trials.

Step 7: Add Automated Tests and a CI Quality Gate

Test the evaluator before trusting it to judge the agent. Add unit tests for complete, partial, and infrastructure outcomes plus known aggregation values.

Create tests/test_evaluation.py:

from pathlib import Path

import pytest

from evaluation import ORDER_TASK, evaluate_trial, summarize

def test_complete_trial(tmp_path: Path) -> None:
    result = evaluate_trial(ORDER_TASK, 1, tmp_path, "success")
    assert result.completed is True
    assert result.score == pytest.approx(1.0)
    assert result.failed_criteria == ()

def test_partial_trial_is_not_complete(tmp_path: Path) -> None:
    result = evaluate_trial(ORDER_TASK, 1, tmp_path, "partial")
    assert result.completed is False
    assert result.score == pytest.approx(0.6)
    assert result.failed_criteria == ("status_confirmed",)

def test_infrastructure_failure_is_visible(tmp_path: Path) -> None:
    result = evaluate_trial(ORDER_TASK, 1, tmp_path, "tool_error")
    assert result.completed is False
    assert result.failure_stage == "infrastructure"

def test_summary_uses_strict_completion(tmp_path: Path) -> None:
    results = [
        evaluate_trial(ORDER_TASK, 1, tmp_path / "a", "success"),
        evaluate_trial(ORDER_TASK, 2, tmp_path / "b", "partial"),
    ]
    summary = summarize(results)
    assert summary.completion_rate == pytest.approx(0.5)
    assert summary.mean_score == pytest.approx(0.8)

Then create quality_gate.py:

import json
from pathlib import Path

report = json.loads(Path("report.json").read_text(encoding="utf-8"))
summary = report["summary"]
minimum_rate = 0.70
minimum_trials = 10

if summary["trials"] < minimum_trials:
    raise SystemExit("FAIL: insufficient evaluation trials")
if summary["completion_rate"] < minimum_rate:
    raise SystemExit("FAIL: completion rate below release threshold")
print("PASS: AI agent completion quality gate")

Choose thresholds from product risk and a known baseline. A lower confidence bound is a stricter gate than the observed rate, but it may require many more trials. Also consider a no-regression rule for critical task families and a maximum infrastructure-failure rate. Never tune the threshold after viewing a candidate build. For a production workflow beyond this minimal gate, follow the guide to adding CI to a test framework.

Verify Step 7: Run python -m pytest -q, then python run_evaluation.py && python quality_gate.py. Pytest should report four passing tests, and the gate should print PASS. Change minimum_rate to 0.71 temporarily and confirm that the gate fails, then restore 0.70.

How to Measure AI Agent Task Completion Rate in Production

Offline evaluation gives reproducible release evidence. Production measurement shows real traffic, changing dependencies, and user behavior. Use the same versioned completion criteria where possible, but add privacy controls and delayed outcome checks. Some tasks are only verifiable after a payment settles, a user approves a draft, or a downstream system processes an event. Apply a test data strategy for safe, representative coverage when those checks depend on accounts, records, or regulated data.

Define the production denominator explicitly. A useful formula is verified completed tasks / eligible terminal tasks. Track started, abandoned, refused, invalid, timed out, and still-pending tasks separately. Do not remove failures just because they are inconvenient. Correct policy refusals should be visible and should not be interpreted as capability defects.

For each metric point, retain these dimensions without storing sensitive content:

• Agent, model, prompt, tool, and evaluator versions.
• Task family and risk tier.
• Terminal reason and completion-criterion IDs.
• Retry and human-intervention indicators.
• Latency, token usage, and tool-call counts.
• Experiment cohort and deployment region when relevant.

Alert on sustained changes, not isolated noise. Compare matched task mixes because a harder traffic mix can reduce the aggregate rate even when the agent itself has not regressed. Investigate segment-level changes before rolling back. For multi-agent systems, the overall task may fail even when each specialist looks healthy, so evaluate the protocol with the multi-agent handoff quality tutorial.

Troubleshooting

The agent says it succeeded, but the evaluator fails -> Trust independently observed state. Inspect criterion evidence, timestamps, and the target environment. Update a check only if the expected contract was genuinely wrong.

Completion rate changes dramatically between runs -> Increase repeated trials, control task mix, record seeds and configuration, and inspect shared dependency failures. Report the confidence interval instead of comparing rounded percentages alone.

Old artifacts make failed trials pass -> Allocate an isolated workspace and unique external records for every trial. Add setup and cleanup assertions, and fail closed when cleanup is incomplete.

The weighted score is high while completion rate is low -> Keep both metrics. The pattern means agents often get close but miss at least one required criterion. Rank failed criterion IDs to find the common blocker.

The judge is flaky or model-based -> Prefer deterministic state checks for hard requirements. If semantic judgment is unavoidable, pin the judge configuration, use a rubric, test agreement against human labels, and repeat disputed cases.

CI fails because a vendor was unavailable -> Preserve the infrastructure label and retry according to a predeclared policy. Publish outage counts. Do not silently delete failed trials after seeing the candidate's score.

Where To Go Next

Use the complete AI agent testing guide to place completion rate inside a full test strategy. Then deepen the failure-specific suites that your report identifies:

• Test missing goals and invalid action order with the AI agent planning failure workflow.
• Validate delegation context, acceptance, and ownership with the multi-agent handoff evaluation guide.
• Catch repeated actions, premature stopping, and runaway retries with the agent loop termination tutorial.

Your next practical improvement should be a second task family with different side effects. Add a read-only research task, a state-changing workflow, or a recovery scenario. Keep completion contracts versioned, rerun the baseline, and compare matched suites before making a release decision.

Interview Questions and Answers

Q: What is AI agent task completion rate?

It is the proportion of eligible evaluated trials in which every required completion criterion passes. A credible report includes the numerator, denominator, task mix, evaluator version, and uncertainty interval.

Q: Why is an agent's final response not sufficient evidence?

The response expresses the agent's belief, not the actual state of files, databases, APIs, or user constraints. Independent checks can detect missing side effects, incorrect values, and premature stopping.

Q: How is completion rate different from a weighted score?

Completion rate is binary per trial and answers whether all mandatory outcomes passed. A weighted score gives partial credit and helps diagnose near misses, but it must not redefine an incomplete task as successful.

Q: What belongs in the denominator?

Include all eligible terminal trials under the policy declared before execution. Report pending, invalid, policy-refused, and infrastructure-failed runs separately, and explain whether each category is included.

Q: Why run the same task multiple times?

Models and external tools can produce variable outcomes. Repetition estimates that variability and supports an uncertainty interval, while one pass only proves that a single path worked once.

Q: How do you prevent evaluator leakage?

Keep hidden checks and expected values outside the agent context, isolate the evaluator from agent-written status, and version all criteria. Periodically compare automated labels with independent human reviews.

Q: How should a team set a release threshold?

Base it on task risk, an established baseline, sample size, and the cost of failure. Declare it before evaluating the candidate and add critical-family gates so an aggregate score cannot hide a severe regression.

Common Mistakes

• Counting done messages instead of validating final state.
• Treating a partial-credit average as strict completion rate.
• Using only easy happy-path prompts in the benchmark.
• Comparing releases that used different tasks or evaluator versions.
• Omitting timeouts and infrastructure failures without a documented rule.
• Reusing workspaces, accounts, or records across trials.
• Reporting a percentage without its numerator, denominator, and uncertainty.
• Letting the agent see hidden expected values or modify judge output.
• Increasing timeouts or retries until a weak candidate passes.

Conclusion

To measure AI agent task completion rate reliably, begin with observable completion contracts, independently judge isolated trials, and count only trials that satisfy every mandatory criterion. Report strict completion, weighted diagnostic score, sample size, confidence interval, task mix, and failure stages together.

Run the harness in CI and production with versioned criteria. Start by replacing run_sample_agent with your real adapter, preserve the deterministic tests, and add one representative task family at a time.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• AI Agent Testing Complete Guide (2026)
• Test AI Agent Loop Termination: A Practical Tutorial
• Test AI Agent Planning Failures Step by Step
• Testing an AI agent with LangSmith (2026)
• Testing tool calling in an AI agent (2026)
• AI assisted API test generation (2026)