Measuring cost per test with LLMs (2026)

TL;DR

Attribute every paid model request, retry, tool, retrieval operation, and infrastructure unit to a completed test execution. Preserve raw usage, calculate with a versioned rate card, and gate cost only alongside quality, latency, and safety.

Key Takeaways

• Define one completed test execution before assigning model, tool, retrieval, and retry costs.
• Store raw usage with a versioned rate card so price changes do not require rerunning tests.
• Report total, median, p95, maximum, and retry rate instead of relying on one average.
• Use cost per passing or grounded case to prevent low-quality configurations from looking efficient.
• Compare matched cases with constant rates and minimum sample sizes before failing CI.
• Keep sensitive prompt content out of finance telemetry unless controlled debugging requires it.

Measuring cost per test with LLMs means assigning every executed evaluation case its share of model tokens, tool calls, retrieval, infrastructure, and retry cost, then relating that spend to test value. The useful number is not the provider invoice divided by an arbitrary test count. It is a reproducible per-case metric that can explain why one scenario, model, or prompt costs more than another.

This guide gives QA and SDET teams a practical cost model, a runnable Python collector for the OpenAI Responses API, aggregation rules, CI gates, and a quality-adjusted view. It treats cost as an engineering signal, not as a reason to select the cheapest model blindly.

TL;DR

The core formula is cost_per_test = attributable_cost / completed_test_executions. Keep raw usage beside the calculated currency value so a price change can be applied without rerunning the suite.

1. What Measuring Cost per Test With LLMs Actually Means

A conventional API test has a fairly stable compute profile. An LLM evaluation can vary because prompts have different lengths, responses may contain hidden reasoning tokens, retrieval can add context, agents can call tools, and a failed assertion may trigger a retry. Cost therefore belongs to the execution record, not only to the suite summary.

Define one test execution as one attempt of a named case against a declared system configuration. The identity should include the dataset case ID, application version, prompt version, model identifier, retrieval index version, and test runner build. If the runner repeats a case three times to estimate variance, record three executions. If infrastructure retries a request after a transport error, attach that attempt to the same logical execution and expose the retry cost separately.

Use two related metrics:

1. Attempt cost measures every paid request, including failed or retried calls.
2. Completed test cost assigns all attempts required to reach a final assertion result to that test execution.

This separation answers two different questions. Attempt cost explains provider consumption. Completed test cost explains the budget required to obtain a usable QA result. A team that reports only successful calls can hide money lost to timeouts, rate-limit handling, malformed output, or flaky orchestration.

Do not confuse test generation cost with test execution cost. Generating a test case, expected answer, or synthetic document is a dataset-building activity. Track it as evaluation development cost unless the generation happens on every run. The distinction keeps recurring CI expense comparable over time.

2. Build a Complete Cost Boundary Before Calculating

The most important design decision is the boundary around a test. Start at the runner accepting a case and end when it persists a final verdict. Every metered dependency inside that boundary is attributable. Shared services outside it can be allocated with a documented rule.

The table is a starting taxonomy, not a universal invoice. A RAG regression run might use generation, embeddings, reranking, and storage reads. A prompt unit test may use only one model response. A browser agent might add screenshots, computer actions, and a sandbox. Preserve component-level columns even when the current value is zero.

Avoid folding engineering salaries into the request-level metric. Human effort matters for total cost of quality, but mixing a monthly labor estimate into a CI execution number makes the latter noisy and politically difficult to use. Report recurring machine cost, human review cost, and one-time development cost as separate layers.

For the broader evaluation design that this metric supports, see measuring hallucination rate in LLM systems. Cost without a trustworthy quality definition encourages the wrong behavior.

3. Design an Execution Record That Survives Price Changes

A currency total is derived data. The durable evidence is usage plus configuration. Store an immutable record for each test execution, then calculate cost through a versioned price catalog. At minimum, capture:

• run_id, case_id, attempt, started_at, and duration_ms
• application, prompt, dataset, retrieval index, and model versions
• input, cached input, output, and total tokens
• tool names and billable quantities
• provider request ID when it is safe to retain
• assertion outcome, error class, and retry reason
• rate-card version and calculated component costs

Use decimal arithmetic for money. Binary floating-point is acceptable for a dashboard estimate, but it can accumulate rounding differences over millions of executions. Calculate at the smallest available usage unit and round only for display. Store currency explicitly because regional endpoints and providers may invoice differently.

Keep the model identifier returned by the provider, not only the alias requested by the client. An alias can move. A snapshot or resolved version helps explain a sudden token or behavior shift. Also store the prompt bytes or a content hash. Token counts can change when a model family uses a different tokenizer, so characters are a useful secondary diagnostic.

Never put full sensitive prompts into a cost event by default. Cost telemetry normally needs counts, hashes, tags, and IDs. If a failed case needs content for debugging, send it through the controlled trace-retention path described in monitoring LLM apps in production. This keeps inexpensive finance telemetry from becoming an accidental data store.

4. Version the Price Catalog and Allocation Rules

Provider prices are configuration, not constants in test code. Put rates in a dated catalog reviewed by the platform or FinOps owner. Each entry should identify provider, model, region or processing mode when relevant, unit, currency, effective time, and source. Never copy a price from an old ticket into a test assertion.

A simple catalog can be JSON or a database table. For text models, keep separate rates for uncached input, cached input, and output. For tools, retain the provider's billing unit, such as call, image, second, or token. For internal infrastructure, define a unit rate that the organization can reproduce. If no credible unit rate exists, report usage quantities rather than inventing a dollar value.

Use the rate active when the request occurred for finance reconciliation. Use a normalized comparison rate when evaluating engineering efficiency across months. These produce different but valid views:

• Actual cost answers, "What were we charged for this run?"
• Constant-rate cost answers, "Did our application become more or less efficient?"
• Forecast cost answers, "What will this suite cost at projected volume?"

Document allocation rules next to the catalog. Offline document embeddings, for example, could be charged to the release that created the index, amortized across expected queries, or excluded from per-query cost and shown as a separate fixed expense. Any rule can work if readers can see it and it remains stable. Changing the rule requires a new version and a backfill decision.

Finally, never place a mutable web price directly in a live CI calculation. Cache and approve catalog updates. Otherwise an external price update can fail an unchanged pull request, which makes the gate non-deterministic.

5. Implement Measuring Cost per Test With LLMs in Python

The following script makes one real OpenAI Responses API call, reads documented usage fields, calculates component cost from environment-supplied rates, and prints a JSON execution record. Install the current openai package, export OPENAI_API_KEY, set your approved model, and supply the current rates from your own catalog. Requiring rates as inputs prevents this article from freezing a price that may change.

import json
import os
import time
import uuid
from decimal import Decimal

from openai import OpenAI


def env_decimal(name: str) -> Decimal:
    value = os.environ.get(name)
    if value is None:
        raise RuntimeError(f"Set {name} to the approved USD rate per 1M tokens")
    return Decimal(value)


def token_cost(tokens: int, usd_per_million: Decimal) -> Decimal:
    return Decimal(tokens) * usd_per_million / Decimal(1_000_000)


client = OpenAI()
model = os.environ.get("OPENAI_MODEL", "gpt-5.6-luna")
case_id = "refund-policy-grounding-001"
started = time.perf_counter()

response = client.responses.create(
    model=model,
    instructions=(
        "Answer only from the supplied policy. If the policy does not answer, "
        "say NOT_FOUND. Keep the answer under 60 words."
    ),
    input=(
        "Policy: Refund requests are accepted within 30 calendar days of delivery.\n"
        "Question: Can a customer request a refund 20 days after delivery?"
    ),
)

duration_ms = round((time.perf_counter() - started) * 1000)
usage = response.usage
details = usage.input_tokens_details
cached_tokens = details.cached_tokens
uncached_tokens = usage.input_tokens - cached_tokens

input_cost = token_cost(uncached_tokens, env_decimal("INPUT_USD_PER_M"))
cached_cost = token_cost(cached_tokens, env_decimal("CACHED_INPUT_USD_PER_M"))
output_cost = token_cost(usage.output_tokens, env_decimal("OUTPUT_USD_PER_M"))
total_cost = input_cost + cached_cost + output_cost

record = {
    "run_id": str(uuid.uuid4()),
    "case_id": case_id,
    "attempt": 1,
    "model_requested": model,
    "response_id": response.id,
    "duration_ms": duration_ms,
    "input_tokens": usage.input_tokens,
    "cached_input_tokens": cached_tokens,
    "output_tokens": usage.output_tokens,
    "total_tokens": usage.total_tokens,
    "input_cost_usd": str(input_cost),
    "cached_input_cost_usd": str(cached_cost),
    "output_cost_usd": str(output_cost),
    "total_cost_usd": str(total_cost),
    "passed": "30" in response.output_text,
}

print(json.dumps(record, indent=2))

Run it with values from the approved rate card:

export OPENAI_MODEL="gpt-5.6-luna"
export INPUT_USD_PER_M="<current-input-rate>"
export CACHED_INPUT_USD_PER_M="<current-cached-input-rate>"
export OUTPUT_USD_PER_M="<current-output-rate>"
python measure_case.py

The placeholders deliberately force catalog ownership. The program uses response.output_text, response.usage.input_tokens, output_tokens, total_tokens, and input_tokens_details.cached_tokens, all current Responses API fields. In production, add error handling that writes an attempt record even when no response object is returned.

6. Aggregate Distributions Instead of Trusting One Average

Mean cost alone hides expensive tails. LLM output length can be skewed, agent tool loops can create outliers, and a small number of retries may dominate the bill. For each stable segment, publish count, total, median, p90 or p95, maximum, retry rate, and token components. Use the mean for budget forecasting, the median for a typical case, and p95 for tail control.

Never mix unlike workloads in one headline. Segment at least by suite, model, prompt version, scenario class, and success status. A twenty-page contract review should not make a short classification test look inefficient. Conversely, a large number of cheap smoke cases must not hide a costly agent scenario.

This dependency-free Python example summarizes decimal cost strings exported by the collector:

import json
import statistics
import sys
from decimal import Decimal


def percentile(values: list[Decimal], fraction: Decimal) -> Decimal:
    ordered = sorted(values)
    if not ordered:
        raise ValueError("No completed test costs")
    index = int((len(ordered) - 1) * fraction)
    return ordered[index]


records = [json.loads(line) for line in sys.stdin if line.strip()]
costs = [Decimal(row["total_cost_usd"]) for row in records]

summary = {
    "completed_tests": len(costs),
    "total_cost_usd": str(sum(costs, Decimal("0"))),
    "mean_cost_usd": str(sum(costs, Decimal("0")) / len(costs)),
    "median_cost_usd": str(statistics.median(costs)),
    "p95_cost_usd": str(percentile(costs, Decimal("0.95"))),
    "max_cost_usd": str(max(costs)),
}
print(json.dumps(summary, indent=2))

For a regression decision, compare matched cases against the previous baseline. A paired difference reduces noise because each new execution is compared with the same dataset case under the old configuration. When stochastic variation is material, run repeated samples and show confidence intervals or bootstrap intervals. Do not claim a regression from two unrelated small runs.

7. Add Quality-Adjusted Cost and Cost per Useful Result

The cheapest response is worthless if it fails the business requirement. Pair monetary cost with a quality gate, then calculate cost per useful result. A simple metric is:

cost_per_pass = total_suite_cost / number_of_cases_meeting_all_quality_gates

Suppose configuration A costs less per raw execution but frequently fails grounding. Configuration B costs more per request but passes more cases. Cost per passing case may favor B. This is why procurement-style model comparisons can mislead an engineering team.

Define "useful" before viewing results. It could require exact task accuracy, citation correctness, groundedness, schema validity, safety, and latency below an SLO. Use a conjunctive release gate for critical requirements. Avoid rolling everything into one opaque weighted score, because a high style score could compensate for an unsafe answer.

Useful companion metrics include:

• cost per grounded answer
• cost per correctly completed tool workflow
• cost per accepted human review
• incremental cost per one percentage point of quality gained
• marginal cost of enabling a reranker, larger model, or verification pass

For a RAG system, break cost and quality down by retrieval and generation. The end-to-end RAG chatbot testing guide explains how to isolate retrieval misses from generation failures. If retrieval supplies the wrong evidence, spending more on the generator can increase confidence without improving correctness.

When presenting a model choice, plot or table quality against mean and p95 cost. Identify Pareto-efficient configurations, those for which no alternative is both cheaper and better. The final selection still depends on risk and latency, but the tradeoff becomes explicit.

8. Set Cost Budgets and CI Regression Gates

A useful CI gate protects against accidental growth while allowing intentional experiments. Start with a versioned baseline produced from the same fixed dataset, region, processing mode, and concurrency policy. Compare constant-rate cost so an external catalog update does not masquerade as a code regression.

Use several guardrails:

1. a hard per-case maximum catches runaway prompts and tool loops;
2. a suite total cap protects the CI budget;
3. a paired median or mean change detects broad inflation;
4. a p95 gate protects the tail;
5. a retry-rate gate catches orchestration waste;
6. quality gates prevent cost-only optimization.

Thresholds should reflect normal variance and business impact. An illustrative policy might warn when the matched-case mean rises more than a chosen percentage and fail only when both the relative and absolute increases exceed agreed limits. The absolute floor prevents a tiny baseline from creating noisy percentage alarms. These numbers must come from your data, not from a generic article.

Make baseline updates reviewable. The pull request should show old and new totals, token deltas, quality deltas, changed prompt bytes, and the reason for approval. Store the accepted baseline with prompt, model, dataset, and rate-card versions. Never let CI silently overwrite it.

Separate PR smoke economics from nightly coverage. A pull request can run a stratified low-cost subset with strict per-case caps. A scheduled run can estimate distributions across the full corpus. Production shadow tests need their own budget because they may include real traffic patterns and observability retention.

9. Compare Approaches to Measuring Cost per Test With LLMs

Teams often begin with invoice division because it is easy. Maturity comes from moving toward attributable, versioned execution records without building an accounting platform before the first useful dashboard.

Use provider-reported usage as the source of truth for a successful request. A local tokenizer is valuable for early rejection and forecast checks, but it should not replace returned usage in cost reconciliation. For a transport failure where no usage is returned, mark cost as unknown until provider logs or billing exports resolve it. Zero is not a safe assumption.

Trace-based allocation is worth the effort when one logical test fans out into query rewriting, retrieval, reranking, generation, moderation, and verification. Give every component span the same run_id and case_id. The test record can then sum metered events without guessing from timestamps.

Choose the simplest approach that preserves the decision you need to make. A prompt regression suite can start with response usage records. An autonomous support agent needs trace-level tool and loop attribution. Both should retain raw units and a versioned catalog.

10. Turn the Metric Into Engineering Decisions

A dashboard is useful only when it names the lever that changed. Show cost beside input length, cached share, output length, tool count, retries, latency, and quality. Then an engineer can distinguish a longer system prompt from a model change or a retrieval fan-out increase.

Review the largest contributors first. Common levers include removing duplicated context, limiting retrieved chunks, caching stable prompt prefixes, selecting a smaller approved model for simple cases, reducing unnecessary verification calls, bounding agent steps, and shortening verbose outputs. Apply one change at a time and rerun matched cases. The prompt engineering cheat sheet for QA can help make prompts more precise without weakening assertions.

Forecast using traffic classes, not one blended mean. Multiply the mean completed-test cost for each scenario class by expected executions, then add fixed indexing and observability costs. Include a sensitivity range for output length and retry rate. This gives an owner a defensible low, expected, and high plan without pretending model behavior is deterministic.

Establish ownership. QA owns case semantics and release gates. Platform engineering owns telemetry completeness and rate-card integration. Product owns traffic forecasts and acceptable quality. FinOps or finance validates reconciliation. No single team should both define quality and approve a cost-driven reduction without visibility.

Finally, retain a compact evidence trail for every optimization: hypothesis, baseline, code or prompt change, usage delta, quality delta, and decision. That history is more valuable than a screenshot because it teaches the organization which savings generalize and which merely shifted failure elsewhere.

Interview Questions and Answers

Q: How would you define cost per test for an LLM evaluation suite?

I define it as the sum of attributable metered costs for all attempts needed to produce one completed test verdict. The numerator can include generation, cached input, embeddings, tools, retrieval, and allocated compute. I keep raw usage and the rate-card version so the currency value is auditable. I also report cost per passing case because raw execution cost can reward low-quality output.

Q: Why is average cost per test insufficient?

LLM costs usually have a skewed distribution because output lengths, retries, and agent tool loops vary. The mean is useful for forecasting, but it can hide the typical case and expensive tail. I report total, median, p95, maximum, and retry rate by stable workload segment. I investigate outliers rather than deleting them without a documented rule.

Q: Would you estimate tokens locally or trust API usage?

I use local estimation for preflight budgets and prompt-size checks. For completed API calls, provider-reported usage is the reconciliation source because it reflects the actual request and model accounting. I store both when useful and alert on large differences. If a failed request has no returned usage, I record unknown cost rather than assuming zero.

Q: How do cached tokens affect the calculation?

Cached input must be separated from uncached input because the rate can differ. I calculate uncached tokens as total input tokens minus cached tokens, then apply each catalog rate independently. I keep the cache-hit share as an optimization metric. Tests should not assume a cache hit unless the response usage confirms it.

Q: How would you add a cost gate to CI without creating flaky builds?

I use a fixed, versioned dataset and matched cases against a reviewed baseline. The gate requires a minimum sample size and both a relative and absolute regression, while p95 and hard per-case caps catch tails. I calculate with a constant comparison rate and run quality gates in parallel. Baseline changes require review and never happen automatically.

Q: What is quality-adjusted cost?

It relates spend to an acceptable result, for example total suite cost divided by cases that pass groundedness, correctness, safety, and latency gates. It prevents a cheap but inaccurate model from appearing efficient. I prefer transparent component gates to one weighted score. The definition of acceptable is agreed before comparing configurations.

Q: How would you allocate offline RAG indexing cost?

I would first report it as a separate fixed release cost. If the business needs a per-query total, I would amortize it using a documented expected query volume and show that assumption. I would version the allocation rule and avoid mixing it silently into online generation cost. This keeps operational comparisons reproducible.

Common Mistakes

• Dividing the whole provider bill by the test count. The invoice can include development, staging, production, and unrelated teams. Attribute usage with execution IDs.
• Counting only passing requests. Retries and failed attempts consume budget. Attach them to the logical test and expose their cost.
• Hardcoding prices in assertions. Rates change. Use an approved, effective-dated catalog and retain raw units.
• Using one blended average. Segment by model, suite, scenario, version, and outcome. Publish tail metrics.
• Ignoring cached input. Treat cached and uncached tokens as distinct quantities and apply the appropriate catalog entries.
• Optimizing cost without a quality floor. Pair every financial gate with correctness, grounding, safety, and latency requirements.
• Treating unknown cost as zero. A timeout may still have consumed provider resources. Preserve an unknown state until reconciled.
• Auto-updating baselines. This normalizes regressions. Require an evidence-rich review that shows both cost and quality deltas.
• Storing sensitive prompt text in finance events. Use hashes and identifiers in low-retention cost telemetry, with controlled traces for debugging.

Conclusion

Measuring cost per test with LLMs is an attribution problem before it is a calculation problem. Define the execution boundary, capture provider usage and every metered dependency, apply a versioned rate card, and report distributions plus cost per useful result. That gives QA a metric that is auditable, comparable, and hard to game.

Start with one stable evaluation suite and the execution record in this guide. Baseline its median, p95, retry rate, and quality-adjusted cost. Then add a reviewed CI gate and optimize the largest measured component, one controlled change at a time.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• AI test data generation with Faker and LLMs (2026)
• Automating Jira to test case with n8n (2026)
• Building a test case generator with LangChain (2026)
• Connecting Claude to your test suite with MCP (2026)
• Generating test cases from a PRD with AI (2026)
• Managing test cases in Jira with Xray (2026)