Measuring LLM latency and cost in tests (2026)

TL;DR

Instrument the complete model call with a monotonic clock, capture time to first token for streams, read actual usage fields, and calculate cost from a versioned rate card. Compare percentiles and cost per successful task on matched workloads, with retries, failures, caching, and quality outcomes visible.

Key Takeaways

• Measure time to first token, end-to-end duration, inter-token gaps, and errors because one average cannot describe user experience.
• Use a monotonic clock around the client boundary and preserve provider usage fields with every observation.
• Keep model rates in reviewed configuration rather than hardcoding prices in tests or dashboards.
• Report latency percentiles by prompt size, output size, route, tool path, and concurrency, including failed and throttled requests.
• Calculate cost per successful task and per quality-qualified task, not only cost per API call.
• Separate cold starts, warm requests, streaming, retries, and cached input so comparisons remain fair.
• Gate releases on statistically and operationally meaningful regressions using a stable baseline environment.

Measuring LLM latency and cost in tests requires more than wrapping one request with a timer. A defensible test captures the user-visible latency boundary, time to first token for streaming, token usage, retries, errors, model and prompt versions, workload shape, and a dated rate configuration. It then reports distributions and cost per useful outcome.

LLM performance varies with input length, requested output, model class, reasoning settings, tools, retrieval, concurrency, cache state, service load, and network path. This guide builds a reproducible Python measurement harness around the current OpenAI Responses API, while keeping the model and prices configurable so the same method remains valid when offerings change.

TL;DR

Use a monotonic clock, warm up intentionally, retain raw samples, calculate percentiles from the unrounded data, and never copy an old price into source code as if it were permanent.

1. Define the Boundary for Measuring LLM Latency and Cost in Tests

Name the system boundary before collecting a number. A direct provider test may begin immediately before the SDK call and end after the response is parsed. An application test may include authentication, retrieval, prompt construction, tool calls, model inference, moderation, post-processing, persistence, and the network between the user and your service. Both measurements are useful, but they answer different questions.

Use client wall-clock time for user-visible performance. Provider-reported server timing, when available, can help diagnosis but normally excludes part of the client path. Capture both without substituting one for the other. Use time.perf_counter_ns() or time.perf_counter() in Python because it is monotonic and suitable for measuring durations. Wall-clock timestamps are useful for correlation, not elapsed-time arithmetic.

Define success as well. A 200 response with empty text, malformed JSON, a refusal on an allowed request, or an incorrect tool call may not be a successful task. Cost per request can look excellent while users repeatedly retry. Track transport success, contract success, and quality-qualified success separately.

Finally, define the workload. Record input characters or tokens, expected output range, streaming mode, tools, model configuration, region or endpoint, and concurrency. A comparison is fair only when the baseline and candidate receive matched cases under comparable conditions.

2. Understand the Latency Timeline

For a non-streaming request, end-to-end duration is the primary observable because the user receives nothing until completion. For a streaming request, TTFT describes how quickly useful output begins, while total duration describes when the task finishes. Inter-token gaps reveal a stream that starts quickly but pauses uncomfortably.

The timeline may include DNS, connection establishment, TLS, queueing, prompt processing, model generation, tool execution, and response transfer. Connection reuse can make the first request slower than later calls. A model or local runtime may also have a cold load. Record warm-up policy rather than deleting inconvenient first samples.

Report percentiles such as p50, p90, p95, and p99 when sample size supports them. The mean is sensitive to outliers and says little about the tail. A percentile still needs context: p95 from 20 samples is a coarse order statistic, and percentiles from different workload mixes are not directly comparable.

For streams, define "first token" operationally. SDK events may include metadata, reasoning, tool calls, or empty deltas before user-visible text. If the interface displays only text, stop TTFT on the first non-empty output text delta. If a tool-status indicator is displayed earlier, measure a separate time to first visible event.

3. Capture Non-Streaming Latency and Usage

The current OpenAI Python library uses client.responses.create() for the Responses API. The code below makes the model an environment variable, measures the SDK boundary, reads the documented usage object, and returns a structured observation. It does not contain a model price or assume that usage is always present.

from dataclasses import asdict, dataclass
from datetime import datetime, timezone
import json
import os
from pathlib import Path
from time import perf_counter_ns
from openai import OpenAI


@dataclass(frozen=True)
class Observation:
    timestamp: str
    case_id: str
    model: str
    ok: bool
    latency_ms: float
    input_tokens: int
    output_tokens: int
    response_id: str


client = OpenAI(timeout=120.0, max_retries=0)


def measure(case_id: str, prompt: str) -> Observation:
    model = os.environ["OPENAI_MODEL"]
    started = perf_counter_ns()
    response = client.responses.create(model=model, input=prompt)
    latency_ms = (perf_counter_ns() - started) / 1_000_000
    usage = response.usage
    return Observation(
        timestamp=datetime.now(timezone.utc).isoformat(),
        case_id=case_id,
        model=response.model,
        ok=bool(response.output_text.strip()),
        latency_ms=latency_ms,
        input_tokens=usage.input_tokens if usage else 0,
        output_tokens=usage.output_tokens if usage else 0,
        response_id=response.id,
    )


sample = measure("short-summary-001", "Summarize: Quality is a team responsibility.")
with Path("llm-observations.jsonl").open("a", encoding="utf-8") as output:
    output.write(json.dumps(asdict(sample)) + "\n")
print(sample)

Install with python -m pip install openai. Set OPENAI_API_KEY and OPENAI_MODEL in the environment. Keep retries disabled in the measurement client so the observation represents one attempt. If the application uses retries, instrument each attempt and the total logical request separately.

Do not store raw private prompts or outputs in a general performance artifact. Case IDs and workload features are usually sufficient. Restricted diagnostic output can use a separate retention and access policy.

4. Measure TTFT With Streaming Events

The OpenAI Python SDK provides a Responses streaming context manager. Iterate events, stop the TTFT clock on the first non-empty response.output_text.delta, and call get_final_response() to obtain final usage. The code also records gaps between visible deltas.

from dataclasses import dataclass
import os
from time import perf_counter_ns
from openai import OpenAI


@dataclass(frozen=True)
class StreamTiming:
    ttft_ms: float
    total_ms: float
    max_gap_ms: float
    input_tokens: int
    output_tokens: int


def measure_stream(prompt: str) -> StreamTiming:
    client = OpenAI(timeout=120.0, max_retries=0)
    started = perf_counter_ns()
    first_text_at: int | None = None
    previous_text_at: int | None = None
    gaps_ms: list[float] = []

    with client.responses.stream(
        model=os.environ["OPENAI_MODEL"],
        input=prompt,
    ) as stream:
        for event in stream:
            if event.type == "response.output_text.delta" and event.delta:
                now = perf_counter_ns()
                if first_text_at is None:
                    first_text_at = now
                if previous_text_at is not None:
                    gaps_ms.append((now - previous_text_at) / 1_000_000)
                previous_text_at = now
        response = stream.get_final_response()

    finished = perf_counter_ns()
    if first_text_at is None:
        raise AssertionError("Stream completed without visible output text")
    usage = response.usage
    return StreamTiming(
        ttft_ms=(first_text_at - started) / 1_000_000,
        total_ms=(finished - started) / 1_000_000,
        max_gap_ms=max(gaps_ms, default=0.0),
        input_tokens=usage.input_tokens if usage else 0,
        output_tokens=usage.output_tokens if usage else 0,
    )


print(measure_stream("Explain boundary value analysis in three bullets."))

Consuming the stream without printing keeps terminal overhead out of the timing. In a browser test, separately measure when the UI renders its first visible text, because application buffering can erase the benefit of provider streaming. Capture stream errors and incomplete responses as failures, not as impressively short totals.

5. Calculate Cost From a Versioned Rate Card

Provider prices, model names, cached-token rules, batch rates, and reasoning-token treatment can change. Put rates in a reviewed configuration that includes provider, model, effective date, currency, billing unit, and source URL. The test should fail if the observed model has no active rate rather than silently applying a similar model's price.

For a simple model with separate input and output prices per million tokens, the arithmetic is deterministic. Use Decimal for currency calculations and retain more precision than the final display.

from dataclasses import dataclass
from decimal import Decimal
import os


MILLION = Decimal("1000000")


@dataclass(frozen=True)
class Rates:
    input_usd_per_million: Decimal
    output_usd_per_million: Decimal


def rates_from_environment() -> Rates:
    return Rates(
        input_usd_per_million=Decimal(os.environ["INPUT_USD_PER_MILLION"]),
        output_usd_per_million=Decimal(os.environ["OUTPUT_USD_PER_MILLION"]),
    )


def estimated_cost(input_tokens: int, output_tokens: int, rates: Rates) -> Decimal:
    input_cost = Decimal(input_tokens) * rates.input_usd_per_million / MILLION
    output_cost = Decimal(output_tokens) * rates.output_usd_per_million / MILLION
    return input_cost + output_cost


rates = rates_from_environment()
print(estimated_cost(input_tokens=1250, output_tokens=340, rates=rates))

The token counts above are illustrative inputs to the function, not a pricing claim. Your billing model may include cached input, uncached input, audio, image, tool, storage, or other categories. Extend the rate schema to match the provider's current invoice semantics and reconcile estimates against actual billing exports.

Keep currency conversion separate and date-stamped. A stable USD estimate should not appear to drift because a dashboard applies today's foreign-exchange rate to last month's usage.

6. Design a Representative Performance Test

Build cases by workload factors rather than random prose. Useful factors include input-token bands, expected output-token bands, task type, number of retrieved passages, number and type of tools, structured-output schema complexity, language, image presence, and conversation length. Preserve the production distribution for a capacity forecast, and maintain worst-case suites for limits.

Warm-up must be explicit. A direct hosted API test may warm DNS, TLS, and connection pools. A local server may load model weights and allocate memory. Run and label cold-start tests separately. For steady-state testing, perform a documented warm-up that is excluded from summary statistics but retained in raw observations.

Control output length where the product permits it. If candidate A generates twice as many tokens as candidate B, its total latency and cost are not a pure inference-speed comparison. Report quality and length together rather than truncating output merely to improve performance.

Use realistic arrival patterns. Closed-loop load generators wait for a response before sending the next request and may understate queue growth. Open arrival-rate tests better represent independent users, but they require careful capacity controls. Increase load gradually, monitor throttle and error responses, and stop before a test harms a shared service.

Run from a stable location and record client host, network, region, SDK version, connection policy, concurrency, and service settings. One laptop Wi-Fi run is a debugging observation, not a release benchmark.

7. Summarize Percentiles Without Hiding Failures

Retain one record per attempt. Summary code should operate on unrounded values and state its percentile method. Python's standard statistics.quantiles has method choices that can surprise teams with small samples, so a simple nearest-rank function is often easier to explain for service reports.

from math import ceil
from statistics import mean


def nearest_rank(values: list[float], percentile: float) -> float:
    if not values:
        raise ValueError("values must not be empty")
    if not 0 < percentile <= 100:
        raise ValueError("percentile must be in (0, 100]")
    ordered = sorted(values)
    rank = ceil(percentile / 100 * len(ordered))
    return ordered[rank - 1]


def summary(latencies_ms: list[float]) -> dict[str, float]:
    return {
        "count": float(len(latencies_ms)),
        "mean_ms": mean(latencies_ms),
        "p50_ms": nearest_rank(latencies_ms, 50),
        "p95_ms": nearest_rank(latencies_ms, 95),
        "p99_ms": nearest_rank(latencies_ms, 99),
    }

Do not calculate latency percentiles only from successful responses and omit the failure count. Report attempted requests, completed responses, timeouts, throttles, server errors, parse failures, empty outputs, and quality-qualified successes. Depending on the product, a timeout may be represented as a deadline value in a user-experience distribution, but the report must explain that convention.

Break summaries down by important slices. An overall p95 can improve simply because the test mix contains more short prompts. A matched, paired case comparison is more sensitive for regressions. Plotting latency against input and output tokens also reveals whether the candidate changed scaling behavior.

8. Set CI Gates for Measuring LLM Latency and Cost in Tests

Performance gates need stable baselines and tolerance for natural variation. Avoid asserting that every live request completes under a tight duration in a shared environment. That produces flaky pipelines and encourages blind retries. Use a dedicated test environment for release gates, a sufficient repeated sample, and a comparison against a recent approved baseline.

Define absolute objectives from the user experience and relative regression limits from measurement variability. A policy might require no increase beyond an agreed p95 tolerance, no new timeout class, and an estimated cost-per-success within budget. Set the actual values from your service objectives and empirical runs. This article does not invent universal thresholds.

Keep quality in the gate. A candidate can reduce cost by returning shorter but incomplete answers. Join each performance observation to deterministic and semantic results, then compute cost per contract-valid and quality-qualified response. The evaluation design in LLM evaluation interview questions for QA explains why dimensions should remain visible even when a combined decision is needed.

Use a two-stage pipeline. A small smoke benchmark can detect obvious changes on a pull request. A larger controlled benchmark can run before release or on schedule. If a gate fails, preserve raw records, compare workload composition, identify the affected slices, and rerun only according to a predefined confirmation policy.

9. Diagnose Latency and Cost Regressions

Start by confirming that the comparison is matched. Check model identifier, prompt and system instructions, tool definitions, retrieval count, output limit, streaming mode, reasoning settings, SDK, endpoint, region, connection reuse, concurrency, and dataset version. A changed prompt that adds thousands of input tokens is not an unexplained provider slowdown.

Decompose the application trace. Measure retrieval, reranking, prompt assembly, provider TTFT, generation, tool calls, validation, and UI rendering. If provider TTFT is stable but end-to-end time rises, the fault is elsewhere. If output tokens increase, inspect verbosity and stop conditions. If request count rises, inspect retries, agent loops, and fallback routes.

Cost regressions often come from more than model price. Look for duplicated context, larger conversation histories, lower cache hits, repeated tool loops, parse retries, failed responses, and users retrying low-quality answers. Normalize cost by successful business outcome.

For RAG applications, retrieval changes can alter both prompt length and answer quality. Evaluate them together using measuring RAG faithfulness and relevancy. For private local inference, no per-token provider invoice may exist, but the latency methodology still applies. The local LLM setup for private test data covers warm load and hardware recording.

Create a short regression report with evidence, suspected phase, affected slice, business impact, and a controlled experiment. Do not blame the model until the trace supports that conclusion.

10. Monitor Production and Reconcile Estimates

Synthetic tests cannot reproduce every production route, payload, network, or service load. Emit safe production telemetry for total latency, TTFT where measurable, token categories, model and prompt versions, status, tool count, retry count, and quality proxy. Do not log raw sensitive content merely to diagnose cost.

Use histograms with suitable buckets or a metrics system that can aggregate distributions across instances. Averaging precomputed p95 values from multiple hosts does not produce a global p95. Preserve consistent units and avoid high-cardinality labels such as user ID or prompt text.

Set alerts on sustained objectives, not isolated slow calls. Segment by route and workload so a change in traffic mix does not masquerade as a regression. Track budget burn and forecast using recent volume, but show uncertainty for launches and seasonal traffic.

Reconcile the estimator with provider billing or internal chargeback. Differences may reveal missing token categories, retries counted in one system, cache discounts, asynchronous jobs, rounding, taxes, or reporting windows. Version the rate card when a price changes and do not retroactively rewrite historical estimates without preserving the prior basis.

Close the loop by adding representative production shapes to the safe benchmark. If users increasingly submit long documents or invoke three tools, the test mix must evolve. Keep a locked core for regression comparability and a rotating sample for distribution drift.

Interview Questions and Answers

Q: Why is average LLM latency insufficient?

The mean can hide a slow tail and changes with workload mix. I report percentiles, errors, and timeouts by input size, output size, route, and concurrency. For streams, I separate TTFT from total duration because they describe different parts of the user experience.

Q: How do you measure time to first token?

I start a monotonic timer immediately before the request and stop it on the first non-empty output text delta that the user could see. I also measure the UI render boundary when testing the application. Metadata events do not count unless the product displays them.

Q: Why disable retries in a benchmark client?

It makes each observation correspond to one attempt and exposes transient failures. If the production application retries, I instrument each attempt plus the total logical request. Otherwise retries can hide reliability problems while increasing latency and cost.

Q: How do you calculate LLM request cost?

I multiply actual usage categories by a versioned, effective-dated rate card and preserve full precision. The schema reflects current provider billing, including cached or special token categories when applicable. I reconcile estimates against invoices rather than assuming simple input and output arithmetic is always complete.

Q: How would you performance-test a streaming chatbot?

I measure TTFT, total duration, inter-token gaps, completion rate, and user-visible render timing under representative prompt and concurrency shapes. I label cold and warm behavior, preserve failures, and monitor output tokens and quality. Load increases gradually with service safeguards.

Q: What is cost per successful task?

It is total inference cost divided by responses that meet the defined success contract, not merely requests returning HTTP 200. A stricter version uses quality-qualified outcomes. It exposes cheap calls that create retries, escalations, or unusable answers.

Q: How do you make an LLM performance gate less flaky?

I use a controlled environment, matched cases, adequate samples, explicit warm-up, and a recent baseline. The gate compares distributions with empirically justified tolerance and treats errors separately. Tight per-request assertions against a shared live API are reserved for generous functional deadlines, not microbenchmarks.

Q: What would you inspect after a cost regression?

I inspect model and rate versions, input and output tokens, duplicated context, conversation growth, cache behavior, retries, agent loops, tool calls, failures, and workload mix. I then compare cost per quality-qualified outcome. The provider price is only one possible cause.

Common Mistakes

• Timing with a wall clock that can jump instead of a monotonic performance clock.
• Measuring only total duration for a streaming interface and ignoring TTFT or gaps.
• Reporting an average without percentiles, failures, workload shape, or sample size.
• Comparing cold baseline calls with warm candidate calls.
• Letting SDK retries hide attempts, throttles, added latency, and added cost.
• Hardcoding a price from a blog post and applying it to every model or billing category.
• Counting every HTTP 200 response as a successful user task.
• Improving cost by truncating useful output without measuring quality.
• Averaging percentiles from separate workers as if they formed one global distribution.
• Logging private prompts in a general performance report.

Conclusion

Measuring LLM latency and cost in tests is an instrumentation and experiment-design problem. Define the boundary and success criteria, use monotonic timing, capture TTFT and total duration, preserve actual usage, apply a dated rate card, and compare matched workload distributions with failures visible.

Implement the small observation harness first, then add representative slices, streaming timing, quality qualification, CI baselines, and production reconciliation. The result is a performance signal that can support release and budget decisions instead of a single stopwatch number with no context.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Run tests in headed mode in Cypress (2026)
• How to Run tests in headed mode in Playwright (2026)
• How to Run tests in headed mode in Selenium (2026)
• How to Skip and group tests in Cypress (2026)
• How to Skip and group tests in Playwright (2026)
• How to Skip and group tests in Selenium (2026)