Load testing an LLM API (2026)

TL;DR

Load testing an LLM API requires token-aware workloads, not just requests per second. Use realistic prompt and completion budgets, an arrival-rate scenario, separate streaming and non-streaming measurements, and correlate latency, errors, token throughput, quality, cost, and backend saturation to find safe capacity.

Key Takeaways

• Model request arrivals and token sizes separately because equal request rates can create very different inference demand.
• Measure queue time, time to first token, end-to-end latency, inter-token behavior, token throughput, errors, and quality together.
• Use an open arrival model for offered load and track dropped iterations so the load generator does not hide overload.
• Create workload classes from privacy-safe production distributions instead of repeating one short prompt.
• Run baseline, load, spike, soak, and recovery scenarios against an authorized environment with explicit cost and safety limits.
• Correlate client metrics with gateway, scheduler, accelerator, cache, and dependency telemetry before naming a bottleneck.
• Set release thresholds from service objectives and observed baselines, not from generic benchmark numbers.

Load testing an LLM API requires more than sending concurrent POST requests and reading average response time. LLM inference cost and latency depend heavily on input tokens, generated tokens, batching, queueing, cache behavior, model choice, and streaming, so the workload must model those variables explicitly.

The goal is not a flashy maximum request rate. It is a defensible operating envelope: the traffic mix a service can sustain while meeting latency, availability, quality, and cost objectives, plus evidence of how it degrades and recovers. This guide shows a practical k6 workflow and the QA reasoning needed around it.

TL;DR

Start small, cap spend and duration, and obtain explicit authorization. Never aim an uncontrolled performance test at a shared public endpoint or production users.

1. Define Success Before Load Testing an LLM API

Write a performance charter that names the endpoint, model and configuration, deployment, tenant, expected traffic mix, service objectives, data restrictions, budget, abort conditions, monitoring owners, and recovery expectation. Without this boundary, a test can consume costly capacity or produce results that do not answer a product question.

LLM API performance testing and AI inference capacity testing share this charter, but the latter usually adds scheduler, accelerator, batching, and model-worker evidence.

Separate objectives. A baseline test measures one request at a time and exposes inherent latency. A load test verifies expected traffic. A stress test finds degradation and the breaking region. A spike test examines sudden arrival changes. A soak test looks for leaks, fragmentation, cache drift, throttling, and slow dependency failure. A failover or recovery test examines behavior during and after disruption.

Define the unit of capacity. Requests per second is useful at the gateway, but inference demand is better described with input tokens per time, output tokens per time, active sequences, model, and length distribution. A service processing ten short classifications is not doing the same work as ten long document summaries.

Success also includes answer validity. Under pressure, a system might truncate output, omit required JSON fields, skip retrieval, choose a fallback model, or return a cached answer incorrectly. Add a small set of deterministic contract checks and offline quality samples. For functional API foundations, review API performance testing basics before introducing model-specific metrics.

2. Model Token-Aware Workloads

Create workload classes from privacy-safe production telemetry or a reviewed forecast. Useful dimensions include route, model, input token bucket, requested output limit, streaming flag, retrieval use, tool availability, cache eligibility, tenant class, and conversation length. Store representative synthetic prompts for each class rather than replaying sensitive user text.

Tokenize with the exact tokenizer used by the model or service when possible. Character count is only a rough proxy and varies by language, code, and formatting. Record the actual usage values returned by the endpoint if available, while remembering that providers may define cached or reasoning tokens separately.

Preserve distributions and correlations. Long inputs may request shorter outputs, one route may stream while another returns JSON, and premium tenants may use a different model. Randomizing each field independently can create unrealistic combinations. A weighted scenario file can select a prebuilt class with valid relationships.

Test at least small, typical, large, and boundary token sizes. Boundary cases include the application's maximum accepted context, maximum output budget, and combined context rules. Do not fill long prompts with meaningless repeated characters if the inference stack compresses, tokenizes, or caches them differently from realistic text. Generate approved synthetic documents with stable seeds.

3. Choose an Open or Closed Load Model

In a closed model, a fixed number of virtual users start a new iteration after the previous one completes. When the service slows, those users submit fewer requests. This can hide overload because offered traffic automatically falls with latency. Closed models are still useful for modeling a fixed population that truly waits before sending again.

In an open model, iterations start at a target arrival rate independent of response time, subject to available load-generator virtual users. This is often a better representation for API traffic arriving from many callers. Grafana k6 provides constant-arrival-rate and ramping-arrival-rate executors for this purpose.

Watch dropped_iterations. If k6 cannot allocate enough virtual users to start scheduled iterations, the generator did not deliver the intended load. A chart showing low server error rates is misleading if a material portion of arrivals were dropped locally. Increase preallocated or maximum virtual users carefully, or distribute generators after confirming their own CPU, memory, network, and file descriptor headroom.

Run the generator close enough to avoid irrelevant internet noise but not on the same constrained host as the service. Synchronize clocks for trace correlation. Tag requests by workload class and endpoint, not by high-cardinality prompt, user, or conversation ID.

4. Measure LLM Latency Correctly

End-to-end latency starts before the client sends the request and ends after the complete response is received. For non-streaming calls, it approximates the user's wait. Break it into connection, TLS, gateway, queue, prompt processing, token generation, transfer, and client parsing when instrumentation allows.

For streaming, time to first token is the interval from request start until the client receives the first meaningful generated-token event. HTTP time to first byte may instead capture headers, a heartbeat, or an empty server-sent event. Validate the protocol and parser before naming a metric TTFT. Time to first meaningful content may be the more relevant user measure.

Time to first token testing must therefore parse the deployed protocol and reject empty control events as content.

Track inter-token latency or stream gaps for conversational applications. An acceptable first token followed by long pauses still feels broken. Record stream completion, malformed events, duplicated chunks, missing termination markers, and client cancellations. End-to-end latency remains relevant because downstream parsing or tool execution waits for completion.

Report percentiles and the sample count for each workload class. The mean hides queues and tail behavior. Avoid combining short and long prompts into one percentile without segmentation. Compare warm and cold conditions, cache hit and miss, and steady-state windows separately.

5. Measure Throughput, Errors, Quality, and Cost

Define output token throughput precisely. Per-request generation rate is output tokens divided by the generation interval for that request. Aggregate throughput is total generated output tokens per wall-clock second across the service. These answer different questions. If only total response time is available, label the derived rate accordingly because prompt processing and queue time are included.

A tokens per second benchmark is meaningful only when its tokenizer, interval, workload, model, and aggregation method are stated.

Segment errors into connection failures, client timeouts, cancellations, 400 contract errors, 401 or 403 authorization failures, 429 throttling, 5xx service errors, malformed streams, invalid schemas, and incomplete outputs. Expected throttling may protect the system, but it still needs a caller retry contract and product objective.

Measure quality with a controlled sample rather than grading every high-volume response synchronously. Deterministic checks can validate JSON, required citations, non-empty content, language, and maximum length. Offline evaluation can assess relevance, groundedness, or task success on sampled traces. A low-latency fallback that violates product quality is not a successful capacity strategy.

Estimate cost or resource consumption using actual metering and infrastructure telemetry. Do not assume public list prices or a universal tokens-to-hardware conversion. Track input, cached input, output, and any provider-specific usage categories returned by the system. Set explicit test budgets and alerts before execution.

6. Build a Runnable k6 LLM Load Test

This script targets an authorized OpenAI-compatible chat endpoint supplied through environment variables. It uses documented k6 HTTP, check, custom metric, and constant-arrival-rate APIs. The example measures full response duration, output usage when returned, and invalid response bodies. It does not claim to measure true streaming TTFT.

import http from 'k6/http';
import { check } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const apiUrl = __ENV.LLM_API_URL;
const apiKey = __ENV.LLM_API_KEY;
const model = __ENV.LLM_MODEL;
const targetRate = Number(__ENV.RATE || '2');
const maxP95Ms = Number(__ENV.MAX_P95_MS || '12000');

if (!apiUrl || !apiKey || !model) {
  throw new Error('Set LLM_API_URL, LLM_API_KEY, and LLM_MODEL');
}

const responseDuration = new Trend('llm_response_duration_ms', true);
const outputTokens = new Trend('llm_output_tokens');
const invalidResponse = new Rate('llm_invalid_response');

export const options = {
  scenarios: {
    typical_chat: {
      executor: 'constant-arrival-rate',
      rate: targetRate,
      timeUnit: '1s',
      duration: __ENV.DURATION || '1m',
      preAllocatedVUs: Number(__ENV.PRE_ALLOCATED_VUS || '10'),
      maxVUs: Number(__ENV.MAX_VUS || '50'),
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    llm_invalid_response: ['rate<0.01'],
    llm_response_duration_ms: ['p(95)<' + maxP95Ms],
    dropped_iterations: ['count==0'],
  },
};

const prompts = [
  'Summarize three risks of retrying a non-idempotent API request.',
  'Return four test ideas for a paginated search endpoint.',
  'Explain the difference between authentication and authorization.',
];

export default function () {
  const prompt = prompts[Math.floor(Math.random() * prompts.length)];
  const payload = JSON.stringify({
    model: model,
    messages: [{ role: 'user', content: prompt }],
    max_tokens: Number(__ENV.MAX_OUTPUT_TOKENS || '160'),
    stream: false,
  });

  const response = http.post(apiUrl, payload, {
    headers: {
      Authorization: 'Bearer ' + apiKey,
      'Content-Type': 'application/json',
    },
    tags: { name: 'llm_chat_completion', workload: 'typical_chat' },
    timeout: __ENV.REQUEST_TIMEOUT || '60s',
  });

  responseDuration.add(response.timings.duration);
  let valid = false;
  if (response.status === 200) {
    try {
      const body = response.json();
      valid = Boolean(body.choices && body.choices[0] && body.choices[0].message);
      if (body.usage && Number.isFinite(body.usage.completion_tokens)) {
        outputTokens.add(body.usage.completion_tokens);
      }
    } catch (error) {
      valid = false;
    }
  }
  invalidResponse.add(!valid);
  check(response, {
    'status is 200': (result) => result.status === 200,
    'response contract is valid': () => valid,
  });
}

Run it with environment values, for example k6 run -e LLM_API_URL=https://authorized.example/v1/chat/completions -e LLM_API_KEY=... -e LLM_MODEL=... llm-load.js. The threshold values are illustrative configuration placeholders, not universal objectives. Set them from the service's approved SLO and baseline.

7. Design Baseline, Load, Spike, and Soak Scenarios

Begin with connectivity and one-request functional checks. Then run a single-worker baseline for each workload class to understand minimum observed latency and token behavior. Warm up deliberately and label the warm-up period rather than mixing it into steady-state results.

For the expected-load test, hold the forecast mix long enough to reach stable queue, cache, accelerator, and autoscaling behavior. Ramp arrival rate in controlled steps for capacity discovery. Each step should last long enough for the system to settle, and every run should use the same dataset and configuration when comparing builds.

A spike test increases arrivals faster than normal autoscaling can react. Verify admission control, clear 429 or overload responses, bounded queues, caller retry guidance, and recovery after the spike. A stress test continues beyond the target to identify the knee where latency accelerates or errors rise. Stop before safety or budget limits are exceeded.

Treat the LLM API stress test as a coordinated reliability experiment, with named stop authority and recovery verification.

Soak tests use expected or moderately elevated load for longer periods. Watch memory, connection pools, context caches, scheduler queues, log volume, storage, rate-limit counters, and output drift. Use a duration justified by the suspected failure mode. "Overnight" is not automatically meaningful if the relevant resource cycles hourly or weekly.

8. Test Streaming and Cancellation Separately

Do not infer streaming performance from a non-streaming k6 call. Build a protocol-aware client that parses server-sent events or the service's actual streaming format, timestamps the first content token and subsequent chunks, and verifies the termination event. The tool must not buffer the full response before exposing data to the measurement code.

Streaming LLM performance needs its own scenario and metric pipeline because buffered HTTP timing cannot describe the user-visible token cadence.

Test client cancellation after connection, after first token, and mid-generation. Verify the gateway and inference scheduler release work promptly where the architecture supports cancellation. Otherwise, disconnected clients may continue consuming expensive generation capacity. Correlate cancellation IDs through the stack.

Simulate slow consumers carefully. A client that reads slowly can fill buffers and create backpressure. Check server memory, stream timeouts, proxy limits, and whether one slow stream affects unrelated callers. Verify heartbeat behavior but do not count a heartbeat as generated content.

Streaming correctness tests should detect malformed event boundaries, duplicate tokens, missing chunks, invalid Unicode, reordered events, unexpected metadata, and premature completion. If tools are streamed, validate partial arguments are not executed before the complete, schema-valid tool call is assembled.

9. Correlate Client Results with System Telemetry

Client metrics tell you what users experienced, not why. Correlate test windows and request IDs with gateway rate limits, authentication latency, retrieval calls, prompt construction, scheduler queue depth, batch size, accelerator utilization, memory, cache hit rate, model worker errors, tool dependencies, and autoscaling events.

Look for causal timing. Rising queue wait with stable generation time suggests admission or capacity pressure. Stable gateway time with slower retrieval points elsewhere. High accelerator utilization alone does not prove a bottleneck because batching can intentionally run devices near saturation. Pair utilization with queue, throughput, latency, and error evidence.

Compare by model and workload class. A single aggregate dashboard can hide one long-context route starving short interactive traffic. Test workload isolation and priorities explicitly. If the architecture uses separate pools, confirm routing and fallback behavior under saturation.

Monitor the load generator too: CPU, memory, network, open sockets, DNS, and dropped iterations. Distributed generators need synchronized configuration and consolidated tags. If the generator saturates first, report the run as invalid instead of presenting its throughput as service capacity.

10. Analyze Capacity Without Overclaiming

Plot offered rate, completed rate, token throughput, latency percentiles, errors, throttles, dropped iterations, queue depth, and resource use on the same timeline. The safe operating point is below the region where objectives become unstable, not the final successful request before collapse.

Compare only controlled configurations. Record deployment revision, model identifier and quantization where applicable, hardware pool, replica count, batching settings, context limit, prompt template, retrieval configuration, cache state, test dataset, generator version, region, and time window. A benchmark without this context is difficult to reproduce.

Use multiple runs to understand variance, but do not average away distinct incidents. Report median and range across comparable runs, plus anomalies with evidence. Capacity planning should include headroom for traffic variance, failover, noisy neighbors, model updates, and longer-than-forecast outputs.

Be careful with public comparisons. Different tokenizers, output stopping rules, prompt lengths, hardware, batching, and quality levels make a simple requests-per-second ranking misleading. The most useful benchmark reflects your product workload and objectives.

11. Operationalize Load Testing an LLM API

Keep a tiny performance smoke check in CI for severe regressions, then run controlled load tests in a dedicated pipeline or environment. Functional pipelines are poor places for expensive, noisy stress workloads. Use scheduled or release-triggered runs with budget approval, exclusive environment rules, and automatic teardown.

Store workload definitions and thresholds as code. Keep secret values in the CI secret store, never in scripts or output. Export results to an approved metrics backend or versioned artifact. Add model and deployment metadata so trends remain interpretable.

Set abort conditions for spend, error rate, latency, queue size, infrastructure health, and impact on shared users. Coordinate with operations before stress and failover tests. A test that threatens unrelated tenants is an incident, not good performance engineering.

After each material model, prompt, retrieval, scheduler, accelerator, or gateway change, rerun the relevant baseline and target-load suite. Pair performance evidence with LLM quality evaluation metrics and API reliability test design. Capacity improvements that reduce groundedness or contract compliance need product review, not an automatic celebration.

Interview Questions and Answers

Q: Why is requests per second insufficient for LLM load testing?

Inference demand depends on input and output tokens, model, cache, batching, retrieval, and streaming. Two requests can differ by orders of magnitude in work even when they share an endpoint. I report request rate with token distributions, active sequences, and token throughput.

Q: What is the difference between TTFB and TTFT?

TTFB is when the HTTP client receives the first byte. TTFT is when it receives the first meaningful generated token. Headers, heartbeats, or empty events can make TTFB earlier, so I validate the stream parser before calling a metric TTFT.

Q: Why use a constant-arrival-rate executor?

It maintains offered arrivals independently of service response time, which resembles many API workloads. A closed virtual-user model submits fewer requests as latency rises and can conceal overload. I also track dropped iterations to prove the generator delivered the target.

Q: How do you choose LLM performance test data?

I derive privacy-safe workload classes from production or forecast distributions, preserving relationships among model, token sizes, route, streaming, and retrieval. I use reviewed synthetic prompts with stable seeds and include small, typical, large, and boundary cases.

Q: Which errors do you separate in the report?

I separate network failures, client timeouts, cancellations, contract 4xx, authentication or authorization, throttling, service 5xx, malformed streams, schema failures, and incomplete output. Their causes and user handling are different, so one error rate is insufficient.

Q: How do you identify the bottleneck?

I correlate client latency and throughput with gateway, retrieval, scheduler queue, batch, accelerator, memory, cache, dependency, and autoscaling telemetry. I look for aligned changes and run a controlled experiment before assigning cause. Client timing alone cannot identify the saturated component.

Q: How do you keep an LLM load test safe?

I use an authorized environment, synthetic data, least-privilege credentials, fixed spend and duration limits, monitored abort conditions, and operations coordination. I start with low load and verify generator and service telemetry before increasing arrivals.

Common Mistakes

• Repeating one short prompt and calling the result representative capacity.
• Reporting average response time without token-size segmentation or percentiles.
• Calling HTTP TTFB time to first token without parsing the actual stream.
• Using fixed virtual users when the objective requires a stable offered arrival rate.
• Ignoring dropped iterations and accidentally benchmarking the load generator.
• Measuring speed while omitting malformed, truncated, ungrounded, or fallback output.
• Stressing a shared provider or production environment without explicit authorization and budget controls.
• Combining warm-up, autoscaling, cache fill, and steady state into one aggregate result.
• Naming a backend bottleneck from client metrics without correlated service telemetry.

Conclusion

Load testing an LLM API is a token-aware capacity exercise. Model realistic arrivals and prompt distributions, measure user latency and generation behavior separately, validate output under pressure, and correlate the client view with the complete inference stack.

Start with a one-request baseline and three representative workload classes. Then run an authorized constant-arrival-rate test at expected load, confirm zero generator drops, and identify the first objective that degrades. That evidence gives engineering a useful next optimization target and product teams a safer capacity boundary.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Testing an LLM agent memory (2026)
• Testing an LLM router (2026)
• Testing structured output from an LLM (2026)
• API contract testing with Pact: A Practical Guide (2026)
• API performance testing tutorial (2026)
• API security testing basics: A Practical Guide (2026)