Testing an LLM router (2026)

TL;DR

Testing an LLM router requires three distinct checks: deterministic policy eligibility, routing decision quality, and end-to-end outcome quality. Use labeled prompts, capability constraints, failure injection, load tests, and shadow comparisons to prove the router chooses a valid path and degrades safely.

Key Takeaways

• Define route eligibility and service objectives before measuring model-selection accuracy.
• Test hard policy gates separately from probabilistic intent or complexity classification.
• Use a versioned golden dataset with ambiguous, adversarial, multilingual, long-context, and no-route cases.
• Verify capability, privacy, region, tenant, budget, and tool requirements before optimizing cost.
• Inject provider errors, timeouts, throttling, malformed responses, and partial streams to test fallback semantics.
• Measure end-to-end task quality, latency, cost, fallback rate, and safety by route rather than route accuracy alone.
• Log decision factors and policy versions without storing raw sensitive prompts by default.

Testing an LLM router means proving that every request reaches a model or workflow that is allowed, capable, available, and appropriate for the task. Route accuracy alone is not enough. A router can match a human label yet violate data residency, omit a required tool, exceed a latency budget, or send high-risk work to a model that was never approved for it.

A production test strategy separates deterministic policy gates from probabilistic classification and then evaluates the quality of the completed task. This guide shows how to define route contracts, build a golden dataset, inject provider failures, test cost and latency controls, and create auditable release gates for semantic routers and AI gateways.

TL;DR

Test request -> features -> eligible routes -> selection -> provider execution -> validation -> fallback -> outcome as an observable chain. A routing decision passes only when hard constraints hold and the resulting task meets its quality, latency, safety, and budget objective.

1. Why testing an LLM router is more than classification

An LLM router may select a small or large model, a regional provider, a tool-enabled agent, a retrieval workflow, a specialist model, a cached response, or a refusal path. Some routers use rules, some embeddings, some a classifier model, and many combine all three. The risk does not come only from choosing the wrong intent label. It comes from violating a constraint anywhere in that decision.

Model the harms first. An underpowered route can produce a wrong answer or invalid structured output. An oversized route can create needless cost and latency. A provider chosen without the required region or retention controls can create a compliance incident. A route lacking vision, long context, or function calling can fail mechanically. A retry to a second provider can duplicate a side effect if request semantics are unclear. An attacker may also manipulate text features to force a privileged or costly route.

Write invariants that apply before scoring. A restricted tenant never uses an unapproved provider. Requests with images never reach text-only routes. Tool-required work never reaches a route without the declared tool capability. The route selected at execution time must still be healthy and enabled. A fallback cannot weaken data, safety, or capability requirements. No classifier confidence may override these policies.

Then define optimization goals among eligible routes. Teams may minimize expected cost subject to quality and latency floors, or maximize quality under a per-request budget. Make the priority explicit. Without it, QA receives contradictory bugs such as router used an expensive model and router should have used the strongest model for the same prompt.

2. Inventory routes, features, and decision contracts

Create a route catalog that is independent of marketing model names. Each route should expose a stable internal ID, enabled status, provider adapter, supported modalities, context limit, structured-output support, tool policy, region, data classification, tenant allowlist, timeout, retry policy, cost class, and health state. Model IDs can change without rewriting the test vocabulary.

Define request features just as carefully. Useful fields include tenant, user plan, region, data class, modality, estimated tokens, required capabilities, requested latency class, task category, safety tier, conversation state, and whether an operation has side effects. Features derived from prompt text should retain their version and confidence, while authoritative metadata such as tenant and region must come from trusted application state.

A decision contract should return more than a route name. Include request ID, selected route ID, policy version, classifier version, eligible route IDs, exclusion reason codes, selection score or reason, budget, and fallback plan. Avoid logging hidden chain-of-thought. Compact decision facts are sufficient for reproducibility and safer to retain.

Review precedence explicitly. A useful order is authorization and data policy, required capabilities, route health, context fit, safety tier, customer preference, quality threshold, latency objective, then cost optimization. The order may differ by product, but hard policy should not be mixed with a model-generated free-form recommendation. The agentic tool-calling testing guide uses the same separation between model planning and deterministic authorization.

3. Create a route decision table and failure taxonomy

A decision table converts vague routing logic into reviewable cases. Rows represent request traits and columns represent hard requirements, acceptable routes, preferred route, and expected fallback. Allow a set of acceptable routes where the product genuinely permits alternatives. Forcing a single golden route makes tests brittle and hides the actual contract.

Build a failure taxonomy around eligibility, feature extraction, selection, adapter contracts, provider behavior, output validation, fallback, and telemetry. Examples include missing trusted metadata, wrong token estimation, stale health, unsupported response schema, partial stream, incorrect usage accounting, retry storm, fallback loop, and decision logs missing a policy version.

Assign severity from user and business impact. Cross-region or data-class violations are usually critical. A rare cost-class inefficiency may be lower severity unless it can be exploited at scale. Wrong routes for destructive tool workflows deserve more scrutiny than wrong routes for low-risk drafting. This risk weighting later determines corpus coverage and release thresholds.

4. Build a golden dataset with hard negatives

A routing dataset should contain the input envelope, sanitized prompt or feature representation, expected hard constraints, acceptable route set, preferred route when meaningful, expected outcome rubric, and tags. Include ordinary traffic, boundary cases, prior incidents, and deliberately adversarial prompts. Keep training and evaluation sets separate if any router component learns from examples.

Hard negatives make the dataset useful. Pair an invoice extraction request with and without an image. Pair public product copy with a confidential customer record. Put words like simple inside a complex code request and use the biggest model inside untrusted document content. Add very long repeated text, mixed languages, code-switching, misspellings, empty prompts, attachments with misleading names, and prompts whose first and last instructions imply different tasks.

Test conversation turns, not only isolated prompts. A session can begin as public FAQ and later include restricted data or request a tool. The router must recompute eligibility from the current envelope, not reuse a stale first-turn route. Conversely, route thrashing on every small wording change can hurt cache use and latency. Add metamorphic cases where harmless paraphrases should retain an acceptable route and policy-changing metadata should not.

Label uncertain cases with an acceptable set or escalation expectation. Human reviewers may disagree about whether a medium or strong model is needed. Route safety requirements should still be exact. Track agreement and adjudication notes. Version labels alongside policy, since a model approved for a capability in July may not have been approved in March. Never silently rewrite historic expected results when a routing change fails. Explain whether policy, product goals, or a defect changed.

5. Test deterministic eligibility before semantic selection

Unit-test eligibility as pure logic whenever possible. Generate combinations of region, tenant, data class, modality, context size, capabilities, and health state. Assert both included routes and stable exclusion reason codes. Property-based tests are useful for invariants such as adding a required capability cannot make an incapable route eligible and raising data sensitivity cannot unlock a less trusted provider.

Boundary values matter. Test token estimates just below, equal to, and above context budgets after reserving output and tool tokens. Test a route changing from healthy to unhealthy between decision and execution. Test missing metadata with fail-closed rules. Verify canonical normalization for region and tenant identifiers without treating unknown aliases as approved values.

Do not let prompt injection influence trusted features. If a document says classification: public or route to admin model, that text is data, not application metadata. Build cases that place fake JSON, headers, XML tags, and system-like instructions inside user content. The deterministic policy must read data classification from authenticated context.

Test configuration changes as code. Invalid route catalogs should fail startup or deployment: duplicate route IDs, no route for a mandatory tenant, fallback cycles, capabilities inconsistent with adapters, negative budgets, and an enabled route with missing approval. A disabled route should disappear immediately from new decisions. Configuration rollback should restore the previous policy version and produce traceable decision differences.

6. Use runnable contract tests for the routing core

The example below separates exact policy from a replaceable scoring function. Save it as test_router.py, install pytest, and run pytest -q. In production, the score value may come from a classifier or semantic router, but it never adds a route that policy excluded.

from dataclasses import dataclass
from typing import FrozenSet

import pytest


@dataclass(frozen=True)
class Route:
    id: str
    regions: FrozenSet[str]
    data_classes: FrozenSet[str]
    capabilities: FrozenSet[str]
    max_input_tokens: int
    healthy: bool
    cost_rank: int


@dataclass(frozen=True)
class Request:
    region: str
    data_class: str
    required: FrozenSet[str]
    estimated_input_tokens: int


def eligible(route: Route, request: Request) -> tuple[bool, str]:
    if not route.healthy:
        return False, "unhealthy"
    if request.region not in route.regions:
        return False, "region"
    if request.data_class not in route.data_classes:
        return False, "data_class"
    if not request.required.issubset(route.capabilities):
        return False, "capability"
    if request.estimated_input_tokens > route.max_input_tokens:
        return False, "context"
    return True, "eligible"


def choose(routes: list[Route], request: Request, scores: dict[str, float]) -> Route:
    allowed = [route for route in routes if eligible(route, request)[0]]
    if not allowed:
        raise LookupError("no eligible route")
    return min(allowed, key=lambda route: (-scores.get(route.id, 0.0), route.cost_rank, route.id))


ROUTES = [
    Route("economy", frozenset({"us", "eu"}), frozenset({"public"}),
          frozenset({"text"}), 16_000, True, 1),
    Route("vision-eu", frozenset({"eu"}), frozenset({"public", "restricted"}),
          frozenset({"text", "vision", "json"}), 64_000, True, 3),
]


def test_high_score_cannot_bypass_data_policy():
    request = Request("eu", "restricted", frozenset({"text"}), 2_000)
    selected = choose(ROUTES, request, {"economy": 1.0, "vision-eu": 0.4})
    assert selected.id == "vision-eu"


def test_required_capabilities_are_conjunctive():
    request = Request("eu", "public", frozenset({"vision", "json"}), 2_000)
    assert choose(ROUTES, request, {"economy": 1.0, "vision-eu": 0.2}).id == "vision-eu"


def test_no_policy_weakening_when_all_routes_are_ineligible():
    request = Request("ap", "restricted", frozenset({"vision"}), 2_000)
    with pytest.raises(LookupError, match="no eligible route"):
        choose(ROUTES, request, {"economy": 1.0, "vision-eu": 1.0})


@pytest.mark.parametrize("tokens, expected", [(64_000, True), (64_001, False)])
def test_context_boundary(tokens: int, expected: bool):
    request = Request("eu", "public", frozenset({"vision"}), tokens)
    assert eligible(ROUTES[1], request)[0] is expected

Extend the test at the adapter boundary with a local stub server or provider SDK mock that reproduces documented HTTP status, timeout, streaming, and response shapes. Mock your wrapper, not imaginary provider methods. Keep a small sandbox integration suite that makes real calls to each enabled route to detect credential, deployment, schema, and capability drift. Mark it separately so ordinary unit tests stay fast and deterministic.

7. Evaluate semantic routing and end-to-end quality

A probabilistic router needs more than exact accuracy. Calculate top-one accuracy only for cases with one defensible route. For acceptable sets, use set membership and report the preferred-route rate separately. Use a weighted confusion matrix so sending a hard task to an incapable route costs more than sending a simple task to a stronger route. Evaluate calibration if confidence controls escalation or fallback.

Slice results by task, language, length, modality, tenant, risk, ambiguity, and conversation depth. Overall accuracy can hide complete failure on a small but important region. Compare against simple baselines such as all traffic to the approved default or a deterministic ruleset. A sophisticated classifier that does not improve user outcomes over the baseline adds operational risk without value.

Then run the selected route and score the actual task. For JSON extraction, validate schema and field accuracy. For code changes, run tests and static checks. For question answering, verify grounded claims and citations. For tool use, verify allowed calls and side effects. The AI-assisted API test generation guide shows why generated artifacts need executable oracles rather than a fluency score.

Use repeated trials where model variance affects outcome, but preserve the exact route decision and provider response metadata for each trial. Compare four systems: current router with current models, candidate router with current models, current router with candidate models, and candidate router with candidate models. This factorial view helps distinguish routing gains from endpoint model changes. A router is successful when it maintains required quality while improving the stated cost or latency objective, not merely when it predicts internal labels.

8. Test fallbacks, retries, and output validation

Fallback behavior deserves a state-machine test plan. Inject connection errors, DNS failure, authentication failure, quota exhaustion, rate limits, provider server errors, first-byte timeout, midstream disconnect, malformed JSON, missing usage, safety refusal, and valid but low-quality output. Define which events are retryable, which route can follow, and how many total attempts are allowed.

Fallback must preserve eligibility. A restricted request cannot fail over to an unapproved public route. A vision request cannot fall back to text unless the product explicitly extracts text through an approved deterministic path. A long-context request cannot be silently truncated to fit. Test the complete fallback chain from configuration and reject cycles at load time.

Separate transport retry from semantic retry. A request that never reached a provider may be safe to retry. A streamed response that failed after a tool call or external side effect may not be. Carry an idempotency key through your own orchestration and ensure adapters do not duplicate committed actions. Verify that partial text is not shown as a complete answer and that the UI communicates recovery or failure consistently.

Output validation can trigger bounded repair or fallback. Test a schema-invalid response, a repair that succeeds, repeated invalid output, and a valid structure with semantically unsafe content. Count all attempts against latency and cost budgets. Preserve the first failure in telemetry so a later successful fallback does not erase reliability evidence. If all eligible routes fail, return an explicit unavailable outcome rather than fabricating a result or weakening policy.

9. Validate cost, latency, cache, and quota controls

Cost tests use controlled token and request fixtures rather than fabricated public pricing. Store pricing and internal cost weights as versioned configuration. Given known input, output, cache, image, and tool usage reported by the adapter, assert the calculated internal charge and budget decision. Test missing usage, negative values, integer overflow, currency precision, and pricing changes during a run.

Measure latency as a distribution at each boundary: feature extraction, eligibility, classifier, queue, provider time to first byte, generation, validation, and fallback. A cheap model chosen after a slow classifier may miss the endpoint objective. Test cold starts, connection pool exhaustion, long streams, cancellation, and client disconnects. Cancellation should stop avoidable downstream work and usage accounting should reflect what actually occurred.

Cache routing is another route with its own contract. Test key construction, tenant isolation, prompt and policy versioning, expiration, invalidation, and semantic-cache false positives. Sensitive requests may prohibit caching entirely. A cached response must meet current authorization and safety policy, not only the policy from when it was created. Seed near-duplicate prompts with materially different entities or negation to expose unsafe semantic matches.

Quota tests cover per-user, tenant, route, provider, and global limits. Exercise exact boundaries and concurrent requests. Verify atomic reservation or reconciliation so a burst cannot exceed budget through races. Decide whether quota exhaustion downgrades to a cheaper eligible route, queues, or returns an error. The answer must be visible and testable, not hidden in exception handling.

10. Attack router security and abuse controls

Treat route selection as a privilege boundary when stronger models, tools, private providers, or higher budgets are involved. Fuzz untrusted prompt content with fake feature headers, route names, model IDs, XML tags, role markers, and encoded instructions. None should overwrite authenticated metadata or configuration. Normalize only where safe, and test confusable Unicode in tenant, region, and route identifiers.

Attempt cost amplification with repeated huge prompts, nested attachments, requests crafted to score as complex, forced fallbacks, and disconnects after generation begins. Verify rate limits, request-size limits, token estimation margins, concurrency caps, and maximum attempts. Do not make the router cheaper by sending suspicious traffic to an incapable model if the safe behavior is rejection.

Test data minimization. A classifier may not need the full prompt. If features or redacted excerpts suffice, verify that restricted fields never reach a routing vendor. Decision logs should avoid raw prompts by default and expose stable reason codes, policy versions, sizes, hashes where appropriate, and approved trace references. Test access controls on route dashboards because they can reveal customer activity and cost.

Probe model enumeration and information leakage through errors. External messages should not reveal hidden route names, providers, account quotas, or detailed health. Internal traces can retain operational facts under access control. Test that safety refusals cannot be bypassed by routing to a different endpoint, and that fallback uses a shared minimum safety policy. The AI test data masking guide provides additional cases for protecting sensitive evaluation and trace data.

11. Load test route stability and recovery

Load profiles should resemble request diversity, not only a constant stream of tiny prompts. Mix short and long text, attachments, streaming and nonstreaming calls, tenant priorities, cache hits, and routes with different service times. Measure decision throughput separately from provider capacity so one does not conceal the other. Use provider stubs for repeatable high load, then limited sandbox tests for real integration behavior.

Look for queue starvation, head-of-line blocking, connection pool limits, stale health, synchronized retries, circuit-breaker oscillation, and hot-tenant impact. When a route slows, verify how traffic shifts and whether the fallback has capacity. A failover that moves all traffic instantly can collapse the healthy provider. Test gradual shedding, per-route concurrency, retry jitter, and recovery thresholds.

Health signals need hysteresis. Inject alternating successes and failures around thresholds and assert that the route does not flap on every sample. After recovery, send probe traffic before restoring full volume. Ensure requests already assigned to an unhealthy route have a documented outcome and that new requests use the latest state.

Soak tests reveal memory leaks in stream handling, growing decision caches, metric cardinality, and leaked connections after cancellation. Reconcile request counts among gateway, router, adapters, and providers. Every request should end in one terminal state: completed, safely refused, failed, canceled, or expired. Unknown terminal states are a reliability defect even if aggregate success appears healthy.

12. Release safely after testing an LLM router

Run deterministic tests on every policy and adapter change. Run the labeled routing corpus when feature extraction, classifiers, embeddings, route catalogs, or prompts change. Run outcome evaluations for affected tasks. Contract-test every adapter and schedule real sandbox probes. Add failure-injection, load, and security suites at risk-appropriate intervals.

Before changing live decisions, shadow the candidate on sampled production envelopes where privacy policy allows it. The candidate records a decision but does not execute. Compare eligibility violations, route distribution, estimated cost class, confidence, and disagreement slices. Shadowing cannot prove endpoint task quality, so combine it with offline replay in a controlled environment.

Use a gradual rollout keyed by stable request or tenant assignment. Monitor policy-denial changes, no-route rate, fallback attempts, provider errors, route distribution, p50 and tail latency, task success, schema validity, safety events, and normalized internal cost. Break down metrics by route and risk slice. Cap metric labels so request or tenant IDs do not create unbounded cardinality.

Rollback should restore policy, classifier, and route catalog as a coherent version. Preserve decision traces across rollback for incident replay. Any policy violation should stop rollout immediately. Quality and cost regressions use predeclared thresholds and observation windows. A router is not finished when it selects a model. It is finished when the request reaches a safe terminal outcome with evidence explaining why that path was permitted and chosen.

Interview Questions and Answers

Q: How would you test an LLM router?

I would separate exact eligibility rules, probabilistic route selection, provider adapter contracts, and end-to-end task quality. I would use a versioned dataset with acceptable route sets, policy constraints, adversarial prompts, and outcome rubrics. I would also inject provider failures and measure safety, quality, latency, and cost by route.

Q: Why is router accuracy an incomplete metric?

A human label may allow several models, and matching it does not prove the selected route was compliant or capable. The endpoint may also produce a poor result. I use hard violation counts, weighted routing error, and actual task success alongside efficiency measures.

Q: How do you test a router fallback?

I model fallback as a bounded state machine and inject specific transport, throttling, timeout, stream, and validation failures. Every next route must satisfy the original region, data, capability, safety, and budget constraints. I verify attempt limits, idempotency, accounting, and the final user-visible outcome.

Q: What belongs in a routing golden dataset?

Each case includes the trusted request envelope, prompt or derived features, hard constraints, acceptable routes, a preferred route if one exists, an outcome rubric, and slice tags. It should contain boundaries, hard negatives, multilingual input, long context, conversation transitions, and prior incidents.

Q: How would you test cost optimization without sacrificing quality?

I compare the candidate with the current router on the same task corpus. The candidate must retain per-slice quality and policy floors before cost improvement counts. Usage calculations use versioned internal pricing and are reconciled across retries, cache, tools, and fallbacks.

Q: How do you stop prompt injection from controlling the route?

Trusted attributes such as tenant, data class, and region come from authenticated application state. Prompt text is treated as untrusted data, even when it contains headers or JSON that resemble routing metadata. Deterministic policy gates run before any model recommendation can influence eligibility.

Q: What would you monitor after a router release?

I monitor no-route rate, policy exclusions, route mix, fallback depth, provider errors, schema validity, task success, latency distributions, safety events, and normalized cost. I slice by route, task, tenant class, region, and risk without logging raw sensitive prompts by default.

Q: How do you test model-router configuration?

I validate unique IDs, capability consistency, approvals, fallback acyclicity, limits, and coverage for mandatory traffic. I test enable, disable, rollback, and health changes as versioned configuration operations. Invalid catalogs fail before serving traffic.

Common Mistakes

• Measuring only top-one route accuracy. Add policy violations, acceptable-set scoring, outcome quality, latency, safety, and cost.
• Letting a classifier decide compliance. Keep tenant, region, data, capability, and health eligibility deterministic.
• Using one obvious prompt per intent. Add hard negatives, ambiguity, adversarial text, conversation transitions, and boundary values.
• Allowing fallback to weaken requirements. Re-evaluate every candidate against the original hard constraints.
• Ignoring partial streams and side effects. Test cancellation, retry safety, duplicate tools, accounting, and user-visible partial output.
• Treating the route catalog as harmless configuration. Validate it, version it, test rollback, and reject cycles or missing approvals.
• Optimizing average latency only. Measure time to first byte, total latency, and tail behavior for each stage and route.
• Logging entire prompts for debugging. Retain minimal decision evidence, redact sensitive data, and control trace access.

Conclusion

Testing an LLM router requires exact policy checks, realistic decision evaluations, adapter contract tests, and end-to-end outcome evidence. The core rule is simple: optimize only among routes that are authorized, capable, healthy, and large enough for the request.

Start with a stable route catalog and a decision table, implement the pure eligibility tests, then add a labeled corpus and failure-injection suite. Release through shadow comparison and gradual rollout, with any policy violation treated as a stop condition rather than an acceptable routing error.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Load testing an LLM API (2026)
• Testing an LLM agent memory (2026)
• Testing structured output from an LLM (2026)
• Evaluating an LLM app with DeepEval metrics (2026)
• Red teaming an LLM chatbot (2026)
• Testing an AI agent with LangSmith (2026)