Testing token limits and truncation (2026)

TL;DR

Testing token limits and truncation requires boundary tests on both prompt and completion budgets, explicit inspection of provider completion status, and application-level completeness assertions. Never infer success only because some text arrived.

Key Takeaways

• Test input capacity, requested output budget, hidden overhead, and application limits as separate boundaries.
• Use provider response status and incomplete details as the primary truncation signal, not punctuation heuristics.
• Assert completion contracts such as final sentinels, valid schemas, record counts, and required closing sections.
• Generate boundary cases from measured token counts and keep tokenizer or provider configuration versioned.
• Test streaming interruption, retries, continuation, duplication, and idempotency independently.
• Treat silent input clipping as a critical correctness defect and preserve high-priority context explicitly.
• Run deterministic budget tests on every commit and a small provider-backed boundary suite before release.

Testing token limits and truncation means proving that an LLM application behaves correctly when input, output, tool results, retrieval context, and conversation history approach or exceed their budgets. A robust test suite detects incomplete responses explicitly, rejects silently clipped data, and verifies that retry or continuation logic does not duplicate or corrupt work.

Token limits are not one number. The provider model has a context window and an output cap, while your gateway, prompt builder, agent framework, database, and UI may impose smaller limits. This guide shows QA and SDET engineers how to map those boundaries, design reproducible cases, test current Responses API status fields, validate streams and structured output, and turn the findings into safe CI gates.

TL;DR

Build tests around below -> at -> above each application-owned limit. For provider-owned limits, use a measured neighborhood rather than assuming an undocumented exact boundary. Inspect machine-readable completion metadata first, then add domain-level completeness assertions.

1. What Testing Token Limits and Truncation Must Prove

Testing token limits and truncation starts with a failure model. Input overflow may be rejected, automatically shortened, summarized by middleware, or accepted after older conversation turns are removed. Output exhaustion may return an explicitly incomplete response, stop at a valid-looking sentence, cut a JSON object, omit the final list entries, or leave an agent waiting for a tool call that never arrives. Each behavior has a different oracle.

Separate five questions. Did the exact intended input reach the model? Did the provider accept it? Did the model finish under the requested output budget? Did the application receive the provider's terminal state? Did the final artifact satisfy the business contract? A single assertion on nonempty text answers none of these adequately.

Define completion in product terms. A support summary may require five named sections. A test generator may require exactly one result per requirement ID. A migration plan may need a final risks array. A conversational answer may be allowed to stop after a complete paragraph, while a financial action proposal must be rejected if any source or approval field is absent.

Make truncation observable. Persist a request ID, model identifier, prompt revision, input and output token usage when provided, configured output budget, response status, incomplete reason, stream terminal event, retry count, and the application decision. Do not log sensitive prompt text by default. Hash or classify payloads when full content cannot be retained.

This mindset complements testing structured output from an LLM, where schema validity is one layer of a larger completion contract.

2. Map Every Token Budget and Ownership Boundary

Create a budget map before writing tests. The model context window is only the outermost constraint. Instructions, user input, prior turns, retrieved documents, tool schemas, tool outputs, images, reasoning, and generated output all compete for capacity according to provider-specific accounting. Your application can also cap characters, documents, conversation turns, or serialized bytes before a provider sees the request.

Label limits as hard, soft, or observed. A hard application limit is under your control and should have exact N-1, N, and N+1 cases. A soft budget triggers summarization or prioritization and needs semantic retention tests. An observed provider boundary should be probed with current documentation and telemetry, not copied forever into a test constant.

Version the budget map with the prompt template, model configuration, tokenizer or counting strategy, retrieval settings, tool schemas, and application release. A new tool description or safety instruction can reduce available user capacity even if the model did not change. Treat that as a contract change and rerun the boundary suite.

3. Build Boundary Inputs Without Brittle Token Assumptions

Token counts are model-specific, and character counts are not a safe substitute. English words, source code, emoji, CJK text, JSON, whitespace, and repeated punctuation tokenize differently. Test data should therefore include both deterministic application limits and measured provider-facing cases.

For a character limit you own, property tests can generate exact boundaries. For token-aware prompt assembly, expose a production counting function or adapter and test its policy. The following runnable Python example tests a simple priority-preserving word budget. It is not presented as a provider tokenizer. Its purpose is to demonstrate a deterministic application contract.

# budget.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Segment:
    name: str
    text: str
    priority: int

def select_segments(segments: list[Segment], max_words: int) -> list[Segment]:
    if max_words < 0:
        raise ValueError("max_words must be nonnegative")

    selected: list[Segment] = []
    used = 0
    for segment in sorted(segments, key=lambda item: item.priority):
        size = len(segment.text.split())
        if used + size <= max_words:
            selected.append(segment)
            used += size
    return selected

# test_budget.py
from budget import Segment, select_segments

POLICY = Segment("policy", "never disclose private account data", 0)
DETAILS = Segment("details", "customer requests a billing explanation", 1)
CHAT = Segment("history", "hello thanks for your earlier help", 2)

def test_exact_boundary_retains_required_segments() -> None:
    chosen = select_segments([CHAT, DETAILS, POLICY], max_words=10)
    assert [item.name for item in chosen] == ["policy", "details"]

def test_one_below_does_not_partially_clip_a_segment() -> None:
    chosen = select_segments([DETAILS, POLICY], max_words=9)
    assert [item.name for item in chosen] == ["policy"]

Add multilingual, minified JSON, code, long unbroken strings, whitespace-heavy text, and adversarial repetition. Always store the actual measured count and counting implementation version with failures.

4. Test Input Overflow and Silent Context Loss

Input overflow is safer when it fails loudly than when a layer silently removes information. Test every supported policy: reject, trim, summarize, retrieve fewer chunks, compress tool results, or start a new conversation. The expected behavior must be deterministic enough to explain which content was preserved and why.

Plant unique sentinels in different prompt regions, such as POLICY_KEEP_7Q, USER_FACT_2M, and OLDEST_TURN_9K. Ask a task whose correct answer depends on those facts, but also inspect the built request directly. Behavioral success alone cannot prove that all intended content was present. Middleware may reconstruct an answer from other context.

For trimming policies, assert invariants:

• system and safety instructions are never removed by user-controlled text,
• atomic records are kept or removed whole, never cut mid-field,
• source IDs remain paired with their content,
• recent turns are not always more important than contractual facts,
• summarization preserves decisions, constraints, unresolved questions, and provenance,
• the user is told when their requested scope cannot fit,
• the original artifact remains available for audit or later retrieval.

Test one very large source and many small sources with the same total size. Test a large tool result after a long conversation. Test Unicode expansion at byte-limited gateways. Test encoded attachments and image inputs separately because multimodal accounting may not map cleanly to text tokens.

A useful metamorphic test removes irrelevant padding and expects the decision to remain stable. Another moves a critical fact from the beginning to the middle or end and expects equal retention. These cases expose position-sensitive loss and naive head or tail clipping.

5. Detect Output Truncation With Current API Status

A provider's machine-readable status is the first oracle for output exhaustion. With the OpenAI Responses API, a response can have status == "incomplete", and incomplete_details.reason explains why. The exact reason set is provider-controlled, so application code should record unknown values rather than crash. A max_output_tokens request setting limits generated output, but it does not promise the business artifact will fit.

This runnable integration example requires the current openai Python package, OPENAI_API_KEY, and an OPENAI_MODEL that your project has enabled. It deliberately requests a tiny output budget and asserts that the application never labels an incomplete response as complete.

# truncation_probe.py
import os
from dataclasses import dataclass
from openai import OpenAI

@dataclass(frozen=True)
class CompletionResult:
    text: str
    complete: bool
    reason: str | None

def request_report(max_output_tokens: int) -> CompletionResult:
    client = OpenAI()
    response = client.responses.create(
        model=os.environ["OPENAI_MODEL"],
        input=(
            "Write a detailed incident report with sections named Summary, "
            "Impact, Timeline, Root Cause, Corrective Actions, and Owners."
        ),
        max_output_tokens=max_output_tokens,
    )
    details = response.incomplete_details
    reason = details.reason if details is not None else None
    return CompletionResult(
        text=response.output_text,
        complete=response.status == "completed",
        reason=reason,
    )

if __name__ == "__main__":
    result = request_report(max_output_tokens=64)
    print(result)
    if not result.complete:
        raise SystemExit(2)

Do not assert that every small budget always yields the same exact reason across every model. Assert the safety property: incomplete provider state never becomes application success. In a controlled provider contract suite, record status, reason, output length, usage, and whether required sections are present.

6. Add Domain Completeness and Structured Output Oracles

A completed provider response may still be incomplete for your product. The model can decide it has finished after returning four of five records, omit a required conclusion, or compress details below a useful threshold. Conversely, a response can end on a period while its provider status is incomplete. Punctuation, balanced Markdown fences, and phrases like done are weak signals.

Choose an application contract that can be verified. For prose, require named sections and facts grounded in the input. For batch generation, require every input ID exactly once. For tool plans, require terminal state plus all mandatory approvals. For structured output, parse the whole object, validate its schema, and check cross-field rules. A valid prefix must never be treated as a valid artifact.

A final sentinel can help when you control the format. Ask the model to end with a fixed marker such as REPORT_COMPLETE, then strip it before display. The marker is an extra check, not a replacement for provider status or content validation. A model can emit the marker early, and a malicious source can inject it. Scope the sentinel to generated output and combine it with structural invariants.

Test continuation carefully. Save the exact partial output and ask the model to continue from a stable boundary. The merge layer must remove overlap, preserve order, and reject contradictions. Structured objects are usually safer to regenerate from scratch with a larger budget than to splice fragments. For expensive artifacts, resume at record boundaries using stable IDs.

The golden dataset guide for LLM evals explains how to preserve expected IDs, facts, and risk slices for these semantic completeness checks.

7. Test Streaming Completion, Cancellation, and Recovery

Streaming creates two independent outcomes: content delivery and response completion. A client can receive many text deltas and still lose the terminal event because of a network interruption, process restart, proxy timeout, or user cancellation. The UI must not infer completion from a quiet socket.

Model the stream consumer as a state machine. Typical states are created, receiving, completed, incomplete, failed, and cancelled. Only an authoritative terminal event may enter completed. Unexpected connection close from receiving becomes failed or unknown, never success. Persist the provider response ID and last confirmed sequence information when the API exposes it.

Test at least these sequences:

1. normal deltas followed by completion,
2. deltas followed by an explicit incomplete event,
3. connection close before any content,
4. connection close after a valid-looking paragraph,
5. duplicated delta or terminal delivery,
6. out-of-order or malformed event handled by the client library or adapter,
7. user cancellation while a tool or model is running,
8. reconnect or retry after partial display,
9. server completion after the user has navigated away,
10. two browser tabs observing the same response.

The recovery path must be idempotent. If a retry can create tickets, send email, or update state, carry stable operation IDs and check side effects before repeating work. For text-only output, decide whether to replace, continue, or label partial content. Never append a full retry blindly to the previous prefix.

UI tests should verify the visible status, retry control, copy behavior, accessibility announcement, and persistence after refresh. The backend and browser must agree on whether an artifact is complete.

8. Cover Agent, RAG, and Tool-Result Budget Pressure

Agents add hidden budget consumers. Tool definitions, arguments, results, intermediate reasoning items, retrieved evidence, and repeated observations may all share context. A tool can return a megabyte of JSON when only three fields are relevant. A loop can repeat nearly identical results until the final answer has too little space.

Create adversarial tool doubles that return empty results, maximum-size valid results, highly repetitive data, malformed Unicode, thousands of records, nested objects, secret-like fields, and pagination tokens. Assert that the agent adapter filters fields, enforces byte and record limits, preserves provenance, and instructs the model how to request the next page. The raw result should not be silently clipped inside a JSON string.

For RAG, vary chunk count, chunk size, overlap, ranking confidence, duplicated sources, and query expansion. Measure which source IDs actually reach the model. A retrieval test that only checks the vector store response misses prompt-level eviction. Reserve budget for instructions and final output before filling the remaining space with evidence.

Test long conversations with a late tool call, not only one-turn agents. Confirm that authorization rules and the user's current goal survive summary or compaction. If a summary replaces old turns, compare it against a deterministic checklist of decisions, entities, constraints, pending approvals, and irreversible actions.

The deeper orchestration risks are covered in testing an AI agent with LangSmith. Use trace evidence to separate model selection errors from context assembly and budget defects.

9. Define Safe Retry, Continuation, and User Experience Policies

A retry policy should depend on failure class. Retrying an overloaded service can help. Retrying the same prompt with the same output cap after confirmed token exhaustion usually repeats the defect. Increasing a budget may work, but it changes cost, latency, and sometimes model behavior. Summarizing input may remove evidence. Continuing output may duplicate content. Every recovery has tradeoffs that deserve tests.

Set maximum attempts and a total request budget. Surface a clear message after exhaustion, with what completed, what did not, and what the user can do. Do not expose provider internals that confuse users, but preserve them in secured telemetry.

For high-risk actions, fail closed. A partial explanation can be displayed as partial, but a partial tool plan must not execute missing approvals. A truncated data export should not be offered as complete. A summary missing one incident ID should fail a reconciliation check.

Test accessibility and localization of error states. Longer translated messages can expose UI truncation unrelated to tokens, while right-to-left layouts can hide retry controls. Product completeness includes the recovery experience.

10. Automate Testing Token Limits and Truncation in CI

Testing token limits and truncation in CI works best as a pyramid. Run pure prompt-builder, budget allocator, schema, merge, state-machine, and UI-state tests on every commit. Run a small provider-backed suite when the prompt, model, SDK, tools, or budget configuration changes. Run broader long-context evaluations on a schedule because they are slower and can consume substantial capacity.

Create named slices such as input_below_limit, input_policy_boundary, tool_result_pressure, output_minimum_viable, structured_output_cutoff, stream_disconnect, and multilingual_density. Report pass rates by slice rather than one average. A system that passes short English prompts but loses policy text under CJK or JSON-heavy input is not healthy.

Keep thresholds evidence-based. Exact application limits need exact expectations. Model-backed semantic retention may use a reviewed dataset and candidate-versus-baseline comparison. Provider status handling should be deterministic: any incomplete or unknown terminal state must fail the completion contract.

Store diagnostic artifacts without leaking secrets: configuration hashes, selected source IDs, segment sizes, token usage, response status, incomplete reason, event sequence, and application decision. Alert on changes in truncation rate, not raw output length alone. A shorter response may be better, while a nominally long response may omit the required ending.

Include budget review in change control. New tool schemas, prompt instructions, retrieved fields, or agent turns can consume space even when feature tests pass. The production LLM guardrails testing guide provides a related framework for failing safely when model output cannot be trusted.

Interview Questions and Answers

Q: What is the difference between a context limit and an output limit?

A context limit constrains the information available to a request, which may include instructions, history, tools, evidence, and generated content depending on the API. An output limit caps what the model may generate. I test them separately because input overflow, input eviction, and output exhaustion have different signals and recovery paths.

Q: How do you detect LLM truncation reliably?

I first inspect provider status, incomplete details, terminal stream events, and errors. Then I apply a domain completeness contract such as valid schema, required sections, exact record IDs, or a terminal task state. I do not rely on punctuation, output length, or a valid-looking last sentence.

Q: Why are character-count tests insufficient for token limits?

Tokenization varies by model and content. Code, multilingual text, emoji, JSON, whitespace, and long strings can have different token density. Character limits remain useful for application-owned boundaries, but provider-facing tests should use the production counter or observed usage and record its version.

Q: How would you test silent prompt clipping?

I place unique sentinels and required facts in instructions, early history, middle context, recent turns, and retrieved chunks. I inspect the assembled provider request and verify behavior that depends on each retained fact. I also move facts between positions to expose head, tail, or middle loss.

Q: What should happen when a streamed response disconnects after a complete-looking paragraph?

The application should remain incomplete unless it received an authoritative completed event or confirmed status. It can show the received text as partial, preserve the response ID, and offer a safe recovery. It must prevent duplicate side effects if retrying an agent operation.

Q: When is continuation safer than regeneration?

Continuation is reasonable for append-only content with stable record or section boundaries. Regeneration is often safer for a single structured object because fragment merging can create invalid or contradictory data. The merge policy must remove overlap, preserve ordering, and prove completeness.

Q: How would you keep token boundary tests stable across model upgrades?

I separate exact application-policy tests from provider observation tests. I version model, SDK, prompt, tool schema, counting adapter, and datasets, then probe a neighborhood around measured boundaries. Upgrade review focuses on safety invariants and retained information, not preserving one historical count.

Common Mistakes

• Checking only whether text is nonempty: Partial output can be long and convincing.
• Assuming a period means completion: A token-exhausted response can end at a sentence boundary.
• Treating characters as tokens: Content and model tokenization make the ratio unstable.
• Testing only the advertised model window: Gateways, prompt builders, consumers, and UIs can impose smaller caps.
• Silently trimming user or policy content: Make retention policy explicit and observable.
• Retrying with the same exhausted budget: Change the strategy or fail clearly.
• Appending retries without deduplication: Continuation can repeat text or side effects.
• Ignoring tool and retrieval payloads: They are common sources of context pressure.
• Marking a stream complete on socket close: Require a terminal success state.
• Updating hardcoded token boundaries blindly: Revalidate counts and behavior when model or prompt configuration changes.

Conclusion

Testing token limits and truncation is boundary testing across the entire LLM delivery path. Map every owner, generate measured edge cases, inspect explicit provider and streaming states, and enforce product-level completeness before consumers or users see success.

Start by adding one deterministic prompt-budget test, one forced small-output integration probe, and one interrupted-stream test. Those three cases quickly reveal whether your system fails clearly or turns partial work into a dangerous false success.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Pairwise and orthogonal array testing (2026)
• Testing bearer token refresh (2026)
• Testing bulk and batch endpoints (2026)
• Testing embeddings and vector search quality (2026)
• Testing pagination and filtering (2026)
• Testing sorting and search APIs (2026)