How to Test LLM Context Window Degradation (2026)

TL;DR

To test LLM context window degradation, generate controlled documents at several token lengths, place one answer-bearing record at multiple relative positions, query the model, and score exact retrieval. Plot accuracy by length and position, then enforce a regression budget against a versioned baseline.

Key Takeaways

• Measure task accuracy across token lengths and evidence positions instead of trusting the advertised context limit.
• Use deterministic synthetic records so every expected answer is known before the model runs.
• Sweep evidence through the beginning, middle, and end because aggregate length alone hides positional failures.
• Record prompt tokens, latency, refusal, parse errors, and exact accuracy for every case.
• Repeat cases and compare confidence intervals before treating a small score change as a regression.
• Gate releases on a versioned baseline, a critical slice floor, and an explicit allowed drop.
• Test the complete application path separately from direct model calls to expose retrieval and prompt-assembly defects.

To test LLM context window degradation, do not send one enormous prompt and declare success when the answer looks plausible. Build a controlled matrix that changes context length and evidence position independently, gives every case a known answer, and records exact accuracy, latency, token usage, and failure type. The resulting degradation curve shows your application's usable context, which is usually a more meaningful boundary than a provider's maximum accepted token count.

This tutorial builds a Python harness against the official OpenAI Responses API. The design transfers to other providers because the dataset, scoring, and gate are provider-independent. It complements a broader LLM evaluation pipeline guide, but focuses on long-context behavior and positional sensitivity.

What You Will Build

You will create a small, auditable benchmark that:

• generates deterministic synthetic records without copying benchmark answers from the internet,
• approximates specified token budgets with the model's tokenizer,
• places a unique target record at 10%, 50%, and 90% of the context,
• calls a real model through client.responses.create,
• writes case-level JSONL evidence and a Markdown summary,
• fails CI when critical slices fall below an explicit accuracy floor.

The benchmark asks for an eight-character authorization code from one numbered record. Distractor records use the same shape, so keyword search alone is insufficient. The output contract permits only the code or NOT_FOUND, making exact scoring reliable.

Prerequisites

Use Python 3.12.x, openai==1.109.0, tiktoken==0.11.0, and pytest==8.4.1. The example uses gpt-5-mini; set LONG_CONTEXT_MODEL to another Responses API model your account can access. Provider limits and model availability can change, so keep the selected model in the run artifact.

Create an isolated environment and install exact versions:

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install openai==1.109.0 tiktoken==0.11.0 pytest==8.4.1

Export credentials without putting them in source control:

export OPENAI_API_KEY="your-api-key"
export LONG_CONTEXT_MODEL="gpt-5-mini"

Verification: run python --version && python -c "import openai,tiktoken; print(openai.__version__, tiktoken.__version__)". Expect Python 3.12.x followed by 1.109.0 0.11.0.

Use smaller budgets while developing. Expand only after checking the selected model's documented input limit and accounting for instructions, the question, and output tokens.

Step 1: Define the Case and Result Contracts

Create context_eval.py with immutable case metadata and a result record. Keeping generation inputs in each result makes a failure reproducible without storing the full prompt.

from __future__ import annotations

import argparse
import hashlib
import json
import os
import random
import statistics
import time
from dataclasses import asdict, dataclass
from pathlib import Path

import tiktoken
from openai import OpenAI

MODEL = os.getenv("LONG_CONTEXT_MODEL", "gpt-5-mini")
ENCODING = tiktoken.get_encoding("o200k_base")

@dataclass(frozen=True)
class Case:
    case_id: str
    target_tokens: int
    target_position: float
    repeat: int
    seed: int
    record_id: int
    expected: str

@dataclass
class Result:
    case_id: str
    model: str
    target_tokens: int
    actual_tokens: int
    target_position: float
    repeat: int
    expected: str
    actual: str
    passed: bool
    failure_type: str
    latency_ms: int
    input_tokens: int | None
    output_tokens: int | None
    prompt_sha256: str

o200k_base is used for local sizing. The API-reported token count remains authoritative because request formatting and model tokenization can differ. The prompt hash proves which exact input produced a result without publishing the bulky context.

Verification: run python -m py_compile context_eval.py. No output and exit code 0 confirm that imports and dataclasses compile.

Step 2: Generate Controlled Long Contexts

Append the following functions. Each record has a stable ID, an authorization code, and filler. One target record is inserted at a calculated record index. Codes come from SHA-256, so a seed always regenerates the same oracle.

def code_for(seed: int, record_id: int) -> str:
    raw = f"{seed}:{record_id}".encode()
    return hashlib.sha256(raw).hexdigest()[:8].upper()

def make_record(seed: int, record_id: int) -> str:
    code = code_for(seed, record_id)
    filler = (
        "Status verified. Region north. Tier standard. "
        "The audit trail contains no exception."
    )
    return (
        f"RECORD {record_id:05d} | AUTH_CODE {code} | {filler}"
    )

def build_context(case: Case) -> str:
    sample = make_record(case.seed, 0) + "\n"
    tokens_per_record = len(ENCODING.encode(sample))
    record_count = max(20, case.target_tokens // tokens_per_record)
    target_index = min(
        record_count - 1,
        max(0, round((record_count - 1) * case.target_position)),
    )

    records = []
    next_distractor = 10_000
    for index in range(record_count):
        record_id = case.record_id if index == target_index else next_distractor
        records.append(make_record(case.seed, record_id))
        next_distractor += 1
    return "\n".join(records)

def make_cases() -> list[Case]:
    cases = []
    for target_tokens in (2_000, 8_000, 24_000):
        for position in (0.10, 0.50, 0.90):
            for repeat in range(2):
                seed = target_tokens + int(position * 100) + repeat
                record_id = 700 + repeat
                cases.append(Case(
                    case_id=f"t{target_tokens}-p{position:.2f}-r{repeat}",
                    target_tokens=target_tokens,
                    target_position=position,
                    repeat=repeat,
                    seed=seed,
                    record_id=record_id,
                    expected=code_for(seed, record_id),
                ))
    return cases

This is a retrieval probe, not a complete reasoning benchmark. That narrowness is useful: when accuracy falls, you know the relevant fact existed in the prompt and the answer was unambiguous. Later, add separate suites for multi-hop joins, chronology, summarization, and instruction conflicts.

Verification: run python -c "from context_eval import *; c=make_cases()[0]; x=build_context(c); assert c.expected in x; print(len(ENCODING.encode(x)), c.expected)". Expect a token count near 2,000 and one eight-character code.

Step 3: Call the Responses API and Classify Failures

Append a runner that submits one case. It uses the documented OpenAI client and responses.create method. Set temperature only when the selected model supports it; the minimal call below avoids model-specific sampling parameters.

def normalize_answer(text: str) -> str:
    return text.strip().upper().replace("`", "")

def run_case(client: OpenAI, case: Case) -> Result:
    context = build_context(case)
    prompt = (
        "Read the records below. Return only the AUTH_CODE belonging to "
        f"RECORD {case.record_id:05d}. If that record is absent, return NOT_FOUND."
        " Do not explain.\n\n" + context
    )
    actual_tokens = len(ENCODING.encode(prompt))
    started = time.perf_counter()
    try:
        response = client.responses.create(
            model=MODEL,
            input=prompt,
            max_output_tokens=32,
        )
        actual = normalize_answer(response.output_text)
        passed = actual == case.expected
        failure_type = "none" if passed else (
            "not_found" if actual == "NOT_FOUND" else "wrong_answer"
        )
        usage = response.usage
        return Result(
            case_id=case.case_id, model=MODEL,
            target_tokens=case.target_tokens, actual_tokens=actual_tokens,
            target_position=case.target_position, repeat=case.repeat,
            expected=case.expected, actual=actual, passed=passed,
            failure_type=failure_type,
            latency_ms=round((time.perf_counter() - started) * 1000),
            input_tokens=getattr(usage, "input_tokens", None),
            output_tokens=getattr(usage, "output_tokens", None),
            prompt_sha256=hashlib.sha256(prompt.encode()).hexdigest(),
        )
    except Exception as exc:
        return Result(
            case_id=case.case_id, model=MODEL,
            target_tokens=case.target_tokens, actual_tokens=actual_tokens,
            target_position=case.target_position, repeat=case.repeat,
            expected=case.expected, actual=type(exc).__name__, passed=False,
            failure_type="provider_error",
            latency_ms=round((time.perf_counter() - started) * 1000),
            input_tokens=None, output_tokens=None,
            prompt_sha256=hashlib.sha256(prompt.encode()).hexdigest(),
        )

Do not count a rate limit or transport timeout as a wrong model answer. Separating provider_error prevents infrastructure instability from masquerading as context degradation. In a production harness, catch documented SDK exception subclasses to create finer categories while still preserving a safe error type rather than secrets or request content.

Verification: run python -c "from context_eval import normalize_answer; assert normalize_answer('AB12CD34') == 'AB12CD34'". This local check costs nothing. If credentials are ready, the full run arrives in Step 5.

Step 4: Summarize the Degradation Curve

Append aggregation and report functions. Accuracy is calculated only over completed model responses. Provider errors are displayed separately so an outage cannot improve or lower the behavioral denominator silently.

def group_summary(results: list[Result]) -> list[dict]:
    groups: dict[tuple[int, float], list[Result]] = {}
    for result in results:
        groups.setdefault(
            (result.target_tokens, result.target_position), []
        ).append(result)

    summary = []
    for (tokens, position), rows in sorted(groups.items()):
        completed = [r for r in rows if r.failure_type != "provider_error"]
        accuracy = (
            sum(r.passed for r in completed) / len(completed)
            if completed else 0.0
        )
        latencies = [r.latency_ms for r in completed]
        summary.append({
            "target_tokens": tokens,
            "position": position,
            "accuracy": round(accuracy, 3),
            "completed": len(completed),
            "provider_errors": len(rows) - len(completed),
            "median_latency_ms": (
                round(statistics.median(latencies)) if latencies else None
            ),
        })
    return summary

def write_report(results: list[Result], path: Path) -> None:
    lines = [
        "# Long-context evaluation\n",
        "| Tokens | Position | Accuracy | Completed | Provider errors | Median ms |",
        "|---:|---:|---:|---:|---:|---:|",
    ]
    for row in group_summary(results):
        lines.append(
            f"| {row['target_tokens']} | {row['position']:.0%} | "
            f"{row['accuracy']:.1%} | {row['completed']} | "
            f"{row['provider_errors']} | {row['median_latency_ms']} |"
        )
    path.write_text("\n".join(lines) + "\n", encoding="utf-8")

Read the table horizontally and vertically. A fall across token budgets at all positions suggests length-related degradation. A trough at 50% with stronger scores at 10% and 90% suggests a positional effect often called lost in the middle. Rising provider errors indicate capacity or rate-limit trouble, not evidence that the model forgot the target.

Two repeats make the tutorial affordable but do not support a stable release decision. For a real gate, select repetitions from observed variability and report a binomial confidence interval. Avoid declaring a two-percentage-point victory when the interval around each estimate is wider than the difference.

Verification: run python -c "from context_eval import group_summary; assert group_summary([]) == []; print('aggregation ok')". Expect aggregation ok.

Step 5: Execute the Matrix and Preserve Evidence

Append the command-line entry point. It writes one JSON object per line immediately after each call, so partial evidence survives an interrupted run. The --limit option provides a cheap smoke check.

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--limit", type=int, default=None)
    parser.add_argument("--output", default="artifacts/context-results.jsonl")
    args = parser.parse_args()

    cases = make_cases()[:args.limit]
    output = Path(args.output)
    output.parent.mkdir(parents=True, exist_ok=True)
    client = OpenAI()
    results = []
    with output.open("w", encoding="utf-8") as stream:
        for case in cases:
            result = run_case(client, case)
            results.append(result)
            stream.write(json.dumps(asdict(result)) + "\n")
            stream.flush()
            print(case.case_id, result.failure_type, result.latency_ms)

    write_report(results, output.with_suffix(".md"))
    return 0 if all(r.failure_type != "provider_error" for r in results) else 2

if __name__ == "__main__":
    raise SystemExit(main())

Run one case before spending the full matrix budget:

python context_eval.py --limit 1
python -m json.tool artifacts/context-results.jsonl

Then run all 18 cases:

python context_eval.py
sed -n '1,20p' artifacts/context-results.md

Verification: the smoke command should print a case ID, a failure classification, and latency. The JSON object must include the model, expected and actual values, token counts, and prompt hash. The full report must contain nine slice rows. Do not proceed if the first case is a provider error or its API token count exceeds the model limit.

Step 6: Interpret Results Without Overclaiming

A degradation threshold is a product decision, not a universal model property. If your support workflow normally sends 6,000 tokens, failure at 100,000 may be irrelevant. If a legal review workflow depends on evidence placed anywhere in 80,000 tokens, one middle-position miss can be release-critical. Define the supported envelope from production traces, privacy-safe length histograms, and risk.

Compare at least four views:

1. Accuracy by actual input tokens. Target size is only a generator setting; API usage tells you what the provider processed.
2. Accuracy by relative position. Position effects disappear in an overall score when easy edge cases outnumber middle cases.
3. Failure taxonomy. Wrong code, NOT_FOUND, malformed output, refusal, truncation, and provider error suggest different causes. Extend the simple classifier when those outcomes occur.
4. Latency and cost. A model may remain accurate while becoming too slow or expensive for the user journey. Treat these as separate service objectives rather than mixing everything into one quality score.

Do not infer causation from one matrix. Prompt wording, record similarity, tokenizer choice, model revision, and API configuration all affect results. Re-run the same manifest, preserve model and prompt hashes, and change one factor at a time. For disciplined datasets, follow the adversarial RAG evaluation dataset tutorial. When subjective reasoning is tested, calibrate an LLM judge against human labels rather than replacing exact oracles with unverified grading.

Verification: inspect artifacts/context-results.md and confirm every target length contains all three positions. Also confirm completed + provider_errors equals two for every slice.

Step 7: Add a Repeatable Regression Gate

Store an approved baseline summary in version control only after reviewing its case-level evidence. The gate below checks a critical absolute floor and limits degradation relative to the baseline. Create gate_context_eval.py:

import json
import sys
from dataclasses import fields
from pathlib import Path

from context_eval import Result, group_summary

def load_results(path: str) -> list[Result]:
    names = {field.name for field in fields(Result)}
    rows = []
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        value = json.loads(line)
        rows.append(Result(**{key: value[key] for key in names}))
    return rows

def keyed(rows: list[dict]) -> dict[tuple[int, float], dict]:
    return {(r["target_tokens"], r["position"]): r for r in rows}

def main() -> int:
    candidate = keyed(group_summary(load_results(sys.argv[1])))
    baseline = keyed(json.loads(Path(sys.argv[2]).read_text()))
    failures = []
    for key, old in baseline.items():
        new = candidate.get(key)
        if not new or new["completed"] == 0:
            failures.append(f"{key}: no completed results")
            continue
        if new["provider_errors"] > 0:
            failures.append(f"{key}: provider errors present")
        if new["accuracy"] < 0.90:
            failures.append(f"{key}: below absolute floor")
        if old["accuracy"] - new["accuracy"] > 0.05:
            failures.append(f"{key}: dropped more than 0.05")
    print("\n".join(failures) if failures else "context gate passed")
    return 1 if failures else 0

if __name__ == "__main__":
    raise SystemExit(main())

Export baseline-context-summary.json from a reviewed run using group_summary, then execute:

python gate_context_eval.py artifacts/context-results.jsonl baseline-context-summary.json

The illustrative 90% floor requires more repetitions than this tutorial's two. With two cases, possible slice scores are only 0%, 50%, and 100%. Increase repetitions before adopting the gate, select a floor from your workflow risk, and require minimum sample counts. A mature comparison should also pin the dataset revision, model alias or snapshot when available, tokenizer, SDK lockfile, and application revision.

Run this suite before a model, system prompt, retrieval, chunking, or prompt-assembly release. The promptfoo CI evaluation tutorial shows how to preserve reports and distinguish gate failures in a delivery pipeline.

Verification: run the gate once with the candidate summary copied as the baseline and expect context gate passed. Change one baseline slice accuracy from 1.0 to 0.0 only in a temporary test fixture, reverse the candidate and baseline arguments as appropriate, and confirm exit code 1 for a drop.

How to Test LLM Context Window Degradation Beyond Retrieval

Single-record retrieval isolates attention to one fact, but real applications combine capabilities. Add a separate suite for each claim rather than making one case judge everything. A multi-hop case can place a customer ID near the beginning and its entitlement near the end, then require a join. A chronology case can distribute dated events and ask which valid event came last. A conflict case can place stale and current policy with provenance and ask the model to prefer the effective version.

For RAG systems, distinguish the retrieval boundary from the generation boundary. First test whether the retriever returns the answer-bearing chunk. Then inject known-good chunks directly into the generator and run the positional matrix. Finally exercise the complete application. Otherwise, a missing answer could mean retrieval recall failure, context truncation, incorrect ordering, prompt injection, or model utilization failure. The broader AI agent testing guide covers tool calls and multi-step state beyond raw context use.

Use production-shaped distributions without copying personal or confidential data. Match document lengths, markup, languages, table density, duplicated passages, and noise ratios with synthetic content. Test token budgets around actual percentiles and operational boundaries, not only neat powers of two. Add cases just below and just above any application truncation threshold.

Also vary target position in absolute and relative terms. A fact at token 4,000 is 50% of an 8,000-token prompt but only 5% of an 80,000-token prompt. Both representations reveal different mechanisms. Record the number of tokens before the target, within the target, and after it if exact placement matters.

How to Test LLM Context Window Degradation Safely in CI

Keep a tiny deterministic smoke set on pull requests and a statistically stronger matrix on a schedule or release candidate. Long prompts multiply token cost, runtime, and rate-limit exposure. Estimate calls and input tokens before execution, cap concurrency, and stop when provider errors invalidate the run. Never make repeated calls until a preferred answer appears.

CI credentials should have the least available privilege and an explicit spend boundary. Do not upload raw production prompts as artifacts. This tutorial stores synthetic outputs and prompt hashes; an enterprise application may need redaction, access-controlled artifacts, and short retention. Ensure logs do not include SDK exception bodies that might echo request content.

Separate three outcomes in the job: harness invalid, provider unavailable, and behavioral regression. A syntax error or missing baseline means no evaluation occurred. A 429 response means evidence is incomplete. An exact wrong answer means the evaluated behavior failed. Different exit codes and artifact annotations help the right owner respond.

Before trusting CI, seed sentinel fixtures. One should pass, one should contain no target, one should contain a wrong target code, and one should simulate a provider error through a fake client. These prove that scoring, absence behavior, classification, and job status work. Review changes to datasets and baselines like test code, since relaxing an oracle can hide the same regression as changing application logic.

Interview Questions and Answers

The structured interview section below contains model answers for six common questions. In an interview, emphasize experimental control: vary length and position separately, use a deterministic oracle, distinguish behavioral failures from infrastructure failures, and define the supported envelope from product risk. Mention that an advertised context window states an acceptance limit, not uniform task accuracy across every position and workload.

Common Mistakes

• Testing only one maximum-length prompt and missing the shape of the degradation curve.
• Placing evidence only at the end, where recency can make results look stronger.
• Using approximate string similarity when the expected value supports exact scoring.
• Mixing retrieval misses, truncation, provider errors, and wrong answers into one failure count.
• Comparing models with different prompts, datasets, output limits, or repetition counts.
• Treating tokenizer estimates as the API's authoritative billed input count.
• Running too few repetitions while interpreting tiny percentage differences as meaningful.
• Averaging away a critical middle-position failure with many easy short cases.
• Updating the candidate and baseline simultaneously without independent review.
• Publishing sensitive prompts, outputs, or exception text in CI artifacts.
• Assuming longer accepted input means the application should always send more context.
• Using a model judge for a fact that deterministic code can verify exactly.

Troubleshooting

Problem: the API rejects the request as too large -> Lower target_tokens and reserve space for instructions and output. Local tokenizer estimates are planning aids; use API usage and the selected model's current documented limits. Also inspect application wrappers that may add hidden system or tool text.

Problem: every middle-position case returns NOT_FOUND -> Confirm the target is actually present by regenerating the prompt from its seed and searching for both record ID and expected code. Then run the same case at 10% and 90%. A position-specific recovery supports a positional hypothesis, while universal failure suggests prompt or data defects.

Problem: output contains an explanation around the code -> Keep the strict instruction and classify the response as a format failure instead of silently extracting any eight-character string. If production accepts structured output, replace the plain response with a documented JSON Schema output configuration and validate the complete object.

Problem: results change on repeated runs -> Increase repetitions, preserve every output, and calculate uncertainty by slice. Check whether the model alias, retrieval index, prompt builder, or provider configuration changed. Do not choose the best run.

Problem: long cases frequently receive rate limits or timeouts -> Reduce concurrency, request the correct account limits, schedule the broad suite, and apply bounded retries only to documented transient errors. Mark exhausted retries as provider errors, never wrong answers or passes.

Problem: accuracy stays perfect at every tested size -> That is valid evidence for this simple probe, not proof of all long-context reasoning. Increase distractor similarity, add multi-hop and conflict suites, test operational boundaries, and keep the original retrieval probe as a sentinel.

Where To Go Next

Start with the 18-case matrix, inspect every result, and decide which token-position envelope your product promises. Increase repetitions and add production-shaped noise before turning the illustrative threshold into a release gate.

Then extend the system deliberately:

• Use the LLM evaluation pipeline complete guide to connect datasets, metrics, reports, and ownership.
• Build difficult grounded cases with the adversarial RAG dataset tutorial.
• Add automated release evidence through evals in CI with promptfoo.
• Validate subjective graders using LLM judge and human label calibration.
• Expand from context retrieval to tools and state with the AI agent testing guide.

You can also use the QA practice workspace to rehearse how you would explain the experimental design, failure taxonomy, and regression decision in an interview.

Conclusion

A useful context-window test measures reliable task performance, not whether an endpoint accepts a large request. Generate known evidence, sweep length and position, score exact behavior, preserve case-level artifacts, and separate infrastructure errors from model failures. That process reveals where your application's context becomes unreliable and which slices cause the decline.

Treat the curve as a versioned product contract. Re-run it when the model, prompt, retrieval, ordering, truncation, or SDK changes, and expand the suite one capability at a time. That produces actionable regression evidence instead of a context-limit marketing number.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Add CI to a test framework (2026)
• How to Add logging to a test framework (2026)
• How to Add reporting to a test framework (2026)
• How to Use Gatling to Load Test a GraphQL API (2026)
• How to Use k6 to Load Test HTTP3 Endpoints (2026)
• How to Use k6 to Test gRPC Streaming APIs (2026)