Resource library

QA How-To

How to Debug and Fix Flaky Tests

Diagnose and fix flaky tests with repeatable evidence, failure classification, timing and data controls, quarantine rules, and reliability metrics.

1,984 words | Article schema | FAQ schema | Breadcrumb schema

Overview

A flaky test passes and fails against equivalent code, which turns a binary signal into a probability. The immediate cost is reruns, but the larger cost is behavioral: engineers stop trusting red builds, real regressions get labeled as noise, and risky changes merge after someone clicks retry. Treating flakiness as routine maintenance lets that damage compound across the entire delivery system.

This guide presents a forensic workflow for intermittent failures. You will preserve evidence, measure the pattern, classify the likely system boundary, reproduce under controlled variation, replace timing guesses, isolate data, and verify a repair. It also explains quarantine and retry policies that protect the pipeline signal without hiding unresolved product races. The method works across UI, API, and integration suites.

Define Flakiness Precisely

A test is flaky when the same test, code, data contract, and intended environment can produce different outcomes. That definition matters because an unstable external environment, a deliberately changing feature flag, and a product race may all appear as intermittent red. Do not assume the automation is guilty. The failure can reveal nondeterminism in the product, test, infrastructure, dependency, or data lifecycle.

Measure first-attempt outcomes. If a test fails once and passes on retry, the original failure still counts. Track attempts, failures, retries, duration, worker, browser, environment, commit, and failure signature. A test with a 1 percent failure rate looks harmless until it runs across hundreds of builds. Suite reliability is multiplicative, so many mildly flaky tests create a consistently noisy pipeline.

Record the first result separately from the final retried result.
Group failures by exception, assertion, screenshot region, or log signature.
Distinguish test nondeterminism from real intermittent product behavior.
Measure developer delay and rerun volume, not only test count.

Preserve Evidence Before Rerunning

A rerun often destroys the only useful state. Configure the runner to retain the first-failure screenshot, trace, video when interaction timing matters, browser console, network summary, test log, application correlation IDs, and environment metadata. For service tests, capture sanitized request and response details plus downstream trace links. Record the random seed, clock, locale, timezone, worker count, and test-order seed where supported.

Make artifact collection run even when setup or teardown fails. Name artifacts with run and attempt identifiers so a passing retry cannot overwrite them. Avoid unlimited debug logging, which can change timing and bury the event. Prefer structured events around state transitions: request started, response received, element became enabled, record version changed. Redact secrets and customer data before storing evidence.

Capture the failed attempt, not only the final attempt.
Link client actions to server work through correlation IDs.
Record exact browser, runner image, dependencies, and resource allocation.
Preserve seeds and order so randomization can be replayed.

Classify the Failure Before Editing Code

Use a small set of categories to guide investigation: synchronization and timing, shared or stale data, order dependence, unstable selectors, external service variability, resource exhaustion, time and timezone, random generation, environment drift, and genuine product concurrency. Examine where the expected state should have been created and whether that state existed. A missing button could come from slow rendering, a failed setup API, wrong user permission, or a feature flag.

Create a brief evidence statement: On Firefox worker 6, the save request completed with 200 at 14:03:11.422, but the DOM remained disabled until after the five-second assertion timeout. This is more useful than Save test is flaky. Compare several failures for a shared signature. Different symptoms under one test name may require separate issues, owners, and targeted fixes.

Reproduce With Controlled Variation

Run the smallest failing test repeatedly against a fixed build. Then vary one dimension at a time: headless versus headed, single worker versus full parallelism, fast versus throttled CPU, clean versus reused account, one browser versus another, local versus CI container, and isolated versus full-suite order. A command such as `for i in {1..50}; do npm test -- save-profile.spec.ts || break; done` is useful only if every iteration records its seed and state.

Use test-order randomization and repeat-each features to expose hidden coupling. Stress the suspicious boundary instead of only adding repetitions. If polling may race, inject response delay around the threshold. If parallel workers may share a user, deliberately run the test twenty times concurrently. If midnight matters, use a controllable clock. A reproduction harness should increase the probability of the suspected mechanism, not merely hope the failure returns.

Reduce to the narrowest test and fixed application build.
Change one variable, record the result, then restore it.
Amplify the suspected race with scheduling, delay, or concurrency.
Replay the exact seed and test order from a failed run.

Replace Sleeps With Observable Synchronization

Fixed sleeps are guesses about another system's completion time. A two-second delay is wasteful when work finishes in 100 milliseconds and unreliable when it takes 2.1 seconds. Wait for the state the user or service actually requires: a response with the correct correlation ID, a visible confirmation, a disabled button becoming enabled, a job reaching Completed, or a database version changing. Set a bounded timeout and include the last observed state in the failure.

Be precise about what readiness means. Waiting for network idle can hang on analytics or pass before asynchronous UI work finishes. Waiting for an element to exist can pass while an overlay blocks it. Prefer framework assertions that retry the full relevant condition, such as `await expect(page.getByRole('status')).toHaveText('Saved')`. For eventually consistent services, poll with an interval and deadline, then report the observed sequence rather than adding a large global timeout.

Wait on business state, not elapsed time.
Use bounded polling for documented eventual consistency.
Assert actionability, such as visible and enabled, before interaction.
Keep timeouts local to the slow operation and include diagnostics.

Eliminate Shared State and Order Dependence

Each test should create or lease its own mutable data and clean it using returned identifiers. Permanent shared accounts accumulate settings, tokens, carts, and rate limits. Random usernames reduce uniqueness collisions but do not prevent a cleanup job from deleting another worker's record. Add a run namespace and ownership tag, then scope all searches and cleanup to it, including failure recovery jobs.

Reset browser storage, cookies, mocks, feature flags, fake clocks, and global variables according to framework isolation rules. Avoid tests that depend on a previous test creating an order. If setup is expensive, create an immutable template and clone it per test, or perform API setup in each worker. Prove isolation by shuffling order and running the same spec in parallel. A suite that passes only serially contains hidden contracts.

Harden Selectors and Interactions

Selectors should describe stable user semantics or an explicit test contract. Prefer role, accessible name, label, and dedicated test identifiers over CSS position, generated class, or text that changes with data. Still, a stable selector does not guarantee stable interaction. Confirm the intended element is unique, visible, enabled, within the current frame, and not moving under an animation before clicking.

Avoid force-clicking as a generic repair because it bypasses the same conditions a user faces. If animation is cosmetic, disable it in test configuration or wait for the final state. If a virtualized list recycles rows, locate the item after it scrolls into view rather than retaining a stale handle. If the UI rerenders on every keystroke, interact through framework locators that resolve at action time.

Prefer semantic locators and unique accessible names.
Resolve elements close to the action instead of caching DOM handles.
Wait for overlays and motion to end through observable state.
Treat force actions as narrowly justified exceptions.

Investigate Infrastructure and Product Races

If failures cluster on one worker or at peak CI time, inspect CPU throttling, memory pressure, disk, file descriptors, container eviction, DNS, and network errors. Compare runner metrics with test timestamps. Pin tool and browser versions, use lockfiles, and make environment configuration visible. Increasing every timeout may reduce symptoms while preserving an overloaded runner that will fail again as the suite grows.

Product races deserve product fixes. Examples include duplicate submission after a timeout, stale reads from a replica, two updates overwriting each other, or a notification arriving before a subscription is ready. Use server traces, controlled delay, and concurrency tests to demonstrate the ordering. Add idempotency, version checks, atomic operations, or explicit readiness in the product, then keep a regression test that forces the original interleaving.

Use Quarantine and Retries as Controls

Quarantine protects the trusted blocking lane while investigation occurs. A quarantined test should still run, report visibly, and have an owner, failure evidence, issue link, and expiration date. Do not move an entire file when only one scenario is affected. Critical coverage needs a temporary replacement, perhaps an API check or focused manual release step, until the test returns to reliable service.

Retries can gather evidence and reduce interruption, but cap them and expose the first failure. Never let retry success transform the dashboard into an uncomplicated pass. Use different artifacts for each attempt and track retry-pass rate. Remove the retry exception after the root cause is fixed. If a low-value test remains expensive to stabilize, delete it openly and cover the risk at a better layer.

Quarantine narrowly with an owner and deadline.
Keep quarantined results visible and replace critical lost coverage.
Count pass-on-retry as flaky, not clean.
Review stale quarantines and retry rules every sprint.

Verify the Fix and Prevent Recurrence

A single green run does not verify an intermittent fix. Run the reproduction harness enough times to make the previous failure rate meaningfully unlikely, then restore normal concurrency and full-suite order. Compare pre-fix and post-fix first-attempt rates using the same environment. Review nearby tests for the same pattern, because one shared helper or fixture may create a family of flakes.

Track suite first-pass rate, flaky tests introduced and resolved, quarantine age, rerun minutes, and top failure categories. Set a reliability objective for the merge gate and pause new tests when the budget is exceeded. Include deterministic data, clock control, state-based waits, and artifact requirements in framework templates and reviews. Prevention turns each investigation into a platform improvement instead of a one-test patch.

Frequently Asked Questions

What is a flaky test?

A flaky test produces different outcomes against equivalent code and intended conditions. The nondeterminism may come from the test, product, data, environment, dependency, timing, or infrastructure, so the automation should not be blamed without evidence.

What causes flaky automated tests?

Common causes include fixed waits, shared mutable data, test-order dependence, unstable selectors, uncontrolled clocks or randomness, external services, environment drift, constrained runners, and genuine concurrency bugs in the application.

Should flaky tests be retried automatically?

Limited retries can collect evidence and reduce immediate disruption, but the first failure must remain visible and count toward reliability metrics. Retry success does not fix the cause and should not turn the run into an ordinary pass.

How do I reproduce a flaky test locally?

Fix the application build, replay the failed seed and order, and repeatedly run the smallest test while preserving artifacts. Vary one dimension at a time, such as concurrency, CPU, browser, data, delay, or clock, to amplify the suspected mechanism.

Is increasing the test timeout a valid flaky test fix?

Only when the documented operation legitimately needs a larger bounded time and the test waits for the correct state. A global timeout increase usually hides slow readiness, infrastructure contention, or an incorrect synchronization condition.

When should a flaky test be quarantined?

Quarantine it when confirmed intermittency is corrupting a trusted blocking signal and cannot be fixed immediately. Keep it running visibly, assign an owner and deadline, preserve or replace critical coverage, and return it only after repeated clean verification.