How to Reduce flaky tests in a CI pipeline (2026)

TL;DR

To reduce flaky tests in a CI pipeline, measure flakiness by test identity, fix non-determinism in waits/data/network first, use tiny retry budgets with visible attempts, and quarantine only with owners and expiries.

Key Takeaways

• Measure flip rate and retry rescue rate before adding more retries.
• Fix timing, data isolation, and network variance ahead of tooling changes.
• Keep retry budgets small and preserve attempt history as flaky, not clean pass.
• Quarantine only with owner, ticket, expiry, and non-blocking execution.
• Pin environment contracts: OS image, browser, timezone, flags, and secrets.
• Rank flakes by CI-minute impact so the team fixes the costliest noise first.
• Split PR gates (deterministic) from nightly real-integration suites.

Flaky tests erode trust in continuous integration. When a pipeline turns red for reasons unrelated to the change under review, teams start ignoring failures, re-running builds blindly, and shipping risk. Learning how to reduce flaky tests in a CI pipeline is therefore not a cosmetic quality metric; it is a reliability and culture problem that directly affects release speed.

This guide is a practical playbook for QA and SDET teams in 2026. You will classify flake sources, instrument detection, fix root causes before adding retries, design quarantine without hiding debt, and wire CI so flakiness becomes visible and owned. The examples use Playwright, GitHub Actions, and common reporting patterns, but the principles apply to Cypress, Selenium, WebdriverIO, and most other runners.

TL;DR

If you only do three things this week: stop global retry as a default, log attempt history, and track pass rate by test identity across the last 50 runs.

1. How to reduce flaky tests in a CI pipeline: definitions

A flaky test is a test that produces both pass and fail results without meaningful changes to the product code under test, the test code, or the intended environment contract. Intermittent failures caused by a real race in production code are still defects; they are not "just flaky tests." The operational definition that matters in CI is: same commit SHA, same suite, different outcome.

Teams often mislabel three different problems as flake:

1. Environment drift: different runners, browser versions, time zones, or secrets.
2. Test non-determinism: shared mutable data, hard sleeps, unordered assertions, race-prone selectors.
3. Product race conditions: true concurrency bugs that only surface under load or timing variance.

How to reduce flaky tests in a CI pipeline starts with honest classification. If the product is racing, fixing the assertion timeout only masks a production incident waiting to happen. If the runner is under-provisioned, no amount of locator cleanup will stabilize the suite.

Build a shared vocabulary with developers: flaky means non-deterministic under fixed inputs; defective means deterministic failure under fixed inputs; unstable infra means the fixed inputs were never actually fixed. That language alone cuts triage time in standups.

2. Measure flakiness before you "fix" anything

You cannot manage what you do not measure. Instrument every CI run so each test identity records:

• suite name and full test title (stable, no timestamps)
• commit SHA, branch, and job attempt number
• attempt index within retries
• final status and duration
• browser, OS, node version, and shard id
• failure message fingerprint (first meaningful stack frame)

A minimal JSON lines log after each job works well:

// scripts/record-result.ts
import { appendFileSync } from "node:fs";

type Attempt = {
  testId: string;
  sha: string;
  attempt: number;
  status: "passed" | "failed" | "skipped" | "timedOut";
  durationMs: number;
  errorFingerprint?: string;
  shard?: string;
};

export function recordAttempt(row: Attempt) {
  appendFileSync("flake-history.ndjson", JSON.stringify(row) + "\n");
}

Aggregate over a rolling window (for example, the last 50 non-canceled runs on main and on PRs separately). Compute:

• Pass rate = passes / (passes + fails)
• Flip rate = transitions between pass and fail on consecutive runs of the same identity
• Retry rescue rate = fails on attempt 1 that pass on a later attempt

Tests with pass rate between roughly 0.5 and 0.99 over many runs are flake candidates. Tests that always fail are broken, not flaky. Tests that always pass need no rescue. Prioritize high flip rate plus high runtime cost, because those burn the most CI minutes and human attention.

Publish a simple weekly table in Slack or your dashboard: top 20 flaky tests by CI-minute impact. Visibility creates ownership.

3. Build a root-cause map of common flake sources

Use a cause taxonomy so every flake ticket maps to one primary bucket:

When you open a flake, attach evidence: screenshot, trace, network HAR if useful, and the fingerprint. Prefer one root cause over a laundry list of "could be" theories. For AI-assisted triage patterns, see AI for flaky test root cause analysis.

4. Stabilize selectors, waits, and assertions (UI)

UI flakes are dominated by race conditions between the application and the test. Replace fixed sleeps with condition-based waiting and stable locators.

Playwright example (current API style):

import { test, expect } from "@playwright/test";

test("checkout shows confirmation after place order", async ({ page }) => {
  await page.goto("/checkout");
  await page.getByTestId("email").fill("buyer@example.com");
  await page.getByRole("button", { name: "Place order" }).click();

  // Wait for a stable business outcome, not a fixed timer.
  await expect(page.getByRole("heading", { name: "Order confirmed" })).toBeVisible();
  await expect(page.getByTestId("order-id")).toHaveText(/ORD-\d+/);
});

Anti-patterns to remove:

// Bad: arbitrary sleep hides real readiness
await page.waitForTimeout(5000);

// Bad: brittle CSS that changes with design tweaks
await page.locator("div.card > span:nth-child(2)").click();

Prefer role, label, and test id locators. Assert on user-visible outcomes. If an animation blinds the click, wait for the element that signals readiness (spinner gone, network idle only when justified, or a specific response). Document intentional networkidle usage; it is often overused and itself flaky under analytics scripts.

For Cypress suites, the same principle applies with command retry-ability: assert state, do not chain arbitrary delays. Related patterns are covered in Cypress handling flaky tests.

5. Isolate data, accounts, and environment contracts

Shared mutable data is the second largest flake factory after timing. Each test (or at least each worker) needs:

• unique primary keys and emails (deterministic seed + worker index)
• isolated auth sessions
• cleanup that does not depend on the previous test having passed
• a frozen clock when the domain is time-sensitive

Example deterministic identity:

function userEmail(testInfo: { workerIndex: number; title: string }) {
  const slug = testInfo.title.toLowerCase().replace(/\W+/g, "-").slice(0, 40);
  return `e2e.w${testInfo.workerIndex}.${slug}@example.test`;
}

Environment contract checklist for CI:

• pin Node, browser, and OS images
• set TZ=UTC and LANG=C.UTF-8 unless locale is under test
• inject secrets only via CI secret stores
• use a known seed database or API fixtures, not shared staging full of human data
• fail fast if required env vars are missing

If two tests mutate the same "admin" user, they will flake under parallel load. Prefer API setup hooks over UI setup when the UI path is not the behavior under test. That reduces surface area and runtime.

6. Control network, third parties, and feature flags

Outbound dependencies inject variance: payment sandboxes, maps, chat widgets, feature flag evaluation, A/B assignment, and CDN cache state. Strategies ranked by reliability:

1. Stub at the network layer for pure UI behavior tests.
2. Contract tests against partner mocks for integration boundaries.
3. Narrow real calls only in a small, monitored smoke suite.

Playwright route example:

await page.route("**/api/payments/authorize", async (route) => {
  await route.fulfill({
    status: 200,
    contentType: "application/json",
    body: JSON.stringify({ status: "authorized", id: "pay_test_1" }),
  });
});

Feature flags should be explicit in CI. Force a known flag set per suite so experiments do not randomly change copy, layout, or flow. Record the flag snapshot in the run metadata so a failure is reproducible.

Do not stub so aggressively that you never test real integrations. Split suites: deterministic PR gate with stubs, nightly integration with real sandboxes and higher tolerance for infra noise.

7. Design retries, re-runs, and flake detection in CI

Retries are a safety net, not a strategy. Unlimited or silent retries train the organization to accept chaos.

Recommended policy:

Playwright config sketch:

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  reporter: [["list"], ["json", { outputFile: "test-results.json" }]],
  use: {
    trace: "on-first-retry",
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
});

GitHub Actions pattern for evidence:

name: e2e
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test
        env:
          TZ: UTC
          CI: "true"
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: |
            playwright-report
            test-results
            flake-history.ndjson

A test that fails once and passes on retry must be labeled flaky in the report, not greenwashed into a clean pass. That distinction is how you reduce flaky tests in a CI pipeline over months instead of weeks of theater.

8. Quarantine without normalizing failure

Quarantine (skip or move to a non-blocking job) is a controlled firebreak. Done poorly, it becomes a junk drawer where broken tests go to die.

Healthy quarantine rules:

1. Every quarantined test has an owner, ticket, and expiry date.
2. Quarantine is automatic only after a threshold (for example, flip rate above X for N days).
3. Quarantined tests still run in a non-blocking "flake" job so you see if they healed.
4. Expiry without fix re-enables the test or deletes it with review.
5. Quarantine count is a team KPI, not a personal shame metric.

Example tag-based skip:

test("search suggests popular terms", { tag: "@quarantine" }, async ({ page }) => {
  // body unchanged; CI config can skip @quarantine on the blocking job
});

Blocking job:

npx playwright test --grep-invert @quarantine

Non-blocking job:

npx playwright test --grep @quarantine || true

Prefer "run and ignore status" with visibility over permanent skip. Permanent skips rot. For process detail, see flaky test quarantine in CI.

9. Parallelism, sharding, and resource isolation

Parallel execution multiplies flake surface if tests share ports, files, browsers, or accounts. Practices that help:

• unique base ports per worker
• separate browser contexts (default in Playwright)
• no shared filesystem temp names without worker index
• shard by timing, not by file count alone, to avoid overloaded shards
• cap workers based on CPU and memory, not "as high as possible"

Playwright sharding in CI:

strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npx playwright test --shard=${{ matrix.shard }}/4

Merge reports carefully so flake history still attributes the same test identity across shards. If two shards write the same artifact path, you will corrupt evidence and create "mystery" flakes.

Load on the system under test also matters. Four workers hammering a tiny review app can create product timeouts that look like test flakes. Scale the environment or throttle concurrency.

10. A practical 30-day reduction program

Week 1: Instrument history, publish top flake list, disable global multi-retry if it is greater than 1.

Week 2: Fix the top 10 by CI-minute impact. Prefer data isolation and wait fixes over rewrites.

Week 3: Introduce quarantine with owners and expiries. Split PR gate vs nightly integration.

Week 4: Add flake budgets (for example, new tests must stay above 99% pass rate for 14 days) and a dashboard. Document the environment contract in the repo README.

Governance that sticks:

• flake budget reviewed in the same meeting as deploy metrics
• "no new sleep" lint or code review checklist item
• require traces on retry for UI suites
• delete tests that assert nothing useful

Celebrate hard green, not lucky green. A suite that is green only after three re-runs is still red for trust purposes.

11. How to reduce flaky tests in a CI pipeline: decision checklist

Use this checklist when a failure is intermittent:

1. Can I reproduce on the same SHA locally with the same env contract?
2. Does the failure fingerprint match a known bucket?
3. Is product behavior wrong under concurrency?
4. Is the test asserting a non-deterministic order or time?
5. Did a dependency or flag change without a pin?
6. Is retry the only "fix" proposed? If yes, reject until a root cause is named.
7. Should this test move to a lower layer (unit/API) where it is deterministic?

Escalate product races to developers with a minimal reproduction. Escalate infra to platform with runner metrics. Keep test-only fixes in the QA backlog with SLAs.

12. Tooling and reporting patterns that keep flakes visible

Dashboards only help if the underlying events are trustworthy. Wire your reporter so each logical test keeps a stable id across retries, shards, and re-runs of the same commit. Export JUnit or JSON for the CI host, and keep a richer internal format for flake analytics.

Practical reporting rules:

• Never overwrite attempt 1 with attempt 2 in storage.
• Include shard id and worker index as dimensions, not as part of the test name.
• Fingerprint errors by normalizing dynamic ids (UUIDs, timestamps) so the same bug clusters.
• Surface "flaky pass" as its own status in PR checks when your platform allows check annotations.

Example post-processing sketch after Playwright JSON output:

// scripts/summarize-flakes.ts
import { readFileSync, writeFileSync } from "node:fs";

type Spec = {
  title: string;
  ok: boolean;
  tests: { results: { status: string }[] }[];
};

const report = JSON.parse(readFileSync("test-results.json", "utf8"));
const flaky: string[] = [];

for (const suite of report.suites ?? []) {
  for (const spec of suite.specs ?? []) {
    const statuses = (spec.tests ?? []).flatMap((t: any) =>
      (t.results ?? []).map((r: any) => r.status)
    );
    const hadFail = statuses.some((s: string) => s === "failed" || s === "timedOut");
    const hadPass = statuses.some((s: string) => s === "passed");
    if (hadFail && hadPass) flaky.push(spec.title);
  }
}

writeFileSync("flaky-this-run.json", JSON.stringify({ flaky }, null, 2));
if (flaky.length) {
  console.log(`Flaky passes detected: ${flaky.length}`);
  for (const t of flaky) console.log(` - ${t}`);
}

Fail the job on unresolved hard failures, but still print flaky passes so reviewers see residual risk. Introduce ML-based flake detection only after you have clean historical labels; models amplify garbage labels.

13. Team habits that permanently lower flake rate

Process beats heroics. Bake flake hygiene into the Definition of Done:

1. New UI tests must use condition waits and isolated data helpers from a shared library.
2. PR authors re-run a failed job at most once; a second failure opens a ticket instead of endless re-runs.
3. On-call for the test suite rotates; the person triages new flakes within one business day.
4. Delete or rewrite tests that only assert framework noise (for example, pure styling without user risk).
5. Review CI minute cost monthly the same way you review cloud spend.

When leadership asks for "more automation," answer with "more reliable automation." A smaller green suite that blocks on real defects beats a huge red suite that everyone ignores. That cultural stance is the durable answer to how to reduce flaky tests in a CI pipeline, not a single tool purchase.

Interview Questions and Answers

Q: How do you define a flaky test?

A flaky test yields both pass and fail outcomes for the same commit and intended environment without intentional code changes. I separate true non-determinism from broken tests and from environment drift, because each needs a different fix.

Q: What is your first step when the CI suite becomes noisy?

I measure: collect per-test history, flip rate, and retry rescue rate, then rank by CI-minute impact. Without ranking, teams thrash on the most recent failure instead of the most expensive one.

Q: Why are retries alone a bad strategy?

Retries convert red builds to green without removing root causes. They also hide product races and inflate CI cost. I allow at most a small retry budget and require attempt history to remain visible as flaky, not as a clean pass.

Q: How do you prevent data-related flakes under parallel runs?

I isolate identities per worker, use deterministic unique keys, avoid shared mutable admin accounts, and clean up independently of pass/fail. Shared staging data is treated as a last resort, not the default.

Q: How should quarantine work?

Quarantine is temporary, owned, time-boxed, and still executed in a non-blocking job. Permanent silent skips create dead code and false confidence.

Q: How do you handle third-party flakiness?

I stub network calls for PR UI gates, keep a small real-integration suite on a schedule, and track partner sandbox health separately so product PRs are not blocked by vendor noise.

Q: What metrics prove flakiness is improving?

Declining flip rate, declining retry rescue rate, fewer quarantined tests, and higher trust (fewer manual re-runs). Green rate alone is insufficient if retries are masking failures.

Common Mistakes

• Treating every intermittent failure as a test problem when the product has a race.
• Setting retries: 3 globally and declaring victory when CI turns green.
• Using waitForTimeout as the default synchronization tool.
• Sharing one test user across parallel workers.
• Quarantining without owners, tickets, or expiry dates.
• Comparing flake rates across different browser matrices without normalizing.
• Uploading artifacts only on success, which deletes the evidence you need.
• Ignoring timezone and locale differences between laptops and CI images.
• Letting analytics and chat widgets hit real endpoints in every UI test.
• Deleting flaky tests without checking whether they guarded a real risk.

Conclusion

Knowing how to reduce flaky tests in a CI pipeline is a systems skill: measure honestly, fix root causes, limit retries, quarantine with discipline, and keep environment contracts explicit. The goal is not a perfectly silent suite forever; it is a suite that fails for real reasons and that the team still trusts enough to block releases.

Start this week with history instrumentation and a top-10 impact list. Fix or quarantine with owners, publish attempt-aware reports, and keep going until green means green. For related CI setup patterns, continue with how to add CI to a test framework and parallel test sharding in CI.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Add CI to a test framework (2026)
• How to Debug a failing test in VS Code in Cypress (2026)
• How to Debug a failing test in VS Code in Playwright (2026)
• How to Debug a failing test in VS Code in Selenium (2026)
• How to Run tests in headed mode in Cypress (2026)
• How to Run tests in headed mode in Playwright (2026)