Flaky test quarantine in CI: Step by Step (2026)

TL;DR

Implement flaky test quarantine in CI as two lanes: a required job excludes @quarantine, while a visible nonblocking job executes only @quarantine tests. Back every entry with evidence, an issue, owner, narrow scope, and expiry; validate that registry in the required job and remove the exception after root cause is fixed.

Key Takeaways

• Quarantine changes CI routing temporarily, it does not skip execution or redefine a product failure as acceptable.
• Require equivalent-run evidence, a linked issue, an owner, narrow scope, a reason, and an expiration date before admission.
• Use Playwright @quarantine tags with separate required and allowed-failure jobs.
• Fail governance validation when a quarantine entry expires even though the quarantined test itself is nonblocking.
• Keep retry-only passes visible and preserve JUnit, traces, screenshots, and environment context for diagnosis.
• Measure active count, age, additions, removals, execution rate, and escapes instead of celebrating pass rate alone.
• Remove the tag, annotations, and registry entry together after the fix is proven in the affected conditions.

A flaky test quarantine in CI is a temporary routing mechanism that removes a proven nondeterministic test from the required quality gate while continuing to execute it in a visible, owned, nonblocking job. A real quarantine needs an issue, evidence, an owner, the smallest affected scope, an expiration date, preserved diagnostics, and an explicit exit path.

Quarantine is not test.skip(), unlimited retries, or permission to merge a product regression. It is a controlled exception that protects confidence in the required suite while the team fixes root cause. This guide implements the pattern with current Playwright tags and GitHub Actions, then shows how to govern it across frameworks.

TL;DR

The core execution is simple: the required job runs npx playwright test --grep-invert @quarantine, while a separate job runs npx playwright test --grep @quarantine under a platform-native nonblocking policy. Governance is what prevents that split from becoming permanent test deletion.

1. Define Flaky Test Quarantine in CI Precisely

A flaky test produces different outcomes without a relevant change to the code, configuration, data, or environment being evaluated. The word "relevant" matters. If one run points at a different service version or shares data with another test, inconsistent results may be deterministic consequences of uncontrolled inputs rather than random framework behavior.

Quarantine changes routing, not truth. The test leaves the required merge gate but continues to run on the same commit and relevant environment. Its failures remain visible, artifacts remain accessible, and a team remains accountable. The required suite becomes more credible because known nondeterminism no longer creates repeated false alarms. The quarantine lane provides evidence for repair.

A quarantine is valid only when all of these are true:

• The team has captured at least two conflicting outcomes under an equivalent test contract or has other strong nondeterministic evidence.
• A product regression has been ruled out or separately accepted by the responsible product owner.
• The affected test, browser, environment, or data scope is identified narrowly.
• An issue records evidence, suspected cause, owner, and next investigation step.
• An expiration date forces review.
• The test still executes on a useful cadence with diagnostics.

Do not quarantine critical security, payment, or data-integrity coverage merely because it is inconvenient. If the product risk cannot be left unblocked, the gate must remain red or the release decision needs explicit human ownership.

For foundational prevention techniques, read how to fix flaky automation tests.

2. Separate Quarantine, Retry, Skip, and Expected Failure

Teams often use these mechanisms interchangeably, but they communicate different facts.

Retries are useful for collecting a trace and measuring retry-only passes. They are dangerous when a green final status hides instability. Playwright can report a test as flaky when it fails first and passes on retry, and its failOnFlakyTests configuration can make such outcomes fail CI. A team adopting quarantine can use this signal to discover candidates, but it should investigate before automatically moving them.

test.skip() and test.fixme() do not execute the body, so they cannot show whether the defect disappeared. test.fail() runs the test with expected-failure semantics and complains if it unexpectedly passes. That can be appropriate for an accepted product state, but it is not a substitute for an isolated nonblocking lane with ownership and trend evidence.

Choose the mechanism that communicates reality. Mislabeling a consistent defect as flaky erases useful product signal.

3. Establish an Evidence-Based Admission Policy

Quarantine should require a lightweight decision record, not an informal chat message. Capture the failing run, passing run, commit SHA, runner image, browser and version, target deployment, worker and shard context, data identity, trace or log, and the observed assertion. Compare inputs before concluding that outcomes conflict.

Use a triage decision tree:

1. Did the application behavior violate the requirement consistently when reproduced? Treat it as a product failure.
2. Did the test use an unstable locator, fixed sleep, missing await, or non-retrying assertion? Treat it as a test defect and fix immediately if small.
3. Did parallel runs collide on shared data? Repair isolation or quarantine only the exact scope while doing so.
4. Did the environment fail readiness, network, capacity, or dependency checks? Route to the environment owner.
5. Is the inconsistency still unexplained after equivalent reruns and diagnostics? Admit a time-bounded quarantine with an investigation hypothesis.

The burden is not to prove cosmic randomness. It is to show that removing the test from the gate is safer than repeated false failures and that product risk is covered another way. A manual verification, lower-level test, feature flag, or narrower stable scenario may provide temporary compensating coverage.

Require reviewer approval from the test owner or service owner. The pull request should link the issue and show the exact selection rule. Quarantining a whole file because one browser-specific case is unstable is too broad.

4. Create a Quarantine Registry With Expiration

Tags control execution, but a registry controls governance. Store a reviewed file such as test-governance/quarantine.json:

[
  {
    "testId": "tests/checkout.spec.ts::submits a card payment",
    "issue": "QA-1842",
    "owner": "checkout-quality",
    "reason": "Payment sandbox callback sometimes arrives after the bounded UI wait",
    "scope": "chromium on staging",
    "quarantinedOn": "2026-07-13",
    "expiresOn": "2026-07-27"
  }
]

The dates above illustrate a 14-day review window, not a universal standard. Choose a short service-level objective that matches team cadence and product risk. Renew only with new evidence, an updated hypothesis, and reviewer approval.

Validate expiration in CI with a small Node script:

import { readFileSync } from 'node:fs';

const entries = JSON.parse(
  readFileSync('test-governance/quarantine.json', 'utf8')
);
const today = new Date().toISOString().slice(0, 10);
const required = [
  'testId',
  'issue',
  'owner',
  'reason',
  'scope',
  'quarantinedOn',
  'expiresOn'
];

const errors = [];

for (const [index, entry] of entries.entries()) {
  for (const field of required) {
    if (!entry[field]) {
      errors.push(`entry ${index} is missing ${field}`);
    }
  }

  if (entry.expiresOn && entry.expiresOn < today) {
    errors.push(
      `${entry.testId ?? `entry ${index}`} expired on ${entry.expiresOn}`
    );
  }
}

if (errors.length > 0) {
  console.error(errors.join('\n'));
  process.exit(1);
}

console.log(`Validated ${entries.length} quarantine entries`);

Save it as scripts/validate-quarantine.mjs and run it in the required job before test selection. It uses current Node standard-library APIs and exits nonzero when policy is invalid.

A mature validator also checks duplicate IDs, ISO date format, allowed owner names, issue URL format, and agreement between tagged tests and registry entries. Build those checks from repository conventions rather than parsing source with fragile regular expressions.

5. Tag Playwright Tests Without Hiding Them

Playwright tags begin with @ and can be selected with --grep or excluded with --grep-invert. Attach a quarantine tag and useful annotations to the smallest test or describe block:

import { test, expect } from '@playwright/test';

test('submits a card payment', {
  tag: ['@checkout', '@quarantine'],
  annotation: [
    {
      type: 'issue',
      description: 'QA-1842'
    },
    {
      type: 'quarantine-expires',
      description: '2026-07-27'
    }
  ]
}, async ({ page }) => {
  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Pay now' }).click();
  await expect(page.getByRole('heading', {
    name: 'Payment confirmed'
  })).toBeVisible();
});

The tag is the executable selector. Annotations put issue and expiry context in the Playwright report. The registry remains the policy source because it can be reviewed, validated, and summarized without executing the suite.

Run the required scope:

npx playwright test --grep-invert @quarantine

Run the quarantine scope:

npx playwright test --grep @quarantine

If only Chromium is affected, do not quarantine Firefox and WebKit. Use a project-specific quarantine tag or a conditional describe structure that keeps stable configurations blocking. Avoid title substring conventions such as [FLAKY] because they mix human labels with selection logic and are easy to mistype.

Do not use test.fixme() for quarantine. It prevents execution and removes the evidence needed to know whether the test is still unstable or repaired.

6. Implement Flaky Test Quarantine in CI as Two Jobs

The following GitHub Actions workflow uses current action majors and two explicit jobs. The required job validates governance and excludes quarantine. The quarantine job executes only quarantined tests and is allowed to fail without failing the workflow.

name: browser-tests

on:
  pull_request:
  push:
    branches:
      - main

permissions:
  contents: read

jobs:
  required:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: '22'
          cache: npm
      - run: npm ci
      - run: node scripts/validate-quarantine.mjs
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep-invert @quarantine
      - name: Upload required diagnostics
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: required-playwright-report
          path: |
            playwright-report
            test-results
          if-no-files-found: warn
          retention-days: 14

  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: '22'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep @quarantine
      - name: Upload quarantine diagnostics
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: quarantine-playwright-report
          path: |
            playwright-report
            test-results
          if-no-files-found: warn
          retention-days: 14

Make only the required job a required branch check. Job-level continue-on-error keeps the quarantine failure visible while allowing the overall workflow policy to continue. If your organization prefers a separate workflow for nonblocking tests, retain the same selectors and reporting contract.

A suite with zero quarantined tests may cause the runner to return a no-tests status. Either schedule the quarantine job only when the validated registry is nonempty or handle that exact no-tests condition explicitly. Never swallow every exit code, because configuration and browser-launch failures still need visibility.

7. Preserve Failure Signal and Diagnostic Quality

A nonblocking job can still become invisible if it has no alert, report, or owner. Keep its actual test step red inside the allowed-failure job. Publish JUnit and Playwright HTML output, traces on retry, screenshots on failure, and the registry as an artifact or summary.

Configure Playwright to surface retry-only passes:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  failOnFlakyTests: Boolean(process.env.CI),
  reporter: [
    ['line'],
    ['junit', { outputFile: 'test-results/junit.xml' }],
    ['html', { outputFolder: 'playwright-report', open: 'never' }]
  ],
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure'
  }
});

For the required job, failOnFlakyTests prevents a retry-only pass from silently becoming acceptable. For a quarantine lane, you may keep it enabled because the job itself is allowed to fail. That records instability while preserving workflow progress.

Route quarantine failures to the registered owner through the organization's normal alerting or issue automation. Avoid creating a new defect on every failed run. Update one canonical issue with links or a summarized cadence so evidence remains coherent.

Artifacts can contain sensitive DOM text, network payloads, and screenshots. Use synthetic accounts and limited retention. Diagnostic quality is not measured by the volume of files; it is measured by whether the owner can distinguish synchronization, data, product, environment, and runner failures.

8. Automate Reviews, Expiration, and Ownership

The required job should fail when a registry entry expires, even though the quarantined test itself is nonblocking. This changes the enforcement question from "Did the flaky test happen to pass today?" to "Has the team reviewed this exception by the agreed date?"

Add a scheduled governance workflow that:

• Validates every registry entry.
• Lists entries by owner and age.
• Detects issues that are closed while the tag remains.
• Detects tagged tests missing from the registry.
• Detects registry entries whose test ID no longer exists.
• Sends one actionable summary to owners.
• Reports entries added, removed, renewed, and expired.

Use repository-native ownership such as CODEOWNERS, a team catalog, or a validated owner list. Free-text names become stale. The issue should contain a hypothesis and next experiment, not only "test is flaky."

Renewal should be a pull request that updates the date, explains work completed, links fresh evidence, and receives approval. Automatic renewal defeats expiration. If a test remains quarantined through several review cycles, escalate to the engineering manager or product risk owner and consider replacing the scenario with a more stable coverage layer.

Keep the registry small enough to read in review. Once it becomes large, generate a dashboard from it but retain the repository file as the reviewed source of policy.

9. Measure Quarantine Health Without Gaming Pass Rate

A quarantine program needs measures that reward removal and honest signal:

Do not celebrate a higher required-suite pass rate if the team achieved it by moving many tests out. Pair gate stability with quarantine inventory, coverage risk, and escape analysis.

Avoid one global flaky percentage without context. A test that fails only on WebKit carries a different repair path from a test that collides under parallel data. Segment by owner, capability, environment, cause category, and age.

Publish trends to the same engineering review that examines build time and production defects. Quarantine is a delivery-system control, not a private QA cleanup list. Product and platform teams often own the underlying dependency or environment issue.

The goal is not zero observed flakiness at any cost. The goal is rapid detection, honest routing, root-cause repair, and a required gate that engineers trust.

10. Repair Root Cause and Exit Quarantine Safely

Start repair from the strongest evidence. Playwright traces show action timing, locator resolution, DOM snapshots, console output, and network activity. Compare a failing run with a passing run and identify the first meaningful divergence.

Common repair categories include:

• Synchronization: replace fixed sleeps with web-first assertions or event waits.
• Locators: prefer role, label, text, or stable test contracts over brittle CSS chains.
• Data: create unique records per test and clean them independently.
• Environment: wait for readiness and isolate external dependency failures.
• Concurrency: remove shared mutable accounts, files, queues, or database rows.
• Product race: fix the application and keep a regression test.
• Resource pressure: bound workers and provision capacity instead of extending every timeout.

After the fix, remove the tag, annotation, and registry entry in the same pull request. Run the test repeatedly across the affected browser, shard, and environment, then let it rejoin the required gate. There is no universal repetition count. Choose evidence proportional to observed frequency and business risk.

Watch the test after reintegration. A recurrence should reopen or link the issue and revisit the hypothesis, not automatically renew the old quarantine. The first fix may have treated a symptom.

Use Playwright web-first assertions and test data management in CI for two frequent root-cause areas.

Interview Questions and Answers

Q: What is the purpose of flaky test quarantine?

It temporarily removes proven nondeterminism from the required gate while continuing to execute the test with ownership and diagnostics. It protects trust in CI without deleting coverage. The exception must expire and lead to repair.

Q: How is quarantine different from retry?

A retry repeats execution inside the same run and may reveal a retry-only pass. Quarantine routes the test to a separate nonblocking lane across runs. Retries collect evidence, while quarantine changes gate policy under explicit governance.

Q: What evidence is required before quarantine?

I compare failing and passing runs under equivalent commit, environment, browser, data, and concurrency inputs. I preserve the trace, log, assertion, and run links and rule out a consistent product defect. The issue records the suspected cause and next experiment.

Q: Why should the quarantined test keep running?

Continued execution reveals failure patterns, captures evidence, and shows when a fix may have worked. A skip cannot provide that signal. Execution also prevents the test from silently rotting while dependencies and product behavior change.

Q: How do you prevent quarantine from becoming permanent?

Use a reviewed registry with owner, issue, reason, scope, admission date, and expiry. Fail the required governance validation on expiry. Require new evidence and approval for renewal, and report inventory and age to engineering leadership.

Q: How should CI represent the two lanes?

The required job excludes the quarantine tag and remains a branch check. A separate job selects the tag, retains its real failures and artifacts, but uses the CI platform's allowed-failure policy. Only the required job protects merge.

Q: Can a product bug be quarantined?

Not merely because it fails consistently. A product defect should remain blocking when its risk is unacceptable. If the organization explicitly accepts the product risk, expected-failure or release-risk policy should record that decision separately from flaky-test quarantine.

Q: What metrics show that quarantine is healthy?

I track active count, age, additions versus removals, expired entries, execution rate, retry-only passes, and escaped product defects. A higher gate pass rate alone is misleading because it can be improved by excluding more tests.

The structured interviewQnA field below provides model answers. In an interview, emphasize that quarantine is a governed risk control, not a framework annotation.

Common Mistakes

• Quarantining after one unexplained failure without comparing inputs.
• Moving a consistent product defect out of the required gate.
• Using test.skip() or test.fixme() and calling it quarantine.
• Adding unlimited retries until the final status becomes green.
• Quarantining an entire file or browser matrix for one narrow failure.
• Omitting the issue, owner, expiry, or repair hypothesis.
• Letting the quarantine job swallow every exit code and appear healthy.
• Failing to upload traces, JUnit, screenshots, and environment context.
• Allowing expired registry entries to remain without a policy failure.
• Auto-renewing quarantine dates.
• Counting a rising required-suite pass rate without reporting excluded coverage.
• Removing the tag but leaving stale registry or annotation data.
• Returning a test to the gate after one lucky pass.
• Retaining sensitive browser artifacts longer than necessary.

Conclusion

Flaky test quarantine in CI succeeds when it is narrow, visible, temporary, and enforceable. Use tags to split execution, a required registry validator to enforce ownership and expiry, a nonblocking job that still shows real failures, and diagnostics that support a specific repair hypothesis. Never let the mechanism hide a product defect or silently stop execution.

Implement the registry and two-job workflow with one known candidate, then intentionally expire its entry in a branch. If CI blocks the stale policy, the quarantine lane reports failures, and the required suite remains trustworthy, the control is working. The final measure of success is removing the tag after root cause is fixed.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Parallel test sharding in CI: Step by Step (2026)
• Allure report in CI: Step by Step (2026)
• Azure DevOps test pipelines: Step by Step (2026)
• CircleCI for test automation: Step by Step (2026)
• GitLab CI for test automation: Step by Step (2026)
• Grafana dashboards for test metrics: Step by Step (2026)