Quarantine Flaky Tests in a CI Pipeline

TL;DR

Quarantine flaky tests by moving them into a separate, non-blocking CI job while preserving execution, reports, traces, ownership, and expiry. Keep the trusted suite as the required merge check, and automatically reject stale quarantine records.

Key Takeaways

• Quarantine changes a test's release-gating status, not its visibility or ownership.
• Keep quarantined test IDs in a reviewed manifest with an owner, issue, reason, and expiry date.
• Run healthy and quarantined tests in separate CI jobs so only trusted failures block merges.
• Upload reports and traces from both jobs, even when a quarantined test fails.
• Fail CI when quarantine records expire or reference missing tests.
• Use retries as diagnostic evidence, not as a substitute for quarantine governance.

A safe quarantine flaky tests CI pipeline workflow separates release-gating status from test execution. Trusted tests stay blocking, while quarantined tests run in a visible, non-blocking job that always publishes evidence and points to an owner and repair issue.

This tutorial builds that workflow with Playwright, TypeScript, and GitHub Actions. It fits into the broader Test Automation CI/CD Complete Guide for 2026, which explains how test selection, credentials, environments, evidence, and deployment gates work together.

You will not silence failures, add unlimited retries, or delete unstable coverage. You will create a temporary control that protects delivery while keeping every flaky test measurable and accountable.

What You Will Build

By the end, you will have:

• A reviewed JSON manifest containing each quarantined test ID, owner, issue, reason, and expiry date.
• A small TypeScript selector that converts the manifest into a Playwright --grep-invert or --grep pattern.
• Separate trusted and quarantine scripts that execute complementary test sets.
• A GitHub Actions workflow where trusted failures block a pull request and quarantine failures remain visible without blocking it.
• HTML reports, JUnit XML, traces, and logs retained as CI artifacts.
• A policy check that fails when quarantine entries expire, duplicate IDs, or no longer match a test.

The design is portable. GitLab CI, CircleCI, Jenkins, and Azure Pipelines can use the same manifest and npm scripts. Only the CI orchestration syntax changes.

Prerequisites

Use Node.js 22 LTS or a later supported LTS, npm, Git, and a GitHub repository with Actions enabled. The sample uses @playwright/test and TypeScript. Install them in an existing project:

npm install --save-dev @playwright/test typescript tsx
npx playwright install --with-deps chromium

Create these directories if they do not exist:

mkdir -p tests scripts .github/workflows test-results

You also need permission to edit repository branch protection or rulesets if the trusted job will become a required status check. Do that only after the workflow succeeds on a branch.

Use a unique marker in every test title, such as @QF-101. The marker is a stable machine-readable test ID, not a quarantine tag. A test becomes quarantined only when its ID appears in the manifest.

Step 1: Create Stable Test IDs and a Reproducible Failure

Start with two tests. The first is deterministic. The second simulates intermittent behavior only when an environment variable is set, so the tutorial remains predictable by default.

// tests/account.spec.ts
import { expect, test } from '@playwright/test';

test('@QF-100 user can view the account page', async ({ page }) => {
  await page.setContent('<main><h1>Account</h1></main>');
  await expect(page.getByRole('heading', { name: 'Account' })).toBeVisible();
});

test('@QF-101 user sees a completed import', async ({ page }) => {
  const shouldSimulateFailure = process.env.SIMULATE_FLAKE === '1';
  const status = shouldSimulateFailure ? 'Processing' : 'Completed';

  await page.setContent(`<main><p data-testid="status">${status}</p></main>`);
  await expect(page.getByTestId('status')).toHaveText('Completed');
});

Create a Playwright configuration that saves useful evidence on failure. One retry in CI helps identify a flaky signature, but a retry does not convert a failing test into a trusted test.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: Boolean(process.env.CI),
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 2 : undefined,
  reporter: [
    ['line'],
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['junit', { outputFile: 'test-results/junit.xml' }]
  ],
  use: {
    browserName: 'chromium',
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure'
  }
});

Run the stable baseline:

npx playwright test

Verify: both tests pass and test-results/junit.xml exists. Then run SIMULATE_FLAKE=1 npx playwright test. @QF-101 should fail on the initial attempt and retry, while @QF-100 passes. The HTML report should contain the failed assertion and trace attachment.

Step 2: Define the Quarantine Manifest

Create quarantine.json at the repository root. Treat it as production configuration: changes require review, and every record must explain who will fix the test and when the exception ends.

{
  "schemaVersion": 1,
  "tests": [
    {
      "id": "QF-101",
      "owner": "quality-platform",
      "issue": "https://github.com/example/acme/issues/842",
      "reason": "Import status event occasionally arrives after the UI timeout",
      "expiresOn": "2026-07-29"
    }
  ]
}

Use an HTTPS issue URL rather than free text such as JIRA-123, unless your organization validates another URL form. The expiry should be close enough to force review. Two weeks is a reasonable starting policy, but choose a window that matches your team cadence.

Do not store a file path or line number as the primary identity. Files move, tests are refactored, and line numbers change. The explicit QF-101 marker survives those edits and is easy to locate with rg '@QF-101' tests.

Document the operating rule in your contribution guide: adding a record needs a reproduced failure, an issue, an owner, and an expiry. Removing a record happens in the same pull request that proves the repaired test is stable.

Verify: run node -e "const q=require('./quarantine.json'); console.log(q.tests[0].id)". The output must be QF-101. Also search for the ID in the suite. Exactly one test title should contain it.

Step 3: Build the Suite Selector

Playwright accepts regular expressions through --grep and --grep-invert. Build the expression from the reviewed manifest instead of duplicating IDs in workflow YAML.

// scripts/quarantine-pattern.ts
import { readFileSync } from 'node:fs';

type Entry = {
  id: string;
  owner: string;
  issue: string;
  reason: string;
  expiresOn: string;
};

type Manifest = { schemaVersion: number; tests: Entry[] };

const manifest = JSON.parse(
  readFileSync(new URL('../quarantine.json', import.meta.url), 'utf8')
) as Manifest;

const escapeRegExp = (value: string): string =>
  value.replace(/[.*+?^${}()|[\]\\]/g, '\\
amp;');

const ids = manifest.tests.map((entry) => `@${escapeRegExp(entry.id)}`);

if (ids.length === 0) {
  console.log('(?!)');
} else {
  console.log(`(?:${ids.join('|')})`);
}

The empty-list pattern (?!) never matches. That makes the quarantine command succeed with zero selected tests when the manifest is empty, while --grep-invert '(?!)' still selects every trusted test.

Add complementary scripts to package.json:

{
  "scripts": {
    "test:trusted": "playwright test --grep-invert \"$(tsx scripts/quarantine-pattern.ts)\"",
    "test:quarantine": "playwright test --grep \"$(tsx scripts/quarantine-pattern.ts)\" --pass-with-no-tests"
  }
}

These scripts use POSIX command substitution, which works on GitHub-hosted Ubuntu runners. If developers use Windows PowerShell, provide a small Node wrapper or run the scripts through WSL. Avoid pretending the shell syntax is portable.

Verify: run npm run test:trusted. Only @QF-100 should appear. Run npm run test:quarantine; only @QF-101 should appear. Finally run SIMULATE_FLAKE=1 npm run test:trusted; it should still pass because the quarantined ID is excluded.

Step 4: Validate Ownership, Expiry, and Test Coverage

A quarantine list becomes a graveyard unless CI rejects stale records. Create a policy script that validates required fields, unique IDs, valid dates, HTTPS issue links, future expiry, and a matching test title.

// scripts/validate-quarantine.ts
import { readFileSync } from 'node:fs';
import { globSync } from 'node:fs';

type Entry = { id: string; owner: string; issue: string; reason: string; expiresOn: string };
type Manifest = { schemaVersion: number; tests: Entry[] };

const manifest = JSON.parse(readFileSync('quarantine.json', 'utf8')) as Manifest;
const today = new Date();
today.setUTCHours(0, 0, 0, 0);
const source = globSync('tests/**/*.{ts,tsx,js,mjs,cjs}')
  .map((file) => readFileSync(file, 'utf8'))
  .join('\n');
const errors: string[] = [];
const seen = new Set<string>();

if (manifest.schemaVersion !== 1) errors.push('schemaVersion must be 1');

for (const entry of manifest.tests) {
  if (!/^QF-[0-9]+$/.test(entry.id)) errors.push(`${entry.id}: invalid ID`);
  if (seen.has(entry.id)) errors.push(`${entry.id}: duplicate ID`);
  seen.add(entry.id);
  if (!entry.owner.trim()) errors.push(`${entry.id}: owner is required`);
  if (!entry.reason.trim()) errors.push(`${entry.id}: reason is required`);
  if (!entry.issue.startsWith('https://')) errors.push(`${entry.id}: issue must be HTTPS`);

  const expiry = new Date(`${entry.expiresOn}T00:00:00Z`);
  if (Number.isNaN(expiry.valueOf())) errors.push(`${entry.id}: invalid expiresOn`);
  else if (expiry < today) errors.push(`${entry.id}: quarantine expired`);

  const matches = source.split(`@${entry.id}`).length - 1;
  if (matches !== 1) errors.push(`${entry.id}: expected one test marker, found ${matches}`);
}

if (errors.length > 0) {
  console.error(errors.join('\n'));
  process.exit(1);
}
console.log(`Validated ${manifest.tests.length} quarantine record(s)`);

Node.js 22 provides globSync in node:fs. Pinning Node 22 in CI makes that dependency explicit. Add "quarantine:validate": "tsx scripts/validate-quarantine.ts" to the scripts object.

The date comparison allows an entry through on its expiry date and rejects it the next day. If you want expiry to mean the first invalid day, change < to <= and document the convention. Consistency matters more than either interpretation.

Verify: run npm run quarantine:validate. Expect Validated 1 quarantine record(s). Temporarily change expiresOn to yesterday and rerun. The command must exit nonzero with QF-101: quarantine expired; restore the future date afterward.

Step 5: Quarantine Flaky Tests CI Pipeline Setup With GitHub Actions

Create a workflow with three jobs. Policy validation runs first. Trusted and quarantine jobs then execute the complementary selections.

# .github/workflows/tests.yml
name: Tests

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  quarantine-policy:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run quarantine:validate

  trusted-tests:
    needs: quarantine-policy
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run test:trusted
      - name: Upload trusted report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: trusted-playwright-report
          path: |
            playwright-report/
            test-results/
          if-no-files-found: warn
          retention-days: 14

  quarantine-tests:
    needs: quarantine-policy
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run test:quarantine
      - name: Upload quarantine report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: quarantine-playwright-report
          path: |
            playwright-report/
            test-results/
          if-no-files-found: warn
          retention-days: 14

Job-level continue-on-error: true keeps the quarantine job visible while preventing its failure from failing the workflow. Do not put || true on the test command. That erases the test step's failure signal and makes the job look healthy.

For sensitive test environments, replace long-lived cloud secrets with the workflow described in the GitHub Actions OIDC test environments tutorial. Keep permissions minimal and grant id-token: write only to the job that needs federation.

Verify: open a pull request. The policy and trusted jobs should pass. With SIMULATE_FLAKE configured for the quarantine job, its test step should fail, the job should show a tolerated failure, and both report artifacts should be downloadable from the workflow run.

Step 6: Preserve Evidence and Make Failures Actionable

Quarantine is safe only when failures remain observable. The workflow already uploads reports with if: always(), but artifact names, retention, and notification routes need operational discipline.

Keep trusted and quarantine output separate. If both jobs upload an artifact named playwright-report, investigators can open the wrong evidence or overwrite files in downstream aggregation. Include the suite and, for matrix jobs, the browser or shard in each artifact name.

JUnit XML supports dashboards and trend analysis. The Playwright HTML report supports human investigation. Traces, screenshots, and video explain the browser state. Preserve all three layers when the storage cost is acceptable. The guide to publishing test evidence as CI artifacts shows a fuller retention and naming strategy.

Add a scheduled workflow trigger if pull request traffic is low:

on:
  schedule:
    - cron: '17 3 * * *'
  workflow_dispatch:

A daily run exposes persistent quarantine failures and proves when a fix has stabilized. Use a non-round minute such as 17 to avoid the busiest scheduler boundary. A schedule runs from the default branch, so make sure its manifest and tests are current there.

Route failures to the owner from the manifest through your existing alerting system. Avoid creating a new issue on every run. Update the linked issue with a run URL or aggregate repeated occurrences in a dashboard.

Verify: force @QF-101 to fail, run the workflow manually, and open quarantine-playwright-report. Confirm the report identifies the test ID, retry result, failed expectation, trace, commit SHA, and workflow run. Confirm the linked issue is accessible to the owner.

Step 7: Configure the Merge Gate Correctly

In the repository ruleset, require quarantine-policy and trusted-tests. Do not require quarantine-tests, because its purpose is to report an acknowledged unstable signal without blocking every change.

This does not mean quarantine failures are optional work. They are non-blocking for a bounded period, but their records have mandatory owners and expiry dates. The policy job blocks a pull request if someone tries to extend or misuse that exception incorrectly.

Protect against accidental coverage loss. A contributor could add a manifest ID that matches a broad title fragment or reuse an ID. The validator prevents duplicates and requires exactly one marker. Code review should also confirm that the issue contains failure evidence and that the quarantined assertion covers meaningful behavior.

Use environments or deployment jobs for release gates beyond pull requests. A production deployment can depend on the trusted test job and a separate environment approval. If your suite is large, organize it with the same parent-child principles described in GitLab CI child pipelines for test suites, even when your current runner is GitHub Actions.

Verify: create a branch where @QF-100 fails. The pull request must be blocked by trusted-tests. Restore it and make only @QF-101 fail. The trusted check and policy check should pass, and the quarantine result should remain visible without blocking merge. Finally, expire the manifest record and confirm policy validation blocks the pull request.

Step 8: Repair, Prove Stability, and Remove Quarantine

Fix the underlying cause, not the symptom. Common causes include shared test data, clock assumptions, asynchronous events, selector ambiguity, leaked browser state, network dependencies, and inadequate readiness signals. Replace arbitrary sleeps with assertions against a user-visible or API-level condition.

After the repair, keep the test in quarantine for a defined observation period. Run it repeatedly in the same CI environment where it failed. Do not claim stability from a single green run. Your evidence can be a sequence of scheduled runs, targeted repeats, and stress runs that exercise the suspected boundary.

Playwright can repeat a test locally or in a temporary CI job:

npx playwright test --grep '@QF-101' --repeat-each=20 --workers=1

The count of 20 is illustrative, not a universal statistical threshold. Choose a confidence policy based on test frequency, impact, and historical failure rate. Serial execution helps isolate test logic, while an additional parallel run can reveal shared-state defects.

Remove the manifest entry in the same pull request that links the fix and stability evidence. The next trusted run automatically includes the test because --grep-invert no longer excludes its ID. Keep the stable ID in the title so future regressions are traceable. Close the repair issue only after the test passes as part of the trusted suite.

Verify: remove QF-101 from quarantine.json, run npm run quarantine:validate, and run npm run test:trusted. Both @QF-100 and @QF-101 must execute. Run npm run test:quarantine; it should exit successfully with no tests selected.

Troubleshooting

Problem: the trusted job runs a quarantined test -> print tsx scripts/quarantine-pattern.ts, then compare the output with the exact marker in the test title. Check shell quoting and ensure the ID includes the @ prefix only once.

Problem: the quarantine job passes without executing anything -> run npx playwright test --list --grep '@QF-101'. If no test appears, the manifest ID and title differ or the test directory is wrong. The policy validator should catch a missing marker, but --list reveals Playwright filtering directly.

Problem: artifact upload says no files were found -> keep if: always() on the upload step and confirm reporter paths match the workflow paths. Installation or TypeScript startup failures can happen before Playwright creates a report, so retain console logs as the first source of evidence.

Problem: an expired record still passes validation -> confirm the runner clock is UTC and the date string uses YYYY-MM-DD. Review whether your policy treats the expiry date as valid through that day or invalid at its start.

Problem: the entire workflow looks green despite quarantine failures -> job-level tolerance can make the overall workflow succeed. Inspect the quarantine test step, publish JUnit results to your reporting system, and alert the owner. Do not rely only on the workflow conclusion color.

Problem: branch protection cannot find the trusted check -> run the workflow once on a pull request, then select the exact generated job name in the ruleset. Keep job IDs and names stable after making them required.

Quarantine Flaky Tests CI Pipeline Best Practices

Do make quarantine temporary, reviewed, and measurable. Require an owner, linked repair work, exact test ID, expiry, failure evidence, and a removal condition. Keep the quarantined test running on every relevant change or on a frequent schedule.

Do not quarantine a failing test before determining whether it exposes a real product defect. A test that fails consistently against incorrect behavior is not flaky. Blocking it may be the correct action.

Do not use retries as the only control. Retries can reveal intermittency and gather traces, but they increase runtime and can hide deteriorating reliability when only the final result is reviewed. Report the initial failure and retry outcome.

Do not skip or comment out quarantined tests. A skipped test generates no evidence that the defect persists or that a repair worked. A separate non-blocking execution preserves that signal.

Do not let one manifest entry match multiple tests. Broad tags such as @flaky provide poor ownership and removal semantics. Stable, unique IDs make validation and audit history precise.

Finally, watch the size and age of the quarantine set. A rising count, repeated extensions, or the same ownership hotspot signals a system problem that should be addressed in planning, test architecture, or environment reliability.

Where To Go Next

You now have a bounded exception mechanism, not a permanent second-class suite. Use the complete test automation CI/CD guide to connect it to deployment gates, runner strategy, and feedback design.

Next, strengthen adjacent parts of the pipeline:

• Use GitHub Actions OIDC for secure test environments so test jobs obtain short-lived cloud credentials.
• Apply GitLab child pipeline patterns for test suites when independent suites need separate ownership and execution.
• Publish test evidence as durable CI artifacts so failures can be investigated after runners disappear.

Once those controls are in place, add a small reliability dashboard: active quarantine count, entries near expiry, failures by owner, time in quarantine, and removal rate. Use the data to shorten quarantine duration, not to normalize instability.

Interview Questions and Answers

Q: What does it mean to quarantine a flaky test?

It means the test continues to run and report evidence, but its known unstable result no longer blocks the trusted release gate for a limited period. The quarantine record must have an owner, reason, repair issue, and expiry.

Q: Why is skipping a test different from quarantining it?

Skipping stops execution, so the team loses failure frequency, diagnostic artifacts, and proof of recovery. Quarantine preserves execution in a separate visible job while changing only the gating behavior.

Q: Should retries replace quarantine?

No. Retries are diagnostic and can expose an intermittent signature. Quarantine is a governance decision about an acknowledged unreliable signal, with ownership and an exit condition.

Q: Which CI checks should be required for merging?

Require quarantine policy validation and the trusted test suite. Keep the quarantine execution visible but non-blocking, and monitor its underlying test-step results through reports or alerts.

Q: How do you prevent quarantined tests from being forgotten?

Validate expiry dates in CI, assign owners, link repair issues, run the tests continuously, and report age and failure trends. Expired records should block changes until they are removed, fixed, or explicitly reviewed.

Q: How do you return a test to the trusted suite?

Fix the root cause, gather stability evidence in the original CI environment, remove its manifest entry, and verify it executes in the blocking trusted job. Keep its stable ID for traceability.

Conclusion

To quarantine flaky tests in a CI pipeline safely, isolate the gating decision, not the test. Execute the unstable tests in a separate job, preserve their evidence, and make every exception expire under a named owner.

Start with one flaky test and prove the complete lifecycle: reproduce, quarantine, observe, repair, validate stability, and restore it to the trusted suite. A quarantine system succeeds when tests leave it quickly and the merge gate becomes more credible.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Reduce flaky tests in a CI pipeline (2026)
• Flaky test quarantine in CI: Step by Step (2026)
• How to Debug a failing test in VS Code in Cypress (2026)
• How to Debug a failing test in VS Code in Playwright (2026)
• How to Debug a failing test in VS Code in Selenium (2026)
• How to Run tests in headed mode in Cypress (2026)