Design a Flaky Test Detection System Interview

TL;DR

In a design flaky test detection system interview, propose an asynchronous pipeline that consumes immutable test attempts, groups comparable executions, computes a versioned and explainable flakiness score, and opens a governed triage workflow. Separate product failures from infrastructure noise, require enough evidence, and never let automatic quarantine silently erase coverage.

Key Takeaways

• Define flakiness as inconsistent outcomes for the same test and comparable code state, not simply any retried pass.
• Preserve immutable attempt history and classify product, test, and infrastructure failures separately.
• Combine transition evidence, failure diversity, and sample confidence into an explainable score.
• Keep detection asynchronous so analytics outages never block test result ingestion.
• Use quarantine as a time-bounded workflow with ownership, evidence, and exit criteria.
• Measure false positives, time to triage, recurrence, and quarantined coverage alongside flaky-test counts.

A strong design flaky test detection system interview answer does more than count retries. Build a system that identifies inconsistent outcomes for the same test under comparable conditions, explains the evidence, and routes suspected tests through a safe ownership and quarantine workflow.

This tutorial gives you a runnable TypeScript detector and a production evolution path. Use the broader Senior SDET system design interview complete guide to frame requirements, estimates, APIs, reliability, and tradeoffs before you zoom into flakiness.

You will model immutable attempts, normalize failure signals, calculate a confidence-aware score, expose an API, and add triage controls. The local implementation is deliberately small, but every boundary maps to a database, event stream, worker, and dashboard at scale.

What You Will Build

By the end, you will have:

• A Node.js and TypeScript service that accepts test attempt events.
• A detector that compares recent outcomes within a controlled environment key.
• An explainable score based on pass-fail transitions, failure diversity, and sample confidence.
• HTTP endpoints for evidence, triage state, quarantine, and recovery.
• A design for durable ingestion, asynchronous scoring, observability, and safe rollout.

The key boundary is simple: the execution system produces facts, while the flaky-test service derives classifications. Never let the classifier rewrite raw results. A changed algorithm should be able to replay history and produce a new version without losing the original evidence.

Prerequisites

Use Node.js 22 LTS or newer and npm 10 or newer. The tutorial uses TypeScript 5, Fastify 5, and Zod 4. Create a separate practice directory so these commands do not alter your application repository.

mkdir flaky-detector && cd flaky-detector
npm init -y
npm install fastify zod
npm install -D typescript tsx @types/node
npx tsc --init --target ES2022 --module NodeNext --moduleResolution NodeNext --strict
mkdir src

Add "type": "module" to package.json and scripts for "start": "tsx src/server.ts" and "check": "tsc --noEmit". You also need curl for API verification.

Verify the setup by running node --version, npm --version, and npx tsx --version. Each command should print a version and exit successfully.

Step 1: Frame the Design Flaky Test Detection System Interview Requirements

Start with a precise definition. A flaky test produces different valid outcomes when the relevant code, test definition, environment, and input are materially equivalent. A failed attempt followed by a passed retry is evidence, not proof. Infrastructure loss, an expired credential, or a real code fix can produce the same sequence.

Write the functional requirements before drawing components:

1. Ingest immutable test attempts with stable test identity and environment context.
2. Normalize errors and classify assertion, test-code, infrastructure, and unknown failures.
3. Calculate an explainable score over a versioned observation window.
4. Show evidence and correlations to an owner.
5. Support acknowledge, quarantine, assign, resolve, and reopen transitions.
6. Recompute classifications when policy or detector logic changes.

Add nonfunctional requirements. Result ingestion must remain available when scoring is down. Duplicate events must not double count. A tenant must not see another tenant's tests. Detection should be eventually consistent, while workflow changes require conditional writes. The system must expose classification freshness and algorithm version.

Estimate with symbols if the interviewer gives no numbers. At A attempts per second and B bytes of normalized metadata per attempt, daily storage is A * B * 86,400. If each update reads W recent attempts, naive work is A * W; incremental aggregates can reduce that cost. State that retention and score latency come from product requirements, not guesswork.

Verification: Confirm you can distinguish a test failure, a flaky test, and an infrastructure failure. Name the owner of every state transition. If automatic quarantine has no guardrail or audit trail, the requirements are incomplete.

Step 2: Model Immutable Attempts and Comparable Cohorts

Create src/domain.ts. Stable identity and comparison context prevent unrelated executions from contaminating the score.

import { z } from "zod";

export const AttemptSchema = z.object({
  eventId: z.uuid(),
  tenantId: z.string().min(1),
  testId: z.string().min(1),
  runId: z.string().min(1),
  commitSha: z.string().min(7),
  environment: z.string().min(1),
  runner: z.string().min(1),
  attempt: z.number().int().positive(),
  outcome: z.enum(["passed", "failed", "timed_out", "canceled"]),
  failureType: z.enum(["assertion", "test_code", "infrastructure", "unknown"]).optional(),
  errorMessage: z.string().max(10_000).optional(),
  durationMs: z.number().int().nonnegative(),
  finishedAt: z.iso.datetime()
});

export type Attempt = z.infer<typeof AttemptSchema>;

export function cohortKey(a: Attempt): string {
  return `${a.tenantId}:${a.testId}:${a.environment}:${a.runner}`;
}

Do not use display names as identity because teams rename tests and duplicate names across suites. Generate testId from repository, suite path, parameter identity, and a durable framework identifier, or register tests in a catalog. Version identity migrations so history can be joined deliberately.

The cohort key compares the same environment and runner. Production models may add browser major version, operating system, device model, locale, feature flags, or dataset version. Too broad a cohort creates false positives. Too narrow a cohort never accumulates enough evidence. Keep commit SHA as evidence rather than always putting it in the key, because a multi-commit window is useful after excluding known behavioral changes.

Canceled attempts provide operational evidence but should not count as passes or failures. Infrastructure failures should feed infrastructure health models and correlations, not automatically lower test reliability.

Verification: Run npm run check. It should return without diagnostics. Temporarily change outcome to skipped in a typed fixture and confirm TypeScript rejects it. Explain why two browsers belong to separate cohorts unless the test contract proves them equivalent.

Step 3: Normalize Failures and Compute an Explainable Score

Create src/detector.ts. This detector uses recent non-infrastructure outcomes, counts adjacent pass-fail transitions, measures normalized failure diversity, and discounts small samples. The formula is a policy example, not a universal statistical truth.

import type { Attempt } from "./domain.js";

export type Evidence = {
  score: number; samples: number; transitions: number;
  passRate: number; fingerprints: string[]; algorithmVersion: string;
  classification: "insufficient_data" | "stable" | "suspected_flaky";
};

export function fingerprint(message = ""): string {
  return message.toLowerCase()
    .replace(/[0-9a-f]{8}-[0-9a-f-]{27,}/g, "<uuid>")
    .replace(/\b\d{4,}\b/g, "<number>")
    .replace(/\s+/g, " "  ).trim().slice(0, 300);
}

export function detect(input: Attempt[]): Evidence {
  const attempts = input
    .filter((a) => a.outcome === "passed" ||
      (a.outcome === "failed" && a.failureType !== "infrastructure"))
    .sort((a, b) => a.finishedAt.localeCompare(b.finishedAt))
    .slice(-50);
  const outcomes = attempts.map((a) => a.outcome);
  const transitions = outcomes.slice(1)
    .filter((value, i) => value !== outcomes[i]).length;
  const failures = attempts.filter((a) => a.outcome === "failed");
  const fingerprints = [...new Set(failures.map((a) => fingerprint(a.errorMessage)))];
  const passRate = attempts.length === 0 ? 0 :
    outcomes.filter((x) => x === "passed").length / attempts.length;
  const transitionRate = attempts.length < 2 ? 0 : transitions / (attempts.length - 1);
  const mixedOutcome = passRate > 0 && passRate < 1 ? 1 : 0;
  const diversity = Math.min(fingerprints.length / 3, 1);
  const confidence = Math.min(attempts.length / 10, 1);
  const score = Number((100 * confidence * mixedOutcome *
    (0.75 * transitionRate + 0.25 * diversity)).toFixed(1));
  return { score, samples: attempts.length, transitions, passRate, fingerprints,
    algorithmVersion: "transition-v1",
    classification: attempts.length < 5 ? "insufficient_data" :
      score >= 30 ? "suspected_flaky" : "stable" };
}

The response exposes its ingredients so an engineer can challenge the result. In production, store algorithm version, window bounds, excluded-event counts, and the evidence event IDs. Evaluate thresholds on labeled historical data by tenant or test family. A probabilistic model can come later, but it still needs explanations and drift monitoring.

Verification: Pass ten alternating passed and assertion-failed attempts to detect; expect suspected_flaky and a high transition count. Pass ten failures with one fingerprint; expect stable because there is no outcome transition. Pass four mixed attempts; expect insufficient_data.

Step 4: Build Idempotent Ingestion and Evidence APIs

Create src/server.ts. The in-memory maps make the contract runnable; replace them with durable storage and conditional writes in production.

import Fastify from "fastify";
import { z } from "zod";
import { AttemptSchema, cohortKey, type Attempt } from "./domain.js";
import { detect } from "./detector.js";

const app = Fastify({ logger: true });
const events = new Map<string, Attempt>();
const cohorts = new Map<string, Attempt[]>();

app.post("/v1/attempts", async (request, reply) => {
  const parsed = AttemptSchema.safeParse(request.body);
  if (!parsed.success) return reply.code(400).send({ error: parsed.error.flatten() });
  const prior = events.get(parsed.data.eventId);
  if (prior) return reply.code(200).send({ accepted: true, duplicate: true });
  events.set(parsed.data.eventId, parsed.data);
  const key = cohortKey(parsed.data);
  cohorts.set(key, [...(cohorts.get(key) ?? []), parsed.data]);
  return reply.code(202).send({ accepted: true, cohort: key });
});

app.get("/v1/tests/:testId/evidence", async (request, reply) => {
  const { testId } = request.params as { testId: string };
  const environment = String((request.query as { environment?: string }).environment ?? "ci");
  const runner = String((request.query as { runner?: string }).runner ?? "linux-chromium");
  const tenantId = String(request.headers["x-tenant-id"] ?? "");
  if (!tenantId) return reply.code(401).send({ error: "tenant required" });
  const key = `${tenantId}:${testId}:${environment}:${runner}`;
  return { testId, environment, runner, evidence: detect(cohorts.get(key) ?? []) };
});

app.listen({ port: 3000, host: "127.0.0.1" }).catch((error) => {
  app.log.error(error); process.exit(1);
});

A production ingestion transaction inserts the unique event ID and an outbox record atomically. A consumer updates incremental cohort statistics and writes a classification snapshot. Store the full attempt separately so replay remains possible. If the same event ID arrives with different content, return a conflict instead of silently accepting it.

Derive tenant identity from authenticated claims, not a public header. The header keeps only this local exercise simple. Paginate evidence, redact secrets in error text, and authorize test ownership before returning samples.

Verification: Run npm run start. Post a valid attempt twice and expect 202 followed by a duplicate 200. Request its evidence with x-tenant-id and expect one sample. Omit the header and expect 401.

Step 5: Add a Governed Triage and Quarantine Workflow

Detection is not the product outcome. The system must turn evidence into accountable action without allowing a noisy model to hide failures. Add a workflow record to server.ts before listen.

type Workflow = { state: "open" | "quarantined" | "resolved"; owner: string;
  reason: string; version: number; expiresAt?: string };
const workflows = new Map<string, Workflow>();
const ChangeSchema = z.object({
  state: z.enum(["open", "quarantined", "resolved"]),
  owner: z.string().min(1), reason: z.string().min(10),
  expectedVersion: z.number().int().nonnegative(),
  expiresAt: z.iso.datetime().optional()
});

app.put("/v1/tests/:testId/workflow", async (request, reply) => {
  const { testId } = request.params as { testId: string };
  const parsed = ChangeSchema.safeParse(request.body);
  if (!parsed.success) return reply.code(400).send({ error: parsed.error.flatten() });
  const current = workflows.get(testId);
  if ((current?.version ?? 0) !== parsed.data.expectedVersion)
    return reply.code(409).send({ error: "workflow changed" });
  if (parsed.data.state === "quarantined" && !parsed.data.expiresAt)
    return reply.code(400).send({ error: "quarantine expiration required" });
  const next = { ...parsed.data, version: (current?.version ?? 0) + 1 };
  workflows.set(testId, next);
  return next;
});

Production workflow keys include tenant and test ID. Record actor, timestamp, prior state, evidence snapshot, ticket, and reason in an append-only audit log. Require an owner and expiry for quarantine. Continue running quarantined tests on a nonblocking lane so you can see recovery, and alert when expiration approaches.

Do not auto-quarantine on one score alone. Combine minimum samples, score confidence, recent impact, test criticality, and policy. High-risk security or payment tests may never be eligible for automatic quarantine. Resolution should require a sustained passing window or a reviewed code fix, then monitor recurrence.

Verification: Quarantine without expiresAt and expect 400. Submit version 0 with an expiry and expect version 1. Repeat with expected version 0 and expect 409. Confirm that the workflow changes test gating policy but never deletes historical failures.

Step 6: Design the Production Pipeline and Storage

Separate the write path from scoring. Runners send attempts to a stateless ingestion API. The API validates identity and writes an immutable result plus an outbox event in one transaction. A relay publishes events to a partitioned stream keyed by tenant and test ID. Scoring workers consume in order per key, update windowed features, and write versioned classifications.

Use a relational database for test catalog, attempts, classifications, workflow, ownership, and audit records. Put logs, traces, screenshots, and videos in private object storage. A stream or durable queue absorbs bursts and permits replay. A cache can accelerate dashboards, but it is never the evidence source of truth. The test result storage system design tutorial covers idempotent attempt storage and artifact separation in depth.

Partition attempts by time and retain compact normalized facts longer than large artifacts when policy allows. Index (tenant_id, test_id, finished_at desc) for evidence lookup and (tenant_id, classification, calculated_at) for work queues. Store feature snapshots so a reviewer can reproduce why a score changed.

Delivery is at least once. Deduplicate by event ID and make aggregate updates conditional on the last processed sequence. Late events trigger a bounded recomputation of the affected window. A poison event goes to a dead-letter path with tenant-safe diagnostics. If scoring stops, ingestion continues and freshness alarms rise.

Verification: Walk through three failures: duplicate delivery, scorer outage, and a late attempt. For each, identify the durable record, retry behavior, visible freshness, and reconciliation action. No accepted attempt should disappear because a derived analytics component failed.

Step 7: Add Correlation, Evaluation, and Observability

A score becomes useful when it points toward a cause. Join attempts with worker image, browser, device, shard, region, time of day, duration, resource pressure, test data, and recent ownership changes. Correlation with one worker pool suggests infrastructure noise. Diverse failures across healthy workers suggest test nondeterminism. A failure beginning at one commit may be a product regression.

The execution service must emit consistent metadata. See the cross-platform test execution service design for capability-aware workers and fenced attempts. For physical-device correlations, use the device farm scheduler design tutorial to connect reservations, device health, and session artifacts.

Evaluate the detector on a reviewed dataset. Track precision among suspected tests, recall of known flaky tests, false quarantine rate, time from first evidence to triage, time to resolution, recurrence after resolution, and algorithm coverage. Avoid treating a falling flaky-test count as success if quarantine volume or unexecuted coverage rises.

Operational metrics include ingest acceptance and conflict rates, event lag, oldest unprocessed event, score calculation latency, replay backlog, database saturation, dead-letter count, workflow conflicts, and expired quarantines. Segment metrics by tenant without putting tenant IDs in unbounded metric labels. Use traces across ingestion, outbox publication, scoring, and notification.

Roll out a new algorithm in shadow mode. Compute old and new versions side by side, compare classifications, sample disagreements for human review, then gradually switch read policy. Keep rollback simple because score changes can alter CI gates.

Verification: Create a dashboard that distinguishes system health, detector quality, and workflow outcomes. Inject scorer lag and confirm freshness alerts fire while ingestion remains healthy. Replay labeled examples and compare both algorithm versions before promoting the new one.

Troubleshooting

Every retried pass is labeled flaky -> Exclude infrastructure failures, compare equivalent cohorts, require a minimum sample count, and use transitions as evidence rather than a retry flag as proof.

One deterministic failure gets a high score -> Check ordering, duplicate ingestion, fingerprint normalization, and whether unrelated environments share a cohort. A single-outcome series should not be classified as flaky.

Parameterized tests contaminate each other -> Put a stable parameter identity in testId or the cohort key. Do not merge cases merely because they share a display name.

The dashboard changes after a replay -> Persist algorithm version, window bounds, input event IDs, and feature snapshot. Ensure consumers deduplicate and apply deterministic ordering for equal timestamps.

Quarantined tests never return to gating -> Require expiry, owner, ticket, reminders, continued nonblocking execution, and explicit exit evidence. Escalate expired quarantine instead of renewing it silently.

Scoring falls behind during suite bursts -> Partition by stable test key, batch database reads, update incremental features, autoscale on oldest-event age, and keep ingestion independent. Bound replay work so it does not starve current events.

Where To Go Next

Return to the complete Senior SDET system design interview guide and rehearse the full sequence: clarify the definition, estimate volume, draw the asynchronous path, model attempts, explain scoring, then test failures and tradeoffs.

Continue with these connected designs:

• Design a test result storage system for immutable attempts, artifacts, indexes, retention, and outbox delivery.
• Design a cross-platform test execution service for worker capabilities, leases, isolation, and consistent attempt metadata.
• Design a device farm scheduler for scarce-resource reservations and device health correlations.
• Review automation testing interview questions for experienced engineers to practice concise follow-up answers.

Extend the tutorial by moving maps to PostgreSQL, publishing attempt events through a transactional outbox, and adding a small labeled evaluation set. Then rehearse how you would change the threshold safely without rewriting raw evidence.

Interview Questions and Answers for a Design Flaky Test Detection System Interview

Q: What exactly is a flaky test?

A flaky test produces inconsistent outcomes under materially equivalent code, test, data, and environment conditions. A retry that passes is useful evidence but not sufficient proof. The system should exclude known infrastructure failures and preserve comparison context.

Q: Why make scoring asynchronous?

Flakiness is a derived property over history, not part of accepting one test result. An asynchronous pipeline keeps result ingestion reliable during scorer outages and supports replay after algorithm changes. Dashboards must expose freshness because classifications become eventually consistent.

Q: How do you identify the same test over time?

Use a stable catalog ID derived from repository, suite path, parameter identity, and framework metadata. Avoid display names alone. Record explicit identity migrations for renames and refactors so history is joined intentionally.

Q: How would you calculate a flakiness score?

Start with explainable features such as mixed outcomes, adjacent transitions, retry reversals, normalized failure diversity, and sample count. Exclude infrastructure and canceled attempts according to documented policy. Version the formula and calibrate its threshold on labeled historical cases.

Q: How do you avoid false positives after a real product regression?

Respect commit boundaries and deployment changes, detect failure onset, and compare fingerprints. A deterministic failure beginning at one commit should stay a product-failure candidate even if earlier builds passed. Human-readable evidence lets owners correct ambiguous classification.

Q: Should the system automatically quarantine tests?

Only under a governed policy with strong evidence, test criticality checks, an owner, an expiry, and an audit trail. Continue executing quarantined tests outside the blocking gate. Critical tests may require manual approval or may never be eligible.

Q: How do you handle duplicate and out-of-order events?

Deduplicate with a stable event ID and make updates idempotent. Partition streams by tenant and test ID to preserve normal ordering. For late events, recompute the bounded affected window and publish a newer versioned snapshot.

Q: What storage would you choose?

Use relational storage for identity, attempts, workflows, and versioned classifications, plus object storage for large artifacts. Add a durable stream for asynchronous scoring and replay. Introduce specialized analytics storage only when measured query volume requires it.

Q: How do you evaluate detector quality?

Build a labeled set from reviewed flaky, deterministic, and infrastructure cases. Measure precision, recall, false quarantine rate, and calibration by test family. Also track time to triage, resolution, and recurrence because classification alone does not improve reliability.

Q: What happens when the scoring service is unavailable?

Attempt ingestion continues and the durable event backlog grows. Existing classifications remain visible with their calculation time, while a freshness alert warns users. Consumers catch up idempotently after recovery.

Best Practices

• Preserve immutable attempts and recompute derived classifications.
• Keep test identity and environment comparison rules explicit.
• Separate assertion, test-code, infrastructure, canceled, and unknown outcomes.
• Show evidence, confidence, algorithm version, and freshness with every label.
• Calibrate thresholds on reviewed data before affecting CI gates.
• Make quarantine temporary, owned, audited, and continuously exercised.
• Roll out detector changes in shadow mode and review disagreements.
• Measure escaped flakes and lost coverage, not only label counts.

Avoid three interview mistakes: defining every retry as flakiness, promising exactly-once event delivery, and presenting automatic quarantine as harmless. A senior design treats classification uncertainty and test coverage as first-class risks.

Conclusion

To succeed in a design flaky test detection system interview, separate immutable execution facts from versioned classifications. Compare equivalent cohorts, exclude infrastructure noise, calculate an explainable confidence-aware score, and connect the result to a governed triage workflow.

Build the local detector, verify its edge cases, then practice the production evolution from outbox ingestion to partitioned scoring and replay. The strongest answer explains not only how to find suspected flaky tests, but also how to avoid hiding real defects while the organization fixes them.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Design a Test Result Storage System
• Design a Cross-Platform Test Execution Service
• Design Device Farm Scheduler Interview: An SDET System Design Guide
• Senior SDET System Design Interview Complete Guide (2026)
• How to Design a test data strategy (2026)
• Playwright Test Runner Interview Questions and Answers