How to Detect Flaky Tests With OpenTelemetry Traces (2026)

TL;DR

To detect flaky tests with OpenTelemetry traces, create one span for each test attempt, record pass or fail as attributes, and group spans by stable test identity plus source revision. A test is flaky when that group contains both outcomes; child spans then reveal whether latency, retries, or a dependency changed on failing attempts.

Key Takeaways

• Model every test attempt as a span and every meaningful dependency call as a child span.
• Detect a flake only when the same test identity produces both passing and failing outcomes for the same revision.
• Calculate flake score from mixed outcomes, while keeping failure rate and latency spread as separate signals.
• Attach bounded test, CI, retry, and environment attributes that make traces comparable without leaking secrets.
• Use deterministic repeated runs to validate the telemetry pipeline before connecting it to CI.
• Quarantine with an owner and expiry date, then preserve the trace evidence needed to fix the cause.

To detect flaky tests with OpenTelemetry traces, record each execution attempt as a span and compare outcomes for the same test and source revision. A test is flaky only when identical code produces both pass and fail results. The trace adds the evidence that an ordinary test report loses: which operation slowed down, which retry succeeded, and which environment handled the attempt.

This tutorial builds a small TypeScript lab around Node.js, node:test, and the OpenTelemetry JavaScript SDK. You will run a deliberately timing-sensitive test repeatedly, export completed spans in memory, calculate a flake score, and inspect the child spans behind the mixed outcome. The same span model transfers to Playwright, Cypress, Selenium, or an internal runner. For the wider pipeline context, see the test automation CI/CD complete guide.

TL;DR

Do not label every intermittent-looking failure as flaky. A consistently failing assertion is a regression, and a test that fails only on one broken environment may be an infrastructure defect. Group comparable traces first, then classify mixed outcomes.

What You Will Build

You will create a local diagnostic project that:

• starts the OpenTelemetry Node SDK before the test code acquires a tracer;
• wraps each execution in a test.run span with stable identity and outcome attributes;
• creates a dependency.wait child span to expose timing variation;
• repeats one test 20 times with a seeded delay sequence;
• groups finished spans and reports pass count, fail count, failure rate, and flake score;
• prints the slowest failing attempt so you can move from detection to diagnosis.

The lab intentionally uses an InMemorySpanExporter. That exporter is part of the official SDK and makes the example runnable with no collector, account, or network. In production, replace it with OTLP while retaining the same span names and attributes.

Prerequisites

Use Node.js 22 LTS, npm 10 or newer, TypeScript 5.9.3, tsx 4.20.6, @opentelemetry/api 1.9.0, @opentelemetry/sdk-node 0.221.0, and @opentelemetry/sdk-trace-base 2.5.0. The OpenTelemetry Node SDK remains experimental, so pinning compatible versions is safer than allowing independent caret upgrades.

Check the runtime, create an empty directory, and install exact versions:

node --version
npm --version
mkdir otel-flake-lab
cd otel-flake-lab
npm init -y
npm install --save-exact @opentelemetry/api@1.9.0 @opentelemetry/sdk-node@0.221.0 @opentelemetry/sdk-trace-base@2.5.0
npm install --save-dev --save-exact typescript@5.9.3 tsx@4.20.6 @types/node@22

Verification: npm ls @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/sdk-trace-base tsx typescript must show one installed version for each package and no invalid marker. If you are adapting an existing suite, commit its lockfile before instrumentation so an unrelated dependency change cannot contaminate the baseline.

Step 1: Configure the TypeScript Test Project

Create tsconfig.json with Node's ESM-aware compiler settings:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src/**/*.ts"]
}

Add scripts and ESM mode to package.json:

{
  "name": "otel-flake-lab",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "typecheck": "tsc --noEmit",
    "test:traced": "tsx src/run.ts"
  }
}

NodeNext makes TypeScript follow Node's package and extension rules. The lab uses tsx to execute TypeScript directly, while tsc --noEmit catches incorrect OpenTelemetry calls before a run. Keep type: module because the source uses import; changing only one side commonly produces an ESM loader error.

Verification: run npm run typecheck. At this stage, tsc should exit with code 0 even though src is empty. If it reports that no inputs were found, create the src directory and continue to Step 2 before repeating the check.

Step 2: Start OpenTelemetry Before Test Imports

Create src/telemetry.ts:

import { NodeSDK } from '@opentelemetry/sdk-node';
import {
  InMemorySpanExporter,
  SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-base';

export const spanExporter = new InMemorySpanExporter();

const sdk = new NodeSDK({
  serviceName: 'checkout-e2e-tests',
  spanProcessors: [new SimpleSpanProcessor(spanExporter)],
});

sdk.start();

export async function stopTelemetry(): Promise<void> {
  await sdk.shutdown();
}

The SDK must start before the module containing test instrumentation is imported. OpenTelemetry APIs return harmless no-op implementations when no provider is registered, which creates a deceptive green run with zero finished spans. SimpleSpanProcessor exports each completed span immediately. That behavior is convenient for a deterministic lab, but a production runner should usually use the SDK's batch processing defaults with an OTLP exporter.

serviceName identifies the suite as a trace-producing service. In a shared backend, use a stable service name such as payments-api-tests, not the branch or job ID. Put the variable context into resource or span attributes so dashboards can compare runs.

Verification: run npm run typecheck. You should get a clean exit. A compile error mentioning spanProcessor usually means singular configuration copied from an older SDK example; this version uses the plural spanProcessors array.

Step 3: Define a Stable Test Span Contract

Create src/instrumented-test.ts:

import { SpanStatusCode, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('qa.test-runner', '1.0.0');

export type TestOutcome = 'pass' | 'fail';

export interface AttemptResult {
  outcome: TestOutcome;
  observedDelayMs: number;
}

export async function runInstrumentedAttempt(
  testName: string,
  attempt: number,
  revision: string,
  execute: () => Promise<AttemptResult>,
): Promise<AttemptResult> {
  return tracer.startActiveSpan('test.run', {
    attributes: {
      'test.name': testName,
      'test.framework': 'node:test',
      'test.attempt': attempt,
      'vcs.revision': revision,
      'ci.provider': process.env.CI ? 'local-ci' : 'local',
    },
  }, async (span) => {
    try {
      const result = await execute();
      span.setAttributes({
        'test.outcome': result.outcome,
        'test.observed_delay_ms': result.observedDelayMs,
      });

      if (result.outcome === 'fail') {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: 'dependency exceeded test deadline',
        });
      }
      return result;
    } catch (error) {
      span.recordException(error as Error);
      span.setAttribute('test.outcome', 'fail');
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

export async function observedDependency(
  delayMs: number,
): Promise<number> {
  return tracer.startActiveSpan('dependency.wait', async (span) => {
    span.setAttribute('dependency.delay_ms', delayMs);
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    span.end();
    return delayMs;
  });
}

The identity is test.name plus vcs.revision. Attempt number is context, not identity. If you include the attempt, worker, or random run ID in the grouping key, every group has one result and no flake can ever be detected.

The attribute names under test.* are an explicit local contract. Keep their value types stable. Never put access tokens, full request bodies, customer data, or unbounded stack traces into attributes. Use trace links or artifact URLs for bulky evidence.

Verification: run npm run typecheck. It must accept startActiveSpan, setAttributes, recordException, and SpanStatusCode.ERROR. This confirms that the public API, not an invented wrapper, is being used.

Step 4: Generate Reproducible Mixed Outcomes

Create src/run.ts and import telemetry first:

import { spanExporter, stopTelemetry } from './telemetry.js';
import {
  observedDependency,
  runInstrumentedAttempt,
} from './instrumented-test.js';

const testName = 'checkout confirms an accepted payment';
const revision = process.env.GIT_SHA ?? 'local-demo-revision';
const deadlineMs = 30;
const delays = [
  12, 18, 42, 16, 24, 37, 14, 20, 45, 19,
  23, 34, 17, 21, 39, 13, 26, 41, 15, 22,
];

for (const [index, delayMs] of delays.entries()) {
  await runInstrumentedAttempt(
    testName,
    index + 1,
    revision,
    async () => {
      const observedDelayMs = await observedDependency(delayMs);
      return {
        outcome: observedDelayMs <= deadlineMs ? 'pass' : 'fail',
        observedDelayMs,
      };
    },
  );
}

const spans = spanExporter.getFinishedSpans();
console.log(`finished spans: ${spans.length}`);
console.log(`test spans: ${spans.filter((s) => s.name === 'test.run').length}`);
await stopTelemetry();

The delay sequence is fixed on purpose. It creates 20 test spans and 20 child spans, with failures above the illustrative 30 ms deadline. A seeded or fixed disturbance is valuable while validating instrumentation because two engineers should see the same answer. Once the pipeline works, point the wrapper at real tests and remove the synthetic condition.

This example returns a failed outcome instead of throwing so all 20 attempts complete. Real test adapters normally catch the runner's result event and record it, then let the runner retain responsibility for process exit codes. Do not swallow application assertions merely to produce telemetry.

Verification: run npm run test:traced. The final lines must be finished spans: 40 and test spans: 20. A zero count means initialization happened too late. A count below 40 means a span path did not call end().

Step 5: Detect Flaky Tests With OpenTelemetry Traces

Add this analysis code above await stopTelemetry() in src/run.ts:

const testSpans = spans.filter((span) => span.name === 'test.run');
const groups = new Map<string, typeof testSpans>();

for (const span of testSpans) {
  const name = String(span.attributes['test.name']);
  const sha = String(span.attributes['vcs.revision']);
  const key = `${name}@@${sha}`;
  groups.set(key, [...(groups.get(key) ?? []), span]);
}

for (const [key, attempts] of groups) {
  const passed = attempts.filter(
    (span) => span.attributes['test.outcome'] === 'pass',
  ).length;
  const failed = attempts.length - passed;
  const mixedOutcomes = passed > 0 && failed > 0;
  const failureRate = failed / attempts.length;
  const flakeScore = mixedOutcomes
    ? 1 - Math.abs(passed - failed) / attempts.length
    : 0;

  console.log({
    test: key,
    attempts: attempts.length,
    passed,
    failed,
    flaky: mixedOutcomes,
    failureRate: Number(failureRate.toFixed(2)),
    flakeScore: Number(flakeScore.toFixed(2)),
  });
}

mixedOutcomes is the classification rule. The example flake score ranges from 0 to 1 and is highest when outcomes are evenly split. It is a transparent prioritization heuristic, not an OpenTelemetry standard or a probability that the next run fails. Keep failureRate beside it because a test failing 5 percent of the time and one failing 50 percent create different operational pain.

Require a useful sample before automating quarantine. With only one pass and one fail, you have detected inconsistency but estimated its frequency poorly. A practical policy might evaluate the last 20 comparable attempts, require at least two failures, and exclude runs already marked as infrastructure incidents. The exact threshold belongs to your release risk, test duration, and retry volume.

Verification: run npm run test:traced. The object must show 20 attempts, 14 passed, 6 failed, flaky: true, failureRate: 0.3, and flakeScore: 0.6.

Step 6: Correlate Failures With Child Span Timing

Detection answers which test is unstable. Diagnosis asks what differs. Add the following before shutdown:

const failedAttempts = testSpans
  .filter((span) => span.attributes['test.outcome'] === 'fail')
  .map((span) => ({
    attempt: Number(span.attributes['test.attempt']),
    delayMs: Number(span.attributes['test.observed_delay_ms']),
    traceId: span.spanContext().traceId,
  }))
  .sort((a, b) => b.delayMs - a.delayMs);

const childSpans = spans.filter((span) => span.name === 'dependency.wait');
const slowest = failedAttempts[0];
const matchingChildren = childSpans.filter(
  (span) => span.spanContext().traceId === slowest.traceId,
);

console.log('slowest failing attempt', slowest);
console.log(
  'matching child operations',
  matchingChildren.map((span) => ({
    name: span.name,
    delayMs: span.attributes['dependency.delay_ms'],
  })),
);

A trace ID joins the parent attempt to its operations without copying every test attribute onto every child. In the lab, failing attempts all exceed 30 ms, and the slowest child reports 45 ms. In a distributed suite, the same trace may include browser actions, an API request, a message publish, and a database operation. Comparing failed and passed traces can expose a lock wait, cold start, delayed event, throttled endpoint, or missing propagation boundary.

Do not assume the longest child caused the failure. Correlation narrows the hypothesis, but causation still requires code and environment evidence. Check whether the assertion deadline is justified, whether a response event is actually awaited, and whether concurrent tests mutate shared state. The guide to debug test automation race conditions provides a useful investigation pattern even if your runner is TypeScript.

Verification: rerun the lab. slowest failing attempt should contain attempt 9, delay 45, and a 32-character trace ID. matching child operations must contain exactly one dependency.wait item with delay 45.

Step 7: Add CI Context Without Creating Cardinality Problems

Extend the attributes passed to startActiveSpan in runInstrumentedAttempt:

attributes: {
  'test.name': testName,
  'test.framework': 'node:test',
  'test.attempt': attempt,
  'test.retry': Math.max(0, attempt - 1),
  'test.outcome': 'unknown',
  'vcs.revision': revision,
  'ci.pipeline.id': process.env.CI_PIPELINE_ID ?? 'local',
  'ci.job.name': process.env.CI_JOB_NAME ?? 'local-trace-test',
  'test.environment': process.env.TEST_ENVIRONMENT ?? 'local',
},

Run IDs and commit hashes are intentionally high-cardinality values. They are appropriate for trace lookup attributes when your backend can index them selectively, but they are poor metric labels. Derive a low-cardinality metric such as failure count by suite and branch separately. Do not attach timestamp values as attribute keys, and do not encode parameters into span names. Keep the span name test.run; put the specific test in test.name.

For matrix jobs, record browser family, operating system, runtime major version, region, and test shard as bounded values. Those dimensions separate a real test flake from a Chromium-only defect or an overloaded runner pool. Record deployment identity on the service under test when trace propagation reaches it. If one environment fails consistently while another passes consistently, classify the environment before blaming the test.

Verification: execute CI_PIPELINE_ID=demo-42 CI_JOB_NAME=e2e TEST_ENVIRONMENT=staging npm run test:traced. Temporarily log testSpans[0].attributes and confirm the three supplied values appear. Remove that broad attribute log afterward because production values may include identifiers you do not want in console artifacts.

Step 8: Replace Memory Export With OTLP in CI

Install the HTTP/protobuf trace exporter that matches the experimental SDK release:

npm install --save-exact @opentelemetry/exporter-trace-otlp-proto@0.221.0

For a CI-oriented src/telemetry-otlp.ts, use the official exporter and let the SDK batch spans:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';

const sdk = new NodeSDK({
  serviceName: 'checkout-e2e-tests',
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
  }),
});

sdk.start();

export async function stopTelemetry(): Promise<void> {
  await sdk.shutdown();
}

Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to a full traces endpoint such as http://collector:4318/v1/traces. Do not append /v1/traces twice. Prefer environment-based headers for credentials, mask them in CI, and never print the exporter configuration. Always await shutdown() after the runner completes so the batch processor flushes buffered spans before the ephemeral worker exits.

Query your backend by service name, test.name, vcs.revision, and test.outcome. Build a scheduled aggregation that groups attempts across recent pipelines rather than treating retry siblings as the entire history. If your organization uses load tests, the k6 OpenTelemetry trace export tutorial shows how performance traffic can join the same observability platform.

Verification: point the variable at a local or staging collector, run the suite, and query for service.name=checkout-e2e-tests. Confirm one test.run parent and its dependency.wait child share a trace ID. Then stop the collector and verify that telemetry failure does not turn correct tests into false functional failures.

Step 9: Turn Trace Evidence Into a Flake Policy

A useful policy separates detection, containment, and repair. Detection requires mixed outcomes for the same identity and revision. Containment moves a confirmed flake out of a blocking gate only when its disruption exceeds an agreed threshold. Repair assigns an owner, links representative passing and failing trace IDs, and removes quarantine after a clean observation window.

Use separate statuses for suspected, confirmed, quarantined, and resolved. A suspected test may have one inconsistent pair. A confirmed test has enough comparable history to withstand random noise and known infrastructure exclusions. Quarantine should expire automatically, because permanent quarantine quietly converts coverage into theater. The flaky test quarantine in CI guide covers ownership and expiry mechanics, while reducing flaky tests in a CI pipeline addresses suite-level prevention.

Retries must create distinct attempt spans and preserve the original failure. Reporting only the final retry turns a fail-then-pass sequence into a clean pass and destroys the strongest flake signal. Track first-attempt pass rate for release confidence, and track eventual pass rate to understand how much instability retries conceal. If retries become routine, consult Playwright trace on retry patterns for artifact strategy.

Verification: write three policy fixtures from the lab output: all pass should classify stable, all fail should classify consistently failing, and the existing 14/6 sample should classify flaky. Review the labels with the team before allowing the classifier to change a required CI gate.

How to Detect Flaky Tests With OpenTelemetry Traces at Scale

At scale, preserve three levels of context. Resource attributes describe the runner service and deployment. The test span describes identity, revision, attempt, result, framework, and bounded environment dimensions. Child spans describe actions and dependencies. This division makes queries predictable and avoids repeating volatile CI fields on every network operation.

Sampling needs special treatment. Head sampling can discard a trace before its failed outcome is known. For diagnostic suites, retain all error traces and a useful sample of passes, or use a collector with tail sampling. You need passing traces because flakiness is a comparison problem. Keeping only failures proves that something broke, but it cannot show mixed behavior or establish a healthy latency baseline. Configure retention according to volume, privacy, and incident needs.

Normalize parameterized tests carefully. Grouping all data cases under one name can manufacture apparent flakiness when only one input is defective. Grouping by raw random value can create millions of identities. Use a stable case ID or bounded input class, then store the seed as lookup context. Similarly, distinguish retries from independent scheduled runs so a burst of ten retries does not outweigh a month of clean first attempts.

Finally, combine trace evidence with ownership metadata. A detector that opens alerts without repository, team, severity, and last-seen context merely moves noise. Route a confirmed finding to the owning team with two trace links, observed window, sample size, outcomes, affected environments, and quarantine expiry.

Troubleshooting

Problem: getFinishedSpans() returns an empty array -> import and start telemetry.ts before importing the module that calls trace.getTracer. Check that only one compatible copy of @opentelemetry/api exists with npm ls @opentelemetry/api.

Problem: the process exits before spans reach the backend -> await sdk.shutdown() in a finally block after the runner finishes. For long-lived workers, handle SIGTERM and give the exporter enough time to flush without hanging the CI job indefinitely.

Problem: every test appears stable despite visible retry passes -> verify that the first failure is emitted as its own attempt span. Do not overwrite an earlier result when the retry finishes, and do not group by attempt number.

Problem: unrelated failures are grouped as one flaky test -> strengthen identity with suite path and a stable parameter-case ID. Keep revision and relevant environment dimensions in the comparison key when behavior legitimately differs across those boundaries.

Problem: the backend bill or index size grows sharply -> keep span names constant, limit indexed attributes, sample healthy traces, and move large console output or screenshots to artifact storage. Never store secrets or customer payloads as a shortcut to debugging.

Problem: OTLP requests return 404 or connection refused -> check protocol and endpoint together. HTTP/protobuf normally targets port 4318 with /v1/traces; gRPC normally targets port 4317 and uses a different exporter package.

Best Practices

• End spans in finally so thrown assertions still produce complete traces.
• Record the exception and error status, but keep a normalized outcome attribute for aggregation.
• Preserve both first attempts and retries. A final green result must not erase prior failure.
• Compare the same revision and meaningful environment before declaring nondeterminism.
• Store seeds, case IDs, shard, browser, and runner image when they can explain variation.
• Keep pass traces as a baseline, especially when diagnosing latency-driven failures.
• Treat a score as prioritization, not proof. Confirm with representative traces and source inspection.
• Make quarantine temporary, owned, visible, and measurable.

Interview Questions and Answers

The model answers in the interviewQnA section below cover span design, grouping, sampling, retries, and the difference between a flaky test and infrastructure instability. In an interview, lead with the classification rule, then explain how child spans shorten root-cause analysis.

Where To Go Next

Run the lab unchanged and confirm the 14/6 split. Then instrument one unstable test in your own suite, using the same test.run contract and a child span around the operation you suspect. Keep the first rollout observational until you trust grouping, shutdown, and privacy behavior.

Next, compare manual evidence with AI-assisted flaky test root-cause analysis, formalize containment through quarantine for flaky tests in CI, and prepare system-design trade-offs with the flaky test detection system interview guide. You can also use the QAJobFit practice workspace to rehearse how you would explain the design.

Conclusion

The reliable way to detect flaky tests with OpenTelemetry traces is to preserve every attempt, group comparable spans by stable test identity and revision, and flag only groups containing both pass and fail outcomes. A transparent flake score helps prioritize work, but the real value comes from the trace tree that connects a failed assertion to timing, retries, and dependency behavior.

Start with one test and complete spans at shutdown. Once the expected counts and classifications are repeatable, export through OTLP, add bounded CI context, and enforce a quarantine policy with ownership and expiry. That sequence turns intermittent failures from anecdotal rerun requests into evidence your team can inspect and act on.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Reduce flaky tests in a CI pipeline (2026)
• How to Use Cypress handling flaky tests (2026)
• Detecting flaky tests with machine learning (2026)
• Export k6 Traces to OpenTelemetry: k6 Export Traces OpenTelemetry Tutorial
• How to Add accessibility checks to CI (2026)
• How to Add CI to a test framework (2026)