Validate Trace Propagation With OpenTelemetry (2026)

TL;DR

Create one gateway span, inject its client-span context into an outbound HTTP request, extract it in a downstream service, and assert that every span has one trace ID with the expected parent chain. Add valid upstream and malformed traceparent cases, then run the same tests in CI.

Key Takeaways

• Assert the trace ID and parent chain, because finding a traceparent header alone does not prove correct propagation.
• Test one real HTTP boundary with separate tracer providers so each service has an independently identifiable resource.
• Parse traceparent into version, trace ID, parent ID, and flags instead of comparing the whole value to a fixture.
• Cover valid upstream context and malformed context to verify both continuation and safe trace restart behavior.
• Use an in-memory exporter for deterministic integration tests, then keep an OTLP collector smoke check as a separate deployment layer.
• Force-flush providers before reading spans and shut down servers and providers even when an assertion fails.
• Run the propagation suite in CI whenever telemetry bootstrap, HTTP clients, middleware, or service boundaries change.

To validate trace propagation with opentelemetry, send a real request through at least two instrumented service boundaries and assert the exported span graph, not merely the presence of a traceparent header. A passing test should prove that all spans share one trace ID, every child names the correct parent span ID, the sampled flag survives injection, and invalid upstream context is rejected safely.

This tutorial builds that proof with Node.js, the OpenTelemetry JavaScript SDK, the W3C Trace Context propagator, Node's built-in HTTP server, and the built-in test runner. The gateway calls an inventory service over localhost. Each service owns a separate tracer provider and in-memory exporter, which keeps the test deterministic while preserving a genuine HTTP serialization boundary.

Manual spans are intentional here. Automatic HTTP instrumentation is valuable in production, but explicit injection and extraction expose the exact contract a QA engineer must verify. You can reuse the assertions after replacing the manual spans with auto-instrumentation, an OTLP Collector, or another language. For broader preparation around trace-focused testing, review the observability testing interview scenarios.

What You Will Build

You will create a small repository with four pieces:

• two HTTP services, checkout-gateway and inventory-service;
• one independent OpenTelemetry provider and exporter per service;
• three integration tests for a new trace, a valid remote parent, and a malformed remote parent; and
• a GitHub Actions job that installs the lockfile and runs the propagation contract.

The exported graph for the normal request must have this shape:

POST /checkout                    checkout-gateway SERVER
└── GET inventory                 checkout-gateway CLIENT
    └── GET /inventory            inventory-service SERVER

The three spans must share a 32-character trace ID. The span IDs must be unique 16-character values. The outbound traceparent parent ID must equal the gateway client span ID, because that client span represents the call known to the downstream service.

Prerequisites

Use Node.js 24.18.0 LTS and the following current package versions: @opentelemetry/api 1.9.1, @opentelemetry/core 2.10.0, @opentelemetry/resources 2.10.0, and @opentelemetry/sdk-trace-base 2.10.0. The stable SDK packages share the 2.10.0 release line. Do not mix arbitrary 1.x and 2.x SDK packages even though the API package has its own compatible 1.x version.

Confirm Node before creating the project:

node --version
npm --version

Verification: node --version prints v24.18.0. The exact npm patch can differ if your Node installation method updates npm separately, but the Node runtime must support global fetch, ECMAScript modules, and node:test.

This test needs no Docker, collector, trace backend, sleep call, or fixed TCP port. The servers bind to port 0, so the operating system allocates free ports. That makes parallel CI jobs less likely to collide.

Step 1: Define how to validate trace propagation with opentelemetry

Write the invariant before the implementation. W3C traceparent version 00 has four hyphen-separated fields: a two-character version, a 32-character lowercase hexadecimal trace ID, a 16-character lowercase hexadecimal parent ID, and two hexadecimal trace-flags characters. All-zero trace and parent IDs are invalid. Bit zero of the flags indicates whether the caller sampled the trace.

For a request that begins at the gateway, the gateway server span has no parent. The gateway client span is its child. Injection serializes the client span context, so the header's parent ID is the client span ID. Extraction at inventory creates a remote parent context, and the inventory server span becomes a child of that client span.

Do not assert that the entire header equals a hard-coded value. OpenTelemetry generates new IDs on every root request, and newer W3C flags can use bits in addition to the sampled bit. Parse fields and assert semantics instead:

header.traceId == gatewayServer.traceId
header.parentId == gatewayClient.spanId
inventoryServer.parentSpanId == gatewayClient.spanId
(header.flags & 1) == 1

Verification: ask a reviewer to identify the parent of each span from the graph and equations. If the expected parent is described as the gateway server span for the inventory server, correct the model before coding. The outbound client span is the remote parent.

Step 2: Scaffold the pinned Node.js project

Create the directory and save this package.json:

mkdir otel-propagation-test
cd otel-propagation-test

{
  "name": "otel-propagation-test",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "engines": {
    "node": "24.18.0"
  },
  "scripts": {
    "test": "node --test trace-propagation.test.mjs"
  },
  "dependencies": {
    "@opentelemetry/api": "1.9.1",
    "@opentelemetry/core": "2.10.0",
    "@opentelemetry/resources": "2.10.0",
    "@opentelemetry/sdk-trace-base": "2.10.0"
  }
}

Install once and commit both the manifest and generated lockfile:

npm install
npm ls --depth=0

Exact pins make a tutorial reproducible. In a maintained service, use an automated dependency update pull request and rerun this suite before accepting a new SDK line. OpenTelemetry context APIs are stable, but SDK configuration and readable-span properties can change across majors.

Verification: npm ls --depth=0 lists the four requested versions without invalid, missing, or extraneous. Confirm package-lock.json exists. If the install resolves another version, inspect the manifest instead of editing the lockfile by hand.

Step 3: Create isolated tracer providers and span summaries

Save telemetry.mjs. SimpleSpanProcessor exports a span as soon as it ends, while forceFlush() gives the test an explicit synchronization point. InMemorySpanExporter avoids network timing and exposes finished readable spans for structural assertions.

import {
  AlwaysOnSampler,
  BasicTracerProvider,
  InMemorySpanExporter,
  ParentBasedSampler,
  SimpleSpanProcessor
} from '@opentelemetry/sdk-trace-base';
import { resourceFromAttributes } from '@opentelemetry/resources';

export function createTelemetry(serviceName) {
  const exporter = new InMemorySpanExporter();
  const provider = new BasicTracerProvider({
    resource: resourceFromAttributes({ 'service.name': serviceName }),
    sampler: new ParentBasedSampler({ root: new AlwaysOnSampler() }),
    spanProcessors: [new SimpleSpanProcessor(exporter)]
  });

  return {
    exporter,
    provider,
    tracer: provider.getTracer('trace-propagation-test', '1.0.0')
  };
}

export function summarize(span) {
  return {
    service: span.resource.attributes['service.name'],
    name: span.name,
    kind: span.kind,
    traceId: span.spanContext().traceId,
    spanId: span.spanContext().spanId,
    parentSpanId: span.parentSpanContext?.spanId ?? null
  };
}

Create two providers rather than registering one global provider. This lets the test distinguish spans by service.name and avoids global state leaking between test cases. A parent-based sampler with an always-on root records locally created traces and respects a valid remote sampling decision.

Verification: run a module import check:

node -e "import('./telemetry.mjs').then(({ createTelemetry }) => { const t = createTelemetry('probe'); console.log(t.tracer.constructor.name); return t.provider.shutdown(); })"

The command prints a tracer constructor name and exits with status 0. An import error usually means a package version drifted or type: module is missing.

Step 4: Implement injection and extraction across real HTTP

Save services.mjs. The getter accepts Node's incoming header object. The setter writes strings into a new outbound carrier. Both services use the same standards-based propagator instance, but they use different tracer providers.

import http from 'node:http';
import { ROOT_CONTEXT, SpanKind, SpanStatusCode, trace } from '@opentelemetry/api';
import { W3CTraceContextPropagator } from '@opentelemetry/core';

const propagator = new W3CTraceContextPropagator();
const getter = {
  keys: carrier => Object.keys(carrier),
  get: (carrier, key) => carrier[key]
};
const setter = {
  set: (carrier, key, value) => { carrier[key] = value; }
};

async function listen(server) {
  await new Promise((resolve, reject) => {
    server.once('error', reject);
    server.listen(0, '127.0.0.1', resolve);
  });
  const address = server.address();
  return `http://127.0.0.1:${address.port}`;
}

function close(server) {
  return new Promise((resolve, reject) => {
    server.close(error => error ? reject(error) : resolve());
  });
}

export async function startInventory(tracer) {
  const server = http.createServer((request, response) => {
    if (request.method !== 'GET' || request.url !== '/inventory') {
      response.writeHead(404).end();
      return;
    }

    const remoteContext = propagator.extract(ROOT_CONTEXT, request.headers, getter);
    const span = tracer.startSpan(
      'GET /inventory',
      { kind: SpanKind.SERVER },
      remoteContext
    );

    const spanContext = span.spanContext();
    response.writeHead(200, { 'content-type': 'application/json' });
    response.end(JSON.stringify({
      inStock: true,
      traceId: spanContext.traceId,
      spanId: spanContext.spanId,
      receivedTraceparent: request.headers.traceparent ?? null
    }));
    span.end();
  });

  return { url: await listen(server), close: () => close(server) };
}

export async function startGateway(tracer, inventoryUrl) {
  const server = http.createServer(async (request, response) => {
    if (request.method !== 'POST' || request.url !== '/checkout') {
      response.writeHead(404).end();
      return;
    }

    const remoteContext = propagator.extract(ROOT_CONTEXT, request.headers, getter);
    const serverSpan = tracer.startSpan(
      'POST /checkout',
      { kind: SpanKind.SERVER },
      remoteContext
    );
    const serverContext = trace.setSpan(remoteContext, serverSpan);
    const clientSpan = tracer.startSpan(
      'GET inventory',
      { kind: SpanKind.CLIENT },
      serverContext
    );
    const clientContext = trace.setSpan(serverContext, clientSpan);
    const headers = {};
    propagator.inject(clientContext, headers, setter);

    try {
      const inventoryResponse = await fetch(`${inventoryUrl}/inventory`, { headers });
      const inventory = await inventoryResponse.json();
      clientSpan.setStatus({ code: SpanStatusCode.OK });
      serverSpan.setStatus({ code: SpanStatusCode.OK });
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end(JSON.stringify(inventory));
    } catch (error) {
      clientSpan.recordException(error);
      clientSpan.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      serverSpan.setStatus({ code: SpanStatusCode.ERROR, message: 'inventory failed' });
      response.writeHead(502).end();
    } finally {
      clientSpan.end();
      serverSpan.end();
    }
  });

  return { url: await listen(server), close: () => close(server) };
}

Extraction never trusts a header by string shape alone. The propagator validates it and returns a derived context only when it is usable. Injection receives the context containing the client span, which is the detail most frequently lost in handwritten middleware.

Verification: check both modules before starting a server:

node --check telemetry.mjs
node --check services.mjs

Both commands exit silently with code 0. This proves syntax only. The next step proves that a request serializes and reconstructs the intended span context.

Step 5: Assert the complete span graph

Save trace-propagation.test.mjs. Setup starts inventory first, then gives its allocated URL to the gateway. Teardown closes listeners and providers in finally, so a failed assertion does not leave Node waiting on open sockets.

import test from 'node:test';
import assert from 'node:assert/strict';
import { createTelemetry, summarize } from './telemetry.mjs';
import { startGateway, startInventory } from './services.mjs';

async function setup() {
  const gatewayTelemetry = createTelemetry('checkout-gateway');
  const inventoryTelemetry = createTelemetry('inventory-service');
  const inventory = await startInventory(inventoryTelemetry.tracer);
  const gateway = await startGateway(gatewayTelemetry.tracer, inventory.url);
  return { gatewayTelemetry, inventoryTelemetry, inventory, gateway };
}

async function finished(system) {
  await Promise.all([
    system.gatewayTelemetry.provider.forceFlush(),
    system.inventoryTelemetry.provider.forceFlush()
  ]);
  return [
    ...system.gatewayTelemetry.exporter.getFinishedSpans(),
    ...system.inventoryTelemetry.exporter.getFinishedSpans()
  ].map(summarize);
}

async function teardown(system) {
  await Promise.all([system.gateway.close(), system.inventory.close()]);
  await Promise.all([
    system.gatewayTelemetry.provider.shutdown(),
    system.inventoryTelemetry.provider.shutdown()
  ]);
}

function byName(spans) {
  return Object.fromEntries(spans.map(span => [span.name, span]));
}

function parseTraceparent(value) {
  assert.match(value, /^[0-9a-f]{2}-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$/);
  const [version, traceId, parentId, flags] = value.split('-');
  return { version, traceId, parentId, flags: Number.parseInt(flags, 16) };
}

test('propagates one trace through gateway and inventory', async () => {
  const system = await setup();
  try {
    const response = await fetch(`${system.gateway.url}/checkout`, { method: 'POST' });
    assert.equal(response.status, 200);
    const payload = await response.json();
    const spans = await finished(system);
    assert.equal(spans.length, 3);

    const named = byName(spans);
    const traceIds = new Set(spans.map(span => span.traceId));
    assert.equal(traceIds.size, 1);
    assert.equal(named['GET inventory'].parentSpanId, named['POST /checkout'].spanId);
    assert.equal(named['GET /inventory'].parentSpanId, named['GET inventory'].spanId);
    assert.equal(named['POST /checkout'].parentSpanId, null);

    const header = parseTraceparent(payload.receivedTraceparent);
    assert.equal(header.version, '00');
    assert.equal(header.traceId, named['GET inventory'].traceId);
    assert.equal(header.parentId, named['GET inventory'].spanId);
    assert.equal(header.flags & 1, 1);
  } finally {
    await teardown(system);
  }
});

test('continues a valid sampled upstream trace', async () => {
  const system = await setup();
  const traceId = '11111111111111111111111111111111';
  const upstreamSpanId = '2222222222222222';
  try {
    await fetch(`${system.gateway.url}/checkout`, {
      method: 'POST',
      headers: { traceparent: `00-${traceId}-${upstreamSpanId}-01` }
    });
    const spans = await finished(system);
    const named = byName(spans);
    assert.deepEqual(new Set(spans.map(span => span.traceId)), new Set([traceId]));
    assert.equal(named['POST /checkout'].parentSpanId, upstreamSpanId);
  } finally {
    await teardown(system);
  }
});

test('restarts tracing when upstream trace ID is invalid', async () => {
  const system = await setup();
  const invalidTraceId = '00000000000000000000000000000000';
  try {
    await fetch(`${system.gateway.url}/checkout`, {
      method: 'POST',
      headers: { traceparent: `00-${invalidTraceId}-3333333333333333-01` }
    });
    const spans = await finished(system);
    const named = byName(spans);
    assert.equal(new Set(spans.map(span => span.traceId)).size, 1);
    assert.notEqual(named['POST /checkout'].traceId, invalidTraceId);
    assert.equal(named['POST /checkout'].parentSpanId, null);
    assert.equal(named['GET /inventory'].parentSpanId, named['GET inventory'].spanId);
  } finally {
    await teardown(system);
  }
});

The happy-path test checks header syntax, trace continuity, two local parent relationships, the lack of a gateway parent, and sampling. The upstream test proves remote extraction rather than only outbound injection. The malformed case proves an invalid all-zero trace ID cannot force the application to join a bogus trace.

Verification: run the full suite:

npm test

Node's TAP summary reports tests 3, pass 3, fail 0, and the process exits with status 0. To prove the test can fail, temporarily change the expected inventory parent to the gateway server span ID. One test must fail with unequal 16-character IDs. Restore the correct assertion.

Step 6: Validate trace propagation with opentelemetry beyond the happy path

The three tests establish a useful minimum, but production transports add risks. Extend the same harness with one focused case per risk rather than producing a large snapshot of entire spans. Useful cases include an unsampled upstream context ending in flags 00, a valid tracestate, concurrent requests, retries, HTTP redirects, proxy header normalization, and a downstream timeout.

Concurrency deserves special attention when production code relies on automatic context managers. Start 50 requests with unique upstream trace IDs, then group exported spans by trace ID. Every group should contain only its own gateway and inventory spans. A test that checks only span count can miss context bleed between asynchronous requests. This explicit tutorial passes context as function arguments, so it isolates the wire contract from async-local context behavior. Test both layers if the application uses automatic instrumentation.

An unsampled remote parent needs a different expectation. ParentBasedSampler normally honors the remote 00 decision, so spans can be non-recording and absent from the exporter. Assert the response still succeeds and no sampled spans appear. Do not call missing telemetry a propagation bug until you have inspected the sampling flag and policy.

When traffic comes from a performance suite, the k6 OpenTelemetry trace export tutorial shows how load-generated requests contribute trace evidence. Keep performance volume tests separate from this deterministic graph test, because backend ingestion delay should not make a pull-request contract flaky.

Verification: add one case at a time and run node --test --test-name-pattern='upstream' trace-propagation.test.mjs or the relevant test-name fragment. Confirm the new test turns red when you mutate the exact behavior it protects. A test that stays green after its propagation code is removed is not observing the contract.

Step 7: Gate propagation regressions in CI

Save .github/workflows/trace-propagation.yml. Current major versions are actions/checkout@v6 and actions/setup-node@v6. Pin the Node patch used locally and install from the committed lockfile.

name: Trace propagation contract

on:
  pull_request:
    paths:
      - '**/*.mjs'
      - package.json
      - package-lock.json
      - .github/workflows/trace-propagation.yml
  push:
    branches: [main]

jobs:
  propagation:
    runs-on: ubuntu-24.04
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: 24.18.0
          cache: npm
      - run: npm ci
      - run: npm test

npm ci rejects a manifest and lockfile mismatch. The five-minute job timeout catches leaked servers that would otherwise leave the runner hanging. Path filters should include telemetry initialization, shared HTTP clients, gateways, proxy code, and service handlers in a real repository, not only files containing the word trace.

Make this workflow a required check when trace continuity supports incident diagnosis or SLO evidence. The test automation CI/CD guide covers wider pipeline placement, while the DevOps roadmap for QA engineers connects observability checks to deployment skills.

Verification: push the passing suite and confirm the job finishes within five minutes. Then remove propagator.inject(...) on a branch. The inventory span should receive a different trace ID, the first test should fail, and the workflow should block the merge.

Troubleshooting

Problem: inventory has a different trace ID -> Inspect receivedTraceparent first. If it is null, the client did not inject or an intermediary removed the header. If it exists, parse it and compare its trace ID with the gateway client span. Verify extraction uses the incoming headers and that the extracted context is passed as the third argument to startSpan.

Problem: inventory has the gateway server as its parent -> Injection used serverContext instead of clientContext, or no client span was created. Create the client span from the server context, put that span into a new context, and inject the new context. The downstream parent represents the outbound operation.

Problem: getFinishedSpans() returns an empty array -> Confirm every span calls end(), the provider has SimpleSpanProcessor, and the test calls forceFlush() before reading. Also inspect the remote sampled flag. A parent-based sampler can correctly produce no recording spans for an unsampled remote parent.

Problem: the test passes locally but hangs in CI -> Put server and provider cleanup inside finally. Close both listeners before shutting down providers, avoid fixed ports, and keep a job timeout. Run node --test --test-force-exit only as a diagnostic because forced exit can hide a leaked resource.

Problem: parentSpanContext is undefined for a real parent -> Check that the SDK packages share the intended 2.10.0 line. Older examples may read parentSpanId, while current readable spans expose the parent context. Log the summarized span once, then align code and lockfile rather than supporting two shapes silently.

Problem: a malformed header causes a 500 response -> Do not split and trust caller input inside application middleware. Let W3CTraceContextPropagator validate and extract it. Invalid context should be ignored, after which startSpan creates a fresh valid root under the configured sampler.

Interview Questions and Answers

Q: What proves distributed trace propagation is correct?

A valid header is only transport evidence. Strong proof combines one trace ID across services, unique span IDs, the correct parent chain, expected remote-parent behavior, and sampling continuity. I also test malformed input so propagation cannot break the request path.

Q: Why should the downstream server span parent the client span instead of the upstream server span?

The client span represents the outbound request as known by the caller. Its span ID is serialized into traceparent as the parent ID. Parenting the downstream server directly to the upstream server removes the network operation from the causal chain.

Q: How would you test propagation without a trace backend?

I would use independent in-memory exporters and a real transport boundary. After the request completes, I would force-flush each provider and assert trace IDs, span IDs, parents, service resources, and header fields. This is faster and more deterministic than polling a backend.

Q: What changes when the upstream sampled flag is zero?

A parent-based sampler commonly honors that remote decision. The context may still propagate, but the SDK can create non-recording spans that never reach the exporter. The test must distinguish propagation from recording and export.

Q: Why test invalid traceparent values?

Headers are untrusted input and can be truncated, malformed, duplicated, or use forbidden all-zero IDs. A compliant propagator ignores invalid context without throwing. The service should continue the request with a newly generated root when policy allows.

Q: Where does an OpenTelemetry Collector fit in this strategy?

The in-memory suite validates application-side graph construction. A separate deployment smoke test should send OTLP data through the collector and query the chosen backend to prove receivers, processors, exporters, credentials, and retention. Keeping these layers separate localizes failures.

Common Mistakes

• Checking only that traceparent exists, which misses stale, invalid, or incorrectly parented values.
• Reusing one global provider for every fake service, which hides resource attribution bugs.
• Injecting before the client span becomes current, causing the downstream span to skip a node in the graph.
• Asserting fixed generated IDs, which turns randomness into brittle test data.
• Polling a remote backend in every pull request when an in-memory exporter can test the same application contract.
• Forgetting forceFlush() and compensating with arbitrary sleeps.
• Treating an unsampled trace as lost propagation without checking flags and sampler configuration.
• Allowing untrusted callers to force sampling or inject arbitrary baggage without an explicit boundary policy.
• Omitting concurrent-request coverage when AsyncLocalStorage or framework auto-instrumentation carries active context.
• Validating traces visually but never converting the parent graph into executable assertions.

Where To Go Next

Replace the manual HTTP spans with the same instrumentation used by your application, then keep the assertions unchanged. Add a collector smoke layer only after the in-process contract is stable. If test failures themselves need trace diagnosis, follow the guide to detecting flaky tests with OpenTelemetry traces. For browser-grid infrastructure, the Selenium Grid OpenTelemetry monitoring setup shows a different service topology with the same continuity concerns.

Expand coverage at real risk boundaries: API gateway to service, HTTP to messaging, retry worker to dependency, and public edge to trusted network. Name each test after the invariant it protects, and store trace IDs only as diagnostic artifacts rather than long-lived secrets.

Conclusion

To validate trace propagation with opentelemetry reliably, assert a distributed graph: one trace ID, unique spans, correct parent IDs, preserved sampling, accepted valid context, and rejected malformed context. The runnable two-service test provides fast evidence without depending on a collector or vendor UI.

Run it on every telemetry or transport change, then complement it with a deployment smoke test for OTLP delivery. That combination separates application propagation defects from collector, backend, and retention failures.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Detect Flaky Tests With OpenTelemetry Traces (2026)
• Validate ETL Schema Drift With dbt (2026)
• Accessibility testing with Cypress (2026)
• Accessibility testing with Playwright (2026)
• Agentic testing with tool calling (2026)
• AI powered visual testing with vision models (2026)