Design a Cross-Platform Test Execution Service

TL;DR

Build an API-driven control plane that validates a test manifest, expands it into platform-specific jobs, and leases those jobs to isolated workers. Store durable state and artifacts outside workers, use capability-aware scheduling, and make every transition idempotent.

Key Takeaways

• Separate the control plane that accepts and schedules runs from the worker plane that executes tests.
• Model platform capabilities explicitly so the scheduler can match jobs to compatible workers.
• Use leases, heartbeats, and idempotent state transitions to recover safely from worker loss.
• Isolate every attempt and upload logs, reports, screenshots, and traces before cleanup.
• Retry infrastructure failures separately from assertion failures to avoid hiding product defects.
• Design for backpressure, tenant quotas, observability, and security from the first architecture diagram.

A strong design cross platform test execution service interview answer starts with one boundary: the service accepts a test bundle plus a platform matrix, then schedules isolated attempts on compatible workers and returns durable results. It must handle browsers, operating systems, emulators, simulators, and physical devices without coupling the public API to any one test framework.

This tutorial builds a small but runnable control plane in TypeScript. Use it to practice the deeper framework in the Senior SDET system design interview complete guide, then explain how you would replace its in-memory components at production scale.

You will move from requirements to API, scheduler, leases, retry classification, and observability. The code is intentionally local, but its contracts map cleanly to PostgreSQL, a queue, object storage, and Kubernetes workers.

What You Will Build

By the end, you will have:

• A Fastify API that accepts a test manifest and expands a platform matrix into jobs.
• A capability-aware scheduler that prevents a Linux worker from claiming a macOS or Android job.
• Lease and heartbeat rules that recover abandoned work without running it forever.
• Attempt classification that retries infrastructure failures but preserves test failures.
• Status and metrics endpoints that an interviewer can connect to dashboards and alerts.

The architecture has two planes. The control plane owns APIs, durable state, scheduling policy, and audit history. The execution plane contains disposable workers with browsers, SDKs, or attached devices. Workers pull work, run one attempt in isolation, upload artifacts, report a terminal result, and clean the environment.

Prerequisites

Use Node.js 22 LTS or newer and npm 10 or newer. The examples use TypeScript 5, Fastify 5, and Zod 4 APIs available in 2026. Start in an empty practice directory, not inside your application repository.

mkdir cross-platform-runner && cd cross-platform-runner
npm init -y
npm install fastify zod
npm install -D typescript tsx @types/node
npx tsc --init --target ES2022 --module NodeNext --moduleResolution NodeNext --strict
mkdir src

Add scripts to package.json so npm run dev executes tsx watch src/server.ts and npm run start executes tsx src/server.ts. You also need curl in a second terminal. Docker and Kubernetes are not required for this tutorial.

Verify the setup with node --version, npm --version, and npx tsx --version. Each command should print a version and exit with status 0.

Step 1: Frame the Design Cross Platform Test Execution Service Interview Requirements

Start by stating the workload and invariants before drawing boxes. Assume clients submit an immutable bundle URI, command, environment references, timeout, and a platform matrix. A run can contain hundreds of jobs, and each job can have multiple attempts.

Write these functional requirements on the interview whiteboard:

1. Submit, cancel, and inspect a run.
2. Match each job to a worker with the required OS, architecture, browser, runtime, or device.
3. Stream or retrieve logs and artifacts.
4. Retry recoverable infrastructure failures.
5. enforce tenant concurrency and priority fairly.

Add nonfunctional requirements: no two live workers should own the same lease, accepted runs must survive control-plane restarts, workers are untrusted, and overload must produce backpressure instead of collapse. Clarify that exactly-once physical execution is unrealistic under network partitions. Promise at-least-once delivery with idempotent result commits and a single accepted terminal state.

Estimate with symbols when the interviewer gives no traffic numbers. If peak submissions are R runs per second, average matrix width is M, and average duration is D seconds, expected concurrency is roughly R * M * D. Add headroom for retries and platform imbalance. This exposes the real bottleneck: scarce macOS machines or physical devices, not API throughput.

Verify this step by checking that your design can answer four questions: what is a run, what is a job, what is an attempt, and which component owns each state transition. If any answer is ambiguous, fix the model before writing an endpoint.

Step 2: Define the Manifest and State Machine

Create src/domain.ts. Keep requested capabilities data-driven so adding a browser does not require a new endpoint.

import { z } from "zod";

export const PlatformSchema = z.object({
  os: z.enum(["linux", "windows", "macos", "android", "ios"]),
  arch: z.enum(["x64", "arm64"]),
  browser: z.enum(["chromium", "firefox", "webkit"]).optional(),
  browserVersion: z.string().optional(),
  device: z.string().optional()
});

export const ManifestSchema = z.object({
  tenantId: z.string().min(1),
  bundleUri: z.string().url(),
  command: z.array(z.string().min(1)).min(1),
  platforms: z.array(PlatformSchema).min(1).max(100),
  timeoutSeconds: z.number().int().min(10).max(7200).default(1800),
  maxInfrastructureRetries: z.number().int().min(0).max(3).default(1)
});

export type Platform = z.infer<typeof PlatformSchema>;
export type Manifest = z.infer<typeof ManifestSchema>;
export type JobState = "queued" | "leased" | "running" | "passed" | "failed" | "canceled";
export type Job = {
  id: string; runId: string; tenantId: string; platform: Platform;
  state: JobState; attempt: number; maxInfrastructureRetries: number;
  leaseOwner?: string; leaseExpiresAt?: number; result?: string;
};

The run expands into one job per platform entry. An attempt is a time-bounded claim on a job. Legal transitions are queued -> leased -> running -> passed|failed, while cancellation can move nonterminal work to canceled. A lease expiry moves recoverable work back to queued.

In production, validate supported platform combinations against a versioned capability catalog. Reject impossible pairs such as Safari on Linux during submission, not after a worker spends capacity downloading the bundle. Store normalized manifests and never mutate them after acceptance.

Verify with npx tsc --noEmit. It should return without diagnostics. Also try changing an OS literal to solaris; TypeScript should reject it, while runtime Zod validation will reject the same value from an API client.

Step 3: Build an Idempotent Submission API

Create src/server.ts with a minimal store and submission endpoint. An idempotency key prevents a client timeout from creating duplicate runs.

import Fastify from "fastify";
import { randomUUID } from "node:crypto";
import { ManifestSchema, type Job } from "./domain.js";

const app = Fastify({ logger: true });
const jobs = new Map<string, Job>();
const runs = new Map<string, { id: string; jobIds: string[] }>();
const idempotency = new Map<string, string>();

app.post("/v1/runs", async (request, reply) => {
  const key = request.headers["idempotency-key"];
  if (typeof key !== "string") return reply.code(400).send({ error: "idempotency-key required" });
  const prior = idempotency.get(key);
  if (prior) return reply.code(200).send(runs.get(prior));

  const parsed = ManifestSchema.safeParse(request.body);
  if (!parsed.success) return reply.code(400).send({ error: parsed.error.flatten() });
  const runId = randomUUID();
  const jobIds = parsed.data.platforms.map((platform) => {
    const id = randomUUID();
    jobs.set(id, { id, runId, tenantId: parsed.data.tenantId, platform,
      state: "queued", attempt: 0,
      maxInfrastructureRetries: parsed.data.maxInfrastructureRetries });
    return id;
  });
  const run = { id: runId, jobIds };
  runs.set(runId, run);
  idempotency.set(key, runId);
  return reply.code(202).send(run);
});

A production transaction inserts the idempotency key, run, and jobs atomically. Put a unique constraint on (tenant_id, idempotency_key). Return the original response for the same key and payload, but return a conflict if the key is reused with a different payload hash.

Append app.listen({ port: 3000, host: "127.0.0.1" }); temporarily, run npm run start, and submit a manifest twice with the same key.

curl -s -X POST http://127.0.0.1:3000/v1/runs \
  -H 'content-type: application/json' -H 'idempotency-key: demo-1' \
  -d '{"tenantId":"team-a","bundleUri":"https://example.test/bundle.tgz","command":["npm","test"],"platforms":[{"os":"linux","arch":"x64","browser":"chromium"}]}'

Verify both responses contain the same run ID and one job ID. Stop the server, then continue editing before the next step.

Step 4: Match Jobs to Worker Capabilities

Add worker schemas and a pure matcher below the imports in server.ts. Exact matching makes scheduling behavior easy to test.

import { z } from "zod";
import type { Platform } from "./domain.js";

const WorkerSchema = z.object({
  workerId: z.string().min(1),
  platforms: z.array(z.object({
    os: z.string(), arch: z.string(), browsers: z.array(z.string()).default([]),
    devices: z.array(z.string()).default([])
  })).min(1)
});

function supports(worker: z.infer<typeof WorkerSchema>, requested: Platform): boolean {
  return worker.platforms.some((available) =>
    available.os === requested.os && available.arch === requested.arch &&
    (!requested.browser || available.browsers.includes(requested.browser)) &&
    (!requested.device || available.devices.includes(requested.device))
  );
}

Then add a pull endpoint. It reclaims expired leases and chooses one compatible queued job.

app.post("/v1/workers/lease", async (request, reply) => {
  const parsed = WorkerSchema.safeParse(request.body);
  if (!parsed.success) return reply.code(400).send({ error: parsed.error.flatten() });
  const now = Date.now();
  for (const job of jobs.values()) {
    if (job.state === "leased" && (job.leaseExpiresAt ?? 0) <= now) {
      job.state = "queued"; job.leaseOwner = undefined; job.leaseExpiresAt = undefined;
    }
  }
  const job = [...jobs.values()].find((candidate) =>
    candidate.state === "queued" && supports(parsed.data, candidate.platform));
  if (!job) return reply.code(204).send();
  job.state = "leased";
  job.attempt += 1;
  job.leaseOwner = parsed.data.workerId;
  job.leaseExpiresAt = now + 30_000;
  return { job, leaseToken: `${job.id}:${job.attempt}` };
});

The map mutation is safe only in one Node process. Production schedulers must claim atomically, for example with a PostgreSQL transaction using FOR UPDATE SKIP LOCKED, or a queue whose visibility timeout acts as a lease. Apply tenant quotas before selection, then use weighted fair queues or deficit round robin to prevent a large tenant from starving others. Maintain separate pools for scarce devices.

Verify by submitting Linux and macOS jobs. A worker advertising only Linux must receive Linux and then get HTTP 204, never macOS. This capability test is a high-value unit test because a mismatch wastes minutes before failing.

Step 5: Add Heartbeats and Fenced Result Commits

A worker can pause, lose its network, or finish after its lease was reassigned. Use the job ID plus attempt number as a fencing token. Add these endpoints before listen.

const LeaseActionSchema = z.object({ workerId: z.string(), attempt: z.number().int().positive() });

app.post("/v1/jobs/:id/heartbeat", async (request, reply) => {
  const input = LeaseActionSchema.safeParse(request.body);
  const job = jobs.get((request.params as { id: string }).id);
  if (!input.success || !job) return reply.code(404).send({ error: "job or lease not found" });
  if (job.leaseOwner !== input.data.workerId || job.attempt !== input.data.attempt ||
      job.state !== "leased") return reply.code(409).send({ error: "stale lease" });
  job.leaseExpiresAt = Date.now() + 30_000;
  return { leaseExpiresAt: job.leaseExpiresAt };
});

const ResultSchema = LeaseActionSchema.extend({
  outcome: z.enum(["passed", "assertion_failure", "infrastructure_failure"]),
  summary: z.string().max(2000),
  artifactUris: z.array(z.string().url()).default([])
});

app.post("/v1/jobs/:id/result", async (request, reply) => {
  const input = ResultSchema.safeParse(request.body);
  const job = jobs.get((request.params as { id: string }).id);
  if (!input.success || !job) return reply.code(404).send({ error: "job or result invalid" });
  if (job.leaseOwner !== input.data.workerId || job.attempt !== input.data.attempt ||
      job.state !== "leased") return reply.code(409).send({ error: "stale lease" });
  job.result = input.data.summary;
  if (input.data.outcome === "infrastructure_failure" &&
      job.attempt <= job.maxInfrastructureRetries) {
    job.state = "queued"; job.leaseOwner = undefined; job.leaseExpiresAt = undefined;
    return reply.code(202).send({ state: job.state });
  }
  job.state = input.data.outcome === "passed" ? "passed" : "failed";
  return { state: job.state, artifactUris: input.data.artifactUris };
});

Workers should heartbeat at an interval well below the lease duration, with jitter. Result submission must be retryable. A duplicate result for the accepted attempt should return the stored result, while an older attempt receives 409. Production code stores artifacts before committing the result and records an immutable attempt row for diagnosis.

Verify by leasing a job and posting a result with an incorrect attempt. Expect 409. Post an infrastructure failure with the correct attempt and expect queued; lease it again and confirm the attempt increments. Post an assertion failure and confirm it becomes failed, not queued.

Step 6: Isolate Execution and Secure Secrets

The worker must treat bundles and test commands as untrusted. Run each attempt in an ephemeral container or VM with CPU, memory, process, and wall-clock limits. Mobile work needs an emulator, simulator, or exclusive physical-device reservation, but it follows the same lease contract.

A worker loop can stay framework-neutral:

type Lease = { job: Job; leaseToken: string };

async function execute(lease: Lease): Promise<void> {
  const workspace = await createEphemeralWorkspace(lease.job.id);
  try {
    await downloadAndVerifyBundle(workspace);
    const exit = await runSandboxed(workspace, { timeoutMs: 1_800_000 });
    await uploadArtifacts(workspace, lease.job.runId, lease.job.id);
    await commitResult(lease, exit.code === 0 ? "passed" : "assertion_failure");
  } catch (error) {
    await commitResult(lease, classifyInfrastructure(error));
  } finally {
    await destroyWorkspace(workspace);
  }
}

These functions are adapter boundaries, not code to paste into the tutorial server. In production, implement them with your chosen container runtime and object store. The important sequence is create, verify, execute, upload, commit, destroy. Upload incrementally for long runs so a killed worker does not erase every diagnostic.

Issue short-lived, job-scoped credentials. Never place tenant secrets in queue messages, logs, command arguments, or container images. Restrict egress by default, mount the workspace read-write while keeping the base image read-only, run as a non-root identity, and sign approved worker images. Redact common token patterns before log upload.

Verify the design with a deliberately timed-out test. The process tree must stop, partial logs must remain available, the workspace must disappear, and the final classification must be an infrastructure timeout rather than an assertion failure.

Step 7: Expose Status, Cancellation, and Observability

Add a status endpoint that derives run state from jobs. Keep job attempts as the source of truth.

app.get("/v1/runs/:id", async (request, reply) => {
  const run = runs.get((request.params as { id: string }).id);
  if (!run) return reply.code(404).send({ error: "run not found" });
  const runJobs = run.jobIds.map((id) => jobs.get(id)!);
  const terminal = runJobs.every((job) => ["passed", "failed", "canceled"].includes(job.state));
  return { id: run.id, state: terminal ? "completed" : "running", jobs: runJobs };
});

app.post("/v1/runs/:id/cancel", async (request, reply) => {
  const run = runs.get((request.params as { id: string }).id);
  if (!run) return reply.code(404).send({ error: "run not found" });
  for (const id of run.jobIds) {
    const job = jobs.get(id)!;
    if (["queued", "leased", "running"].includes(job.state)) job.state = "canceled";
  }
  return reply.code(202).send({ state: "canceling" });
});

Cancellation is cooperative. Signal a leased worker, reject its later success with the fencing rule, and use a deadline to force termination. Preserve artifacts according to policy. Do not delete evidence immediately because cancellation often follows a hang.

Instrument submission-to-start queue latency by platform, execution duration, pass rate, infrastructure-failure rate, retry count, lease expiry count, scheduler depth, worker utilization, and artifact upload failures. Trace runId, jobId, and attempt across API, scheduler, worker, and artifact service. Avoid tenant IDs and test names as unrestricted metric labels because high cardinality increases cost and harms query performance.

Verify end to end: restart the server, submit a run, lease it, heartbeat, report a pass, and fetch the run. Expect completed and a passed job. Then cancel a second leased run and confirm a late result receives 409. The local maps reset on restart, which is the explicit reason production state belongs in a database.

Troubleshooting

A compatible job returns HTTP 204 -> Normalize capability values and inspect exact OS, architecture, browser, version, and device constraints. Version ranges need a real semantic-version matcher rather than string equality.

Two workers execute the same job -> Make job claim a single conditional database update, use an expiring lease, and fence heartbeats and results with the attempt number. Do not rely on an in-process mutex across replicas.

Jobs remain queued while workers are idle -> Break queue depth down by platform. A global utilization chart can hide exhausted macOS or device pools. Autoscale the constrained pool and reject unsupported matrices at submission.

Retries hide real product failures -> Retry only classified infrastructure failures such as worker loss, browser startup failure, or device disconnect. Assertion failures should remain failed unless the client explicitly requests a separate test-level retry policy. Study the flaky test detection system design before using historical flakiness in policy.

Artifacts disappear after a crash -> Upload logs in chunks, write artifacts to external object storage, and commit an artifact manifest before worker cleanup. Separate metadata and blob design using the test result storage system interview tutorial.

One tenant starves everyone else -> Enforce per-tenant admitted and running limits. Schedule with weighted fairness inside each platform pool, expose quota errors, and age low-priority jobs so they eventually advance.

Interview Questions and Answers

Q: Why separate control and execution planes?

The control plane needs durability, consistent policy, and modest resources. Workers need elastic compute, privileged integrations, and frequent image changes. Separation lets each scale and fail independently while reducing the blast radius of untrusted tests.

Q: How do you prevent duplicate execution?

You cannot guarantee no duplicate physical execution during every partition. Use atomic claims, leases, heartbeats, attempt fencing, and idempotent terminal commits so only one attempt becomes authoritative. Make side effects inside tests the test owner's responsibility or provide isolated environments.

Q: How does scheduling stay fair?

Partition capacity by hard platform constraints, enforce tenant concurrency limits, and use weighted fair queuing within a pool. Priority can change a tenant's order, but should not bypass its quota. Reserve capacity only when service tiers require it.

Q: How would you schedule physical devices?

Treat each device as an exclusive, health-scored resource with capabilities and quarantine state. Lease it to one attempt, reset it before and after use, and remove unhealthy devices from matching. The device farm scheduler system design covers reservation and recovery in depth.

Q: Where should test results live?

Store normalized run, job, and attempt metadata in a relational database. Put large logs, screenshots, videos, and traces in object storage, referenced by immutable artifact records. Apply separate retention policies and signed download authorization.

Q: What happens when a worker dies?

Its heartbeats stop and its lease expires. A reaper or the next scheduler transaction queues the job again if retry policy permits. The old worker cannot commit after recovery because its attempt fencing token is stale.

Q: How do you autoscale workers?

Scale each platform pool on compatible queue depth and queue age, not global CPU alone. Include startup time, current leases, device availability, and maximum tenant demand. Keep a small warm pool when cold starts would violate the queue latency objective.

Q: How do you roll out a new browser image?

Publish an immutable, signed image and advertise its exact capabilities. Canary a small percentage of eligible jobs, compare infrastructure failure and startup metrics, then expand. Retain the prior image for fast rollback and record image digest on every attempt.

Q: How is the public API made idempotent?

Require a tenant-scoped idempotency key on submission and store it with a request hash in the same transaction as the run. A repeated matching request returns the original run. Reuse with a different hash returns a conflict.

Q: Which consistency guarantees matter most?

Job ownership and terminal result acceptance need strong conditional consistency. Status dashboards and aggregate metrics can be eventually consistent. Artifact upload can complete before metadata commit, followed by garbage collection for orphaned blobs.

Common Mistakes and Best Practices

• Do not model every browser and OS as a different service. Use a stable job contract and capability adapters.
• Do not let workers own durable truth. Workers are disposable and may vanish at any instruction.
• Do not claim exactly-once execution. Explain at-least-once work delivery and exactly-once authoritative result acceptance.
• Do not use one FIFO queue for every platform. Head-of-line blocking will leave compatible workers idle.
• Do store an immutable attempt history, including image digest, capability snapshot, timestamps, exit reason, and artifact manifest.
• Do distinguish service retries from framework retries. They solve different failures and need separate budgets.
• Do enforce bundle limits, decompression limits, timeouts, quotas, and egress policy before accepting arbitrary test code.
• Do define service-level objectives for API availability, queue delay by pool, result durability, and cancellation latency.

Where To Go Next

Return to the complete Senior SDET system design interview guide and practice presenting this design in 35 to 45 minutes. Lead with requirements, draw the control and execution planes, then spend most of the discussion on one hard problem such as leases or heterogeneous scheduling.

Deepen the design with these sibling tutorials:

• Design durable test result storage for schemas, artifact retention, and result queries.
• Design a flaky test detection system for history-based classification and quarantine signals.
• Design a device farm scheduler for exclusive reservations, device health, and recovery.

Then replace the in-memory store with PostgreSQL, add transactional claims, and write concurrency tests that race two schedulers. That exercise turns an interview diagram into evidence that you understand failure boundaries.

Conclusion

To design cross platform test execution service interview architecture convincingly, make heterogeneous capabilities, durable ownership, and failure recovery central. The API is the easy part. The signal comes from explaining leases, attempt fencing, isolation, fair scheduling, retry classification, and artifacts with precise state transitions.

Build the local control plane, test stale leases and duplicate submissions, then rehearse the production substitutions aloud. You will have both a runnable example and a design that remains credible when the interviewer adds scale, mobile devices, tenant isolation, or regional failure.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Design a Flaky Test Detection System Interview
• Design a Test Result Storage System
• Design Device Farm Scheduler Interview: An SDET System Design Guide
• How to Design a test data strategy (2026)
• A/B test validation: A Complete Guide for QA (2026)
• Building a synthetic test data generator (2026)