Design Device Farm Scheduler Interview: An SDET System Design Guide

TL;DR

Build a control plane that matches queued mobile test jobs to healthy devices, then grants an exclusive, expiring lease. Use heartbeats, fencing tokens, fair tenant quotas, reset workflows, and quarantine rules to recover safely when workers, devices, or networks fail.

Key Takeaways

• Model devices as exclusive, leased resources whose capabilities and health determine eligibility.
• Separate durable scheduling state from disposable agents that prepare and control devices.
• Use expiring leases, heartbeats, and fencing tokens so abandoned reservations recover safely.
• Schedule within capability pools using tenant quotas, priority, and aging instead of one global FIFO queue.
• Quarantine unhealthy devices automatically and require a reset plus validation before returning them to service.
• Keep immutable attempt history and external artifacts so device cleanup never destroys test evidence.

A strong design device farm scheduler interview answer treats every phone or tablet as a scarce, stateful, exclusive resource. Your scheduler must match capabilities, prevent concurrent ownership, recover expired reservations, protect tenants from noisy neighbors, and keep broken devices out of circulation.

This tutorial builds a runnable TypeScript control plane and connects each choice to the Senior SDET system design interview complete guide. You will implement the smallest credible scheduler first, then explain which parts become transactional databases, queues, device agents, and observability services in production.

The focus is physical Android and iOS hardware. Emulators and simulators can share the job contract, but physical devices add USB failures, battery and thermal limits, dirty state, scarce models, OS fragmentation, and slow resets. Those constraints make the scheduler a useful senior-level design problem.

What You Will Build

By the end, you will have:

• A Fastify API that registers devices and accepts mobile test jobs.
• A capability matcher for platform, OS version, model, locale, and required features.
• An exclusive lease endpoint with priority, aging, and per-tenant fairness.
• Heartbeats and fencing tokens that reject stale owners.
• Release, reset, quarantine, and recovery rules for unreliable hardware.
• Status endpoints and an interview-ready production architecture.

Use this boundary: the control plane owns truth, policy, and history. A device agent runs beside a USB hub or lab host, reports health, prepares one device, executes one leased attempt, uploads artifacts, and releases it. Never let an agent become the only owner of scheduling state.

Prerequisites

Use Node.js 22 LTS or newer and npm 10 or newer. The example uses TypeScript, Fastify, and Zod with APIs current in 2026. You need curl, but you do not need a real phone because the tutorial exercises the control plane.

mkdir device-farm-scheduler && cd device-farm-scheduler
npm init -y
npm install fastify zod
npm install -D typescript tsx @types/node
npx tsc --init --target ES2022 --module NodeNext --moduleResolution NodeNext --strict
mkdir src

Add "type": "module" and scripts "start": "tsx src/server.ts" and "check": "tsc --noEmit" to package.json. Verify the setup with node --version, npm --version, and npx tsx --version. Each command should print a version and exit successfully.

Step 1: Frame the Design Device Farm Scheduler Interview Requirements

Start the interview by defining a job, device, reservation, and attempt. A job requests capabilities and test metadata. A device advertises observed capabilities and health. A reservation is time-bounded exclusive ownership. An attempt records what happened during one reservation.

List the required operations before drawing services:

1. Register devices and update their health.
2. Submit, cancel, and inspect test jobs.
3. Match a queued job to one healthy compatible device.
4. Reserve the device exclusively and renew ownership.
5. Release, reset, or quarantine the device after execution.
6. Preserve logs, video, screenshots, and attempt history.

State the main invariant: at most one current lease can own a device, and only that lease can commit its attempt result. Exactly-once physical execution is not realistic during partitions. The useful guarantee is one authoritative attempt result, backed by conditional state transitions and fencing.

Ask about scale using variables instead of inventing numbers. If arrival rate is R jobs per second and average occupation time is D seconds, baseline concurrent demand is R * D. Split that demand by platform, model, OS, locale, and feature because a global total hides scarce iPhones or foldable Android devices. Include reset time in occupation time.

Verify this step by describing ownership of every transition. The API accepts a job, the scheduler creates a lease, the agent heartbeats and reports a result, and the reset workflow decides whether the device becomes available. If two components can independently make the same device available, the design has a race.

Step 2: Model Devices, Jobs, and Leases

Create src/server.ts. Start with schemas that keep capabilities data-driven and state transitions explicit.

import Fastify from "fastify";
import { randomUUID } from "node:crypto";
import { z } from "zod";

const app = Fastify({ logger: true });

const Capabilities = z.object({
  platform: z.enum(["android", "ios"]),
  osVersion: z.string().min(1),
  model: z.string().min(1),
  locale: z.string().default("en-US"),
  features: z.array(z.string()).default([])
});

type DeviceState = "available" | "leased" | "cleaning" | "quarantined" | "offline";
type Device = z.infer<typeof Capabilities> & {
  id: string; state: DeviceState; agentId: string; failureCount: number;
  leaseId?: string; leaseExpiresAt?: number; version: number;
};
type Job = {
  id: string; tenantId: string; requested: z.infer<typeof Capabilities>;
  priority: number; createdAt: number; state: "queued" | "running" | "passed" | "failed" | "canceled";
  maxMinutes: number;
};
type Lease = {
  id: string; jobId: string; deviceId: string; agentId: string;
  fence: number; expiresAt: number; status: "active" | "released" | "expired";
};

const devices = new Map<string, Device>();
const jobs = new Map<string, Job>();
const leases = new Map<string, Lease>();

Keep requested capabilities separate from observed device facts. A production catalog can add manufacturer, screen geometry, carrier, network profile, biometric support, camera, and accessibility settings without changing the lease protocol. Avoid free-form tags for critical fields because inconsistent spelling silently reduces capacity.

The version is a fencing counter. Every new reservation increments it, and every mutation from an agent must carry the matching value. A late agent may still be physically connected, but it cannot change authoritative control-plane state.

Verify with npm run check. It should finish without TypeScript diagnostics. Temporarily change DeviceState to an invalid value such as busy; the compiler should flag assignments that do not belong to the declared state machine. Restore the valid type before continuing.

Step 3: Register Devices and Submit Jobs

Add validation schemas and two endpoints. Registration is an upsert because agents reconnect, but it must not erase an active lease.

const DeviceInput = Capabilities.extend({
  id: z.string().min(1),
  agentId: z.string().min(1)
});
const JobInput = z.object({
  tenantId: z.string().min(1),
  requested: Capabilities,
  priority: z.number().int().min(0).max(100).default(50),
  maxMinutes: z.number().int().min(1).max(120).default(30)
});

app.put("/v1/devices/:id", async (request, reply) => {
  const input = DeviceInput.safeParse({
    ...(request.body as object), id: (request.params as { id: string }).id
  });
  if (!input.success) return reply.code(400).send(input.error.flatten());
  const old = devices.get(input.data.id);
  const device: Device = old
    ? { ...old, ...input.data }
    : { ...input.data, state: "available", failureCount: 0, version: 0 };
  devices.set(device.id, device);
  return reply.code(old ? 200 : 201).send(device);
});

app.post("/v1/jobs", async (request, reply) => {
  const input = JobInput.safeParse(request.body);
  if (!input.success) return reply.code(400).send(input.error.flatten());
  const job: Job = {
    id: randomUUID(), ...input.data, createdAt: Date.now(), state: "queued"
  };
  jobs.set(job.id, job);
  return reply.code(202).send(job);
});

Production registration should authenticate each agent with a workload identity and keep a stable device identity derived from managed inventory, not a user-supplied name. Store both configured facts and recently observed facts. If the reported serial number changes on a lab port, alert rather than silently treating it as the previous phone.

Add await app.listen({ port: 3000, host: "127.0.0.1" }); at the bottom, then run npm run start. Verify registration in another terminal:

curl -s -X PUT http://127.0.0.1:3000/v1/devices/pixel-01 \
  -H 'content-type: application/json' \
  -d '{"agentId":"lab-a","platform":"android","osVersion":"16","model":"Pixel","locale":"en-US","features":["camera","biometric"]}'

Expect HTTP 201 and state equal to available. Submit a job with the same requested object and expect HTTP 202 plus a generated job ID. Save that ID for the scheduling step.

Step 4: Match Capabilities and Schedule Fairly

Add a matcher that treats required features as a subset. Keep OS matching exact in this tutorial. A production system can use a tested semantic-version range library, but should never compare versions lexically.

function compatible(device: Device, job: Job): boolean {
  const wanted = job.requested;
  return device.state === "available" &&
    device.platform === wanted.platform &&
    device.osVersion === wanted.osVersion &&
    device.model === wanted.model &&
    device.locale === wanted.locale &&
    wanted.features.every((feature) => device.features.includes(feature));
}

function runningByTenant(tenantId: string): number {
  return [...jobs.values()].filter(
    (job) => job.tenantId === tenantId && job.state === "running"
  ).length;
}

function score(job: Job, now: number): number {
  const ageMinutes = Math.floor((now - job.createdAt) / 60_000);
  return job.priority + Math.min(ageMinutes, 60);
}

const LeaseRequest = z.object({ agentId: z.string().min(1) });

app.post("/v1/devices/:id/lease", async (request, reply) => {
  const input = LeaseRequest.safeParse(request.body);
  const device = devices.get((request.params as { id: string }).id);
  if (!input.success || !device) return reply.code(404).send({ error: "device not found" });
  if (device.agentId !== input.data.agentId || device.state !== "available")
    return reply.code(409).send({ error: "device unavailable" });

  const now = Date.now();
  const eligible = [...jobs.values()]
    .filter((job) => job.state === "queued" && compatible(device, job))
    .filter((job) => runningByTenant(job.tenantId) < 2)
    .sort((a, b) => score(b, now) - score(a, now) || a.createdAt - b.createdAt);
  const job = eligible[0];
  if (!job) return reply.code(204).send();

  device.version += 1;
  const lease: Lease = {
    id: randomUUID(), jobId: job.id, deviceId: device.id,
    agentId: input.data.agentId, fence: device.version,
    expiresAt: now + 30_000, status: "active"
  };
  device.state = "leased"; device.leaseId = lease.id;
  device.leaseExpiresAt = lease.expiresAt; job.state = "running";
  leases.set(lease.id, lease);
  return reply.code(200).send({ lease, job });
});

This selection prevents a tenant with two running jobs from claiming another slot. Aging adds one point per queued minute up to a cap, so old work gradually competes with high-priority work. In production, apply quotas and ordering inside a database transaction that locks the device and selected job. An in-memory check is not safe across replicas.

Verify by posting the lease request with agentId set to lab-a. The response should contain the queued job, a lease ID, fence: 1, and an expiry. A second request for the same device should return 409. Register a device with the wrong model and confirm it receives 204 rather than an incompatible job.

Step 5: Renew Ownership With Heartbeats and Fencing

A reservation must expire because agents and lab hosts fail. Add a shared ownership check and heartbeat endpoint.

const LeaseAction = z.object({
  agentId: z.string().min(1),
  fence: z.number().int().positive()
});

function owns(device: Device, lease: Lease, agentId: string, fence: number): boolean {
  return lease.status === "active" && device.leaseId === lease.id &&
    device.agentId === agentId && lease.agentId === agentId &&
    device.version === fence && lease.fence === fence;
}

app.post("/v1/leases/:id/heartbeat", async (request, reply) => {
  const input = LeaseAction.safeParse(request.body);
  const lease = leases.get((request.params as { id: string }).id);
  const device = lease ? devices.get(lease.deviceId) : undefined;
  if (!input.success || !lease || !device)
    return reply.code(404).send({ error: "lease not found" });
  if (!owns(device, lease, input.data.agentId, input.data.fence))
    return reply.code(409).send({ error: "stale lease" });
  lease.expiresAt = Date.now() + 30_000;
  device.leaseExpiresAt = lease.expiresAt;
  return { expiresAt: lease.expiresAt };
});

Agents should heartbeat well before expiry and add jitter so thousands of devices do not synchronize. The server, not the agent, chooses authoritative time. A heartbeat only extends ownership; it does not prove the phone is usable, so the agent must also report independent device health telemetry.

The fencing counter solves a subtle failure. Agent A loses connectivity, its lease expires, and Agent B receives a new lease. If A reconnects and posts success, its lower fence is rejected. Lease IDs alone help, but a monotonic device version makes ordering explicit and works well with conditional database updates.

Verify with the lease ID and fence returned in Step 4. Post POST /v1/leases/{id}/heartbeat with agentId: lab-a and fence: 1; expect a later expiry. Repeat with fence 99 and expect HTTP 409.

Step 6: Release, Clean, and Quarantine Devices

A passed test does not make a phone immediately available. First upload evidence, stop applications, clear app data, restore settings, remove test accounts, verify connectivity, and run a health probe. Model that cleanup explicitly.

const ReleaseInput = LeaseAction.extend({
  outcome: z.enum(["passed", "failed", "infrastructure_failure"]),
  artifactUris: z.array(z.string().url()).default([]),
  summary: z.string().max(2000)
});

app.post("/v1/leases/:id/release", async (request, reply) => {
  const input = ReleaseInput.safeParse(request.body);
  const lease = leases.get((request.params as { id: string }).id);
  const device = lease ? devices.get(lease.deviceId) : undefined;
  const job = lease ? jobs.get(lease.jobId) : undefined;
  if (!input.success || !lease || !device || !job)
    return reply.code(404).send({ error: "lease not found" });
  if (!owns(device, lease, input.data.agentId, input.data.fence))
    return reply.code(409).send({ error: "stale lease" });

  lease.status = "released";
  job.state = input.data.outcome === "passed" ? "passed" : "failed";
  device.state = "cleaning";
  device.leaseId = undefined; device.leaseExpiresAt = undefined;
  if (input.data.outcome === "infrastructure_failure") device.failureCount += 1;
  return reply.code(202).send({
    jobState: job.state, deviceState: device.state, artifacts: input.data.artifactUris
  });
});

const CleanInput = z.object({ agentId: z.string(), healthy: z.boolean() });
app.post("/v1/devices/:id/clean-complete", async (request, reply) => {
  const input = CleanInput.safeParse(request.body);
  const device = devices.get((request.params as { id: string }).id);
  if (!input.success || !device || device.agentId !== input.data.agentId)
    return reply.code(404).send({ error: "device not found" });
  if (device.state !== "cleaning") return reply.code(409).send({ error: "not cleaning" });
  device.state = input.data.healthy && device.failureCount < 3
    ? "available" : "quarantined";
  if (device.state === "available") device.failureCount = 0;
  return { state: device.state };
});

Do not put artifact bytes in the scheduler database. Store immutable attempt metadata in a relational store and blobs in object storage, as explained in the test result storage system design tutorial. Decide retry eligibility from failure classification, then create a new attempt rather than overwriting the failed one.

Verify by releasing the active lease with a valid fence. Expect the job to become terminal and the device to enter cleaning. Request another lease and expect 409. Post a healthy clean completion and confirm the device returns to available. Three infrastructure failures should cause quarantine instead.

Step 7: Recover Expired Leases and Add Observability

Add a small reaper function and status endpoints. The reaper represents a periodic background task, but its transition rules belong in the same durable transaction as lease expiry.

function reapExpired(now = Date.now()): number {
  let recovered = 0;
  for (const lease of leases.values()) {
    if (lease.status !== "active" || lease.expiresAt > now) continue;
    const device = devices.get(lease.deviceId);
    const job = jobs.get(lease.jobId);
    lease.status = "expired";
    if (device && device.leaseId === lease.id && device.version === lease.fence) {
      device.state = "cleaning";
      device.leaseId = undefined; device.leaseExpiresAt = undefined;
      device.failureCount += 1;
    }
    if (job?.state === "running") job.state = "queued";
    recovered += 1;
  }
  return recovered;
}

app.post("/internal/reap", async () => ({ recovered: reapExpired() }));
app.get("/v1/status", async () => ({
  jobs: Object.fromEntries(
    [...jobs.values()].map((job) => [job.id, job.state])
  ),
  devices: Object.fromEntries(
    [...devices.values()].map((device) => [device.id, device.state])
  )
}));

The expired device enters cleaning, not available, because the old test process may still be alive and the phone may contain tenant data. The job returns to the queue for another compatible device. In production, cap infrastructure retries and preserve a separate attempt row for the expired lease.

Measure queue age and depth by capability pool, schedule latency, lease expiry rate, reset duration, device utilization, infrastructure failure rate, quarantine count, and agent heartbeat gaps. Trace jobId, attemptId, deviceId, and leaseFence across scheduler, agent, test runner, and artifact service. Restrict device serial numbers and tenant data in logs.

Verify recovery by reducing the lease duration locally, acquiring a lease, waiting past expiry, and calling /internal/reap. Expect recovered: 1, the job to be queued, and the device to be cleaning. A late release from the old lease must return 409 because its status is expired.

Troubleshooting

A compatible device never receives a job -> Log a structured mismatch reason for platform, OS version, model, locale, and each feature. Normalize catalog values at registration, and use a semantic-version library for ranges instead of string comparison.

Two tests run on one phone -> Claim the device and job in one transaction with a conditional device version update. Enforce the lease again at the agent, and reject every heartbeat, release, and cleanup command with a stale fence.

Devices stay leased after a lab outage -> Run redundant reapers, use server timestamps, and make expiry transitions idempotent. Send recovered devices through cleanup before availability, even when the agent says the process already stopped.

High-priority work starves normal jobs -> Cap priority, add queue aging, enforce tenant quotas, and reserve only an explicit fraction of compatible capacity. Inspect oldest queue age per capability pool, not only average wait time.

The farm shows capacity but jobs still wait -> Capacity is fragmented. Break availability down by requested model, OS, locale, and feature. Add admission checks for impossible requests and use demand history when purchasing or upgrading devices.

Infrastructure retries hide application defects -> Separate device or agent failure from assertion failure. Keep original evidence and retry only an infrastructure-classified attempt. Use historical signals from the flaky test detection system design tutorial as evidence, not as permission to turn every failure into a retry.

Interview Questions and Answers

Q: Why use leases instead of a permanent device lock?

A permanent lock survives after its owner crashes and removes capacity indefinitely. A lease expires without heartbeats, allowing controlled recovery. The device still goes through cleanup because expiry does not prove the prior process stopped safely.

Q: Can you guarantee exactly-once test execution?

No, a partitioned agent can keep executing after the control plane reassigns the job. Guarantee one authoritative result with fencing and conditional commits. Reduce duplicate side effects through isolated test accounts and environments.

Q: How do you match a job to a device?

First filter on hard capabilities such as platform, OS range, model, locale, and hardware features. Then rank eligible jobs by quota, priority, aging, and submission time. Record the capability snapshot on the attempt for later diagnosis.

Q: How do you prevent one tenant from consuming the farm?

Enforce queued and running limits per tenant and capability pool. Use weighted fairness after hard compatibility filtering. Offer reservations or dedicated pools only as an explicit service policy.

Q: What happens when a device disconnects mid-test?

The agent reports an infrastructure failure if possible, otherwise the lease expires. Preserve partial artifacts, requeue within the retry budget, increment the device health penalty, and quarantine it when thresholds are crossed. Never classify a USB disconnect as a product assertion failure.

Q: Why is cleanup a separate state?

A completed attempt can leave apps, accounts, settings, files, or background processes behind. Keeping cleaning separate prevents a new tenant from receiving a dirty device. Availability follows a successful reset and health probe, not merely a result callback.

Q: Where should scheduler state live?

Keep jobs, attempts, devices, leases, and transitions in a relational database with conditional updates. Store large artifacts in object storage. Agents should cache enough context to execute but never be the durable source of truth.

Q: How would you scale the scheduler?

Partition logical queues by hard capability pools, but avoid creating an unmanageable queue for every tag combination. Run stateless scheduler replicas that claim work transactionally. Scale agents separately and monitor queue age for scarce pools.

Q: How do reservations differ from immediate jobs?

An immediate job asks for the next compatible device, while a reservation holds capacity for a time window. Reservations need admission control, expiration, no-show policy, and limits so idle bookings do not starve queued work. Both should produce the same fenced lease at execution time.

Q: What security controls matter in a shared device lab?

Use authenticated agents, short-lived scoped credentials, encrypted transport, isolated test accounts, restricted network egress, and complete audit logs. Wipe app data and secrets after every attempt. Protect developer mode, signing material, device enrollment, and artifact access as privileged infrastructure.

Common Mistakes and Best Practices

• Do not use one FIFO list for all devices. Compatibility fragmentation causes head-of-line blocking.
• Do not mark a device available when a test ends. Require cleanup and a health probe.
• Do not trust client priority without caps and tenant policy.
• Do not retry assertion failures as infrastructure failures. Preserve both classifications.
• Do not overwrite attempts. Immutable history explains expired leases, retries, and quarantine.
• Do use server-side time, monotonic fencing, and conditional writes for ownership.
• Do keep device identity stable and record the exact observed capability snapshot.
• Do expose pool-specific queue age, reset duration, thermal state, battery health, and disconnects.
• Do test races between heartbeat, release, cancellation, and expiry before production.

Where To Go Next

Return to the complete Senior SDET system design interview guide and rehearse this answer in a 35 to 45 minute format. Spend the opening minutes on requirements and scale, draw control plane and device agents, then go deep on leases, fairness, cleanup, and failure recovery.

Continue with these sibling tutorials:

• Design a cross-platform test execution service to place this scheduler inside a broader browser and mobile execution platform.
• Design durable test result storage to model attempts, artifact manifests, retention, and query paths.
• Design a flaky test detection system to turn historical outcomes into actionable reliability signals.

Next, replace the maps with PostgreSQL. Race two scheduler processes against one available device and prove that only one transaction wins. Add property-based tests for legal state transitions and load tests for one scarce capability pool. That work gives you concrete evidence when an interviewer asks how you know the scheduler is correct.

Conclusion

To design device farm scheduler interview architecture at a senior level, center the answer on scarce hardware, exclusive leases, capability-aware fairness, cleanup, and recovery. API endpoints matter, but the strongest signal is a precise ownership model that remains safe when an agent pauses, a phone disconnects, or a stale result arrives.

Run the tutorial, test the stale-fence and expired-lease paths, then practice replacing every in-memory operation with one transactional production boundary. You will be ready to explain both a working scheduler and the tradeoffs required for a secure multi-tenant device lab.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Senior SDET System Design Interview Complete Guide (2026)
• Design a Flaky Test Detection System Interview
• Design a Test Result Storage System
• Design a Cross-Platform Test Execution Service
• Mobile device farm testing: A Practical Guide (2026)
• QA and SDET System Design Interview Questions