Evaluate Multi-Agent Handoff Quality

TL;DR

Evaluate a multi-agent handoff as a contract with a goal, context, constraints, evidence, authority, owner, and acceptance criteria. Test completeness, semantic preservation, receiver acceptance, recovery behavior, and the downstream outcome with deterministic fixtures before adding live-model evaluations.

Key Takeaways

• Model every handoff as a typed contract instead of an unstructured message.
• Score required fields separately so a polished summary cannot hide missing context or authority.
• Verify semantic preservation with task invariants, not exact wording.
• Test sender, receiver, and end-to-end responsibility boundaries independently.
• Reject ambiguous ownership and require explicit acceptance or escalation.
• Use trace evidence to connect the handoff to the final task outcome.
• Turn sanitized production handoff failures into deterministic regression fixtures.

To evaluate multi agent handoff quality, inspect what the sender transfers, what the receiver understands, and whether the delegated task reaches a verified outcome. A strong handoff preserves the goal, relevant context, constraints, evidence, authority, ownership, and acceptance criteria without forcing the next agent to guess.

Place this tutorial inside the broader AI agent testing complete guide for 2026. The guide explains the full test strategy, while this tutorial gives you a runnable TypeScript harness for one boundary: delegation between agents. For broader system-level coverage, pair it with the multi-agent system testing guide.

You will use deterministic fixtures and Vitest, so the core checks run without a model or network. Once those checks are stable, you can feed real handoff messages into the same contract and save failures as regression cases.

What You Will Build

You will build a small evaluation project that can:

• represent a handoff as typed, observable data;
• detect missing context, evidence, ownership, limits, and acceptance criteria;
• compare the sender's task invariants with the receiver's interpretation;
• require explicit acceptance, rejection, or escalation;
• score handoff quality without hiding critical failures in an average;
• connect handoff defects to downstream task completion.

The result is framework-neutral. You can place an adapter in front of it for an orchestration framework, a custom message bus, or stored production traces.

Prerequisites

Use Node.js 20 or newer and npm 10 or newer. This tutorial uses TypeScript 5 and Vitest 3. Check the installed versions:

node --version
npm --version

Create a project and install the development dependencies:

mkdir agent-handoff-tests
cd agent-handoff-tests
npm init -y
npm install --save-dev typescript@5 vitest@3 @types/node
npx tsc --init --target ES2022 --module NodeNext --moduleResolution NodeNext --strict
npm pkg set type=module
npm pkg set scripts.test="vitest run"
mkdir -p src test test/fixtures

Run npx tsc --noEmit. It should exit successfully. npm test can report that no tests exist yet, which is expected at this point.

TL;DR

Do not use one average score as the only release gate. A handoff that scores well on style but drops a safety constraint is still unacceptable.

Step 1: Define the Handoff Contract

Create src/handoff.ts. Make the transferred state explicit:

export type Priority = 'low' | 'normal' | 'high';

export type Handoff = {
  handoffId: string;
  fromAgent: string;
  toAgent: string;
  goal: string;
  context: string[];
  constraints: string[];
  evidence: Array<{ id: string; summary: string }>;
  allowedActions: string[];
  prohibitedActions: string[];
  acceptanceCriteria: string[];
  priority: Priority;
  deadline?: string;
};

export type ValidationResult = {
  valid: boolean;
  errors: string[];
};

These fields answer operational questions. What must be done? What facts matter? What must not happen? What proof is available? Who acts next? What observable conditions define success?

Do not transfer private chain-of-thought. Transfer concise facts, tool results, decisions, uncertainties, and source references that the receiving agent is allowed to use. Redact secrets and personal data before the handoff enters logs or evaluation datasets.

Verification: run npx tsc --noEmit. TypeScript should report no errors. If imports later fail under NodeNext, keep .js suffixes in TypeScript source imports.

Step 2: Validate Completeness and Ownership

Append a deterministic validator to src/handoff.ts:

export function validateHandoff(h: Handoff): ValidationResult {
  const errors: string[] = [];
  const required = [
    ['handoffId', h.handoffId],
    ['fromAgent', h.fromAgent],
    ['toAgent', h.toAgent],
    ['goal', h.goal],
  ] as const;

  for (const [name, value] of required) {
    if (!value.trim()) errors.push(`${name} is required`);
  }
  if (h.fromAgent === h.toAgent) errors.push('sender and receiver must differ');
  if (h.context.length === 0) errors.push('context is required');
  if (h.acceptanceCriteria.length === 0) {
    errors.push('acceptance criteria are required');
  }
  if (h.allowedActions.length === 0) errors.push('allowed actions are required');
  if (new Set(h.evidence.map((item) => item.id)).size !== h.evidence.length) {
    errors.push('evidence ids must be unique');
  }
  return { valid: errors.length === 0, errors };
}

Create test/handoff.test.ts with a valid baseline:

import { describe, expect, it } from 'vitest';
import { validateHandoff, type Handoff } from '../src/handoff.js';

export const validHandoff: Handoff = {
  handoffId: 'H-42',
  fromAgent: 'triage-agent',
  toAgent: 'refund-agent',
  goal: 'Prepare refund recommendation for order O-17',
  context: ['Customer reports duplicate charge', 'Order was delivered'],
  constraints: ['Use the duplicate-charge policy', 'Respond in English'],
  evidence: [{ id: 'payment-1', summary: 'Two settled charges of USD 40' }],
  allowedActions: ['read_order', 'draft_recommendation'],
  prohibitedActions: ['issue_refund'],
  acceptanceCriteria: ['Cite payment evidence', 'Recommend one next action'],
  priority: 'high',
  deadline: '2026-07-16T12:00:00Z',
};

describe('validateHandoff', () => {
  it('accepts a complete handoff', () => {
    expect(validateHandoff(validHandoff)).toEqual({ valid: true, errors: [] });
  });

  it('rejects missing ownership and success criteria', () => {
    const broken = structuredClone(validHandoff);
    broken.toAgent = '';
    broken.acceptanceCriteria = [];
    expect(validateHandoff(broken).errors).toEqual([
      'toAgent is required',
      'acceptance criteria are required',
    ]);
  });
});

This is a schema-level test, not a quality guarantee. It prevents silent omission and gives later semantic checks a stable input.

Verification: run npm test. Two tests should pass. Temporarily remove goal from the fixture and confirm that the valid case fails with goal is required.

Step 3: Evaluate Multi Agent Handoff Quality Through Semantic Preservation

Exact text matching punishes useful summarization. Instead, extract task invariants before delegation and compare them with a structured receiver interpretation. Add these types and function to src/handoff.ts:

export type TaskInvariant = {
  key: string;
  expected: string;
  critical: boolean;
};

export type ReceiverInterpretation = {
  values: Record<string, string>;
};

export function compareInvariants(
  expected: TaskInvariant[],
  received: ReceiverInterpretation,
): { preserved: string[]; missing: string[]; changed: string[] } {
  const preserved: string[] = [];
  const missing: string[] = [];
  const changed: string[] = [];

  for (const item of expected) {
    const actual = received.values[item.key];
    if (actual === undefined) missing.push(item.key);
    else if (actual.trim().toLowerCase() !== item.expected.trim().toLowerCase()) {
      changed.push(item.key);
    } else preserved.push(item.key);
  }
  return { preserved, missing, changed };
}

Import compareInvariants and add a boundary test:

it('detects a changed authority limit and missing deadline', () => {
  const result = compareInvariants(
    [
      { key: 'currency', expected: 'USD', critical: true },
      { key: 'maximumRefund', expected: '40', critical: true },
      { key: 'deadline', expected: '2026-07-16T12:00:00Z', critical: false },
    ],
    { values: { currency: 'usd', maximumRefund: '80' } },
  );

  expect(result).toEqual({
    preserved: ['currency'],
    missing: ['deadline'],
    changed: ['maximumRefund'],
  });
});

In a live system, an agent or deterministic adapter can produce ReceiverInterpretation. Keep critical values structured at the source whenever possible. Do not ask a model to rediscover a refund ceiling that your policy engine already knows.

Verification: run npm test. Three tests should pass. The test must fail if maximumRefund changes from 80 to 40, proving that the changed-value branch is exercised.

Step 4: Evaluate Evidence and Traceability

A receiving agent needs enough evidence to verify important claims. Add a claim map to src/handoff.ts:

export type Claim = { text: string; evidenceIds: string[]; critical: boolean };

export function validateClaims(h: Handoff, claims: Claim[]): string[] {
  const known = new Set(h.evidence.map((item) => item.id));
  const errors: string[] = [];

  for (const claim of claims) {
    if (claim.critical && claim.evidenceIds.length === 0) {
      errors.push(`critical claim has no evidence: ${claim.text}`);
    }
    for (const id of claim.evidenceIds) {
      if (!known.has(id)) errors.push(`claim references unknown evidence: ${id}`);
    }
  }
  return errors;
}

Test both an unsupported claim and a broken reference:

import { validateClaims } from '../src/handoff.js';

it('requires traceable evidence for critical claims', () => {
  expect(validateClaims(validHandoff, [
    { text: 'Customer was charged twice', evidenceIds: [], critical: true },
    { text: 'Order exists', evidenceIds: ['order-missing'], critical: false },
  ])).toEqual([
    'critical claim has no evidence: Customer was charged twice',
    'claim references unknown evidence: order-missing',
  ]);
});

An evidence summary is not the evidence itself. Store a safe reference to the underlying tool result, document, or trace event. Ensure that the receiver has permission to resolve it. If access cannot cross the boundary, the sender must include an approved derivation or route the task to an agent with access.

Verification: run npm test. Four tests should pass. Replace the empty evidence list with ['payment-1'] and remove the unknown reference. The assertion should then fail because the validator returns no errors.

Step 5: Enforce Authority and Constraint Boundaries

The receiver must know both what it may do and what it must never do. Add an action evaluator:

export type ActionAttempt = { action: string; agent: string };

export function authorizeAction(h: Handoff, attempt: ActionAttempt): boolean {
  if (attempt.agent !== h.toAgent) return false;
  if (h.prohibitedActions.includes(attempt.action)) return false;
  return h.allowedActions.includes(attempt.action);
}

Add tests for the positive and negative boundaries:

import { authorizeAction } from '../src/handoff.js';

it('allows only delegated actions by the receiver', () => {
  expect(authorizeAction(validHandoff, {
    agent: 'refund-agent', action: 'draft_recommendation',
  })).toBe(true);
  expect(authorizeAction(validHandoff, {
    agent: 'refund-agent', action: 'issue_refund',
  })).toBe(false);
  expect(authorizeAction(validHandoff, {
    agent: 'other-agent', action: 'read_order',
  })).toBe(false);
});

Prompt instructions are not authorization controls. Enforce the same rule at the tool gateway, then log a sanitized denial event containing the handoff ID, receiver, action class, and policy version. The AI agent tool-calling testing tutorial shows how to exercise that enforcement boundary beyond the handoff contract.

Test constraints as data when you can. Represent locale, maximum amount, required approval, or prohibited destination in fields with deterministic checks. Free text remains useful for nuance, but it should not carry the only copy of a critical limit.

Verification: run npm test. Five tests should pass. Swap issue_refund into allowedActions; it must remain denied because an explicit prohibition takes precedence.

Step 6: Require Receiver Acceptance or Escalation

A message being delivered does not mean responsibility changed. Model the receiver's decision explicitly:

export type HandoffDecision = {
  handoffId: string;
  receiver: string;
  status: 'accepted' | 'rejected' | 'needs-clarification';
  reasons: string[];
};

export function validateDecision(h: Handoff, d: HandoffDecision): string[] {
  const errors: string[] = [];
  if (d.handoffId !== h.handoffId) errors.push('handoff id mismatch');
  if (d.receiver !== h.toAgent) errors.push('decision made by wrong receiver');
  if (d.status !== 'accepted' && d.reasons.length === 0) {
    errors.push('non-acceptance requires a reason');
  }
  return errors;
}

Test a clarification path:

import { validateDecision } from '../src/handoff.js';

it('requires a reason when the receiver cannot accept', () => {
  expect(validateDecision(validHandoff, {
    handoffId: 'H-42',
    receiver: 'refund-agent',
    status: 'needs-clarification',
    reasons: [],
  })).toEqual(['non-acceptance requires a reason']);
});

Use a bounded clarification loop. Define who answers, how many cycles are allowed, and when a human receives the task. A receiver should reject work it cannot perform safely instead of accepting and later claiming completion. If your workflow repeatedly bounces responsibility, add the techniques from testing AI agent loop termination.

Verification: run npm test. Six tests should pass. Add ['Missing order region'] to reasons; the expected error assertion should fail because the decision becomes valid.

Step 7: Calculate a Gated Quality Score

A score helps compare versions, but critical defects must override it. Add this evaluator:

export type HandoffSignals = {
  completeness: number;
  fidelity: number;
  traceability: number;
  efficiency: number;
  criticalFailures: string[];
};

export function scoreHandoff(s: HandoffSignals): { score: number; pass: boolean } {
  const values = [s.completeness, s.fidelity, s.traceability, s.efficiency];
  if (values.some((value) => value < 0 || value > 1)) {
    throw new RangeError('signals must be between 0 and 1');
  }
  const score = Math.round((
    s.completeness * 0.3 +
    s.fidelity * 0.3 +
    s.traceability * 0.25 +
    s.efficiency * 0.15
  ) * 100);
  return { score, pass: score >= 85 && s.criticalFailures.length === 0 };
}

Add the critical-failure test:

import { scoreHandoff } from '../src/handoff.js';

it('fails a high score when a critical constraint was lost', () => {
  expect(scoreHandoff({
    completeness: 1, fidelity: 1, traceability: 1, efficiency: 1,
    criticalFailures: ['maximum refund changed'],
  })).toEqual({ score: 100, pass: false });
});

The weights and threshold are illustrative configuration, not universal benchmarks. Calibrate them with reviewed examples from each task type. Keep safety, authority, identity, and required evidence as hard gates.

Verification: run npm test. Seven tests should pass. Remove the critical failure and confirm the expected pass: false assertion fails.

Step 8: Connect the Handoff to Task Completion

Local handoff quality matters because it affects the parent task. Record lifecycle events and assert causal completion:

export type HandoffEvent = {
  type: 'sent' | 'accepted' | 'evidence-used' | 'subtask-completed' | 'merged';
  handoffId: string;
  detail?: string;
};

export function completedWithEvidence(events: HandoffEvent[], id: string): boolean {
  const types = new Set(
    events.filter((event) => event.handoffId === id).map((event) => event.type),
  );
  return ['sent', 'accepted', 'evidence-used', 'subtask-completed', 'merged']
    .every((type) => types.has(type as HandoffEvent['type']));
}

Add a negative test that looks successful locally but never rejoins the parent workflow:

import { completedWithEvidence } from '../src/handoff.js';

it('does not count an unmerged subtask as complete', () => {
  const events = ['sent', 'accepted', 'evidence-used', 'subtask-completed']
    .map((type) => ({ type, handoffId: 'H-42' })) as Parameters<
      typeof completedWithEvidence
    >[0];
  expect(completedWithEvidence(events, 'H-42')).toBe(false);
});

This test catches orphaned success. The receiver finished, but the coordinator never incorporated the result. Define parent completion outside every agent's self-report and track it with AI agent task completion rate measurement.

Verification: run npm test. Eight tests should pass. Add a merged event and confirm the negative expectation fails. Then run npx tsc --noEmit for a complete static check.

Step 9: Add Regression Fixtures and CI Gates

Save a sanitized production failure as test/fixtures/lost-limit.json:

{
  "expected": [
    { "key": "maximumRefund", "expected": "40", "critical": true }
  ],
  "received": { "values": { "maximumRefund": "400" } }
}

Load it without relying on JSON import attributes:

import { readFile } from 'node:fs/promises';

it('replays a lost authority limit', async () => {
  const url = new URL('./fixtures/lost-limit.json', import.meta.url);
  const fixture = JSON.parse(await readFile(url, 'utf8'));
  const result = compareInvariants(fixture.expected, fixture.received);
  expect(result.changed).toContain('maximumRefund');
});

Run deterministic contract, authorization, decision, and fixture tests on every pull request. Run a sampled live-model evaluation on a schedule or before release. Normalize live outputs into the same types, review novel failures, and promote useful counterexamples into fixtures.

Keep three CI gates: no critical contract defects, all required regression scenarios pass, and no unauthorized action reaches a tool. Report completeness, fidelity, clarification rate, rejection rate, orphaned-subtask rate, and verified parent completion separately.

Verification: run npm test and expect nine passing tests. Run npx tsc --noEmit and expect a zero exit code. In CI, pin the Node major version and install from the lockfile with npm ci.

How to Evaluate Multi Agent Handoff Quality in Production

Sample by workflow risk and task type. Include successful, failed, escalated, and abandoned handoffs. If you inspect only successful conversations, you will miss ownership gaps and rejected work.

Capture a sanitized lifecycle: sender, receiver, contract version, policy version, required invariants, evidence references, acceptance decision, tool authorization decisions, completion events, and parent outcome. Do not log raw secrets, private reasoning, or unnecessary customer content. Apply access controls and retention limits to evaluation data.

Review critical failures individually. For noncritical dimensions, compare distributions and failure categories across releases.

When delegation begins with a faulty plan, isolate that upstream problem using the step-by-step guide to testing AI agent planning failures. Handoff tests should reveal boundary loss, not absorb every planner, tool, and completion defect into one vague score.

Troubleshooting

Problem: Good summaries fail exact text checks. -> Compare normalized structured invariants and task outcomes. Reserve exact matching for identifiers, amounts, permissions, deadlines, and other values that must remain exact.

Problem: Every handoff passes schema validation but tasks still fail. -> Add receiver interpretation, evidence-use, authorization, merge, and parent-completion assertions. Presence alone does not prove understanding or use.

Problem: The receiver continually asks for clarification. -> Inspect which required field is repeatedly absent. Add it to the producer contract, set a clarification budget, and escalate when the sender cannot supply it.

Problem: Traces contain sensitive data. -> Redact at the instrumentation boundary. Store safe evidence IDs, hashes, classifications, and policy outcomes instead of raw credentials or personal records.

Problem: A high average score hides a dangerous handoff. -> Make safety constraints, authority, identity, and critical evidence hard gates. Show dimension scores and critical failures beside the total.

Problem: Live-model evaluations are flaky. -> Gate pull requests with deterministic fixtures. Score live samples on invariant preservation, use repeated samples only where justified, and never retry away a semantic failure.

Best Practices and Common Mistakes

• Version the handoff schema and policy independently.
• Keep critical values structured and machine-checkable.
• Require one current owner and an explicit receiver decision.
• Test accepted, rejected, clarification, timeout, and escalation paths.
• Verify that cited evidence exists and is accessible to the receiver.
• Enforce permissions at tools, not only in prompts.
• Trace the delegated result back into parent completion.
• Segment metrics by workflow and risk.
• Preserve sanitized failures as named regression fixtures.

Avoid copying the entire conversation as context. Excess context increases cost and can bury the decisive constraint. Avoid evaluating only message fluency. A polished handoff can still name the wrong owner or change an amount. Also avoid blaming the receiver for missing data that the sender contract never required. Assign each check to the component that can fix it.

Interview Questions and Answers

Q: What defines a high-quality multi-agent handoff?

It transfers the goal, relevant context, constraints, evidence, authority, owner, and acceptance criteria accurately. The receiver explicitly accepts or escalates, uses the evidence, and produces a result that rejoins the parent task.

Q: Why is schema validation insufficient?

Schema validation proves that fields exist and types are valid. It does not prove that a deadline survived, evidence supports a claim, the receiver understood the goal, or the final result was merged. Semantic and outcome checks cover those gaps.

Q: How would you test context preservation without exact text matching?

Extract task invariants before the handoff and compare them with a structured receiver interpretation. Require exact equality for critical identifiers and limits, but allow normalized or rubric-based equivalence for descriptive context.

Q: Which handoff failures should be hard release gates?

Lost safety constraints, changed authority limits, wrong receiver identity, missing mandatory evidence, unauthorized actions, and false completion should normally be hard gates. Style and compactness can remain scored dimensions.

Q: How do you prevent handoff ping-pong?

Require reasons for rejection or clarification, track meaningful state change, limit clarification cycles, and define an escalation owner. A loop budget should end repeated transfers that add no information.

Q: What metrics would you monitor?

Monitor contract completeness, invariant fidelity, clarification and rejection rates, unauthorized attempts, evidence-use rate, orphaned subtasks, and verified parent completion. Segment by task type and risk rather than relying on one global average.

Q: How should nondeterministic handoffs be evaluated in CI?

Use deterministic fixtures for required pull-request gates. Run sampled live-model tests separately, normalize output into a stable contract, score invariants and outcomes, and convert reviewed failures into regression fixtures.

Where To Go Next

Return to the complete AI agent testing guide for 2026 to combine handoff checks with planner, tool, memory, safety, and end-to-end coverage. Then extend this harness with:

• Test AI agent planning failures step by step when the delegated objective or dependencies are already wrong before transfer.
• Test AI agent loop termination for repeated delegation, clarification ping-pong, and budget enforcement.
• Measure AI agent task completion rate to connect accepted subtasks and merged results to business outcomes.

Keep the handoff contract independent of model prose. That makes your evaluation reusable when models, prompts, agent roles, and orchestration frameworks change.

Conclusion

To evaluate multi agent handoff quality, test the boundary as a contract and the workflow as a causal trace. Validate completeness, preserve critical meaning, enforce authority, require receiver acceptance, verify evidence use, and prove that the result contributes to parent completion.

Start with the nine deterministic tests in this tutorial. Add your real task invariants, calibrate noncritical scoring with reviewed examples, and turn every important production failure into a sanitized regression fixture.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Evaluate RAG Citation Correctness with Examples
• Evaluating summarization quality (2026)
• Testing embeddings and vector search quality (2026)
• Testing multi-agent systems (2026)
• Testing semantic search quality (2026)
• A/B test validation: A Complete Guide for QA (2026)