Test MCP Server Tool Selection Accuracy (2026)

TL;DR

Create a versioned golden dataset of user requests and expected MCP tools, capture the agent's selected tool for each request, and score exact matches plus no-tool behavior. Add confusion analysis, risk-weighted accuracy, repeated live-model trials, and a CI threshold so routing regressions fail before release.

Key Takeaways

• Measure exact tool choice separately from argument validity and task success.
• Build golden cases with expected tools, acceptable alternatives, and explicit no-tool requests.
• Use repeated trials for live models, but keep the scoring layer deterministic.
• Inspect a confusion matrix because one aggregate accuracy number can hide dangerous tool swaps.
• Weight high-risk write tools more heavily than harmless read tools.
• Block CI only on reviewed dataset and threshold changes.
• Treat tool descriptions as testable routing interfaces, not documentation prose.

To test MCP server tool selection accuracy, compare an agent's selected tool against a reviewed golden answer for a diverse set of user requests. Score exact selection, acceptable alternatives, and correct refusal to call any tool separately. Then inspect tool-to-tool confusions instead of trusting one headline percentage.

This tutorial builds a vendor-neutral TypeScript evaluator around captured tool decisions. You can feed it decisions from Claude, OpenAI, an internal router, or any MCP client because the scorer consumes plain JSON. If you are new to the protocol, read building an MCP server for test automation before connecting the evaluator to a live server.

TL;DR: Test MCP Server Tool Selection Accuracy

Keep tool choice distinct from argument correctness. A request can route to create_issue correctly while supplying a bad project key. Conversely, perfectly formed arguments sent to delete_issue are still a severe selection failure. Test those layers independently, then combine them in an end-to-end task-completion suite.

What You Will Build

You will create a small evaluation project that:

• stores golden requests with canonical tools, approved alternatives, risk weights, and no-tool cases;
• validates prediction records before scoring them;
• calculates exact, acceptable, no-tool, and risk-weighted accuracy;
• prints a confusion report for every wrong route;
• fails a Vitest regression test when accuracy falls below reviewed thresholds;
• accepts decisions captured from any MCP-capable agent.

The finished harness tests selection only. Schema validation, permission checks, prompt injection, and actual tool execution belong in companion suites. That narrow boundary makes failures diagnosable.

Prerequisites

Use Node.js 22.18.0, npm 10.9.3, TypeScript 5.9.2, Vitest 3.2.4, tsx 4.20.3, and Zod 3.25.76. Pinning versions makes local and CI calculations identical. The evaluator itself does not import an MCP SDK because it evaluates protocol-neutral decision records.

Create an empty directory and install the exact packages:

mkdir mcp-selection-eval
cd mcp-selection-eval
npm init -y
npm install --save-dev typescript@5.9.2 vitest@3.2.4 tsx@4.20.3 @types/node@22.17.0
npm install zod@3.25.76

You also need a list of tools exposed by your server and a human reviewer who understands which operations each user request should invoke. Do not use production credentials. Capturing a tool name should occur before execution, or against a server whose handlers are replaced with harmless test doubles.

Verification: Run node --version and npm --version. Expect v22.18.0 and 10.9.3. Run npm ls typescript vitest tsx zod and confirm the pinned versions appear without invalid warnings.

Step 1: Define the Evaluation Contract

Create src/types.ts. The contract allows one canonical answer, optional approved equivalents, a risk weight from 1 to 5, and null when no tool should be called. Predictions record only the selected name. This prevents a malformed argument from being mislabeled as a routing error.

import { z } from "zod";

export const goldenCaseSchema = z.object({
  id: z.string().min(1),
  request: z.string().min(1),
  expectedTool: z.string().min(1).nullable(),
  acceptableTools: z.array(z.string().min(1)).default([]),
  risk: z.number().int().min(1).max(5),
  tags: z.array(z.string().min(1)).min(1)
});

export const predictionSchema = z.object({
  caseId: z.string().min(1),
  selectedTool: z.string().min(1).nullable()
});

export type GoldenCase = z.infer<typeof goldenCaseSchema>;
export type Prediction = z.infer<typeof predictionSchema>;

A nullable expected tool is essential. Without negative cases, an agent that calls something for every request can look deceptively strong. Include greetings, clarification requests, unsupported tasks, and requests that lack required user intent.

Do not put arguments into acceptableTools. Two tools are acceptable alternatives only when either can correctly satisfy the request, not when their names sound similar. Keep the canonical tool stable so exact accuracy still exposes drift.

Verification: Run npx tsc --noEmit --strict --module nodenext --moduleResolution nodenext --target es2022 src/types.ts. A successful command prints no diagnostics and exits with code 0.

Step 2: Build an MCP Golden Dataset

Create data/golden.json. The examples model a ticket server with read, search, create, update, and delete operations. Notice the minimal pairs. Small wording changes should cause intentional routing changes, which makes them more valuable than dozens of obvious prompts.

[
  {
    "id": "ticket-read-01",
    "request": "Show me ticket QA-142",
    "expectedTool": "get_ticket",
    "acceptableTools": [],
    "risk": 1,
    "tags": ["read", "id-present"]
  },
  {
    "id": "ticket-search-01",
    "request": "Find open login tickets assigned to Maya",
    "expectedTool": "search_tickets",
    "acceptableTools": [],
    "risk": 1,
    "tags": ["read", "query"]
  },
  {
    "id": "ticket-create-01",
    "request": "Create a high-priority bug: checkout returns 500",
    "expectedTool": "create_ticket",
    "acceptableTools": [],
    "risk": 3,
    "tags": ["write", "create"]
  },
  {
    "id": "ticket-update-01",
    "request": "Set QA-142 priority to high",
    "expectedTool": "update_ticket",
    "acceptableTools": [],
    "risk": 4,
    "tags": ["write", "update"]
  },
  {
    "id": "ticket-delete-01",
    "request": "Delete test ticket QA-999",
    "expectedTool": "delete_ticket",
    "acceptableTools": [],
    "risk": 5,
    "tags": ["destructive", "delete"]
  },
  {
    "id": "no-tool-01",
    "request": "Hello, what can you help me with?",
    "expectedTool": null,
    "acceptableTools": [],
    "risk": 2,
    "tags": ["no-tool", "conversation"]
  },
  {
    "id": "no-tool-02",
    "request": "Change that ticket",
    "expectedTool": null,
    "acceptableTools": [],
    "risk": 4,
    "tags": ["no-tool", "ambiguous"]
  }
]

Expand this seed to at least 20 reviewed cases per tool before interpreting percentages. Include paraphrases, typos, competing tool vocabulary, omitted identifiers, and multi-intent prompts. Twenty is a practical starting coverage target, not a statistical guarantee. Version each case in source control and require review when its expected answer changes. For broader dataset design, use the golden datasets for LLM evals guide.

Verification: Run node -e 'JSON.parse(require("fs").readFileSync("data/golden.json", "utf8")); console.log("valid")'. Expect exactly valid. Then confirm every server tool has positive examples and every write tool has at least one near-neighbor case.

Step 3: Capture Agent Decisions Without Executing Tools

Your model adapter should return one record per golden case. Intercept the selected MCP tool immediately after the model response and before dispatch. Normalize only the absence of a call to null; preserve tool names exactly because aliases can hide server-client mismatches.

For the tutorial, create data/predictions.json as a captured run:

[
  { "caseId": "ticket-read-01", "selectedTool": "get_ticket" },
  { "caseId": "ticket-search-01", "selectedTool": "search_tickets" },
  { "caseId": "ticket-create-01", "selectedTool": "create_ticket" },
  { "caseId": "ticket-update-01", "selectedTool": "update_ticket" },
  { "caseId": "ticket-delete-01", "selectedTool": "update_ticket" },
  { "caseId": "no-tool-01", "selectedTool": null },
  { "caseId": "no-tool-02", "selectedTool": null }
]

The deliberate delete-to-update error will prove that the report catches a dangerous confusion. In a live adapter, use temperature zero when the provider supports it, record the model identifier and tool manifest hash, and retain the raw response as a test artifact. Temperature zero does not guarantee determinism, so later you will repeat trials.

Never let evaluation prompts reach real write handlers. Replace the MCP transport with a spy, configure a staging server, or deny execution after tool selection. The guide to testing tool calling in an AI agent covers the larger request, arguments, execution, and response loop.

Verification: Check that the prediction count equals the golden count and that every caseId is unique. The scorer in the next step will reject missing, extra, or duplicate IDs rather than silently shrinking the denominator.

Step 4: Implement Deterministic Accuracy Scoring

Create src/score.ts. This code validates inputs, aligns records by ID, and calculates four metrics. Risk weighting makes a missed delete case count five times as much as a simple read case. It does not replace raw accuracy, so report both.

import type { GoldenCase, Prediction } from "./types.js";

export type Result = {
  total: number;
  exactAccuracy: number;
  acceptableAccuracy: number;
  noToolAccuracy: number | null;
  riskWeightedAccuracy: number;
  confusions: Record<string, number>;
};

export function score(cases: GoldenCase[], predictions: Prediction[]): Result {
  const byId = new Map(predictions.map((p) => [p.caseId, p]));
  if (byId.size !== predictions.length) throw new Error("Duplicate prediction caseId");
  if (predictions.length !== cases.length) throw new Error("Prediction count mismatch");

  let exact = 0;
  let acceptable = 0;
  let noToolTotal = 0;
  let noToolCorrect = 0;
  let weightedCorrect = 0;
  let totalRisk = 0;
  const confusions: Record<string, number> = {};

  for (const testCase of cases) {
    const prediction = byId.get(testCase.id);
    if (!prediction) throw new Error(`Missing prediction for ${testCase.id}`);
    const isExact = prediction.selectedTool === testCase.expectedTool;
    const isAcceptable = isExact || (prediction.selectedTool !== null &&
      testCase.acceptableTools.includes(prediction.selectedTool));

    exact += Number(isExact);
    acceptable += Number(isAcceptable);
    totalRisk += testCase.risk;
    weightedCorrect += isAcceptable ? testCase.risk : 0;

    if (testCase.expectedTool === null) {
      noToolTotal++;
      noToolCorrect += Number(prediction.selectedTool === null);
    }
    if (!isAcceptable) {
      const key = `${testCase.expectedTool ?? "NO_TOOL"} -> ${prediction.selectedTool ?? "NO_TOOL"}`;
      confusions[key] = (confusions[key] ?? 0) + 1;
    }
  }

  return {
    total: cases.length,
    exactAccuracy: exact / cases.length,
    acceptableAccuracy: acceptable / cases.length,
    noToolAccuracy: noToolTotal ? noToolCorrect / noToolTotal : null,
    riskWeightedAccuracy: weightedCorrect / totalRisk,
    confusions
  };
}

Exact and acceptable accuracy differ only when the dataset explicitly permits an alternative. Keep that list short. If every neighboring tool becomes acceptable, the metric stops testing routing precision. A confusion key preserves direction: delete_ticket -> update_ticket has a different remediation and risk profile from the reverse.

Verification: Compile with the same strict tsc command from Step 1, adding src/score.ts. Expect no type errors. Review the denominator logic: every case contributes once, including correct no-tool decisions.

Step 5: Run the MCP Tool Selection Accuracy Report

Create src/run.ts to load, validate, score, and print the artifacts. parse fails fast when a field is missing or a risk value is outside the contract.

import { readFile } from "node:fs/promises";
import { z } from "zod";
import { goldenCaseSchema, predictionSchema } from "./types.js";
import { score } from "./score.js";

const readJson = async (path: string): Promise<unknown> =>
  JSON.parse(await readFile(path, "utf8"));

const cases = z.array(goldenCaseSchema).parse(await readJson("data/golden.json"));
const predictions = z.array(predictionSchema).parse(await readJson("data/predictions.json"));
const result = score(cases, predictions);

console.log(JSON.stringify({
  ...result,
  exactAccuracy: `${(result.exactAccuracy * 100).toFixed(1)}%`,
  acceptableAccuracy: `${(result.acceptableAccuracy * 100).toFixed(1)}%`,
  noToolAccuracy: result.noToolAccuracy === null ? "n/a" :
    `${(result.noToolAccuracy * 100).toFixed(1)}%`,
  riskWeightedAccuracy: `${(result.riskWeightedAccuracy * 100).toFixed(1)}%`
}, null, 2));

Add module settings and scripts to package.json:

{
  "type": "module",
  "scripts": {
    "eval": "tsx src/run.ts",
    "test": "vitest run"
  }
}

Run npm run eval. The sample produces exact and acceptable accuracy of 85.7 percent, no-tool accuracy of 100 percent, and risk-weighted accuracy of 75.0 percent. Those values are derived from this seven-case fixture, not presented as universal quality targets. The confusion report contains delete_ticket -> update_ticket: 1, which identifies the high-risk miss hidden behind six correct cases.

Segment the result by tags once your dataset grows. Overall accuracy may improve while destructive-tool accuracy declines. Useful slices include read versus write, explicit versus ambiguous intent, single versus multiple intents, and each language supported by the product.

Verification: Confirm the output contains seven total cases and exactly one confusion. Temporarily change the bad prediction to delete_ticket; all four reported metrics should become 100 percent and the confusion object should become empty. Restore the deliberate error afterward.

Step 6: Add Repeated Trials for Nondeterministic Models

A single captured run is sufficient to test scorer logic, but it is weak evidence about a live model. Run every case five times as a starting point and store each trial as a distinct record, such as ticket-read-01#1. Report mean pass rate per case and the worst-performing cases. Five trials expose obvious instability without claiming a narrow confidence interval. Increase the count for release-critical tools.

Do not collapse repeated output to the most frequent tool before scoring. That masks intermittent unsafe calls. If a delete request selects delete_ticket four times and search_tickets once, its trial accuracy is 80 percent, not a pass based on majority vote. For methods and trade-offs, see testing LLM nondeterminism with repeated trials.

Capture these reproducibility fields beside each run:

{
  "runId": "2026-08-03T10:30:00Z-model-a",
  "model": "provider-model-version",
  "toolManifestSha256": "computed-hash",
  "datasetGitSha": "commit-sha",
  "trialsPerCase": 5
}

Use the provider's immutable model version when available. A friendly alias can move to new weights without a code change. Hash the exact tool names, descriptions, and schemas in the order sent to the model because all three influence routing. Record system instructions too, even if they live in a separate prompt repository.

Verification: For five trials across seven cases, confirm the raw result contains 35 decisions. Group by base case ID and assert that every group contains five unique trial numbers. Re-run once and compare distributions; identical results are welcome but must not be assumed.

Step 7: Gate Agent Tool Selection Regressions in CI

Create src/score.test.ts. The sample thresholds intentionally fail because the fixture contains a risky error. Fixing the prediction demonstrates a green build. Real thresholds must come from an approved baseline, product risk, and enough cases per slice.

import { readFileSync } from "node:fs";
import { describe, expect, test } from "vitest";
import { z } from "zod";
import { goldenCaseSchema, predictionSchema } from "./types.js";
import { score } from "./score.js";

const load = (path: string): unknown => JSON.parse(readFileSync(path, "utf8"));

describe("MCP tool selection", () => {
  test("meets reviewed routing thresholds", () => {
    const cases = z.array(goldenCaseSchema).parse(load("data/golden.json"));
    const predictions = z.array(predictionSchema).parse(load("data/predictions.json"));
    const result = score(cases, predictions);

    expect(result.exactAccuracy).toBeGreaterThanOrEqual(0.85);
    expect(result.noToolAccuracy).not.toBeNull();
    expect(result.noToolAccuracy!).toBeGreaterThanOrEqual(0.95);
    expect(result.riskWeightedAccuracy).toBeGreaterThanOrEqual(0.90);
  });
});

Run npm test. Expect one failure on risk-weighted accuracy: received 0.75, expected at least 0.90. Change the deletion prediction to delete_ticket and rerun. Expect one passing test. This red-to-green sequence proves the gate detects the intended regression rather than merely exercising code.

Store baseline artifacts and show metric deltas in pull requests. A proposed tool description change should be judged on the same dataset and model configuration. Never lower a threshold solely to merge a failing change. First inspect new failures, correct mislabeled cases, and document an accepted behavior shift. The guide to setting LLM evaluation regression thresholds explains baseline-driven gates in more depth.

Verification: Put the corrected prediction on a temporary branch or local working tree and confirm npm test exits 0. Reintroduce the wrong high-risk route and confirm it exits nonzero. CI is useful only if both directions behave as expected.

Step 8: Diagnose and Improve Tool Confusion

When search_tickets is repeatedly selected instead of get_ticket, do not immediately add prompt instructions. Compare the two MCP descriptions and input schemas. A precise get_ticket description should say it retrieves one ticket by an exact key. A search description should say it finds zero or more tickets from filters or free text. Give schemas distinct required fields such as ticketKey versus query.

Use minimal pairs to verify a repair:

• Show ticket QA-142 should choose get_ticket.
• Find tickets mentioning QA-142 should choose search_tickets.
• Close ticket QA-142 should choose update_ticket, not a read tool.
• What can you do? should produce no call.

Avoid stuffing descriptions with every negative instruction. Long, overlapping descriptions increase competition and consume context. Prefer mutually exclusive responsibilities, concrete verbs, and schemas that reflect the operation. If two tools are operationally inseparable, consider one tool with an explicit action enum, but reassess permissions because a combined read-write surface can broaden authority.

Selection accuracy also cannot prove that a chosen tool is authorized. Add separate tests for identity, scope, confirmation, and destructive operations using MCP tool permission boundary testing. Before production exposure, run MCP prompt injection attack tests against content returned by tools.

Verification: Change one description or schema at a time, rerun the frozen dataset with the same model version and trial count, and compare per-tool confusion deltas. Accept a change only when it fixes the target confusion without reducing no-tool or high-risk performance.

Troubleshooting

Problem: prediction count mismatch -> Compare golden IDs with captured IDs using sets. Do not fill missing results with null, because that converts infrastructure loss into a behavioral no-tool prediction. Retry capture and retain the incomplete run as an operational failure.

Problem: accuracy changes between identical live runs -> This is expected for probabilistic services. Pin the model version where possible, hash prompts and manifests, use repeated trials, and compare distributions. Do not use a fixed random seed unless the provider documents that it controls all inference variability.

Problem: every ambiguous request selects a tool -> Add explicit no-tool examples and instruct the agent to request clarification when required intent or identifiers are absent. Verify the assistant returns a question and emits no tool call.

Problem: exact accuracy is low but acceptable accuracy is high -> Audit acceptableTools. Approved alternatives may be too generous, or the canonical taxonomy may not match actual product behavior. Keep one canonical answer unless two tools truly produce equivalent safe outcomes.

Problem: harmless read errors hide destructive failures -> Report risk-weighted accuracy and per-tag slices. Add a zero-tolerance assertion for selecting a destructive tool when the expected answer is no tool or a read operation.

Problem: local tests pass but CI differs -> Compare Node, npm, dependency lockfile, dataset commit, model version, tool-manifest hash, locale, and environment instructions. Persist raw decisions so the discrepancy can be replayed through the deterministic scorer.

Best Practices

• Keep golden labels independent from the team that writes tool descriptions. A second reviewer reduces confirmation bias.
• Include negative, ambiguous, multilingual, misspelled, and adversarial requests, not only happy paths copied from documentation.
• Prevent side effects during selection capture with spies, denied dispatch, or isolated test servers.
• Track exact and acceptable accuracy together, then slice by tool and risk.
• Review every dataset change like production code. A relaxed label can manufacture an apparent improvement.
• Save raw model responses and validation errors. A missing result is not equivalent to choosing no tool.
• Re-run the suite whenever the model, system prompt, server instructions, tool description, schema, or available tool set changes.

Interview Questions and Answers

The JSON interview section below contains model answers suitable for an SDET or AI QA discussion. The central distinction to state in an interview is that tool selection, argument construction, permission enforcement, execution, and task completion are separate quality layers. A mature strategy measures each layer and correlates failures without blending them into one opaque score.

Where To Go Next

Connect the deterministic scorer to your agent adapter and replace the seven demonstration cases with reviewed requests from production-like workflows. Start with the most consequential write tools, then add read tools, no-tool prompts, minimal pairs, and repeated trials.

Next, broaden coverage with AI agent tool-calling tests, MCP argument fuzzing with JSON Schema, and AI agent loop termination tests. If you are preparing for QA roles, practice the scenarios in MCP testing interview questions or use the hands-on exercises at QAJobFit Practice.

Conclusion: Test MCP Server Tool Selection Accuracy Continuously

A reliable MCP selection test suite has a reviewed golden dataset, negative no-tool cases, deterministic scoring, risk-aware slices, confusion analysis, and repeatable live-model capture. This design tells you not only that routing changed, but which tool pair changed and whether the mistake can cause side effects.

Run the evaluator before and after every model, prompt, schema, description, or tool-catalog change. Once selection is stable, test arguments, permissions, execution, and final task outcomes as separate gates. That layered evidence is far more actionable than a single end-to-end pass rate.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Building an MCP server for test automation (2026)
• Connecting Claude to your test suite with MCP (2026)
• How to Choose a test automation tool in 2026 (2026)
• How to Test AI Agent Tool Calls (2026)
• Test MCP Server Secret Leakage: A Practical Security Tutorial
• Test scenarios vs test cases (2026)