Testing an AI agent with LangSmith (2026)

TL;DR

Trace the complete agent, evaluate it on a curated LangSmith dataset, and score deterministic behavior separately from subjective quality. Gate high-risk failures in CI, then use sampled online evaluation to detect drift and add confirmed failures to regression coverage.

Key Takeaways

• Evaluate final answers, tool choices, arguments, trajectories, safety, latency, and cost as separate dimensions.
• Build small reviewed datasets with observable references and metadata for critical slices.
• Use deterministic evaluators for exact rules and calibrated model judges only for semantic criteria.
• Treat authorization, privacy, and irreversible actions as hard release constraints.
• Compare named candidates against a versioned baseline and inspect regressions example by example.
• Feed sanitized production failures into the offline regression suite.

Testing an AI agent with LangSmith is not the same as checking whether a chatbot produced a plausible sentence. An agent can choose tools, pass arguments, update state, recover from errors, and still return a fluent but wrong answer. A useful test strategy therefore evaluates the final response and the path taken to produce it.

This guide shows a QA or SDET engineer how to build that strategy with LangSmith in 2026. You will create traceable agent calls, curate datasets, write deterministic evaluators, add model-based review only where judgment is necessary, compare experiments, and decide what belongs in continuous integration versus production monitoring.

TL;DR

Start with a small dataset of business-critical tasks. Trace the full agent, score deterministic facts with code, score nuanced quality with a tightly defined rubric, and compare every material prompt, model, or tool change against a named baseline.

1. What Testing an AI Agent With LangSmith Actually Covers

Testing an AI agent with LangSmith covers several observable behaviors, not one generic accuracy score. The final answer is only the outermost layer. A support agent might correctly say that an order was refunded, but it may have called a refund tool without authorization. A research agent might cite the right fact after querying an unapproved source. Both are product failures even if a text-only evaluator gives them full credit.

A practical quality model has five dimensions:

1. Task outcome: Did the agent solve the user request?
2. Trajectory: Did it take an acceptable sequence of reasoning and tool steps?
3. Tool behavior: Did it select the right tool, send valid arguments, and respect permissions?
4. Response quality: Is the answer accurate, grounded, concise, and appropriate for the user?
5. Operational behavior: Did it finish within accepted latency, cost, and retry limits?

LangSmith helps because traces preserve the hierarchy of an agent run. A parent run can contain model calls, retriever calls, and tool calls. Datasets hold repeatable examples, while experiments connect a version of the application to those examples and evaluators. Online evaluation applies related checks to sampled production traces.

Do not reduce this model to a single average. A mean score can hide a catastrophic authorization failure. Treat safety, privacy, and irreversible tool actions as hard constraints. Use softer thresholds for tone or concision. This risk-based separation is also central to agentic testing with tool calling, where the test oracle must inspect both intent and execution.

2. Design a Layered Agent Test Strategy

An effective strategy uses fast tests near the code and richer evaluation at the system boundary. Each layer answers a different question.

Unit tests should cover date calculations, authorization rules, JSON parsing, and tool exceptions without calling a model. Component tests can replace a payment API with a fake while leaving the model and orchestration intact. Offline evaluation should exercise representative natural-language variation. End-to-end tests should be few because they are slower, costlier, and more sensitive to external systems.

Tag every dataset example with useful dimensions such as locale, risk, intent, customer_tier, and expected_tool. Aggregate results by these slices. An overall pass rate may remain stable while refund requests in one locale degrade sharply.

Define release rules before seeing results. For example, no high-risk authorization failures, no secret leakage, and no meaningful regression in the critical-intent slice. The exact numbers must come from your product risk and historical variation, not a copied benchmark. If the team is new to LLM evaluation, measuring LLM latency and cost in tests provides a complementary operational test model.

3. Set Up Tracing for Testing an AI Agent With LangSmith

Install the current SDKs and provide credentials through environment variables. Keep keys out of source control.

python -m venv .venv
source .venv/bin/activate
python -m pip install -U langsmith openai

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="your-langsmith-key"
export OPENAI_API_KEY="your-provider-key"
export LANGSMITH_PROJECT="order-agent-evaluation"

The following compact agent uses the OpenAI client wrapped by LangSmith. It has one read-only tool so the example is safe to run. The model name can be overridden without changing the test code.

# agent.py
import json
import os
from openai import OpenAI
from langsmith import wrappers

client = wrappers.wrap_openai(OpenAI())
MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")

ORDERS = {
    "A100": {"status": "shipped", "eta": "2026-07-16"},
    "B200": {"status": "processing", "eta": "2026-07-19"},
}

TOOLS = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "description": "Read the status and ETA of an order by ID.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
            "additionalProperties": False,
        },
        "strict": True,
    },
}]

def lookup_order(order_id: str) -> dict:
    return ORDERS.get(order_id, {"error": "order_not_found"})

def order_agent(inputs: dict) -> dict:
    messages = [
        {
            "role": "system",
            "content": (
                "You are an order support agent. Use lookup_order for status "
                "questions. Never claim an order state without the tool result."
            ),
        },
        {"role": "user", "content": inputs["question"]},
    ]
    first = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        tools=TOOLS,
        tool_choice="auto",
    )
    message = first.choices[0].message
    calls = message.tool_calls or []
    if not calls:
        return {"answer": message.content or "", "tool_names": []}

    messages.append(message)
    for call in calls:
        args = json.loads(call.function.arguments)
        result = lookup_order(args["order_id"])
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        })

    final = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        tools=TOOLS,
        tool_choice="none",
    )
    return {
        "answer": final.choices[0].message.content or "",
        "tool_names": [call.function.name for call in calls],
    }

Run one smoke call before building an evaluation. Confirm that the LangSmith project shows a parent interaction with both model calls. Redact sensitive fields before tracing, and use synthetic order IDs in nonproduction datasets.

4. Build a Dataset With Strong Test Oracles

A LangSmith dataset is useful only when its examples reveal failures that matter. Begin with manually reviewed cases from requirements, incident reports, support transcripts that have been de-identified, and boundary analysis. Add synthetic paraphrases later, but do not let generated cases replace domain review.

For the order agent, references should describe observable expectations. Avoid a single exact prose answer when many wordings are valid. Store the required fact, expected tool, and risk metadata instead.

# create_dataset.py
from langsmith import Client

client = Client()
dataset = client.create_dataset(
    dataset_name="order-agent-regression-v1",
    description="Reviewed order status, unknown ID, and policy cases.",
)

examples = [
    {
        "inputs": {"question": "Where is order A100?"},
        "outputs": {
            "expected_tool": "lookup_order",
            "required_terms": ["shipped", "2026-07-16"],
        },
        "metadata": {"intent": "order_status", "risk": "medium"},
    },
    {
        "inputs": {"question": "Check order Z999."},
        "outputs": {
            "expected_tool": "lookup_order",
            "required_terms": ["not found"],
        },
        "metadata": {"intent": "unknown_order", "risk": "low"},
    },
    {
        "inputs": {"question": "Write a friendly greeting."},
        "outputs": {"expected_tool": "none", "required_terms": []},
        "metadata": {"intent": "no_tool", "risk": "low"},
    },
]

client.create_examples(dataset_id=dataset.id, examples=examples)
print(dataset.name)

Include positive, negative, boundary, abuse, recovery, and multi-turn cases. For tools, cover missing identifiers, malformed identifiers, duplicate requests, timeouts, empty results, partial results, and permission denials. Keep a separate high-risk dataset for destructive actions so its pass criteria cannot be diluted by easy conversational cases.

Dataset versioning needs discipline. Preserve a stable regression set, add discovered production failures to it, and record why an example changed. If a reference was wrong, fix it transparently rather than deleting an inconvenient failure. Small, trusted datasets usually provide more signal than thousands of weak synthetic examples.

5. Write Deterministic and Model-Based Evaluators

Use deterministic evaluators whenever the expected behavior can be expressed in code. They are fast, explainable, and repeatable. A model-based judge is appropriate for semantic correctness, helpfulness, or tone, but it should complement hard checks rather than replace them.

# eval_agent.py
from langsmith import Client
from agent import order_agent

def correct_tool(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    expected = reference_outputs["expected_tool"]
    actual = outputs.get("tool_names", [])
    passed = (not actual) if expected == "none" else actual == [expected]
    return {"key": "correct_tool", "score": int(passed)}

def contains_required_facts(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    answer = outputs.get("answer", "").lower()
    terms = [term.lower() for term in reference_outputs["required_terms"]]
    score = int(all(term in answer for term in terms))
    return {"key": "required_facts", "score": score}

def no_sensitive_leak(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    answer = outputs.get("answer", "").lower()
    forbidden = ["sk-", "api_key", "system prompt"]
    return {
        "key": "no_sensitive_leak",
        "score": int(not any(token in answer for token in forbidden)),
    }

client = Client()
results = client.evaluate(
    order_agent,
    data="order-agent-regression-v1",
    evaluators=[correct_tool, contains_required_facts, no_sensitive_leak],
    experiment_prefix="order-agent-pr",
    metadata={"candidate": "prompt-v3", "suite": "regression"},
    max_concurrency=2,
)
print(results)

The evaluator keys should remain stable because they become your comparison dimensions. Return a focused score and useful comment when diagnosis requires context. Do not combine tool correctness, factuality, and style into one evaluator. Separate scores tell the engineer what broke.

For an LLM-as-judge, define the exact criterion, input fields, scoring scale, and examples of each rating. Hide candidate identity during pairwise comparison. Calibrate the judge against human-labeled cases and periodically measure disagreement. A judge that prefers longer answers or recognizes its own model family can create misleading gains. The LLM evaluation interview guide for QA engineers covers the tradeoffs interviewers expect you to explain.

6. Test Tool Calls, Arguments, and Agent Trajectories

Trajectory testing asks whether the route was acceptable, not whether every hidden reasoning detail matched a golden sequence. Exact step-by-step matching is brittle because two valid paths may exist. Assert critical invariants instead.

For a read-only status request, useful invariants include:

• lookup_order is called exactly once.
• The order_id comes from the user request, not from model invention.
• No write-capable tool is invoked.
• A not-found result is not rewritten as a real status.
• Tool errors produce a bounded recovery attempt and a transparent response.

Capture these facts from trace inputs and outputs, or expose a sanitized execution summary from the agent target during offline evaluation. Do not parse human prose to infer whether a tool ran. The trace is the stronger oracle.

When several paths are allowed, model them as constraints. A travel agent may call search before pricing or pricing as part of search, but it must verify availability before booking. A support agent can ask for an order ID before calling a tool, but it must never guess one. For high-risk operations, split planning and execution, require an explicit confirmation state, and test that adversarial text cannot skip it.

Error injection is essential. Make fake tools raise timeouts, return invalid schemas, produce an empty list, or report a permission error. Verify that retries have a cap and do not duplicate a non-idempotent action. A fluent apology after charging a customer twice is still a severe failure.

7. Run Offline Experiments and Compare Candidates

An experiment should answer a named change question: Does prompt v3 improve unknown-order handling without hurting normal status requests? Does a model migration preserve tool selection? Does a new retriever reduce unsupported answers? Avoid running anonymous experiments that nobody can reproduce.

Record candidate metadata such as prompt revision, model identifier, tool schema revision, application commit, dataset version, and evaluator version. Keep temperature and other generation settings with the candidate configuration. A result without configuration provenance cannot support a release decision.

Compare at three levels:

1. Hard failures: safety, authorization, schema, and irreversible-action violations.
2. Slice performance: scores for critical intents, languages, risk groups, or tool paths.
3. Aggregate quality: overall factuality, helpfulness, latency, and cost distributions.

Inspect regressions example by example. If a candidate improves average helpfulness but fails one refund authorization case, block it. If results vary across repeated runs, run the uncertain examples more than once and report the distribution. Do not repeatedly tune on the same regression dataset until it becomes an unofficial training set. Maintain a held-out set for final release checks.

LangSmith's comparison view is useful for side-by-side outputs and evaluator feedback. Human reviewers should be able to see the input, reference, candidate outputs, tool path, and evaluator explanation. Their corrections should feed a controlled loop: fix labels, refine evaluators, add failure cases, then rerun both baseline and candidate.

8. Add Online Evaluation and Production Feedback

Offline datasets cannot predict every real request. Online evaluation watches sampled production traces for drift, new intents, tool failures, and safety issues. It is monitoring, not permission to experiment carelessly on users.

Start with deterministic online checks that are cheap and high signal: output schema validity, forbidden tool use, excessive tool-call count, missing citations when required, secret-like patterns, and latency ceilings. Apply model-based evaluators to a controlled sample when human-like judgment is needed. Route severe findings to an alert, while ambiguous findings go to a review queue.

Protect privacy at collection time. Redact credentials, payment data, personal identifiers, and confidential document content before traces leave the application boundary. Restrict project access and define retention rules. A test platform should not become a second ungoverned copy of production data.

Convert confirmed production failures into de-identified regression examples. Record the failure class and expected behavior, then reproduce the issue offline before changing the prompt or agent. This closes the feedback loop without making the regression suite a raw dump of user conversations.

Watch rates and distributions rather than isolated low judge scores. A sudden rise in unknown tool names, repeated calls, empty final answers, or p95 latency can identify a systemic change. Link operational alerts to the trace so the on-call engineer can inspect the complete path rather than guessing from a generic error message.

9. Control Nondeterminism, Latency, and Cost

Model-backed tests will vary even when the code does not. The goal is not to pretend nondeterminism is absent. The goal is to design stable assertions and quantify the remaining variation.

Prefer invariant assertions over exact wording. Check that required facts appear, prohibited claims do not appear, the selected tool is allowed, and arguments satisfy a schema. Keep exact string matches for deterministic adapters and canonical codes. For uncertain high-value examples, repeat the run and define a release rule over the repeated outcomes. Label any thresholds as product decisions, not universal standards.

Separate correctness failures from infrastructure noise. Retry rate limits and transient network failures at the harness layer, with a strict cap and clear reporting. Do not silently rerun a bad semantic output until it passes. That practice hides flakiness instead of measuring it.

Track latency and token usage by step. A final response may be fast on average while one retriever or tool path creates a long tail. Compare candidates on matched examples, preferably under similar load. Cache only where the experiment design allows it, and document whether cached responses are included.

Budget evaluation like any other test environment. Run deterministic unit and component tests on every commit, a focused offline dataset on pull requests, and broader repeated evaluations on a schedule or before release. This test pyramid keeps feedback fast while preserving enough coverage for risky model changes.

10. Put Testing an AI Agent With LangSmith Into CI

Testing an AI agent with LangSmith in CI works best as a staged gate. First run tool and policy unit tests with no external model. Next run a small, tagged dataset for critical intents. Finally, schedule the broad suite separately so a provider incident does not block every unrelated commit.

A minimal workflow can run the evaluation script with secrets supplied by the CI platform:

name: agent-evaluation
on:
  pull_request:

jobs:
  critical-agent-evals:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -U langsmith openai pytest
      - run: pytest -q tests/test_tools.py tests/test_policies.py
      - run: python eval_agent.py
        env:
          LANGSMITH_TRACING: "true"
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          LANGSMITH_PROJECT: order-agent-ci

Make the gate produce a concise summary by evaluator and slice, plus a link to detailed traces. Fail immediately for hard policy violations. For probabilistic quality, require enough evidence before blocking and provide an approved override process that records owner, reason, and expiration.

Do not let the evaluation script mutate production systems. Use read-only tools, fakes, or sandbox accounts. If the agent can send email, issue a refund, or modify a ticket, intercept that action in CI and assert the proposed command. Test execution permission separately in a controlled environment.

Interview Questions and Answers

Q: Why is final-answer accuracy insufficient for an agent?

An agent can reach a correct answer through an unsafe or expensive path. It may call the wrong tool, expose data in arguments, repeat a write, or bypass confirmation. I evaluate outcome, trajectory, tool behavior, safety, and operational constraints separately.

Q: When would you use a deterministic evaluator instead of an LLM-as-judge?

I use deterministic code for schemas, exact facts, tool names, argument boundaries, authorization, and prohibited patterns. It is repeatable and easy to debug. I reserve a model judge for semantic qualities such as relevance or tone, then calibrate it against human labels.

Q: How do you reduce flaky agent evaluations?

I assert invariants instead of exact prose, control the candidate configuration, isolate infrastructure errors, and repeat only the cases where variation matters. I never rerun a semantic failure until it passes. I report distributions and define risk-based gates before reviewing the candidate.

Q: What belongs in a LangSmith dataset?

It should contain representative inputs, observable reference expectations, and metadata for meaningful slices. I include normal paths, boundaries, negative cases, tool errors, safety attacks, and production regressions after de-identification. Every example needs a clear reason for existing.

Q: How would you test a destructive tool?

In offline evaluation I replace it with a fake that records the proposed action. I verify authorization, confirmation state, idempotency key, arguments, and retry behavior without performing the action. A small controlled end-to-end test can validate the real integration separately.

Q: What is the difference between offline and online evaluation?

Offline evaluation runs a candidate against a curated dataset before release and supports direct baseline comparison. Online evaluation scores sampled live traces to find drift and novel failures after release. Confirmed online failures should become sanitized offline regression cases.

Common Mistakes

• Scoring only the final text: This misses unsafe tool calls and wasteful trajectories. Inspect trace structure and tool inputs.
• Using a judge for everything: Exact rules become slower, costlier, and less reproducible. Implement deterministic checks first.
• Treating one run as truth: A probabilistic candidate needs stable invariants and repeated measurement for uncertain critical cases.
• Averaging away severe failures: Report hard safety constraints and critical slices separately from overall quality.
• Collecting raw production data: Redact before tracing, restrict access, and define retention.
• Changing dataset, evaluator, and candidate together: You lose the ability to explain movement. Version each component and compare one intentional change.
• Executing real side effects in CI: Replace write tools with recorders or sandbox implementations.
• Ignoring judge calibration: Compare judge decisions with human labels and investigate systematic bias.

Conclusion

Testing an AI agent with LangSmith requires evidence about what the agent answered, which tools it used, how it handled failures, and whether it stayed inside safety and operational limits. Trace the complete run, build a reviewed dataset, favor deterministic evaluators, and use model-based judgment only for criteria that truly require it.

Start with ten to twenty critical examples and one evaluator per important risk. Run a baseline, inspect every failure, then add the focused suite to CI and feed confirmed production issues back into the dataset. That creates a test system that improves with the agent instead of becoming a dashboard nobody trusts.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Testing tool calling in an AI agent (2026)
• AI powered visual testing with vision models (2026)
• Testing an AI chatbot for tone (2026)
• Testing an AI code assistant (2026)
• Testing an AI defect predictor (2026)
• Testing an AI form filler (2026)