Promptfoo vs LangSmith for LLM Evaluation (2026)

TL;DR

Promptfoo is the stronger default for local, config-driven prompt and model matrices that must run transparently in CI. LangSmith is the stronger default for application-level evaluation tied to traces, managed datasets, collaborative review, and production monitoring. Choose according to where your evidence must live and how engineers investigate a failure.

Key Takeaways

• Choose Promptfoo when a repository-owned, configuration-first evaluation matrix and direct CI gating are the main requirements.
• Choose LangSmith when trace-level debugging, managed datasets, experiment comparison, annotation, and production observability matter most.
• Both tools support deterministic and model-graded checks, so the decisive difference is usually workflow and evidence management, not evaluator vocabulary.
• Promptfoo makes cross-provider prompt and model comparisons especially easy to review as code.
• LangSmith connects offline experiments to application traces and production feedback more naturally.
• A small proof of concept using the same cases and release rules is more reliable than choosing from a feature checklist.
• Teams can use both tools, but they should assign each evaluation one source of truth to prevent conflicting results.

Promptfoo vs LangSmith for LLM evaluation is not a simple open-source tool versus hosted platform contest. Promptfoo excels at repository-owned evaluation matrices, provider comparisons, assertions, red teaming, and command-line gates. LangSmith excels when evaluation must connect to application traces, managed datasets, experiment history, human feedback, and production behavior.

The right choice depends on your test boundary. If you mainly compare prompts or models from code and want every case visible in a pull request, begin with Promptfoo. If you evaluate chains or agents and need to understand the exact model, retriever, and tool calls behind a score, begin with LangSmith. This guide builds the same support-answer evaluation in both products so you can compare real workflows rather than marketing checklists. For a broader test architecture first, read the complete LLM evaluation pipeline guide.

TL;DR

Verdict: use Promptfoo for evaluation-as-code centered on prompts, providers, and assertions. Use LangSmith for evaluation-as-observability centered on application runs and their histories. Neither removes the need for reviewed datasets, stable evaluators, cost controls, or human adjudication.

1. What Promptfoo vs LangSmith for LLM Evaluation Really Compares

Promptfoo and LangSmith overlap, but their natural units of work differ. A Promptfoo evaluation starts with prompts, providers, test variables, and assertions. It expands those inputs into a matrix, executes the cells, and displays or exports the results. The configuration is easy to commit beside application code, which makes prompt changes and expectations visible during review.

A LangSmith evaluation typically starts with an application target, a dataset, and evaluators. The target might be a single model call, a retrieval pipeline, a LangGraph workflow, or an agent. The experiment records outputs and evaluator feedback, while tracing can preserve nested calls that explain why an example failed.

That distinction changes debugging. Suppose a support assistant omits a return-policy exception. In Promptfoo, you inspect the prompt, provider, variables, raw output, and failed assertion. In LangSmith, you can also inspect whether the retriever returned the exception, whether a tool timed out, and which model call discarded the evidence. Promptfoo can evaluate a custom application endpoint, and LangSmith can compare simple prompts, so this is not a hard capability boundary. It is the path of least resistance.

Define your test boundary before comparing licenses or dashboards:

1. Are you testing a prompt template, a model-provider combination, or a deployed application?
2. Must a failure be reproducible from repository files alone?
3. Do reviewers need nested execution traces?
4. Will production failures flow back into the same dataset?
5. Who maintains evaluators, and where will human corrections live?

These questions prevent a team from buying an observability workflow for a YAML-only need or forcing an agent-debugging problem into a flat output matrix.

2. Evaluation Model and Developer Experience

Promptfoo treats configuration as the executable specification. The common loop is edit YAML, run npx promptfoo eval, inspect the terminal matrix, and open npx promptfoo view. Tests can stay inline or come from files and generators. Provider IDs make model substitution explicit, and each case can combine variables with deterministic, semantic, or model-graded assertions.

LangSmith treats an experiment as a recorded execution of a target over dataset examples. You create or select a dataset, run Client.evaluate, attach evaluators, and compare the resulting experiment with a baseline. Alternatively, its pytest integration synchronizes decorated tests, outputs, references, and feedback while retaining normal test failures.

The consequence is visible in version control. A Promptfoo pull request can show the complete prompt, cases, provider matrix, and thresholds in a compact diff. A LangSmith pull request usually shows target and evaluator code, while dataset rows and prior experiment results may live in the service. LangSmith datasets can still be seeded from checked-in files, which is wise when change approval matters.

Neither model is universally cleaner. QA teams comfortable with test data files often learn Promptfoo quickly. ML platform teams that already reason in traces, experiments, annotations, and datasets often find LangSmith more coherent. If your application uses several model calls and tools, a matrix can become too flat. If your scope is twenty prompt variations against three providers, platform setup can feel heavier than the problem.

A useful evaluation framework should also fit the team's existing failure triage. Compare this decision with the risk layers in the AI agent testing complete guide, especially when tool use makes final-text scoring insufficient.

3. Run a Promptfoo Evaluation

Use Node.js 20 or newer and an API key accepted by the provider. npx lets you execute the current package without adding a permanent global install. Create a clean directory and confirm the CLI first.

mkdir support-eval && cd support-eval
node --version
npx promptfoo@latest --version
export OPENAI_API_KEY="replace-with-a-test-key"

Verify: the first command should report Node 20 or newer, and the second should print a Promptfoo version rather than a module error. Never commit the exported key.

Save this as promptfooconfig.yaml. The prompt asks for a concise grounded answer. The two cases test a supported return and an explicit policy boundary. The model ID can be changed to a provider model available in your account without changing the assertions.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Support policy answer regression

prompts:
  - |
    You answer only from this policy: {{policy}}
    If the policy does not answer the question, say "I do not know from the policy."
    Customer question: {{question}}

providers:
  - id: openai:gpt-5-mini
    config:
      temperature: 0

tests:
  - description: unopened item inside return window
    vars:
      policy: Unopened items may be returned within 30 days with a receipt.
      question: Can I return an unopened item after 12 days if I have the receipt?
    assert:
      - type: icontains
        value: return
      - type: icontains
        value: 30 days
      - type: not-icontains
        value: 60 days
  - description: policy does not cover exchanges
    vars:
      policy: Unopened items may be returned within 30 days with a receipt.
      question: Can I exchange an opened item?
    assert:
      - type: icontains
        value: I do not know from the policy

Run the evaluation without cache while establishing a baseline, then export machine-readable output for CI artifacts.

npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache --output results.json
npx promptfoo@latest view -n

Verify: the CLI should show four passing assertions across two cases. If a result fails, inspect the actual response instead of weakening the assertion immediately. view -n starts the local viewer without automatically opening a browser. The JSON report preserves the run for a build artifact.

This example is deliberately deterministic. Add a rubric only for criteria such as completeness or tone that cannot be expressed as stable facts. Provider-backed graders add latency, expense, and a new source of variation.

4. Run the Equivalent LangSmith Evaluation

Use Python 3.11 or newer in an isolated environment. The example uses the current LangSmith client and the OpenAI SDK. It creates a managed dataset, evaluates the same answer function, and applies one deterministic evaluator.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U langsmith openai
export LANGSMITH_API_KEY="replace-with-a-test-key"
export OPENAI_API_KEY="replace-with-a-test-key"
export LANGSMITH_TRACING=true
python -c "import langsmith, openai; print('imports ok')"

Verify: the last command must print imports ok. If an API key belongs to multiple workspaces, set LANGSMITH_WORKSPACE_ID as required by your account.

Save the following as langsmith_eval.py. A unique dataset name avoids accidentally overwriting a reviewed shared set while you learn. The references store required and forbidden phrases instead of one golden response.

import os
import uuid
from langsmith import Client, wrappers
from openai import OpenAI

MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")
client = Client()
llm = wrappers.wrap_openai(OpenAI())

def answer_support(inputs: dict) -> dict:
    response = llm.responses.create(
        model=MODEL,
        input=(
            "Answer only from this policy: " + inputs["policy"] + "\n"
            "If the policy does not answer the question, say "
            "'I do not know from the policy.'\n"
            "Customer question: " + inputs["question"]
        ),
    )
    return {"answer": response.output_text}

def phrase_rules(run, example) -> dict:
    answer = run.outputs.get("answer", "").lower()
    reference = example.outputs or {}
    required = [term.lower() for term in reference.get("required", [])]
    forbidden = [term.lower() for term in reference.get("forbidden", [])]
    passed = all(term in answer for term in required) and not any(
        term in answer for term in forbidden
    )
    return {
        "key": "phrase_rules",
        "score": int(passed),
        "comment": f"required={required}; forbidden={forbidden}",
    }

dataset_name = "support-policy-" + uuid.uuid4().hex[:8]
dataset = client.create_dataset(
    dataset_name=dataset_name,
    description="Two reviewed support policy regression cases",
)
client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {
            "inputs": {
                "policy": "Unopened items may be returned within 30 days with a receipt.",
                "question": "Can I return an unopened item after 12 days if I have the receipt?",
            },
            "outputs": {"required": ["return", "30 days"], "forbidden": ["60 days"]},
        },
        {
            "inputs": {
                "policy": "Unopened items may be returned within 30 days with a receipt.",
                "question": "Can I exchange an opened item?",
            },
            "outputs": {"required": ["I do not know from the policy"], "forbidden": []},
        },
    ],
)
results = client.evaluate(
    answer_support,
    data=dataset_name,
    evaluators=[phrase_rules],
    experiment_prefix="support-policy-baseline",
    metadata={"model": MODEL, "suite": "support-policy"},
)
print(results)

Run it once.

python langsmith_eval.py

Verify: the command should print an experiment result object, and the LangSmith project should contain two rows with phrase_rules feedback. Open each trace and confirm the recorded input, model call, output, and evaluator feedback agree. This trace inspection is the workflow advantage the flat Promptfoo example does not emphasize. For a deeper agent implementation, use the guide to testing an AI agent with LangSmith.

5. Promptfoo vs LangSmith for LLM Evaluation Datasets and Evaluators

Promptfoo datasets naturally live in source-controlled YAML, JSON, CSV, or generator code. That supports branch-specific changes, mandatory code review, and fully local reconstruction. The matrix applies prompts and providers across cases, which is excellent for controlled prompt or model comparisons. Metadata and filtering help restrict a run to a risk slice.

LangSmith datasets are first-class managed objects. Examples can carry inputs, reference outputs, and metadata, then feed repeated experiments. The platform workflow is valuable when subject-matter experts annotate outputs, teams maintain queues, or production failures are promoted into regression cases. Seed managed datasets from an approved repository snapshot if auditors need a durable review trail outside the UI.

Both products support two evaluator families:

• Deterministic checks: required facts, JSON schema, regular expressions, tool names, numeric boundaries, citation presence, and forbidden content.
• Model-based checks: relevance, groundedness, tone, completeness, or rubric conformance where legitimate answers vary semantically.

The syntax matters less than evaluator design. A judge prompt is another program with inputs, outputs, versions, and defects. Calibrate it against labels from qualified reviewers. Track false acceptance and false rejection by risk slice. Pin the grader model when possible, preserve its rubric, and do not let the same model grade its own output without checking bias.

Keep hard safety rules separate from averages. One unauthorized refund must not disappear inside ninety-nine helpful conversations. For semantic scores, define the release decision before running the candidate. If a threshold is chosen after seeing results, it is a negotiation, not an independent gate. The LLM judge interview guide explains how to defend this separation in technical reviews.

6. Comparing Prompts, Models, Chains, and Agents

Promptfoo's matrix is a strong fit for prompt and model selection. Add a second prompt or provider and the runner expands combinations automatically. Each cell retains the same variables and assertions, making side-by-side differences obvious. That is useful for migrations, fallback-provider qualification, response-format checks, and prompt regression.

LangSmith comparison works best when each candidate is a meaningful application revision. Name experiments by hypothesis, such as retriever-k20-reranker-v2, and attach prompt, model, dataset, evaluator, and commit metadata. Compare the candidate with a stable baseline at the individual example level, not only by aggregate score.

Agents raise the value of traces. A final answer can be correct even though the agent called an unapproved tool, retried a write, exposed a secret in arguments, or used a needlessly expensive trajectory. LangSmith's application tracing makes nested model, retriever, and tool activity available beside experiment feedback. Promptfoo can call custom HTTP, JavaScript, or Python providers and assert structured output, so it remains viable when your harness exposes a sanitized execution summary.

For either tool, make the target return observable facts such as answer, tool_names, citation_ids, retry_count, and policy_decision. Never infer tool execution from prose. Test critical ordering as invariants, for example authorization before write and confirmation before purchase, instead of demanding one exact trajectory when several safe routes exist.

Use failure injection outside live systems. Replace email, refund, deletion, and ticket-update tools with recorders. Simulate timeouts, invalid schemas, empty retrieval, permission denial, and partial responses. The agentic tool-calling test guide provides a practical oracle model for these cases.

7. CI/CD, Reports, and Release Gates

Promptfoo fits CI with a direct command and a repository configuration. The following GitHub Actions job installs Node, runs the checked-in suite, and uploads the JSON result even after a failing evaluation.

name: llm-evaluation
on:
  pull_request:

jobs:
  promptfoo:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npx promptfoo@latest eval -c promptfooconfig.yaml --output results.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: promptfoo-results
          path: results.json

Verify: open the Actions log, confirm that failed assertions make the evaluation step fail, and download promptfoo-results to confirm the report exists. In a production repository, pin an reviewed package version instead of latest so a dependency release cannot change gate behavior unexpectedly.

LangSmith offers standard SDK experiments and pytest integration. The pytest route is attractive when QA engineers already use Python assertions. Install langsmith[pytest], mark tracked tests with @pytest.mark.langsmith, and run them with a named suite. Local assertions still control pass or fail while inputs, outputs, references, and feedback synchronize to LangSmith.

python -m pip install -U "langsmith[pytest]"
LANGSMITH_TEST_SUITE="support-policy-pr" pytest --langsmith-output tests/

Verify: pytest must return a nonzero status for a broken assertion, and the named LangSmith test suite should show the same test case and pass feedback. Use LANGSMITH_TEST_TRACKING=false for a dry local run that must not synchronize.

Release gates need tiers. Run deterministic unit checks on each commit, a small critical dataset on pull requests, and broader or repeated model evaluations nightly or before release. Separate provider outages and rate limits from semantic failures. Never rerun a bad answer silently until it passes. Track token use and latency beside correctness using the LLM latency and cost testing guide.

8. Observability, Privacy, and Operating Cost

LangSmith has the advantage when offline evaluation must connect with production traces. A reviewer can move from a failed score to the sequence of application calls, attach feedback, identify a new failure pattern, and turn a sanitized case into regression coverage. That shortens investigation for retrieval and agent systems where the final response alone is weak evidence.

Promptfoo has the advantage when local execution and portable files are architectural requirements. A team can keep cases, prompts, configuration, and reports inside its established source and artifact controls. Promptfoo also offers hosted and self-hosted workflows, but you should evaluate those as deployment decisions rather than assume every Promptfoo run is local.

Privacy depends on configuration, not the logo. Before sending any test or trace data to a service, classify it. Remove credentials, payment data, personal identifiers, proprietary documents, and unrestricted tool payloads. Apply redaction before instrumentation exports data. Restrict access, define retention, document regions and subprocessors, and test deletion procedures. Synthetic IDs are safer for CI than copied production records.

Cost has at least four components: target-model calls, judge-model calls, platform usage, and engineering operations. A free CLI can still create a large provider bill through a Cartesian product. A managed trace platform can save investigation time that dwarfs subscription cost. Calculate expected cases multiplied by prompts, providers, repetitions, and graders. Put a concurrency cap and spend ceiling on scheduled suites.

Also measure human cost. If reviewers export spreadsheets because the chosen interface cannot support annotation, the workflow is incomplete. If engineers ignore a platform because failures cannot be reproduced from a commit, centralization has not produced trust. Pilot the full loop from failed case to corrected regression before standardizing.

9. Which Should You Choose

Choose Promptfoo when most of these statements are true:

• Prompts, provider definitions, cases, and thresholds should be reviewed together in Git.
• The main job is comparing several prompts or models over the same variables.
• A command-line runner and portable report are sufficient for triage.
• Security testing and red-team configuration belong near the evaluation suite.
• The team wants to begin without first designing a managed dataset workflow.

Choose LangSmith when most of these statements are true:

• The target is a chain, retrieval pipeline, or agent whose internal path explains failures.
• Offline experiments must connect to production traces and feedback.
• Domain reviewers need managed datasets, annotation, and shared comparison views.
• The application already uses LangChain or LangGraph, although LangSmith can trace other stacks too.
• Experiment lineage and example-level history matter more than a compact config diff.

Use both only with a written boundary. One defensible arrangement is Promptfoo for pre-merge prompt and provider matrices, and LangSmith for end-to-end application experiments plus production tracing. Keep case IDs aligned, but select one authoritative release result for each gate. Duplicating the same semantic judge in two systems creates disagreement without adding coverage.

Run a two-week proof of concept on one real change. Use twenty to fifty reviewed cases, at least three risk slices, one deterministic evaluator, one calibrated semantic evaluator, and a baseline. Record setup time, run cost, failure diagnosis time, reviewer usability, reproducibility, and data-governance gaps. The winner is the workflow that helps your team make a defensible release decision, not the product with the longest feature table.

10. Migration and Coexistence Strategy

Do not translate configuration line by line. First extract the evaluation contract: case ID, inputs, references, metadata, evaluator version, target version, and release rule. Preserve those meanings while changing representation. A Promptfoo contains assertion might become a LangSmith Python evaluator. A LangSmith dataset reference might become a checked-in YAML test with explicit required facts.

Migrate in three passes. First, run the old and new harnesses against stored deterministic outputs, which isolates evaluator parity from model variation. Second, run both against the same live candidate with identical provider settings. Third, compare disagreements example by example and classify them as data mapping, target execution, evaluator logic, or nondeterminism.

Keep immutable baseline artifacts until the new workflow proves stable across at least one real release decision. Do not change the prompt, model, dataset, and evaluator during migration, because any score movement becomes impossible to attribute. Preserve original case identifiers in metadata so an audit can trace a result across systems.

For coexistence, define ownership in the repository documentation. State which suite blocks a pull request, which platform owns production feedback, where human labels are canonical, and how a confirmed incident becomes a regression case. Give evaluator changes the same review discipline as application code.

A migration is complete when engineers can reproduce a failure, explain a score, locate the approved reference, and compare the current candidate with its baseline. Dashboard availability alone is not acceptance. Practice explaining these decisions with the LLM evaluation interview questions for QA engineers.

Interview Questions and Answers

Q: What is the central difference between Promptfoo and LangSmith?

Promptfoo is naturally configuration-first and matrix-oriented, while LangSmith is naturally application-run and trace-oriented. I choose Promptfoo when repository-owned prompt or provider comparisons dominate. I choose LangSmith when nested execution evidence, managed experiments, and production feedback dominate.

Q: Can Promptfoo evaluate an agent?

Yes. It can call custom providers or application endpoints and apply assertions to returned data. I expose a structured execution summary so assertions can inspect tools and policy decisions rather than infer them from prose. For deep trace investigation across nested calls, LangSmith usually offers the more direct workflow.

Q: Can LangSmith evaluate applications not built with LangChain?

Yes. The target can be an ordinary function, and tracing can be added through SDK wrappers or explicit instrumentation. LangChain and LangGraph integration is convenient, but it is not a requirement for using LangSmith evaluation.

Q: How would you prevent flaky release gates?

I favor deterministic invariants, pin candidate and evaluator configuration, distinguish infrastructure errors from quality failures, and repeat only risk-critical probabilistic cases. I never retry a semantic failure until it happens to pass. Thresholds are defined before reviewing candidate results.

Q: When is an LLM-as-judge appropriate?

It is appropriate when legitimate outputs vary and the criterion requires semantic judgment, such as relevance or tone. I calibrate the judge against expert labels, version the rubric and grader, and monitor disagreements by slice. Exact rules remain deterministic.

Q: Why might a team use both products?

Promptfoo can own fast prompt-provider matrices in pull requests while LangSmith owns application experiments and production traces. This is useful only when responsibilities are explicit. Each gate still needs one authoritative source of truth.

Common Mistakes

• Choosing from feature counts: Run the same target, data, evaluators, and release question through both workflows. Operational fit appears during triage, not on a checklist.
• Comparing unequal runs: Matching dataset names do not guarantee matching prompts, provider settings, grader versions, retries, or caches. Record complete candidate metadata.
• Using only aggregate scores: Inspect critical slices and individual regressions. A severe policy violation cannot be offset by easy successes.
• Treating model judges as objective: Calibrate against human labels and preserve judge explanations. A confident score can still encode bias.
• Copying production conversations into tests: Redact at collection time and use synthetic identifiers. Test stores must not become shadow production databases.
• Running real side effects: Replace write tools with fakes or sandbox implementations and assert the proposed operation.
• Ignoring matrix multiplication: Cases times prompts times providers times repetitions times graders determines call volume. Estimate it before CI runs.
• Letting two systems own one gate: Duplicate truth creates release disputes. Assign one owner and use the other tool for a distinct evidence layer.
• Weakening assertions after failures: First decide whether the target, reference, or evaluator is wrong. Threshold changes require review and a recorded reason.

Conclusion

The Promptfoo vs LangSmith decision becomes clear when you define the object under test and the evidence needed to debug it. Promptfoo is the practical default for transparent evaluation-as-code, prompt and provider matrices, direct assertions, and straightforward CI gates. LangSmith is the practical default for traced application evaluation, managed experiments, collaborative feedback, and a loop from production behavior back to offline tests.

Start with the same small, reviewed dataset in both tools. Measure reproducibility, diagnosis time, reviewer workflow, cost, and governance, then select the system that supports your actual release process. Whichever tool wins, keep deterministic safety checks separate, calibrate semantic judges, version every evaluation component, and turn confirmed failures into durable regression cases.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• LLM Judge vs Human Evaluation for Testing (2026)
• How to Choose an LLM Evaluation Framework (2026)
• Appium 3 vs Maestro for Mobile Testing (2026)
• Appium vs Detox for React Native Testing (2026)
• Cypress vs Playwright for Component Testing (2026)
• Cypress vs Vitest for Component Testing (2026)