Evaluating an AI code reviewer (2026)

TL;DR

Evaluating an AI code reviewer requires a leakage-safe corpus of repository snapshots and diffs with expert-labeled findings, plus clean changes where no comment is expected. Score severity-aware recall, actionable precision, noise, localization, security, latency, and cost, then validate workflow impact in shadow mode before the system posts comments or blocks merges.

Key Takeaways

• Define the reviewer's context, tools, permissions, output schema, and intended action before scoring comments.
• Evaluate on real and carefully seeded diffs, including clean pull requests where silence is correct.
• Use qualified reviewers to label defect existence, severity, location, evidence, and acceptable finding matches.
• Measure severity-weighted recall, actionable precision, noise per pull request, localization, latency, and cost.
• Match findings by defect identity and evidence, not by exact wording or raw embedding similarity alone.
• Test prompt injection, secret handling, untrusted build execution, repository isolation, and comment permissions.
• Run offline shadow evaluation before allowing the reviewer to publish or block pull requests.

Evaluating an AI code reviewer is a defect-detection and developer-workflow measurement problem. A useful evaluation asks whether the reviewer finds material issues, explains them with correct evidence, points to an actionable location, stays quiet on acceptable code, and operates safely inside repository boundaries. Counting fluent comments is not enough.

The correct setup depends on what the reviewer can see and do. A diff-only assistant, a repository-aware reviewer, and an agent that runs tools have different opportunities and risks. This guide defines the evaluation contract, builds a golden corpus, shows a runnable finding-matching harness, covers security and non-determinism, and explains how to move from offline results to a controlled pull-request rollout.

TL;DR

A high recall score on synthetic bugs does not justify merge-blocking authority. Require evidence across realistic repositories, clean diffs, severe defect slices, security tests, and live shadow behavior.

1. Evaluating an AI Code Reviewer as a Defined Product

Start with the product boundary. What event invokes the reviewer? Which files, base revision, head revision, discussions, issue context, dependency manifests, generated files, and repository instructions can it read? Can it retrieve outside documentation, run a build, execute tests, call a code search tool, or post comments?

Write a bounded claim. For example: For Python service pull requests under 800 changed lines, the reviewer proposes non-blocking findings for correctness, security, reliability, and maintainability using the diff plus repository context. It does not evaluate product requirements or execute untrusted code.

Specify outputs. A finding should include category, severity, file, line or range, concise claim, code evidence, impact, and a safe suggested direction. It should support no findings. Free-form prose with no stable identity makes duplicate suppression and evaluation difficult.

Define intended authority:

Do not evaluate all modes as equivalent. A 70 percent actionable precision might support private triage but still create unacceptable public noise. Merge blocking requires an explicit deterministic policy and human override, not a vague model confidence.

Capture the contract in the broader test strategy for AI systems, including exclusions, environments, owners, and release criteria.

2. Model the Finding, Review, and Outcome

A reviewer emits findings, but the evaluation unit is usually a repository state plus a proposed change. Store base commit, head commit, diff, relevant files, build configuration, supported language versions, and allowed tools. A line has meaning only within that snapshot.

Represent a golden finding as a defect identity, not one canonical comment. Useful fields include:

• finding ID and case ID,
• affected file and location range,
• category and severity,
• preconditions for manifestation,
• evidence in the code or contract,
• impact and confidence of the human label,
• acceptable alternate locations,
• acceptable explanation elements,
• source such as incident, expert review, or seeded mutation,
• reviewer and adjudication state.

Some findings span several files and cannot be attached to one changed line. Allow repository-level findings with a primary anchor and supporting locations. Do not mark a correct architectural issue false merely because the comment chose another defensible line.

Separate defect truth from comment quality. First decide whether the issue exists. Then assess whether the AI comment identifies it, localizes it, explains evidence, estimates impact without exaggeration, and suggests a safe direction. A real defect with a harmful fix recommendation is partly correct, not fully actionable.

Store no-finding labels explicitly. An expert-reviewed clean change is essential for measuring silence. Absence of a golden finding in an incompletely reviewed pull request does not mean every AI comment is false. Label completeness must be part of the case metadata.

Finally, capture developer outcome separately: accepted, disputed, duplicate, already known, fixed, not fixed, or superseded. Developer reaction is useful but not automatically ground truth. People can accept bad suggestions or reject valid defects.

3. Build a Realistic and Leakage-Safe Corpus

Use a blend of real historical defects, expert-authored changes, seeded mutations, security examples, and clean refactors. Each source has limitations. Real defects are realistic but may be rare and reveal post-merge information. Seeded defects provide controlled coverage but can be artificial. Clean diffs are needed to expose over-commenting.

Construct cases from a clean base. Apply one or several documented mutations, confirm the repository builds or reaches the intended failing behavior where safe, and have a qualified reviewer verify the defect. Preserve the patch separately from the oracle. Mutations should represent plausible developer mistakes rather than broken syntax that any compiler catches.

Include defect categories relevant to the product:

• incorrect conditions and boundary errors,
• missing authorization or tenant checks,
• injection and unsafe deserialization,
• resource leaks and missing cleanup,
• concurrency and idempotency defects,
• error handling and retry amplification,
• data loss and transaction boundaries,
• API compatibility and schema changes,
• test quality failures that allow false confidence,
• performance problems with a defensible trigger.

Split by repository and change family where possible. If variants of the same mutation or adjacent commits cross development and test, the reviewer or prompt may memorize the pattern. Keep a held-out set of repositories or components to measure generalization.

Remove benchmark oracles from repository files visible to the reviewer. Do not leave comments such as BUG: missing auth check in fixtures. Prevent golden findings from entering retrieval indexes or prompt examples. The process in building golden datasets for evals applies directly to repository snapshots.

Respect licenses, secrets, and customer code policy. A benchmark copied from private repositories is still private, even if identifiers are changed.

4. Establish Expert Labels and Severity Rules

Code review labels require expertise in the language, framework, product contract, and threat model. Two reviewers should independently examine high-risk cases without seeing model output. Adjudicate defect existence and severity before scoring candidates.

Define severity with observable impact and reach:

Do not let style notes dominate precision. Many AI reviewers produce reasonable but low-value suggestions because they are easy to generate. Report defect findings separately from optional notes, and consider suppressing style categories already covered by linters and formatters.

Annotation instructions should tell reviewers which context is allowed, how to treat pre-existing defects, whether only changed lines qualify, how to label speculative risks, and how to match alternate explanations. A finding that is true but unrelated to the change may be useful in one product and prohibited noise in another.

Measure label agreement. Disagreement often exposes missing requirements or severity ambiguity. Record uncertain or insufficient_context rather than forcing a binary label. Exclude unresolved cases from a blocking metric but retain them for research.

Use source-backed rationales. Security findings should name the violated trust boundary or rule. Correctness findings should identify an input or state that produces the problem. This could be buggy is not a golden oracle.

5. Match AI Comments to Golden Findings

Exact text comparison fails because several comments can describe the same defect. Embedding similarity alone also fails because two fluent comments can sound alike while referring to different conditions or impacts. Use structured candidate generation followed by deterministic and expert-aware matching.

A candidate match normally requires compatible file or accepted repository-level scope, overlapping or nearby location, compatible category, and the same defect mechanism. Then evaluate explanation evidence. For ambiguous matches, a blinded human decides.

Use one-to-one matching. One verbose AI comment should not satisfy three distinct defects, and three duplicate comments should not earn three true positives. Duplicate candidates beyond the first count as noise. Maximum bipartite matching is appropriate when many findings overlap. A greedy baseline is acceptable for a simple harness when its limitations are documented.

Classify results:

• true positive: material defect correctly identified,
• partial: correct area but incomplete or overstated mechanism,
• false positive: no supported issue or wrong mechanism,
• duplicate: repeats an already matched finding,
• false negative: golden defect not found,
• out of scope: finding the product is explicitly not designed to produce.

Do not automatically count every out-of-scope comment as harmless. If it is shown to developers, it consumes attention and belongs in noise metrics.

Measure localization separately. A reviewer can understand the defect but anchor the comment on an unchanged or misleading line. GitHub's pull request review API documentation is useful when designing an adapter, but evaluation fixtures should use stable repository-relative locations independent of a live comment ID.

6. Runnable Finding-Matching and Metrics Harness

The following standard-library Python script evaluates exact category and file matches with a configurable line tolerance. It is intentionally transparent and suitable as a baseline.

Create expected.json:

[
  {
    "case_id": "retry-001",
    "findings": [
      {
        "id": "G-1",
        "file": "src/payments.py",
        "line": 42,
        "category": "idempotency",
        "severity": "high",
        "summary": "Retry creates a second charge without an idempotency key"
      }
    ]
  }
]

Create actual.json with the same case and candidate shape:

[
  {
    "case_id": "retry-001",
    "findings": [
      {
        "id": "A-7",
        "file": "src/payments.py",
        "line": 43,
        "category": "idempotency",
        "severity": "high",
        "summary": "Pass a stable idempotency key when retrying the charge"
      }
    ]
  }
]

Save this as score_reviewer.py:

import json
import sys
from dataclasses import dataclass
from pathlib import Path


SEVERITY_WEIGHT = {
    'critical': 8,
    'high': 5,
    'medium': 3,
    'low': 1,
    'note': 0,
}


@dataclass(frozen=True)
class Finding:
    id: str
    file: str
    line: int
    category: str
    severity: str
    summary: str


def load_cases(path: Path) -> dict[str, list[Finding]]:
    payload = json.loads(path.read_text())
    cases = {}
    for case in payload:
        case_id = case['case_id']
        if case_id in cases:
            raise ValueError(f'duplicate case_id: {case_id}')
        cases[case_id] = [Finding(**item) for item in case['findings']]
    return cases


def compatible(expected: Finding, actual: Finding, tolerance: int) -> bool:
    return (
        expected.file == actual.file
        and expected.category == actual.category
        and abs(expected.line - actual.line) <= tolerance
    )


def score(expected_path: Path, actual_path: Path, tolerance: int = 2):
    expected_cases = load_cases(expected_path)
    actual_cases = load_cases(actual_path)
    if set(expected_cases) != set(actual_cases):
        raise ValueError('expected and actual case IDs must match')

    true_positive = 0
    false_positive = 0
    false_negative = 0
    captured_weight = 0
    total_weight = 0

    for case_id, expected_findings in expected_cases.items():
        actual_findings = actual_cases[case_id]
        unmatched = set(range(len(actual_findings)))

        for expected in expected_findings:
            total_weight += SEVERITY_WEIGHT[expected.severity]
            candidates = [
                index for index in unmatched
                if compatible(expected, actual_findings[index], tolerance)
            ]
            if candidates:
                best = min(
                    candidates,
                    key=lambda index: abs(
                        expected.line - actual_findings[index].line
                    ),
                )
                unmatched.remove(best)
                true_positive += 1
                captured_weight += SEVERITY_WEIGHT[expected.severity]
            else:
                false_negative += 1

        false_positive += len(unmatched)

    precision = (
        true_positive / (true_positive + false_positive)
        if true_positive + false_positive else 1.0
    )
    recall = (
        true_positive / (true_positive + false_negative)
        if true_positive + false_negative else 1.0
    )
    weighted_recall = captured_weight / total_weight if total_weight else 1.0

    return {
        'true_positive': true_positive,
        'false_positive': false_positive,
        'false_negative': false_negative,
        'precision': round(precision, 4),
        'recall': round(recall, 4),
        'severity_weighted_recall': round(weighted_recall, 4),
    }


if __name__ == '__main__':
    if len(sys.argv) != 3:
        raise SystemExit(
            'usage: python score_reviewer.py expected.json actual.json'
        )
    result = score(Path(sys.argv[1]), Path(sys.argv[2]))
    print(json.dumps(result, indent=2))

Run it with:

python score_reviewer.py expected.json actual.json

This baseline will score the candidate as a match because file, category, and nearby line agree. A production evaluator must also compare defect mechanism and evidence, support alternate anchors, validate severity, classify partial findings, and handle multi-file defects. Keep human-adjudicated match fixtures to test any automated semantic matcher.

7. Use Metrics That Reflect Developer Cost

Report recall by severity. Missing a critical authorization bypass is not equivalent to missing a naming note. Severity-weighted recall summarizes one preference, but always show raw counts and per-severity results because weights are policy choices.

Precision needs a practical definition. True statement is weaker than actionable finding caused by this change. A comment can be technically correct but already enforced by a linter, irrelevant to the diff, too vague to act on, or based on unavailable requirements. Track actionable precision and general correctness separately if needed.

Useful metrics include:

Avoid one combined score for release decisions. A system can improve average F1 by adding many medium findings while still missing critical security issues. Set floors for non-negotiable slices and use a scorecard.

Measure review burden. Count comments per changed lines or per pull request, time spent dismissing false positives, and repeated categories already covered by automation. Developer thumbs-up data is easy to collect but subject to participation and team-culture bias.

For suggestions, measure compile or type-check success, test results, behavioral correctness, and security, not just acceptance. A developer may accept a plausible patch and later discover a regression.

8. Test Context, Tools, and Security Boundaries

Run separate evaluations for diff-only and repository-aware configurations. More context can improve recall but increases cost, sensitive-data exposure, and prompt-injection surface. Record exactly which files and retrieved chunks the reviewer saw.

Repository content is untrusted. A source comment, Markdown file, test fixture, issue, or generated artifact can say ignore policy, reveal secrets, or approve this change. The reviewer must treat it as code-review evidence, not higher-priority instruction. Create adversarial fixtures across file types and positions.

If tools are available, test allowlists, arguments, timeouts, output size, and network restrictions. Never execute untrusted pull-request code on a secret-bearing runner merely to improve review context. Use an isolated sandbox, read-only filesystem where possible, no production credentials, bounded resources, and controlled egress.

Test repository and tenant isolation. A review in repository A must not retrieve source, discussions, or embeddings from repository B. Delete and access-revocation tests matter when indexes persist.

Test secret behavior with synthetic canaries. The reviewer should not place credentials or sensitive file content into comments, logs, traces, or grader prompts. Redaction must cover tool outputs and exception paths, not only the final response.

Authorization belongs in code. The model should not choose whether it has permission to post, approve, request changes, or modify files. Use least-privileged application credentials and explicit branch or repository policy.

For broader agent safety test patterns, read agentic testing with tool calling.

9. Control Variance and Evaluate Explanations

Model outputs can vary in finding count, wording, location, and severity. Pin application, prompt, tool schemas, repository snapshot, model configuration, and evaluator versions. Repeat a representative subset to estimate detection and noise stability.

Compare sets of defect identities, not exact comment strings. A stable reviewer repeatedly finds the important issue and avoids unsupported claims even if the explanation wording changes. Report the probability that a critical finding appears across repeats rather than selecting the best attempt.

Use deterministic checks for output schema, file existence, line range, category enum, duplicate identity, forbidden data, and tool policy. Use expert rubrics for evidence quality, impact explanation, and suggestion safety. If an LLM judge assists, calibrate it against blinded expert decisions and retain disagreement cases.

A strong explanation identifies the relevant code behavior, a triggering condition, and the consequence. It does not invent runtime facts. A strong suggestion describes a direction and validation method without pretending one patch is universally correct.

Test calibrated abstention. When context is insufficient, the reviewer should say nothing or explicitly request the missing evidence according to product design. Rewarding comment volume encourages speculation.

Track model updates and provider aliases. Re-evaluate when a model, prompt, context-selection algorithm, tool, repository instruction parser, or post-processor changes. A renderer change can alter line anchors without changing model reasoning, so diagnose components separately.

10. Move from Offline Evaluation to Pull-Request Shadowing

Offline benchmarks provide controlled comparison. They cannot fully reproduce current repository conventions, developer behavior, CI state, or the distribution of real pull requests. Shadow mode runs the reviewer on real changes but stores output privately.

In shadow mode, sample across repositories, languages, change sizes, and teams. Have qualified reviewers label a subset of proposed findings and independently inspect sampled no-finding pull requests for misses. Without no-finding review, recall remains unknown.

Compare offline and shadow slices. A reviewer may perform well on compact seeded defects and poorly on dependency updates, generated code, or multi-service changes. Use the gap to improve the corpus, not to discard inconvenient live evidence.

Before advisory rollout, define rate limits, maximum comments per pull request, consolidation, duplicate suppression, timeout behavior, and an easy feedback path. Consider posting one summary with ranked findings rather than many inline comments.

Roll out to volunteer repositories first. Monitor actionable precision, severe misses found through audit, dismissal reasons, latency, cost, and incident reports. Keep the system non-blocking while the team establishes trust.

Blocking requires stronger evidence, deterministic criteria, ownership, an appeal or override path, and fast failure behavior. A model unavailable status should not silently approve a change. Decide whether it fails open or closed according to repository risk and make that state visible.

Use building evals in CI with promptfoo for a general pattern of locked runners, versioned cases, assertion layers, and failure artifacts.

11. Evaluating an AI Code Reviewer Continuously

Create evaluation tiers. Unit tests cover diff parsing, context selection, schema validation, location mapping, comment deduplication, permissions, and redaction. Integration tests exercise repository providers and sandboxes. Offline behavioral suites compare reviewer changes. Shadow audits measure live distribution. Security exercises test adversarial content and tool boundaries.

Version the full manifest: corpus, split, repository snapshots, golden findings, severity policy, matcher, rubric, model, prompt, context selector, tools, and threshold. A metric without this manifest is not reproducible.

Add new regression cases from escaped defects and high-quality disputed comments after sanitization and independent labeling. Keep a fixed holdout for comparison and rotate it when exposure becomes substantial. Do not tune on every live miss and then report the same cases as unbiased improvement.

Monitor drift in languages, frameworks, change size, generated-file rate, repository instructions, and finding categories. Inspect whether noise concentrates on junior-owned repositories or unfamiliar code styles. Fair workflow impact matters even when demographic attributes are not involved.

Review the reviewer's scope. If linters now catch a category deterministically, remove that category from model output or route it differently. If a new tool allows dependency inspection, add corresponding security and correctness cases before expanding the claim.

Maintain a kill switch and rollback. An update that posts secrets, floods comments, attaches to wrong lines, or produces dangerous fixes needs immediate containment. Preserve safe evidence for incident analysis.

Continuous evaluation should reduce developer burden while finding defects that existing automation misses. If the system's main output is more text to triage, its language quality is irrelevant.

Interview Questions and Answers

Q: What are the most important metrics for an AI code reviewer?

I prioritize recall by defect severity, actionable precision, clean-PR comment rate, duplicate noise, localization, evidence quality, suggestion safety, latency, and cost. I use a scorecard because one aggregate can hide a critical miss or excessive noise.

Q: How do you match an AI comment to a golden finding?

I require compatible defect identity, affected scope, location, category, mechanism, and evidence, with allowed alternate anchors. Matching is one-to-one so duplicates do not earn credit. Ambiguous semantic matches receive blinded expert adjudication.

Q: Why include clean pull requests?

Silence is a core behavior. A dataset containing only seeded defects rewards systems that comment everywhere and hides false-positive burden. Expert-reviewed clean and non-defect changes reveal whether the reviewer protects developer attention.

Q: Can developer acceptance be used as ground truth?

It is useful outcome data but not sufficient truth. Developers can accept unsafe suggestions, reject valid findings, or ignore comments because of time. I combine reaction with independent expert review and later defect or fix evidence.

Q: How do you test prompt injection in source code?

I place policy-conflicting instructions in comments, documentation, strings, fixtures, generated files, and retrieved discussions. I verify the reviewer treats them as untrusted content, does not expose canary secrets, stays within tool policy, and records a safe result.

Q: When can an AI reviewer block merges?

Only after strong evidence on realistic held-out and shadow data, especially for the exact blocking categories, plus stable operation and an override path. I prefer deterministic blocking rules. A general model confidence score is not enough authority.

Q: How would you handle non-determinism?

I compare finding identities and behavior rather than exact wording, repeat a controlled subset, and report stability for critical findings and noise. I never run repeatedly and select the best review. Schema, permissions, and tool rules remain deterministic.

Common Mistakes

• Counting comments or fluent explanations as defect-detection success.
• Benchmarking only seeded buggy diffs and omitting clean pull requests.
• Using exact wording to match semantically equivalent findings.
• Using embedding similarity alone to match different defect mechanisms.
• Giving duplicate comments multiple true-positive credits.
• Combining style notes and critical defects into one undifferentiated metric.
• Treating developer acceptance as unquestionable ground truth.
• Letting benchmark oracles appear in repository context or retrieval.
• Executing untrusted pull-request code on a runner with secrets.
• Allowing the model to decide repository authorization or merge status.
• Moving directly from an offline benchmark to public blocking comments.
• Reporting average F1 without severe-miss, clean-PR, and slice analysis.

Conclusion

Evaluating an AI code reviewer requires more than a list of buggy snippets. Define context and authority, build realistic repository snapshots with expert-labeled findings and clean changes, match defect identities one-to-one, and report severity-aware recall alongside actionable precision and workflow cost. Test the security boundary as aggressively as comment quality.

Begin with offline fixtures and the transparent matching harness, then run the reviewer in shadow mode on real pull requests. Expand to advisory comments only when evidence shows it finds material defects without overwhelming developers. Merge-blocking authority should remain a separate, much higher bar with deterministic policy, auditability, and human override.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Evaluating an AI test-case generator (2026)
• Testing an AI code assistant (2026)
• AI code review for Playwright tests (2026)
• Building an AI test report summarizer (2026)
• Evaluating an LLM app with DeepEval metrics (2026)
• Testing an AI agent with LangSmith (2026)