AI test review checklist (2026)

TL;DR

Review AI-generated tests as untrusted code. Confirm purpose and oracle first, then verify APIs, data, state, waits, isolation, security, diagnostics, and maintainability. Merge only after the test demonstrates that it can detect the stated failure.

Key Takeaways

• Confirm the requirement, risk, and detectable failure before reviewing generated syntax.
• Trace every expected result to an authoritative oracle and challenge weak assertions.
• Verify every library API, fixture, option, selector, wait, and helper against the installed stack.
• Review setup, data, state, isolation, cleanup, and parallel behavior as part of the test.
• Treat generated code as untrusted and enforce network, secret, dependency, and environment boundaries.
• Automate mechanical checks while keeping product correctness and risk decisions with reviewers.
• Measure accepted distinct coverage, stability, review effort, and maintenance rather than generated volume.

An AI test review checklist helps reviewers decide whether an AI-generated or AI-edited test is correct, valuable, safe, and maintainable before it enters the suite. Review the test as untrusted code. Verify its requirement, oracle, setup, data, synchronization, isolation, security, and failure diagnostics instead of accepting fluent syntax as evidence.

This 2026 guide provides a repeatable review method for UI, API, unit, integration, data, and AI-system tests. It includes a risk-based checklist, runnable static checks, review questions, approval rules, and metrics that reward durable coverage rather than generated volume.

TL;DR

Use the checklist in this order: purpose -> factual correctness -> oracle -> data and state -> interaction -> isolation -> safety -> diagnostics -> maintainability -> execution evidence -> approval. A syntactically valid test can still be dangerous or meaningless.

1. Scope of an AI Test Review Checklist

The checklist applies when AI proposes a new test, converts a manual scenario, updates selectors, generates fixtures, expands parameter sets, repairs a failure, or summarizes a test change. It also applies to human code heavily completed by an assistant when the author cannot explain every line.

Review responsibility does not move to the model or tool vendor. The author must understand the generated code, state its intended coverage, disclose important assumptions, and run relevant checks. The reviewer independently verifies the important claims. For high-risk controls, use subject-matter review and normal approval rules.

The checklist is not a single universal form. A pure unit test has different environment and security concerns from an API deletion test or a browser checkout journey. Keep a shared core and add technology-specific checks. Scale depth by consequence, uncertainty, change, model autonomy, and blast radius.

Review the diff, not just the final file. AI can remove assertions, weaken waits, broaden mocks, change cleanup, or silently update snapshots while making the test pass. Inspect generated fixtures and helper changes alongside the test because behavior often hides outside the visible case.

2. Confirm Purpose, Source, and Risk

Begin by asking what the test protects. Acceptable sources include an approved requirement, defect, incident, risk, contract, design decision, or explicitly explored behavior confirmed by the responsible owner. Record a stable reference. A prompt request such as add edge cases is not itself an oracle.

State the failure the test should detect. Covers checkout is too broad. Prevents a repeated payment callback from creating a second order is reviewable. Then ask whether the chosen test level can detect it economically. A service-level idempotency test may be stronger and faster than a browser script for the core invariant, with one UI test reserved for customer messaging.

Check novelty. Search for existing tests, shared fixtures, helpers, and equivalent assertions. AI frequently produces duplicates because it sees only selected context. A new case should add a distinct partition, state, role, platform, interface, or oracle. Cosmetic data changes are not new coverage.

Use risk to decide review depth. Authorization, payments, privacy, destructive operations, migrations, concurrency, and production-like data need stronger scrutiny. Link decisions to the risk-based testing guide rather than applying the same ceremony to every assertion.

3. Verify Factual and API Correctness

Confirm that every imported library, method, option, assertion, fixture, command, and configuration field exists in the repository's installed version. AI may combine APIs from different languages or versions, invent convenience methods, or copy an outdated pattern. Use local type checking, official documentation, and source definitions.

Check method semantics. A Playwright locator action and an ElementHandle action are not interchangeable. A mocked HTTP response may need correct content type and body encoding. A database transaction helper may roll back only within one connection. A pytest fixture's scope changes state sharing. Fluent code can hide these distinctions.

Run formatting, lint, type checks, compilation, and the narrow test. Then run the related suite in the same mode CI uses. A test that passes only because the IDE supplies an environment variable is not ready. Review lockfile and configuration changes if the suggestion added a package.

Reject undocumented sleep, retry, or exception patterns even when they happen to pass. Confirm timeouts use supported units and APIs. Check that asynchronous work is awaited and that assertions are not inside an unreturned callback. For API tests, validate URL construction, encoding, authentication, status semantics, and response parsing.

4. Challenge the Test Oracle

The oracle is the review center. Identify the exact assertion that would fail if the protected behavior broke. Then test the assertion mentally or with a controlled mutation. If changing the implementation to a known wrong result still lets the test pass, the test provides weak evidence.

Prefer observable outcomes and invariants over implementation details. For an order retry, assert the same order identity, one charge or creation event, correct user-visible state, and relevant audit behavior. Avoid asserting an internal function call unless that collaboration is the contract at the selected test level.

Check every expected value against an authoritative source. AI often guesses status codes, copy text, default values, sort order, timing, or rounding. If the requirement is ambiguous, raise the question. Do not freeze the model's guess into regression.

Review assertion scope. toBeTruthy() may allow many wrong values. Snapshot approval can hide an unintended bulk change. A response status assertion alone ignores the body and side effects. Conversely, asserting every JSON property or CSS class creates brittle noise. Select properties that represent the risk.

Negative tests require positive evidence of rejection. Confirm the operation did not occur, no unauthorized data leaked, and the error is appropriate. An exception caught and ignored is not a pass.

5. Review Setup, Test Data, and Preconditions

Make preconditions explicit and minimal. The test should create or obtain the state it needs through approved fixtures or APIs, verify critical setup, and avoid relying on execution order. AI-generated tests often assume a seeded account, fixed database row, previous test, local time zone, or feature flag without declaring it.

Use synthetic or approved masked data. Never accept hardcoded production records, real emails, access tokens, session cookies, payment numbers, or internal credentials. Generate unique identifiers where parallel runs can collide. Keep valid, boundary, and invalid data labeled so expected behavior remains obvious.

Check relationships and lifecycle. An order test may require product inventory, customer tenancy, payment state, and cleanup. A test that creates an impossible fixture can pass through mocks while failing to represent production behavior. Validate factories against the domain schema and reuse established builders.

Control time, randomness, locale, and configuration. Seed random generation when exact reproduction matters. Freeze time through supported application or test interfaces, not a global hack that breaks unrelated code. Set locale and time zone where expectations depend on them.

For deeper fixture patterns, see AI test data generation with Faker and LLMs. The generator may suggest data, but validation and privacy controls decide whether it can run.

6. Inspect Selectors, Synchronization, and Interactions

For browser tests, prefer user-facing locators such as role, accessible name, label, placeholder, or an intentional test ID when semantics are insufficient. Reject deeply nested CSS, dynamic generated classes, index-based selection, and broad text that matches unrelated elements. Confirm the locator is unique in the relevant state.

Use event-driven waiting. Playwright actions and assertions include auto-waiting, so arbitrary waitForTimeout calls usually add slowness without proving readiness. Wait for a meaningful UI state, response, event, or application condition. Avoid networkidle as a universal readiness rule for applications with background traffic.

For API and integration tests, synchronize on documented completion signals. Poll with a bounded timeout and diagnostic output when work is eventually consistent. Do not use an unbounded loop or fixed delay. Respect idempotency and rate limits.

Review interaction fidelity. A test that calls internal JavaScript to set application state may bypass validation and events that users trigger. Direct API setup is often appropriate for preconditions, but the action under test should pass through the relevant public boundary. Mocks should isolate a dependency intentionally, not replace the behavior the test claims to cover.

If the test fixes a locator problem, the Playwright strict mode violation guide shows why making a selector specific is better than choosing the first match blindly.

7. Check Isolation, Determinism, and Parallel Safety

A reliable test can run alone, after another test, repeatedly, and in parallel according to suite policy. Review global variables, singleton clients, shared accounts, static filenames, fixed ports, cached sessions, database rows, feature flags, and message queues. Generated code may overlook suite-level concurrency.

Use per-test namespaces or unique IDs. Create cleanup that is safe to retry and scoped to resources created by the test. Prefer platform-supported isolation such as browser contexts, transactions where valid, disposable containers, or tenant fixtures. Do not delete broad data by a shared prefix unless the environment guarantees ownership.

Check determinism without hiding real defects. Seed property inputs for reproduction while allowing controlled seed rotation in a separate job. Bound eventual polling. Capture clock and environment details. Avoid automatic retries as the first response to flakiness. A retry can collect evidence, but it does not make an unreliable test trustworthy.

Run order and repetition checks when risk warrants: the test alone, the file several times, related files in shuffled order, and parallel workers. Diagnose failures before approval. If isolation is impossible because of a known environment constraint, document it and route the case to a suitable serial suite rather than pretending it is parallel-safe.

8. Audit Security, Privacy, and Environment Safety

Treat generated test code as untrusted input. Inspect shell commands, file access, network destinations, package additions, dynamic evaluation, SQL, browser navigation, and cleanup. Reject eval, arbitrary command execution, disabled TLS verification, unrestricted redirects, or absolute URLs sourced from generated data unless a reviewed design requires them.

Secrets must come from approved secret injection and must never appear in code, prompts, fixtures, snapshots, screenshots, traces, or failure messages. Check redaction paths by forcing a safe failure. Use least-privilege accounts and synthetic or approved masked data.

Enforce environment boundaries in code and infrastructure. Destructive tests need explicit tags, isolated hosts, request and concurrency limits, stop conditions, and often manual approval. A comment saying test only is inadequate. Production validation, security probing, and load testing require their own authorized workflows.

Review dependency risk when AI adds packages. Confirm the package name, publisher, source, license, maintenance, lockfile change, and whether an existing dependency already solves the problem. Hallucinated or mistyped package names can create supply-chain risk.

The test should leave the environment in a known state. Cleanup must not expose data in logs or delete resources it does not own. Retain only evidence required by policy and set artifact access appropriately.

9. Evaluate Mocks, Stubs, and Test Doubles

Identify exactly what is real and what is replaced. A test that mocks the system under test or the critical business decision cannot support its stated risk. AI frequently over-mocks to make setup easy, then asserts calls to those mocks, creating a test of its own arrangement.

Use test doubles at clear boundaries. Match the real protocol, status behavior, headers, timing, and error shapes needed by the scenario. Validate stubs against a contract where possible. Keep at least one higher-level test for critical integrations and use sandbox environments for provider behavior that cannot be represented credibly.

Review default behavior. A catch-all mock returning success can hide unexpected requests. Prefer fail-closed handlers that report an unrecognized method, path, or payload. Reset mocks between tests and avoid global interception leaking across workers.

Check assertion value. Verifying that a dependency was called can be useful for a narrow unit collaboration, but it does not prove the user outcome. Assert the resulting state or returned value too. Avoid asserting incidental call order unless order is contractually important.

For generated AI-system tests, distinguish the model under evaluation from any model used to create test data or judge results. Record versions and prevent a generator from seeing held-out expected labels.

10. Review Diagnostics and Maintainability

A failed test should identify the scenario, important inputs, expected property, actual observation, build, and relevant correlation identifiers without leaking secrets. Use descriptive test and parameter names. Add assertion messages or structured attachments when the framework's default output is insufficient.

Keep the test readable. Use Arrange, Act, Assert or another consistent structure. Extract helpers when they represent a stable domain action, not merely to hide generated complexity. Avoid giant fixtures, deeply nested control flow, repeated polling code, and comments that restate syntax.

Check duplication across setup and assertions. Prefer established factories, clients, matchers, and cleanup fixtures. Do not create a second abstraction because the model did not see the first one. Follow repository naming, typing, formatting, and tag conventions.

Review brittleness. Exact timestamps, random IDs, full response snapshots, pixel positions, and incidental text can cause noisy failures. Normalize only nondeterministic fields that are irrelevant to the risk. Never delete assertions just to stabilize the test.

Estimate ownership. A complicated generated test that no teammate understands is technical debt on arrival. The author should be able to explain the failure model, data flow, waits, assertions, and cleanup. Prefer a smaller test whose evidence is clear.

11. Automate Mechanical Review Checks

Static checks can catch prohibited patterns consistently while reviewers focus on meaning. The following Node.js script scans JavaScript and TypeScript test files for several risky strings. It uses documented standard-library APIs and exits nonzero on findings.

import { readFileSync } from 'node:fs';

const forbidden = [
  { pattern: /waitForTimeout\s*\(/, message: 'replace arbitrary waits with an observable condition' },
  { pattern: /\.first\s*\(\)/, message: 'prove locator uniqueness instead of selecting the first match' },
  { pattern: /rejectUnauthorized\s*:\s*false/, message: 'do not disable TLS verification' },
  { pattern: /\beval\s*\(/, message: 'do not evaluate generated code' },
  { pattern: /https?:\/\/(?!localhost)/, message: 'use an allowlisted base URL from configuration' },
];

let failed = false;
for (const filename of process.argv.slice(2)) {
  const source = readFileSync(filename, 'utf8');
  for (const rule of forbidden) {
    if (rule.pattern.test(source)) {
      console.error(`${filename}: ${rule.message}`);
      failed = true;
    }
  }
}

process.exitCode = failed ? 1 : 0;

Save it as review-ai-tests.mjs and run node review-ai-tests.mjs tests/example.spec.ts. Tune rules to repository conventions and use AST-aware lint rules for robust enforcement. A text scan can produce false positives and cannot validate oracles, risk, or data safety.

Add existing formatter, linter, type checker, secret scanner, dependency audit, test repetition, and coverage tools to the gate. Keep generated provenance in metadata, not source comments that expose prompts.

12. Measure AI Test Review Checklist Outcomes

Define decisions clearly. Approve means required evidence exists and normal repository gates pass. Request changes identifies a fixable issue. Reject means the test lacks value, uses an unsupported oracle, duplicates coverage, or creates unacceptable risk. Quarantine is not approval and should require an owner and expiry.

For higher-risk tests, require evidence such as a failing run against the defect or a controlled mutation, followed by a passing run after the fix. This guards against tests that were never capable of detecting the problem. Store concise run evidence in the pull request or CI artifact.

Measure AI-assisted changes by accepted distinct coverage, review time, rejection reasons, post-merge flakiness, diagnostic quality, escaped defects, and maintenance churn. Compare with similar human-authored changes. Do not reward lines generated, number of tests, or percentage of AI suggestions accepted.

Review trends. Many unsupported oracles suggest poor source context. Duplicate cases suggest weak repository retrieval. Security findings suggest inadequate sandboxing. High rewrite effort may mean the task is better done manually. Feed lessons into templates, retrieval, lint rules, and author training.

Keep humans accountable without blaming individuals for tool behavior. Authors need clear boundaries and education, reviewers need time, and teams need permission to reject impressive-looking output.

Interview Questions and Answers

Q: What is the first thing you review in an AI-generated test?

I identify the requirement or risk and the failure the test should detect. Without a valid purpose and oracle, syntax review has little value.

Q: How do you detect a hallucinated test API?

I use the repository's installed types, compiler, linter, source definitions, and official documentation. I also run the test in the same environment as CI and inspect any dependency change.

Q: How do you review the oracle?

I trace every expected value to an authoritative source and ask whether the assertion fails for the important wrong behavior. For high-risk changes, I use a controlled mutation or pre-fix run.

Q: What security risks can generated tests introduce?

They can leak secrets, call arbitrary hosts, run shell commands, disable TLS, use real data, add unsafe dependencies, or perform destructive cleanup. I enforce infrastructure boundaries and inspect the entire diff.

Q: How do you handle flaky generated tests?

I investigate state, waits, data collisions, time, randomness, and environment assumptions. I do not weaken assertions or rely on retries as the primary fix.

Q: When should you reject an AI-generated test?

I reject it when the oracle is unsupported, it duplicates existing coverage, it cannot detect the stated failure, it is unsafe, or its maintenance cost exceeds its risk value. Fluent code is not a reason to merge.

Q: What metrics would you use for AI-assisted test authoring?

I use accepted distinct coverage, review effort, rejection themes, stability, diagnostics, escaped defects, and maintenance churn. Generated test count and acceptance rate can create harmful incentives.

Common Mistakes

• Reviewing syntax before confirming purpose and oracle.
• Trusting a method because the generated code looks idiomatic.
• Accepting status-only or truthy assertions for complex behavior.
• Adding arbitrary sleeps, retries, or first-match selectors to force a pass.
• Using production identifiers, secrets, or shared accounts in fixtures.
• Mocking the behavior the test claims to verify.
• Ignoring helper, snapshot, configuration, and lockfile changes outside the test file.
• Approving duplicate coverage because data values or wording differ.
• Disabling TLS, broadening cleanup, or allowing arbitrary network destinations.
• Merging a test the author cannot explain.
• Measuring productivity by generated lines or test count.

Conclusion

An AI test review checklist protects the suite from plausible but incorrect automation. Start with intent and oracle, then verify APIs, data, state, interactions, isolation, security, diagnostics, and maintenance. Automate mechanical rules, but keep product correctness and risk decisions with accountable reviewers.

Apply the checklist to one AI-assisted pull request and record every rejection or edit reason. Convert recurring mechanical issues into lint rules, improve the context supplied to the assistant, and keep the final standard simple: every merged test must provide credible evidence for a real risk.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Test case review checklist (2026)
• AI assisted API test generation (2026)
• AI code review for Playwright tests (2026)
• AI for exploratory test charters (2026)
• AI for flaky test root cause analysis (2026)
• AI for test coverage gap analysis (2026)