Detecting flaky tests with machine learning (2026)

TL;DR

Detecting flaky tests with machine learning is a supervised ranking problem built on CI history, rerun evidence, and time-safe features. Train an interpretable baseline with chronological validation, choose a threshold from investigation capacity and failure cost, then use the score to prioritize reruns or triage without converting predicted flakiness into an automatic pass.

Key Takeaways

• Define flakiness operationally and preserve unknown labels instead of treating every rerun pass as proof.
• Build features only from information available before the prediction timestamp.
• Use chronological validation because random splits leak future test and infrastructure behavior.
• Start with interpretable heuristics and logistic regression before complex models.
• Evaluate precision, recall, ranking quality, calibration, and saved investigation effort, not accuracy alone.
• Use predictions to prioritize diagnosis and controlled reruns, never to hide failures automatically.
• Monitor label drift, feature drift, slice performance, and feedback loops after deployment.

Detecting flaky tests with machine learning can shorten CI triage when a large suite produces more intermittent failures than engineers can investigate immediately. The model should estimate risk, rank failures, and provide supporting evidence. It should never turn a failing test into a passing build merely because the failure resembles historical flakiness.

The hard part is not selecting an algorithm. It is defining a defensible label, preventing future information from leaking into features, validating on later time periods, and connecting predictions to safe actions. This guide develops an interpretable scikit-learn baseline, explains dataset construction and threshold selection, and shows how to operate the classifier without teaching the organization to ignore real defects.

TL;DR

Machine learning adds value only after test identity, execution events, retries, code revisions, environments, and ownership are reliably observable. Fix missing telemetry before optimizing a classifier.

1. Detecting Flaky Tests with Machine Learning as a Prediction Problem

A flaky test has inconsistent outcomes without a relevant change that explains the difference. That sentence needs an operational definition. A fail followed by a pass is a useful signal, but not conclusive proof. The first attempt may expose a real race condition in the product, the rerun may use different data, or a dependency may recover.

Choose the prediction question precisely. Two common questions are:

1. Before execution, which tests are most likely to produce an intermittent outcome?
2. After a first-attempt failure, which failures are most likely to be flaky?

The second question is often more actionable because the failure has a timestamp, logs, environment, duration, and error signature. Its label and feature set differ from pre-execution selection. Do not mix both in one table without a prediction-time column.

Define the unit as one first-attempt failure, not every retry row. Give it a stable execution ID, test ID, code revision, environment fingerprint, and timestamp. Attach later rerun or investigation evidence as labels, never as same-time features.

Write the intended action before modeling. A score might move a failure into an expedited isolated rerun queue, attach likely causes, alert the owning team, or rank a flakiness backlog. The action determines the cost of false positives and false negatives.

Maintain the core rule: predicted flaky is still failed. Product regressions can be intermittent, and a flaky test can also detect a real defect. The model supports triage, while release policy remains explicit in the test strategy guide.

2. Build Labels You Can Defend

Label quality places a ceiling on model quality. Useful evidence includes repeated outcomes on the same revision and controlled environment, failure-signature variation, isolation runs, a confirmed test defect, infrastructure incident correlation, and owner adjudication.

Use more than two states during curation:

A simplistic rerun passed = flaky rule creates noisy positives. It also creates selection bias because only some failures are rerun. High-profile tests may receive several reruns, while low-priority failures remain unknown. Record the rerun policy and propensity so you understand which population receives labels.

Create a labeling window. For example, evidence may include controlled reruns and owner resolution within an approved number of days. The exact window should match engineering flow. Freeze labels after the window for training snapshots, and version later corrections.

Capture cause categories such as timing or wait, shared state, test data, order dependency, resource exhaustion, network or service instability, clock or locale, environment drift, and actual product race. A binary model can still use cause labels for error analysis.

Review ambiguous high-impact cases independently. A payment test that intermittently exposes duplicate processing should not be dismissed as test flakiness because it passed on retry. The correct label may be product defect with intermittent reproduction.

3. Collect Event Data and Preserve Time

Join data from the test runner, CI orchestrator, source control, artifact store, environment inventory, and incident system. Keep raw events append-only where possible, then build versioned feature snapshots.

At minimum capture:

• execution and test IDs, suite, owner, and file path,
• code revision and branch type,
• start and end timestamps, attempt number, outcome, and duration,
• runner image, operating system, runtime, browser, shard, and worker count,
• failure class, normalized error signature, and artifact references,
• changed files or components known before execution,
• queue depth and resource measurements available at prediction time,
• rerun linkage and controlled-condition metadata,
• final investigation label, cause, reviewer, and date.

Normalize test identity across parameterization and renames. Keep both a logical test ID and an invocation variant. If names change on every generated data value, history fragments. If all variants collapse into one ID, a single unstable parameter may make the whole test look flaky.

Time matters at every join. Repository ownership, test age, quarantine state, and runner image must reflect their values at the prediction timestamp, not today's lookup. Slowly changing dimensions need effective dates.

Missingness is information but can also signal broken telemetry. Distinguish not applicable, not collected, and collection failed where the distinction matters. Monitor missing rates before allowing a model to learn that an instrumentation outage predicts flakiness.

Retain privacy and security boundaries. Logs can include tokens, customer data, URLs, and query contents. Extract approved features, restrict raw artifacts, and avoid making exception text broadly available through analytics.

4. Engineer Leakage-Safe Features

Every feature must be computable at the moment the prediction is made. If scoring happens immediately after the first failure, its duration and error class are available. The subsequent rerun outcome, future quarantine, issue resolution, and later failures are not.

Useful feature groups include:

Compute rolling features with a strict shift. A 30-run failure rate for execution N must use executions before N only. Dataframe rolling functions often include the current row unless explicitly shifted. Write a unit test with tiny known history to prove the calculation.

Avoid raw test name as an early feature. It can memorize historically flaky tests and report impressive validation when the same identities appear in both splits, while failing to generalize to new or changed tests. A historical baseline by test ID is still useful, but compare it explicitly and evaluate cold-start tests separately.

Normalize error signatures carefully. Line numbers, UUIDs, ports, timestamps, and data values can make identical failures look unique. Over-normalization can merge distinct assertions. Keep the transformation version and representative raw-to-normalized examples.

Do not use feature importance as causal proof. A browser feature may correlate with one overloaded runner pool, not with the browser engine. Predictions prioritize investigation, while controlled experiments establish cause.

5. Establish Rules and Statistical Baselines First

Before machine learning, implement baselines that any model must beat. A rule could flag a test whose prior-only outcome flip rate exceeds a reviewed threshold and that has at least a minimum history. Another could route known infrastructure signatures directly to an incident queue.

Useful baselines include:

• global positive rate,
• per-test historical flake rate with minimum support,
• per-signature historical rate,
• recent-window rule,
• timeout plus duration-deviation rule,
• owner or suite backlog ranking.

Baselines reveal whether the problem needs ML. If a simple known-signature table resolves most triage with high precision, maintain that table and use modeling only for the remaining population. A complicated model adds training, serving, monitoring, and explanation cost.

Logistic regression is a strong first classifier for mixed CI features after preprocessing. Its output can rank cases, coefficients are inspectable, and training is fast. Tree ensembles can capture interactions, such as a browser-runner combination becoming risky only at high load, but should earn their complexity through later-period validation.

Compare against the actual operational heuristic, not an artificially weak baseline. If engineers already know that three named tests are unstable, evaluate whether the model improves triage for the rest of the suite and for new tests.

Decide whether you need classification, ranking, or anomaly detection. A triage queue is naturally a ranking problem. New failure signatures with little label history may benefit from anomaly alerts. One binary threshold is not the only useful output.

6. Runnable scikit-learn Baseline

Prepare a CSV containing one labeled first-attempt failure per row. The recent_failure_rate_30 column must be computed from prior executions only. Install the current libraries and pin the tested versions in your project lockfile:

python -m pip install -U pandas numpy scikit-learn joblib

Save this as train_flake_model.py:

import sys
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    precision_recall_curve,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


TARGET = 'is_flaky'
NUMERIC = [
    'duration_ms',
    'recent_failure_rate_30',
    'test_age_days',
    'files_changed',
    'queue_depth',
    'hour_utc',
    'weekday_utc',
]
CATEGORICAL = [
    'runner_image',
    'browser',
    'os_family',
    'test_area',
    'branch_type',
    'failure_class',
]


def choose_threshold(y_true, probabilities, minimum_recall=0.80):
    precision, recall, thresholds = precision_recall_curve(
        y_true, probabilities
    )
    eligible = np.flatnonzero(recall[:-1] >= minimum_recall)
    if len(eligible) == 0:
        return 1.0
    best = eligible[np.argmax(precision[:-1][eligible])]
    return float(thresholds[best])


def main(csv_path: Path) -> None:
    data = pd.read_csv(csv_path, parse_dates=['started_at'])
    data = data.sort_values('started_at').reset_index(drop=True)
    data['hour_utc'] = data['started_at'].dt.hour
    data['weekday_utc'] = data['started_at'].dt.dayofweek

    required = {'started_at', TARGET, *NUMERIC, *CATEGORICAL}
    missing = required.difference(data.columns)
    if missing:
        raise SystemExit(f'missing columns: {sorted(missing)}')
    if len(data) < 100:
        raise SystemExit('need at least 100 labeled rows for this example')

    cut = int(len(data) * 0.80)
    train = data.iloc[:cut]
    validation = data.iloc[cut:]
    if train[TARGET].nunique() < 2 or validation[TARGET].nunique() < 2:
        raise SystemExit('both chronological partitions need both labels')

    numeric = Pipeline([
        ('impute', SimpleImputer(strategy='median')),
        ('scale', StandardScaler()),
    ])
    categorical = Pipeline([
        ('impute', SimpleImputer(strategy='most_frequent')),
        ('encode', OneHotEncoder(handle_unknown='ignore')),
    ])
    preprocessing = ColumnTransformer([
        ('numeric', numeric, NUMERIC),
        ('categorical', categorical, CATEGORICAL),
    ])
    classifier = Pipeline([
        ('features', preprocessing),
        ('model', LogisticRegression(
            class_weight='balanced',
            max_iter=2000,
            random_state=42,
        )),
    ])

    features = NUMERIC + CATEGORICAL
    classifier.fit(train[features], train[TARGET])
    probabilities = classifier.predict_proba(validation[features])[:, 1]
    threshold = choose_threshold(validation[TARGET], probabilities)
    predictions = (probabilities >= threshold).astype(int)

    print(f'validation_average_precision={average_precision_score(validation[TARGET], probabilities):.3f}')
    print(f'selected_threshold={threshold:.3f}')
    print(classification_report(
        validation[TARGET], predictions, zero_division=0
    ))

    joblib.dump(
        {
            'pipeline': classifier,
            'threshold': threshold,
            'feature_columns': features,
            'trained_through': str(train['started_at'].max()),
        },
        'flake_model.joblib',
    )


if __name__ == '__main__':
    if len(sys.argv) != 2:
        raise SystemExit('usage: python train_flake_model.py history.csv')
    main(Path(sys.argv[1]))

Run it with:

python train_flake_model.py history.csv

The code uses a chronological 80 percent and 20 percent split, a ColumnTransformer, OneHotEncoder(handle_unknown='ignore'), and LogisticRegression. The scikit-learn LogisticRegression reference documents the estimator. The illustrative threshold selects the highest precision among points meeting 0.80 recall on validation. Choose the target from your operational cost and verify it on another later period.

7. Validate Like a Time-Dependent System

Random train-test splitting lets future execution patterns teach the past and spreads nearly identical runs across partitions. Use walk-forward or chronological validation: train on an earlier interval and score the next interval, then advance. Reserve the latest untouched period for final verification.

Keep label finalization time in mind. A failure near the split boundary may not have completed its investigation window. Exclude immature labels so they do not become false negatives. Training cutoff should consider both execution timestamp and label availability.

Evaluate slices:

• new tests with little history,
• long-lived tests,
• each runner image and browser,
• critical and non-critical suites,
• changed and unchanged components,
• high-load periods,
• test-owned and infrastructure-owned causes,
• tests currently or formerly quarantined.

Run ablations. Train without raw identity, historical rate, failure class, or load features to understand dependence. If nearly all performance comes from a quarantine flag, the model has learned a downstream decision, not early detection.

Backtest the proposed action, not just classification. Given a fixed budget of 20 investigations per day, how many confirmed flaky failures would the top 20 contain? How many deterministic product failures would be delayed? How much median triage time would change under a replay simulation?

Use confidence intervals or repeated time windows where appropriate, and always report counts. Rare positives can make a single incident swing percentages. Inspect false positives with the highest score and false negatives with the highest impact.

8. Choose Metrics and Thresholds from Costs

Accuracy is usually misleading because most test executions pass and many first-attempt failures may be non-flaky. For post-failure triage, average precision summarizes ranking over the precision-recall curve. Precision answers how many flagged failures are truly flaky. Recall answers how many known flaky failures the workflow captures.

Also measure:

Choose a threshold after defining action. An expedited rerun may tolerate lower precision because its cost is small. Automatically notifying an owner needs higher precision to avoid alert fatigue. Any proposal to suppress a failure should require policy and human evidence, not a classifier threshold.

Class weighting helps training but changes the relationship between raw scores and real prevalence. Check calibration on natural later-period data. If probabilities drive resource allocation, consider a documented calibration method and validate it separately.

Do not optimize only a global score. A false flaky prediction on a security regression has greater cost than one on a cosmetic test. Apply separate routing or stricter rules for critical suites.

Define success against the current process: fewer hours to root cause, reduced uncontrolled reruns, lower quarantine duration, and fewer escaped intermittent product defects.

9. Deploy Scores Without Hiding Failures

Begin in shadow mode. Score failures, store explanations and model metadata, but do not change build status or routing. Compare predictions with later adjudication and observe volume by team. Fix data and threshold issues before influencing work.

A safe production response can attach a risk band and evidence such as prior-only flip rate, duration deviation, environment cluster, and normalized failure class. Avoid claiming cause. Likely associated with runner image X is a hypothesis for investigation, not proof that the runner caused the failure.

Recommended actions by score might be:

• high risk: schedule one controlled isolated rerun and page the test owner queue,
• medium risk: preserve artifacts and prioritize normal triage,
• low risk: follow deterministic-failure path with no delay,
• known infrastructure incident: route through incident correlation outside the classifier.

Bound retries. One or two diagnostically different reruns can gather evidence. Repeating until pass consumes capacity and hides reproducibility. Preserve the first failure and every retry as linked events.

Quarantine needs human approval, owner, reason, issue, expiration, and visible execution. The model can recommend investigation but should not silently remove coverage. The test plan guide can capture exit criteria and failure-handling policy.

Serve the model with a versioned feature contract. Reject missing critical fields or use validated imputations. Record model version, feature snapshot timestamp, score, threshold, action, and later disposition for audit and retraining.

10. Detecting Flaky Tests with Machine Learning in Production

Monitor data quality first: event delay, missing features, unknown categories, label backlog, join failures, and duplicate executions. A healthy endpoint with broken feature history can return confidently poor rankings.

Monitor feature and prediction drift by slice. A new runner image, browser version, parallelization policy, or test framework can change associations. Drift is a review trigger, not automatic evidence that model quality fell. Evaluate a newly labeled later-period set.

Watch feedback loops. If high-scored failures receive more reruns, they gain labels faster and dominate future training. If low-scored failures are ignored, deterministic defects can remain mislabeled as unknown. Sample and investigate across score bands to estimate bias.

Track action outcomes: controlled rerun result, root cause, time to owner response, quarantine decision, fix type, and recurrence. These close the loop only after quality checks. Never use the model's own action as the new ground-truth label.

Retrain on a schedule driven by enough new adjudicated data and meaningful drift, not merely because a week passed. Compare candidate and current models on fixed recent periods and critical slices. Require rollback artifacts and keep the previous model available.

Use a model card or operational document that states target, population, features, exclusions, training window, metrics, thresholds, owners, known limitations, and prohibited uses. Review it after material pipeline changes.

Machine learning should make flakiness more visible and fixable. If the dashboard becomes a justification for accepting red builds, the system has optimized the wrong outcome.

Interview Questions and Answers

Q: How would you define the label for a flaky test classifier?

I would use a written operational rule based on controlled repeated outcomes, unchanged relevant code and environment, failure evidence, and owner adjudication. I would keep infrastructure failure and unknown separate. A single rerun pass is evidence, not automatic truth.

Q: What is target leakage in this problem?

It is using information unavailable at scoring time, such as the later rerun result, future quarantine state, completed investigation, or rolling history that includes the current outcome. Leakage makes offline metrics look excellent while production performance collapses.

Q: Why use a chronological split?

The system predicts future CI behavior from past history. A random split mixes runner versions, repeated test executions, and later information across partitions. Later-period validation better represents deployment and reveals drift.

Q: Which model would you start with?

I would compare a rules baseline and per-test history with logistic regression over time-safe features. It is fast, inspectable, and easy to operate. I would adopt a more complex model only if it improves later-period and slice outcomes enough to justify the cost.

Q: Which metric matters most?

It depends on the action. For a bounded investigation queue, precision at K and recall at capacity are highly actionable. I also track average precision, calibration, severe false positives, triage time, and rerun cost.

Q: Should a high flakiness score pass the build?

No. Intermittent product defects and flaky tests can look similar, and both require attention. The score can prioritize a controlled rerun or investigation, while build and release policy remains deterministic and accountable.

Q: How would you monitor the deployed model?

I monitor feature availability, unknown categories, drift, prediction volume, slice metrics, label delay, and operational outcomes. I sample all score bands to detect feedback bias and compare new candidates on later adjudicated periods before promotion.

Common Mistakes

• Labeling every fail-then-pass sequence as flaky.
• Mixing test flakiness, infrastructure incidents, and intermittent product defects.
• Building rolling features that accidentally include the current or future row.
• Randomly splitting repeated CI executions across train and validation.
• Using current quarantine or issue state as a prediction-time feature.
• Reporting accuracy on a highly imbalanced target.
• Comparing only against a weak global baseline instead of current heuristics.
• Treating feature importance as proof of root cause.
• Auto-passing builds or silently quarantining tests from a model score.
• Retrying until green and then storing only the final attempt.
• Ignoring cold-start tests, critical suites, and new environments.
• Retraining on labels created by the model's own routing policy without bias checks.

Conclusion

Detecting flaky tests with machine learning is valuable when it ranks uncertain failures for faster, better investigation. Build trustworthy labels, preserve event time, create prior-only features, compare with strong baselines, and validate on future periods. Choose metrics and thresholds from the action's real costs.

Start in shadow mode with logistic regression and a controlled rerun workflow. Keep every failure visible, keep quarantine human-owned, and measure whether the system reduces investigation time and recurring flakiness. The best classifier is not the one that makes CI look greener. It is the one that helps teams fix instability without missing real defects.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Generating API tests from OpenAPI with AI (2026)
• Generating Selenium tests with AI (2026)
• How to Reduce flaky tests in a CI pipeline (2026)
• How to Use Cypress handling flaky tests (2026)
• Accessibility testing with Cypress (2026)
• Accessibility testing with Playwright (2026)