Java for Testers: retry analyzer in TestNG (2026)

TL;DR

A java testers retry analyzer in TestNG implements `IRetryAnalyzer`, applies a strict attempt limit, and retries only failures classified as transient. Register it explicitly or through `IAnnotationTransformer`, isolate counters for parallel invocations, and preserve the first failure in reports so a pass on retry is tracked as flakiness rather than silently counted as healthy.

Key Takeaways

• Implement `IRetryAnalyzer.retry(ITestResult)` and return true only for a small, bounded number of eligible failures.
• Treat a retry as a new attempt for evidence collection, not as proof that the original failure did not matter.
• Attach an analyzer directly with `retryAnalyzer` or centrally through an `IAnnotationTransformer` registered before discovery.
• Keep retry counters isolated by logical invocation when tests or DataProvider rows run in parallel.
• Never retry assertion failures, invalid test data, or deterministic setup defects by default.
• Report both the first failed attempt and the final outcome so flaky passes remain visible.
• Use `testng-failed.xml` for deliberate failed-test reruns when an automatic retry is not appropriate.

Java testers retry analyzer in TestNG searches usually begin with a five-line counter, but a production retry policy needs more care. The IRetryAnalyzer interface lets TestNG ask whether a failed test should run again. Your implementation should answer yes only for an eligible transient failure and only within a small attempt budget.

A retry can reduce noise from a genuinely unstable external dependency, but it can also hide broken assertions, race conditions, and poor synchronization. This guide shows the correct TestNG APIs, automatic wiring, parallel-safe state, reporting, and the decision process that keeps retries from becoming a quality loophole.

TL;DR

Start with zero retries. Add one only when you can name the transient condition, collect evidence on both attempts, and assign ownership to the underlying instability.

1. Java Testers Retry Analyzer in TestNG: How the Lifecycle Works

IRetryAnalyzer has one method:

boolean retry(ITestResult result);

TestNG calls it after an associated test method fails. Returning true schedules another invocation. Returning false leaves the failure final. The analyzer receives ITestResult, which provides the throwable, method metadata, parameters, attributes, start and end times, and status for the attempt. It does not fix the state that caused the failure and it does not automatically restore browser or test data.

The word "retry" can mean three different operations in a team, so distinguish them:

1. An IRetryAnalyzer automatically repeats a failed invocation during the same TestNG run.
2. TestNG's generated testng-failed.xml starts a separate run containing failed methods and required dependencies.
3. CI job rerun repeats a broader process, often on a fresh agent.

Those operations have different isolation and reporting behavior. An analyzer usually reuses the same JVM and suite lifecycle. A @BeforeMethod normally runs again for the retry, but suite or class state may remain. If a failed UI attempt leaves an open modal, modified account, or abandoned order, the next attempt must receive clean state through setup and teardown.

Before adding retries to browser failures, review the flaky test debugging workflow. Most click and wait failures need better synchronization, not another attempt.

2. Build the Smallest Correct IRetryAnalyzer

This implementation allows one additional attempt. It uses an AtomicInteger because TestNG may execute tests in parallel and a plain increment is not atomic. The analyzer instance must be scoped appropriately, which the next sections address.

package example.retry;

import java.util.concurrent.atomic.AtomicInteger;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public final class OneRetryAnalyzer implements IRetryAnalyzer {
    private static final int MAX_RETRIES = 1;
    private final AtomicInteger retries = new AtomicInteger();

    @Override
    public boolean retry(ITestResult result) {
        int current = retries.getAndIncrement();
        boolean retry = current < MAX_RETRIES;
        if (retry) {
            System.err.printf(
                    "Retrying %s after attempt %d failed: %s%n",
                    result.getMethod().getQualifiedName(),
                    current + 1,
                    result.getThrowable());
        }
        return retry;
    }
}

Attach it directly to a test:

package example.tests;

import example.retry.OneRetryAnalyzer;
import org.testng.Assert;
import org.testng.annotations.Test;

public class HealthCheckTest {
    private int calls;

    @Test(retryAnalyzer = OneRetryAnalyzer.class)
    public void eventuallyResponds() {
        calls++;
        Assert.assertTrue(calls >= 2, "Illustrative first-attempt failure");
    }
}

This example demonstrates the lifecycle and passes on the second call. Do not copy the stateful failure pattern into real tests. A real test should assert a product outcome and receive isolated setup for every attempt.

Also note the terminology: MAX_RETRIES = 1 means at most two total attempts. Name metrics as attemptNumber and retryNumber carefully because teams often confuse one retry with one total execution.

3. Decide Which Failures Are Eligible

A blanket retry on every throwable is easy to implement and expensive to operate. Classify failures using a small allowlist. Good candidates may include a documented transient grid session-creation exception, a temporary connection reset from a controlled test dependency, or a rate-limited sandbox with a known recovery policy. Poor candidates include AssertionError, NullPointerException, invalid selectors, malformed test data, and authentication failures caused by wrong credentials.

Classification by exception class alone is imperfect. Selenium wraps several causes, messages vary, and a connection exception could represent a deterministic wrong host. Combine type checks with a narrow known operation or a custom framework exception whose constructor preserves the root cause.

Do not retry a failure merely because it occurred in CI. First compare artifacts, worker count, configuration, and shared records. The Selenium timeout diagnosis guide can help separate synchronization defects from infrastructure loss.

4. Implement Selective Retry With an Explicit Policy

Separate the retry decision from the counter so both are testable. This example retries one TransientTestInfrastructureException, but never an assertion failure. A framework can throw the custom exception only after it has gathered relevant evidence.

package example.retry;

public final class TransientTestInfrastructureException extends RuntimeException {
    public TransientTestInfrastructureException(String message, Throwable cause) {
        super(message, cause);
    }
}

package example.retry;

import java.util.concurrent.atomic.AtomicInteger;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public final class SelectiveRetryAnalyzer implements IRetryAnalyzer {
    private static final int MAX_RETRIES = 1;
    private final AtomicInteger retries = new AtomicInteger();

    @Override
    public boolean retry(ITestResult result) {
        Throwable failure = result.getThrowable();
        if (!(failure instanceof TransientTestInfrastructureException)) {
            return false;
        }
        return retries.getAndIncrement() < MAX_RETRIES;
    }
}

The custom exception must not become a universal wrapper. If a page assertion fails, preserve its AssertionError. If the remote grid rejects a session due to a known temporary capacity response, the driver creation layer can translate that narrowly and include the original cause.

Make the maximum retry count configuration-driven only if operations genuinely need it. Validate the integer at startup and cap it in code. An accidental retries=100 can multiply load during an incident. Prefer separate values for fast API checks and expensive UI scenarios only when observed behavior justifies the distinction.

A retry policy is production code. Add tests confirming eligible first failures return true, second failures return false, and ineligible failures never retry.

5. Apply a Retry Analyzer Centrally With IAnnotationTransformer

Adding retryAnalyzer to every @Test is repetitive when the suite has a deliberate global policy. IAnnotationTransformer can set the analyzer while TestNG reads annotations. Do not register an annotation transformer with @Listeners; TestNG needs the transformer before it processes that annotation. Register it in testng.xml, on the TestNG command line, or through the programmatic API.

package example.retry;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

public final class RetryTransformer implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation,
                          Class testClass,
                          Constructor testConstructor,
                          Method testMethod) {
        if (testMethod == null) {
            return;
        }
        if (annotation.getRetryAnalyzerClass() == null) {
            annotation.setRetryAnalyzer(SelectiveRetryAnalyzer.class);
        }
    }
}

Register it in the suite:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Regression">
  <listeners>
    <listener class-name="example.retry.RetryTransformer"/>
  </listeners>
  <test name="UI">
    <packages>
      <package name="example.tests"/>
    </packages>
  </test>
</suite>

The null check on testMethod matters because transformers can be invoked for annotation forms that are not ordinary test methods. Checking getRetryAnalyzerClass() respects a test's explicit analyzer rather than overwriting it.

Central wiring is not permission for blanket retries. The SelectiveRetryAnalyzer still rejects normal failures. You can also use a custom marker annotation or group to scope the transformer, but keep rules discoverable. A developer reading a test should be able to learn why it can retry.

6. Make Retry State Safe for DataProviders and Parallel Tests

The simple counter is correct only when one analyzer instance represents one logical test invocation. With DataProviders, factories, invocation counts, or central transformers, assumptions about instance reuse become risky. One row may consume the retry budget intended for another, and a single atomic counter prevents data races without providing logical isolation.

A robust design keys counters by the test method plus parameters and uses a concurrent map. Parameters must have stable, nonsecret identifiers. Do not put passwords or large mutable objects into a log key.

package example.retry;

import java.util.Arrays;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicInteger;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public final class PerInvocationRetryAnalyzer implements IRetryAnalyzer {
    private static final int MAX_RETRIES = 1;
    private final ConcurrentHashMap<String, AtomicInteger> counts =
            new ConcurrentHashMap<>();

    @Override
    public boolean retry(ITestResult result) {
        if (!(result.getThrowable()
                instanceof TransientTestInfrastructureException)) {
            return false;
        }

        String key = result.getMethod().getQualifiedName()
                + "|" + Arrays.deepToString(result.getParameters());
        AtomicInteger counter = counts.computeIfAbsent(
                key, ignored -> new AtomicInteger());
        return counter.getAndIncrement() < MAX_RETRIES;
    }
}

For a large suite, clean completed keys through a listener or use a run-scoped analyzer so the map does not retain thousands of parameter strings indefinitely. Better yet, pass a compact case ID as a DataProvider argument and build the key from it.

Parallel retry safety also depends on the test itself. WebDriver, mutable page state, downloads, report filenames, and test accounts must be isolated by thread or invocation. See the TestNG groups and parallel guide before enabling parallel retry traffic.

7. Java Testers Retry Analyzer in TestNG Reporting

A final green status can erase the operational signal that the first attempt failed. Your reporting layer should display total attempts, failure type and message for every attempt, artifacts, final status, retry reason, worker or thread, and a stable test case ID. Track "passed after retry" separately from a first-attempt pass.

Use an ITestListener to add attempt metadata and capture failure evidence. The exact screenshot integration depends on your driver lifecycle, but TestNG attributes are standard and useful for passing metadata between components.

package example.reporting;

import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicInteger;
import org.testng.ITestListener;
import org.testng.ITestResult;

public final class AttemptListener implements ITestListener {
    private final ConcurrentHashMap<String, AtomicInteger> attempts =
            new ConcurrentHashMap<>();

    @Override
    public void onTestStart(ITestResult result) {
        String name = result.getMethod().getQualifiedName();
        int attempt = attempts.computeIfAbsent(
                name, ignored -> new AtomicInteger()).incrementAndGet();
        result.setAttribute("attempt.number", attempt);
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.err.printf(
                "FAILED %s attempt=%s throwable=%s%n",
                result.getMethod().getQualifiedName(),
                result.getAttribute("attempt.number"),
                result.getThrowable());
    }
}

For parameterized parallel methods, use the same stable invocation key strategy as the analyzer. The example is intentionally small and records method-level attempts.

Avoid changing an earlier result from failed to skipped merely to make a report green. Report adapters differ in how they aggregate repeated invocations, and status mutation can hide evidence. Preserve raw TestNG results, then calculate a separate final logical outcome in your dashboard.

8. Reset Browser, API, and Data State Between Attempts

Retries are meaningful only if each attempt starts from a defined state. Put per-attempt setup in @BeforeMethod and cleanup in @AfterMethod(alwaysRun = true). Do not rely on a failed test reaching its own cleanup lines.

package example.tests;

import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;

public abstract class UiTestBase {
    @BeforeMethod(alwaysRun = true)
    public void createIsolatedContext() {
        // Create a new driver and unique scenario data for this invocation.
    }

    @AfterMethod(alwaysRun = true)
    public void cleanIsolatedContext() {
        try {
            // Capture final evidence and remove scenario data.
        } finally {
            // Quit the invocation's driver even after a test failure.
        }
    }
}

The comments mark framework-specific extension points; the TestNG lifecycle code itself is valid. Use alwaysRun = true for teardown so group filtering or a failed dependency does not unnecessarily prevent cleanup. Keep screenshot capture before quit.

A retry against the same changed database record can pass for the wrong reason. For example, the first attempt may create an order and time out before seeing confirmation. The retry creates a duplicate, or it finds the original and passes. Use idempotent setup, unique IDs, API cleanup, or a query that can distinguish the attempt's record.

Do not restart an entire shared Selenium Grid or service from a test retry. That changes external state for other workers. Infrastructure recovery belongs in platform automation with its own locks and observability.

9. Test the Retry Analyzer Deterministically

Unit-test the decision policy without running a flaky test. Mockito or another maintained mocking library can create ITestResult, but a small policy class that accepts a Throwable is even easier to test. Keep TestNG integration coverage to one controlled example.

package example.retry;

public final class RetryPolicy {
    public boolean isEligible(Throwable failure) {
        return failure instanceof TransientTestInfrastructureException;
    }
}

package example.retry;

import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
import org.testng.annotations.Test;

public class RetryPolicyTest {
    private final RetryPolicy policy = new RetryPolicy();

    @Test
    public void acceptsKnownTransientInfrastructureFailure() {
        assertTrue(policy.isEligible(
                new TransientTestInfrastructureException("grid busy", null)));
    }

    @Test
    public void rejectsAssertionFailure() {
        assertFalse(policy.isEligible(new AssertionError("wrong total")));
    }
}

For integration coverage, create a test that fails exactly once using a temporary file or an injected fake counter, assert that two starts were reported, and isolate it from normal regression metrics. Do not depend on a real network outage. A test of retry behavior must be deterministic even though the feature handles transient events.

Also test central registration. Run a suite containing one unannotated eligible test and confirm the transformer attaches the analyzer. This catches an easy configuration error, especially when someone incorrectly moves the transformer to @Listeners.

10. Establish a Retry Policy and Exit Criteria

Write down the policy before enabling retries across a repository. It should name eligible failure categories, maximum additional attempts, excluded groups, artifact requirements, ownership, metrics, quarantine behavior, and a date or condition for removing each exception.

A sensible pipeline can use these outcomes:

• Passed on the first attempt: healthy.
• Passed after retry: build may continue, but record a flaky event and notify the owning team.
• Failed after retry: failed build with evidence from every attempt.
• Ineligible failure: fail immediately.
• Retry infrastructure itself failed: fail and label as framework or environment error.

Set a quality gate based on your context. Some teams fail the build on any retry pass in protected branches while allowing it in an exploratory environment. Others allow a small known list but require a defect and expiry. The essential point is that green does not mean invisible.

Review retry metrics by stable test ID and root-cause category. Remove the analyzer once the dependency or test is fixed. If the count only grows, the policy is absorbing defects rather than reducing noise. Use the generated testng-failed.xml when an engineer wants a controlled rerun after inspecting the first run. That path is often better for deterministic failures because it preserves the original failure signal.

Interview Questions and Answers

Q: What does IRetryAnalyzer do in TestNG?

It receives the failed attempt's ITestResult and returns whether TestNG should invoke that test again. A correct implementation limits retries and applies an explicit eligibility policy.

Q: How do you attach a retry analyzer to a test?

Set retryAnalyzer = MyAnalyzer.class on @Test. For a suite-wide policy, set it through IAnnotationTransformer registered in XML, the command line, or TestNG's programmatic API.

Q: Why is a plain integer counter risky?

It is not atomic under parallel execution and may be shared across DataProvider rows or invocations depending on analyzer lifecycle. A thread-safe per-invocation key avoids one test consuming another test's retry budget.

Q: Should assertion failures be retried?

Not by default. An assertion reports that the observed product outcome violated the expected contract. Retrying it hides real defects unless a separately proven transient dependency is the actual cause.

Q: What is the difference between retry analysis and testng-failed.xml?

The analyzer repeats an invocation within the same TestNG run. testng-failed.xml is generated after a run and can launch a separate rerun of failed methods plus necessary dependencies.

Q: How should retries appear in reports?

Reports should preserve every failed attempt, identify the retry reason, attach evidence, and mark a final pass as passed after retry. They should not rewrite the event as an ordinary first-attempt pass.

Q: Why must teardown use alwaysRun = true?

A failed attempt still owns browser, file, and test-data resources. Always-run teardown increases the chance that each retry begins cleanly and prevents resource leaks after failures or skips.

Common Mistakes

• Retrying every throwable with no failure classification.
• Setting three retries and calling it three attempts, when it creates four total attempts.
• Sharing one integer counter across parallel parameter rows.
• Registering IAnnotationTransformer through @Listeners, where it is discovered too late.
• Overwriting a test's explicit analyzer in the transformer.
• Marking original failures as skipped to make a report green.
• Reusing a dirty browser session or database record on the next attempt.
• Retrying assertions, bad selectors, null pointers, or invalid credentials.
• Capturing evidence only on the final attempt.
• Increasing CI load during an outage through unbounded retries.
• Keeping a retry forever after the original issue is fixed.
• Using a retry pass as the only suite health metric.

Conclusion

A java testers retry analyzer in TestNG is valuable when it classifies a narrow transient failure, permits a bounded extra attempt, and preserves evidence from the original failure. The implementation is small, but lifecycle, parallel isolation, cleanup, and reporting determine whether it improves or weakens the suite.

Begin with one eligible exception and one additional attempt. Register the analyzer deliberately, report passed-after-retry as flakiness, and review every retry trend. The best retry analyzer is one whose usage shrinks as synchronization, test data, and infrastructure improve.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Java for Testers: Builder pattern for test data (2026)
• Java for Testers: exception handling in automation (2026)
• Java for Testers: Factory pattern in automation (2026)
• Java for Testers: Generics for frameworks (2026)
• Java for Testers: Log4j2 in framework (2026)
• Java for Testers: Maven profiles for suites (2026)