Unexpected test runs count with retry analyzer

[tbarabanov]

TestNG Version

7.6.1

Expected behavior

tests run count must depend only on parameters size and rules defined by retry analyzer

Actual behavior

tests run count depends on parallel flag of dataProvider annotation

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Arrays;
import java.util.Iterator;

public class RATest {


    public static class RA implements IRetryAnalyzer {

        int cnt = 0;
        static final int threshold = 2;


        @Override
        public synchronized boolean retry(ITestResult result) {
            System.out.println("retry:[" + hashCode() + "](" + Arrays.toString(result.getParameters()) + ")" + Thread.currentThread().getId());
            return cnt++ < threshold;
        }
    }


    @DataProvider(name = "test data", parallel = true) // parallel = false
    public Iterator<String> data() {
        return Arrays.asList("1", "2").iterator();
    }


    @Test(dataProvider = "test data", retryAnalyzer = RA.class)
    public void foo(String ignored) throws IOException {
        throw new FileNotFoundException("this can be retried");
    }


}

here is the output for parallel=true (5 times)

retry:[356127017]([1])16
retry:[1514394209]([2])17 - a new instance of retry analyzer !
retry:[356127017]([2])17
retry:[356127017]([1])16
retry:[356127017]([2])17

[ERROR] Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 3.891 s <<< FAILURE! - in TestSuite
[ERROR] Tests run: 5, Failures: 2, Errors: 0, Skipped: 3

here is the output for parallel=false (6 times)

retry:[1615523464]([1])15
retry:[1615523464]([1])15
retry:[1615523464]([1])15
retry:[132034104]([2])15
retry:[132034104]([2])15
retry:[132034104]([2])15

[ERROR] Tests run: 6, Failures: 2, Errors: 0, Skipped: 4, Time elapsed: 3.379 s <<< FAILURE! - in TestSuite
[ERROR] Tests run: 6, Failures: 2, Errors: 0, Skipped: 4

[tbarabanov]

Hi, folks. Are there any updates? IMO sporadic retryAnalyzer instantiation is not what most users expect to happen. As well as different count of test run.

[juherr]

Hi. Thanks for the report, it looks like an issue.

Could you write what are the expected outputs in both cases?

[tbarabanov]

It is hard to say because the answer depends on what "test" is. If test is steps + params (passed to test method, I would say a single method call) then I expect a new instance of RetryAnalyzer to be created two times and then 6 runs is the answer. If test is steps + all parameters (two parameters from dataProvider) then I expect a new instance of RetryAnalyzer to be created only once thus 1 run for the first param from dataProvider + two retries and one run for second param (zero retries because retries are exhausted) - 4 in total.