testng-annotations

The testng-annotations project contains some extra annotations that are useful when running tests in TestNG.

Requirements:

• Minimum Java version: 11
• TestNG 7.x
  • NOTE: TestNG upgraded to Java 11 starting from 7.6.0.
  • If you need to port this project to Java 8, you won't be able to go beyond TestNG 7.5.

Quick Start

1. Add dependency:

<dependency>
    <groupId>io.github.cpjust</groupId>
    <artifactId>testng-annotations</artifactId>
    <version>1.3.0</version>
    <scope>test</scope>
</dependency>

2. Register listeners (pick one):

• Add to test class:

@Listeners({IncludeOnEnvListener.class, ExcludeOnEnvListener.class})

• OR add to src/test/resources/META-INF/services/org.testng.ITestNGListener:

io.github.cpjust.testng_annotations.listeners.IncludeOnEnvListener
io.github.cpjust.testng_annotations.listeners.ExcludeOnEnvListener
io.github.cpjust.testng_annotations.listeners.annotation_transformers.AllAnnotationTransformers

3. Use annotations in tests:

// Skip in production
@ExcludeOnEnv("prod")
@Test
public void productionSafeTest() {
    // Won't run when -Denv=prod
}

// Only run in dev/qa
@IncludeOnEnv(value = {"dev", "qa"}, propertyName = "environment")
@Test
public void nonProductionTest() {
    // Only runs with -Denvironment=dev or -Denvironment=qa
}

// Using CSV string with delimiter
@ExcludeOnEnv(value = {"dev,qa,prod"}, delimiter = ",")
@Test
public void csvDelimitedTest() {
    // Won't run when -Denv=dev, -Denv=qa, or -Denv=prod
}

// Complex example with inheritance
@ExcludeOnEnv("ci")
public class BaseTest {
    // Class-wide exclusion
}

public class MyTests extends BaseTest {
    @IncludeOnEnv("staging")
    @Test // Combines class and method rules
    public void stagingOnlyTest() {
        // Runs if:
        // - Not in CI (-Denv=ci)
        // - In staging (-Denv=staging)
    }
}

// Using @ValueSource for parameterized tests
@Test
@ValueSource(strings = {"test1", "test2"})
public void testWithStrings(String value) {
    // Test runs twice, once with "test1" and once with "test2"
}

@Test
@ValueSource(ints = {1, 2, 3})
public void testWithInts(int value) {
    // Test runs three times with values 1, 2, and 3
}

Annotations

@ExcludeOnEnv

This annotation will exclude tests if the current environment (as defined by a Java property) matches one of the environments to be excluded. This annotation will not just mark a test as skipped, it will not even attempt to run the test and the test will not appear in the list of tests that were run if the test was excluded. NOTE: The environment names are compared case-insensitively.

CSV string support:
You can use the delimiter attribute to split values in the value array as CSV strings. For example:

@ExcludeOnEnv(value = {"dev,qa,prod"}, delimiter = ",")

This is equivalent to @ExcludeOnEnv(value = {"dev", "qa", "prod"}).

Ex. If a test is annotated with @ExcludeOnEnv(value = {"Stage", "Prod"}, propertyName = "environment") and you run with the -Denvironment=Prod option, the test will be excluded. If you omit the propertyName attribute, it will use "env" as the default property to check.

Tests should be excluded using the following rules:

	No class annotation:	Include by class:	Exclude by class:
No test annotation:	INCLUDE	INCLUDE	EXCLUDE
Include by test:	INCLUDE	INCLUDE	EXCLUDE
Exclude by test:	EXCLUDE	EXCLUDE	EXCLUDE

@IncludeOnEnv

This annotation will include tests if the current environment (as defined by a Java property) matches one of the environments to be included. This annotation will not just mark a test as skipped, it will not even attempt to run the test and the test will not appear in the list of tests that were run if the test was not included. NOTE: The environment names are compared case-insensitively.

CSV string support:
You can use the delimiter attribute to split values in the value array as CSV strings. For example:

@IncludeOnEnv(value = {"dev,qa,prod"}, delimiter = ",")

This is equivalent to @IncludeOnEnv(value = {"dev", "qa", "prod"}).

Ex. If a test is annotated with @IncludeOnEnv(value = {"Stage", "Prod"}, propertyName = "environment") and you run with the -Denvironment=Prod option, the test will be included. If you omit the propertyName attribute, it will use "env" as the default property to check.

Tests should be included using the following rules:

	No class annotation:	Include by class:	Exclude by class:
No test annotation:	INCLUDE	INCLUDE	EXCLUDE
Include by test:	INCLUDE	INCLUDE	INCLUDE
Exclude by test:	EXCLUDE	EXCLUDE	EXCLUDE

@ValueSource

Provides a simple way to parameterize tests with literal values. Similar to JUnit's ValueSource, this annotation runs the test method once for each provided value. This annotation is less powerful than TestNG's native DataProvider, but is easier to use for simple cases, and keeps the test data with the test method.

Supported value types:

• strings(): String values
• chars(): Char values
• booleans(): Boolean values
• bytes(): Byte values
• shorts(): Short values
• ints(): Integer values
• longs(): Long values
• floats(): Float values
• doubles(): Double values
• classes(): Class values

Requirements:

• Test method must have exactly one parameter.
• Parameter type must match the value type being provided.
• At least one value must be provided.

Example:

@Test
@ValueSource(strings = {"hello", "world"})
public void testStrings(String value) {
    assertNotNull(value);
}

@Test
@ValueSource(ints = {1, 2, 3})
public void testInts(int value) {
    assertTrue(value > 0);
}

@Test
@ValueSource(bytes = {1, 2})
public void testBytes(byte value) {
    // ...
}

@Test
@ValueSource(chars = {'a', 'b'})
public void testChars(char value) {
    // ...
}

@Test
@ValueSource(shorts = {10, 20})
public void testShorts(short value) {
    // ...
}

@Test
@ValueSource(floats = {1.5f, 2.5f})
public void testFloats(float value) {
    // ...
}

@Test
@ValueSource(classes = {String.class, Integer.class})
public void testClasses(Class<?> value) {
    // ...
}

---

@NullSource

Indicates that the annotated test method should be invoked with a single null argument.
This is useful for testing how your code handles null inputs.
Similar to JUnit's @NullSource.

Supported parameter types: Any single-parameter test method (except primitive types like: int, long, double).

Example:

@Test
@NullSource
public void testNullInput(String value) {
    assertNull(value);
}

---

@EmptySource

Indicates that the annotated test method should be invoked with a single empty argument.
This can be an empty String (""), empty array, or empty collection (e.g., List, Set, Queue, Map).
Similar to JUnit's @EmptySource.

Supported parameter types:

• String (injects "")
• Array types (injects empty array)
• Collection types (List, Set, Queue, Map - injects empty instance)

Example:

@Test
@EmptySource
public void testEmptyString(String value) {
    assertEquals("", value);
}

@Test
@EmptySource
public void testEmptyList(List<String> list) {
    assertTrue(list.isEmpty());
}

---

@NullAndEmptySource

Indicates that the annotated test method should be invoked twice: once with null and once with an empty value (as described in @EmptySource).
Equivalent to using both @NullSource and @EmptySource on the same method.
Similar to JUnit's @NullAndEmptySource.

Supported parameter types: Same as @EmptySource.

Example:

@Test
@NullAndEmptySource
public void testNullAndEmptyList(List<String> list) {
    // Runs twice: once with null, once with empty list
    if (list == null) {
        // handle null
    } else {
        assertTrue(list.isEmpty());
    }
}

---

Combining @NullSource, @EmptySource, @NullAndEmptySource, and @ValueSource

You can combine these annotations on a single test method to cover a wide range of input scenarios.
The test will be invoked once for each unique value provided by the annotations (duplicates are removed).

Example:

@Test
@NullSource
@EmptySource
@ValueSource(strings = {"foo", "bar"})
public void testAllCases(String value) {
    // Runs with: null, "", "foo", "bar"
}

Or, using @NullAndEmptySource:

@Test
@NullAndEmptySource
@ValueSource(strings = {"foo"})
public void testNullEmptyAndFoo(String value) {
    // Runs with: null, "", "foo"
}

---

@CsvSource

Provides a way to parameterize tests with comma-separated (CSV) values, similar to JUnit's CsvSource. Each string in the value array represents a row of arguments for the test method. The default delimiter is a comma, but you can specify a different delimiter if needed.

Requirements:

• Test method must have as many parameters as there are columns in each CSV row.
• At least one CSV row must be provided.
• Values can be quoted using single quotes (by default) to include delimiters or special characters as part of the value.
• Whitespace around unquoted values is trimmed.
• You cannot use a newline '\n', carriage return '\r' or the quote character as a delimiter.

Parameters:

• value: Array of CSV strings, each representing a row of arguments.
• delimiter: (Optional) The delimiter character to use (default is ',').
• quoteCharacter: (Optional) The character used to quote values (default is single quote ').
• trimWhitespace: (Optional) Whether to trim whitespace around unquoted values (default is true).

Examples:

@Test
@CsvSource({"foo,bar", "baz,qux"})
public void testWithTwoStrings(String a, String b) {
    // Runs twice: ("foo", "bar") and ("baz", "qux")
}

@Test
@CsvSource(value = {"1;2", "3;4"}, delimiter = ';')
public void testWithIntsAsStrings(String a, String b) {
    // Runs twice: ("1", "2") and ("3", "4")
}

@Test
@CsvSource({"'hello, world',42"})
public void testWithQuotedComma(String a, String b) {
    // Runs once: ("hello, world", "42")
}

---

EnumSource Annotation

The @EnumSource annotation allows you to provide enum constants as parameters to your test methods. This is useful for parameterized tests where you want to test all or specific constants of an enum.

Example Usage

@Listeners(EnumSourceListener.class)
public class MyEnumTests {

    private enum MyEnum {
        CONSTANT_ONE,
        CONSTANT_TWO,
        CONSTANT_THREE
    }

    @Test
    @EnumSource(MyEnum.class)
    public void testAllEnumConstants(MyEnum value) {
        // Test logic here
    }

    @Test
    @EnumSource(value = MyEnum.class, names = {"CONSTANT_ONE", "CONSTANT_TWO"})
    public void testSpecificEnumConstants(MyEnum value) {
        // Test logic here
    }
}

NOTE: The test method parameter must be declared with the exact enum type specified in @EnumSource (for example, MyEnum). Supertypes such as java.lang.Enum or java.lang.Object are not supported.

Modes: mode() (INCLUDE | EXCLUDE | MATCH_ANY | MATCH_ALL)

@EnumSource supports an optional mode() property that changes how the names() values are interpreted. The default is Mode.INCLUDE which preserves the previous behavior.

• INCLUDE (default): Treat names() as exact constant names to include. If names() is empty then all enum constants are included.

  Example (include two specific constants):

  @EnumSource(value = MyEnum.class, names = {"CONSTANT_ONE", "CONSTANT_TWO"}, mode = EnumSource.Mode.INCLUDE)
  public void includeConstants(MyEnum value) { }

• EXCLUDE: Treat names() as exact constant names to exclude. If names() is empty no constants are excluded (i.e., all constants are provided).

  Example (exclude one constant):

  @EnumSource(value = MyEnum.class, names = {"CONSTANT_ONE"}, mode = EnumSource.Mode.EXCLUDE)
  public void excludeConstants(MyEnum value) { }

• MATCH_ANY: Treat each entry in names() as a Java regular expression. A constant is included if at least one pattern matches its name. If names() is empty all constants are included.

  Example (match any constant whose name starts with "CONSTANT_O" or "CONSTANT_T" followed by any two characters):

  @EnumSource(value = MyEnum.class, names = {"^CONSTANT_O??$", "^CONSTANT_T??$"}, mode = EnumSource.Mode.MATCH_ANY)
  public void matchAnyRegex(MyEnum value) { }

• MATCH_ALL: Treat each entry in names() as a Java regular expression. A constant is included only if every pattern matches its name. If names() is empty all constants are included.

  Example (a name must match both patterns):

  @EnumSource(value = MyEnum.class, names = {"CONSTANT_.*", ".*_THREE"}, mode = EnumSource.Mode.MATCH_ALL)
  public void matchAllRegex(MyEnum value) { }

Notes about regex and behavior:

• MATCH_ANY and MATCH_ALL use Java regular expressions (java.util.regex.Pattern). The matching implementation uses Pattern.matcher(name).find() which allows substring matches; if you need a full-match use ^ and $ anchors in your pattern.
• Regular expression evaluation is case-sensitive by default (use (?i) if you need case-insensitive matching).
• If a provided pattern is syntactically invalid a IllegalArgumentException is thrown when the data provider is created.
• Empty names() is treated as "no filtering" for all modes (INCLUDE => include all; EXCLUDE => exclude none; MATCH_* => include all).

---

⚠️ Annotation Combination Restrictions

You cannot combine @CsvSource with any ValueSource annotation (@ValueSource, @NullSource, @EmptySource, or @NullAndEmptySource) on the same test method.

You also cannot specify a dataProvider in the @Test annotation if you use any of @CsvSource, @ValueSource, @NullSource, @EmptySource, or @NullAndEmptySource on the same method.

If a test method is annotated with both @CsvSource and any ValueSource annotation, or with a dataProvider and any of these source annotations, an error will occur and the test will not run. This is to prevent confusion, as only one data source can be used per test method.

Example (not allowed):

@Test(dataProvider = "intProvider")
@ValueSource(ints = {1, 2, 3})
public void testValueSourceAndDataProvider_throwsException(int value) { ... } // This will cause an error

@Test
@CsvSource({"foo,bar"})
@ValueSource(strings = {"baz"})
public void testWithBoth(String value) { ... } // This will cause an error

To fix: Use only one of the annotations or dataProvider per test method.

---

Listeners:

ExcludeOnEnvListener

This is the listener for TestNG tests that are annotated with @ExcludeOnEnv. To register this listener, either define it in the src/test/resources/META-INF/services/org.testng.ITestNGListener file (by adding io.github.cpjust.testng_annotations.listeners.ExcludeOnEnvListener to the file) or add the @Listeners({ExcludeOnEnvListener.class}) annotation to the test class.

IncludeOnEnvListener

This is the listener for TestNG tests that are annotated with @IncludeOnEnv. To register this listener, either define it in the src/test/resources/META-INF/services/org.testng.ITestNGListener file (by adding io.github.cpjust.testng_annotations.listeners.IncludeOnEnvListener to the file) or add the @Listeners({IncludeOnEnvListener.class}) annotation to the test class.

CsvSourceListener

This is the listener for TestNG tests that are annotated with @CsvSource. To register this listener, either define it in the src/test/resources/META-INF/services/org.testng.ITestNGListener file (by adding io.github.cpjust.testng_annotations.listeners.annotation_transformers.CsvSourceListener to the file) or add the listener to the testng.xml file.

Notes on annotation implementations:

After implementing the IAnnotationTransformer & IMethodInterceptor interfaces, getting TestNG to actually run them was tricky.

• IAnnotationTransformer only executes if it's defined in the src/test/resources/META-INF/services/org.testng.ITestNGListener file.
• IMethodInterceptor executes if it's defined in the src/test/resources/META-INF/services/org.testng.ITestNGListener file or if you add the @Listeners({ExcludeOnEnvListener.class}) annotation to the test class.
• The env system property is only picked up if defined with -Denv on the command line or in the @BeforeSuite, but not with @BeforeClass.

ValueSourceListener

This is the listener for TestNG tests that are annotated with @ValueSource, @NullSource, @EmptySource and @NullAndEmptySource. To register this listener, do one of the following:

Registration options: (pick one)

• Add to src/test/resources/META-INF/services/org.testng.ITestNGListener (RECOMMENDED):

io.github.cpjust.testng_annotations.listeners.annotation_transformers.ValueSourceListener

• or register it in the testng.xml file. Ex.

<listeners>
    <listener class-name="io.github.cpjust.testng_annotations.listeners.annotation_transformers.ValueSourceListener"/>
</listeners>

Usage notes:

• The listener automatically provides the correct data provider for methods annotated with @ValueSource.
• If you do not register the listener globally, you must specify the data provider in your @Test annotation:

  @Test(dataProvider = "valueSourceProvider", dataProviderClass = ValueSourceListener.class)
  @ValueSource(strings = {"foo", "bar"})
  public void testWithStrings(String value) { ... }

• If registered globally, you can simply use @Test and @ValueSource together.
• You cannot register this listener in the @Listeners annotation.

---

Combining CsvSourceListener and ValueSourceListener

You cannot register both listeners in the src/test/resources/META-INF/services/org.testng.ITestNGListener file. To use both, you must register io.github.cpjust.testng_annotations.listeners.annotation_transformers.AllAnnotationTransformers in the file instead, which will delegate to both listeners.

---

Building and deploying:

To just build, run mvn clean install

To build with signed artifacts, run mvn clean install -Psign-artifacts

You should follow the instructions on https://central.sonatype.org/publish/publish-maven/#gpg-signed-components if you run into problems. You may need to add configuration similar to the following to your settings.xml file:

<profiles>
  <profile>
    <id>testng_annotations_profile_id</id>
    <activation>
      <activeByDefault>true</activeByDefault>
    </activation>
    <properties>
      <gpg.executable>gpg</gpg.executable>
      <gpg.passphrase>xxx your passphrase here xxx</gpg.passphrase>
      <gpg.keyname>0x12345678</gpg.keyname>
    </properties>
  </profile>
</profiles>

Where the value of the gpg.keyname is found in the 'sig 3' line of the output of this command: gpg --list-signatures --keyid-format 0xshort

NOTE: If the signing fails with a "Bad passphrase" error, and you have the right passphrase and keyname, you may need to also set a MAVEN_GPG_PASSPHRASE environment variable to your passphrase value. Ex. export MAVEN_GPG_PASSPHRASE="your passphrase"

Deploy with mvn clean deploy -Psign-artifacts

Troubleshooting

Issue	Solution
Listeners not working	Verify registration via either: - @Listeners annotation on test class, OR - META-INF/services/org.testng.ITestNGListener file
Environment not detected	Set properties: - Via command line: -Denv=value - In @BeforeSuite method Note: @BeforeClass is too late!
Unexpected test execution	Check annotation precedence rules above Verify property name matches in annotations
IllegalArgumentException	Verify you aren't passing a blank or empty string as tha value or propertyName annotation properties
Signing failures	1. Verify GPG is installed 2. Set MAVEN_GPG_PASSPHRASE env var 3. Check settings.xml config
The @Test priority attribute is ignored	This is a bug in TestNG versions below 7.5
Other issues	Try upgrading/downgrading TestNG version Check Java version compatibility

Best Practices

1. Consistent Environments:

• Standardize on a property name (e.g. always use "env")
• Document expected values (dev, qa, prod)

2. Annotation Strategy:

• Use class-level for broad rules
• Method-level for exceptions
• Avoid mixing @Include and @Exclude
• Do not use @ValueSource and TestNG DataProvider on the same method

3. Debugging:

@BeforeSuite
public void logEnvironment() {
    System.getProperties().forEach((k,v) ->
        log.info("Prop: {}={}", k, v));
}

4. CI Integration:

# Sample GitHub Actions
jobs:
  test:
    env:
      ENV: ci
    steps:
      - run: mvn test -Denv=$ENV