agjendem opened a new issue, #3544: URL: https://github.com/apache/maven-surefire/issues/3544
### Affected version 3.6.0 (maven-surefire-plugin and maven-failsafe-plugin). The code involved is the same on master and on the `surefire-3.6.1` tag. ### Bug description With `runOrder=random` on the JUnit Platform and the default `forkCount=1`, the test classes are shuffled twice with the same seed. The second shuffle applies the same permutation again, so the class order is σ² instead of σ. Some orders can never occur: two test classes always run in alphabetical order, whatever the seed. **Reproducer** Three test classes `demo.ATest`, `demo.BTest` and `demo.CTest`, each with one test that prints its class name, and this `pom.xml`: <details><summary>pom.xml and ATest.java</summary> ```xml <project xmlns="http://maven.apache.org/POM/4.0.0";> <modelVersion>4.0.0</modelVersion> <groupId>demo</groupId> <artifactId>surefire-random-order</artifactId> <version>1.0</version> <properties> <maven.compiler.release>17</maven.compiler.release> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.junit</groupId> <artifactId>junit-bom</artifactId> <version>5.13.4</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.6.0</version> </plugin> </plugins> </build> </project> ``` ```java package demo; import org.junit.jupiter.api.Test; class ATest { @Test void run() { System.out.println("ORDER A"); } } ``` </details> For each of 20 large seeds: ``` mvn surefire:test -Dtest=ATest,BTest -Dsurefire.runOrder=random -Dsurefire.runOrder.random.seed=<seed> ``` As a control, the same seeds with only JUnit's random class orderer (default `runOrder`): ``` mvn surefire:test -Dtest=ATest,BTest "-DargLine=-Djunit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer\$Random -Djunit.jupiter.execution.order.random.seed=<seed>" ``` | Classes | `runOrder=random` | Only JUnit `ClassOrderer$Random` | |---|---|---| | A, B | AB ×20 | AB ×9, BA ×11 | | A, B, C | ABC ×17, CAB ×3 | ACB ×6, BCA ×6, BAC ×4, CAB ×3, CBA ×1 | The seeds must be large. For small seeds such as 1–20, `java.util.Random` returns the same first `nextInt(2)`, so the control gives BA every time. **Cause** (links to the `surefire-3.6.0` tag) 1. `DefaultRunOrderCalculator` creates `new Random(seed)` ([L52-L53](https://github.com/apache/maven-surefire/blob/0ff622b160253f362511a72d29a1f8067631f9d3/surefire-api/src/main/java/org/apache/maven/surefire/api/util/DefaultRunOrderCalculator.java#L52-L53)), sorts the classes and shuffles them ([L70-L72](https://github.com/apache/maven-surefire/blob/0ff622b160253f362511a72d29a1f8067631f9d3/surefire-api/src/main/java/org/apache/maven/surefire/api/util/DefaultRunOrderCalculator.java#L70-L72)). 2. `JUnitPlatformProvider.setupRunOrder()` sets `ClassOrderer$Random` and passes the same seed as `junit.jupiter.execution.order.random.seed` ([L168-L174](https://github.com/apache/maven-surefire/blob/0ff622b160253f362511a72d29a1f8067631f9d3/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L168-L174)). 3. With `forkCount=1` and `reuseForks=true`, the defaults ([`ForkStarter.isForkOnce()`](https://github.com/apache/maven-surefire/blob/0ff622b160253f362511a72d29a1f8067631f9d3/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/booterclient/ForkStarter.java#L338-L340)), the provider sends all classes to JUnit in one discovery request, in the shuffled order ([L278-L283](https://github.com/apache/maven-surefire/blob/0ff622b160253f362511a72d29a1f8067631f9d3/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/JUnitPlatformProvider.java#L278-L283)). 4. Jupiter's `ClassOrderer.Random` calls `Collections.shuffle(classDescriptors, new Random(seed))`. `Collections.shuffle` with a fresh `Random(seed)` applies the same permutation σ to every list of the same size. Step 4 applies σ to the result of step 1, so the final order is σ∘σ. A simulation of `java.util.Random` and `Collections.shuffle` with σ∘σ gives exactly the 40 orders in the `runOrder=random` column. With two classes, σ² is always the identity. In general only squares of permutations can occur, and the alphabetical order comes up with probability (number of involutions)/n!: 1 for n=2, 2/3 for n=3 and 5/12 for n=4. This came with #3421 (SUREFIRE-859), which both sorts the classes before the shuffle and forwards the seed to Jupiter. Each change is right on its own, but together they shuffle twice with one seed. **Why it matters** `runOrder=random` is how you look for order-dependent tests. Because of σ², a module with two test classes never runs them in the other order, and with more classes many orders never occur. We found this while we looked for order-dependent tests in a large multi-module build: the class orders we saw matched σ², not σ. **Possible fixes** - When all classes go to JUnit in one request with `ClassOrderer$Random`, keep the alphabetical sort but skip Surefire's own shuffle, so that Jupiter shuffles once. The path with one class per request (`LazyTestsToRun`) still needs Surefire's shuffle. - Or give Jupiter a seed that is derived from the Surefire seed, for example mixed with a constant, so that one seed still reproduces the whole order. ### AI assistance I used generative AI assistance while working on this report. I reviewed and verified the analysis and the results, and remain responsible for the report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
