Tibor17 commented on PR #3230: URL: https://github.com/apache/maven-surefire/pull/3230#issuecomment-3648486572
Thx for reporting it. I want to make an audit of current status of the project. Now I will create one issue which would aggregate many issues together, and then I would split this issue in several independent pull-requests but all of them would play this game. The reason is the historical design and idea how the test status is observed, delivered and finally processed. All the issues with re-run, flaky tests, parallel tests have a common denominator. Hopefully, the important code is shared across all the test providers, so it would not be so much difficult. Basically, I would follow the plan when we had 3.0.0 milestones because the road map defined the steps for this reason. This does not exist on the project web page but I have it in my local copy. The fixes made in February 2025 may only pretend that the parallel tests are deterministically reported. The same is with re-run and flaky tests, or combinations. Historically, we understood that the test classes are executed in a sequence. We also understood that the test methods also run in sequence and not re-acquiring the run. Even though we did not assume any reordering would happen. This is perhaps 12 years old era. Nothing has changed since then. As a result, two Report classes which make a guess about the status of particular test is complicated and there are two classes with this level of complexity which can be reduced if the Test Provider transfers such identifiers of the test which exactly say how the test should be reported straight ahead and then the logic would be simple and perfectly deterministic. The common problem is the identification of the test run itself. Many other identifiers which should identify the particular run of the test, as well as the mode of the run, and place in JVM are not transferred, we have no notion about them, and we only make a guess in the logic and of course the guess must be therefore wrong! So, many people who report and say "a report of parallel xyz does not work", or "report with flaky tests does not work", or "re-run is broken" is caused by this, and every time you fix one, you break another feature. Of course this has a solution but you cannot fix microscopical problem and suggest it's done. So far the methodology was like this like in many open sourced or commercial projects, quite normal... I have done this fix in my PR/branch but it's old, so I should to extract the changes and motivation from this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
