Hi Rob, This behaviour is documented in the release note.
The tests for randomness use a random seed. So they fail with a given probability. We have a set number of reruns configured. A few failures get logged as a flake. If they fail 3 times then the build will fail. With a p-value of 0.001 this will be 1 in a billion. But there are thousands of tests. The build fail rate is closer to 1 in 1000 but I haven't done the analysis on it for a while. Hope that helps. Alex On Mon, 8 Jul 2024, 21:30 Rob Tompkins, <chtom...@gmail.com> wrote: > I was running the build on java 21 and got this: > Results: > [INFO] > [WARNING] Flakes: > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextBytesPartialBuffer(UniformRandomProvider)[8] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextBytesPartialBuffer:124 > org.apache.commons.rng.core.source32.Well44497b ==> expected: <true> but > was: <false> > [INFO] Run 2: PASS > [INFO] > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextFloat(UniformRandomProvider)[25] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextFloat:167->checkNextFloat:418->checkNextInRange:528 > org.apache.commons.rng.core.source32.XoShiRo128PlusPlus: Too many failures > for n = 1234, sample size = 1000 (12 out of 500 tests failed, chi2 > > 21.666=[25.107, 26.161, 22.291, 21.885, 23.420, 32.030, 23.780, 21.808, > 22.042, 22.531, 28.331, 25.669]) > [INFO] Run 2: PASS > [INFO] > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextFloat(UniformRandomProvider)[30] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextFloat:167->checkNextFloat:418->checkNextInRange:528 > org.apache.commons.rng.core.source64.TwoCmres (Cmres: [0xedce446814d3b3d9L, > 33, 330658535] + Cmres: [0xc5b3cf786c806df7L, 33, 331932042]): Too many > failures for n = 1234, sample size = 1000 (14 out of 500 tests failed, chi2 > > 21.666=[22.259, 23.260, 22.523, 30.913, 23.325, 22.569, 32.086, 24.000, > 22.940, 23.656, 39.190, 23.015, 23.873, 24.451]) > [INFO] Run 2: PASS > [INFO] > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextIntegerInRange(UniformRandomProvider)[25] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextIntegerInRange:142->checkNextIntegerInRange:389->checkNextInRange:528 > org.apache.commons.rng.core.source32.XoShiRo128PlusPlus: Too many failures > for n = 32, sample size = 1000 (13 out of 500 tests failed, chi2 > > 21.666=[23.824, 25.531, 22.309, 23.461, 23.315, 22.693, 21.968, 21.749, > 30.803, 23.157, 26.877, 22.016, 21.896]) > [INFO] Run 2: PASS > [INFO] > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextIntegerInRange(UniformRandomProvider)[2] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextIntegerInRange:139->checkNextIntegerInRange:389->checkNextInRange:528 > org.apache.commons.rng.core.source32.MersenneTwister: Too many failures for > n = 10, sample size = 1000 (13 out of 500 tests failed, chi2 > > 21.666=[23.920, 21.720, 22.440, 24.280, 23.920, 27.320, 22.680, 22.160, > 34.440, 27.020, 29.680, 22.260, 22.400]) > [INFO] Run 2: PASS > [INFO] > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextLongInRange(UniformRandomProvider)[14] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextLongInRange:155->checkNextLongInRange:404->checkNextInRange:528 > org.apache.commons.rng.core.source32.XoShiRo128Plus: Too many failures for > n = 31, sample size = 1000 (12 out of 500 tests failed, chi2 > > 21.666=[24.651, 24.155, 21.915, 21.778, 22.584, 21.667, 22.667, 24.093, > 31.256, 25.519, 22.101, 21.750]) > [INFO] Run 2: PASS > [INFO] > [WARNING] > org.apache.commons.rng.core.ProvidersCommonParametricTest.testUniformNextLongInRange(UniformRandomProvider)[16] > [ERROR] Run 1: > ProvidersCommonParametricTest.testUniformNextLongInRange:156->checkNextLongInRange:404->checkNextInRange:528 > org.apache.commons.rng.core.source32.PcgXshRr32: Too many failures for n = > 32, sample size = 1000 (12 out of 500 tests failed, chi2 > 21.666=[24.699, > 23.464, 25.043, 31.397, 31.739, 22.288, 24.637, 22.547, 22.483, 21.939, > 22.795, 21.973]) > [INFO] Run 2: PASS > [INFO] > [INFO] > [WARNING] Tests run: 2058, Failures: 0, Errors: 0, Skipped: 6, Flakes: 7 > > Any thoughts? > > > On Jul 8, 2024, at 4:21 PM, Alex Herbert <alex.d.herb...@gmail.com> > wrote: > > > > Hi Gary, > > > > Thanks for the feedback. I'll fix the docs module tomorrow. It is only > for > > an aggregate javadoc report for the site. It depends on all the other jar > > modules and builds a single aggregate javadoc. This requires it to be a > jar > > artifact in the pom. But there is no output jar. I'll disable japicmp to > > fix this. Really japicmp should handle a non-existent jar but it doesn't. > > > > The email I sent out is edited from the one built by the commons release > > plugin. But I did not edit the site link (oops). The site is not staged > > correctly for a multi-module jar. So I built the site separately and put > it > > in a personal Apache site. This makes it easy to see all the site > including > > the examples (which are not officially released but documented in the > site). > > > > Note: I cancelled this vote as some intentionally private constructors > were > > public. > > > > Alex > > > > > > On Mon, 8 Jul 2024 at 18:52, Gary D. Gregory <ggreg...@apache.org> > wrote: > > > >> Oops, actually 'mvn clean install -DskipTests japicmp:cmp' fails with > >> > >> [INFO] Reactor Summary for Apache Commons RNG 1.6: > >> [INFO] > >> [INFO] Apache Commons RNG ................................. SUCCESS [ > >> 21.207 s] > >> [INFO] Apache Commons RNG Client API ...................... SUCCESS [ > >> 24.159 s] > >> [INFO] Apache Commons RNG Core ............................ SUCCESS [ > >> 25.444 s] > >> [INFO] Apache Commons RNG Simple .......................... SUCCESS [ > >> 20.091 s] > >> [INFO] Apache Commons RNG Sampling ........................ SUCCESS [ > >> 42.508 s] > >> [INFO] Apache Commons RNG (Bill of Materials) ............. SUCCESS [ > >> 3.610 s] > >> [INFO] Apache Commons RNG Documentation ................... FAILURE [ > >> 3.601 s] > >> [INFO] > >> ------------------------------------------------------------------------ > >> [INFO] BUILD FAILURE > >> [INFO] > >> ------------------------------------------------------------------------ > >> [INFO] Total time: 02:22 min > >> [INFO] Finished at: 2024-07-08T13:42:52-04:00 > >> [INFO] > >> ------------------------------------------------------------------------ > >> [ERROR] Failed to execute goal > >> com.github.siom79.japicmp:japicmp-maven-plugin:0.21.2:cmp (default-cli) > on > >> project commons-rng-docs: The following artifacts could not be resolved: > >> org.apache.commons:commons-rng-docs:jar:1.5 (absent): Could not find > >> artifact org.apache.commons:commons-rng-docs:jar:1.5 in central ( > >> https://repo.maven.apache.org/maven2) -> [Help 1] > >> > >> The POM for that module is probably missing some configuration. > >> > >> Gary > >> > >> On 2024/07/08 17:50:41 "Gary D. Gregory" wrote: > >>> Hello, > >>> > >>> +1 > >>> > >>> This email contains this link that returns a 404 for > >>> > >>> > >> > https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/site/changes-report.html > >>> > >>> I see that for some reason, the site is in a personal folder instead of > >> https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/site, so I > >> imagine this is some search and replace error. > >>> > >>> ASC OK. > >>> SHA512 OK. > >>> Running 'mvn' OK. > >>> Running 'mvn clean install -DskipTests japicmp:cmp' OK. > >>> > >>> Using: > >>> > >>> Apache Maven 3.9.8 (36645f6c9b5079805ea5009217e36f2cffd34256) > >>> Maven home: C:\java\apache-maven-3.9.8 > >>> Java version: 17.0.11, vendor: Eclipse Adoptium, runtime: C:\Program > >> Files\Eclipse Adoptium\jdk-17.0.11.9-hotspot > >>> Default locale: en_US, platform encoding: Cp1252 > >>> OS name: "windows 10", version: "10.0", arch: "amd64", family: > "windows" > >>> > >>> Gary > >>> > >>> On 2024/07/05 12:21:37 Alex Herbert wrote: > >>>> We have fixed a few bugs and added enhancements since Apache Commons > >> RNG > >>>> 1.5 was released, so I would like to release Apache Commons RNG 1.6. > >>>> > >>>> Apache Commons RNG 1.6 RC1 is available for review here: > >>>> https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1 (svn > >>>> revision 70143) > >>>> > >>>> The Git tag commons-rng-1.6-RC1 commit for this RC is > >> commons-rng-1.6-RC1 > >>>> which you can browse here: > >>>> > >>>> > >> > https://gitbox.apache.org/repos/asf?p=commons-rng.git;a=commit;h=commons-rng-1.6-RC1 > >>>> You may checkout this tag using: > >>>> git clone https://gitbox.apache.org/repos/asf/commons-rng.git > >> --branch > >>>> commons-rng-1.6-RC1 commons-rng-1.6-RC1 > >>>> > >>>> Maven artifacts are here: > >>>> > >>>> > >> > https://repository.apache.org/content/repositories/orgapachecommons-/org/apache/commons/commons-rng/1.6/ > >>>> > >>>> These are the artifacts and their hashes: > >>>> > >>>> #Release SHA-512s > >>>> #Fri Jul 05 12:42:04 BST 2024 > >>>> > >> > commons-rng-1.6-bin.tar.gz=bd6475337043085fe48e0da4a10bbc06157b0a5cf0496e9784aedcdd27e60141575e127a49ae6dab21f2ec404e3f9b834c56155abfe58b0d8c7761f543bffdb8 > >>>> > >> > commons-rng-1.6-bin.zip=dbbc2fe27c024302dfce1d62414aa3718a8bfcdfd639e6acfae5828f624ea4bbcd01511e16b429be6873e8952354eb542f90b6ff62f3a00a5e55af9a5d7480bf > >>>> > >> > commons-rng-1.6-src.tar.gz=6c1f89aaf3889296d830bcfd3f05ff650670ca3b8aa5b3c395b59801615f4b98e076f6731c6c7922adc348fa5f0367e1dec0bcb176318a589b61961b064666c1 > >>>> > >> > commons-rng-1.6-src.zip=af96ba992540583e515a642a61ab03b98201f8f8e4b3ce7512f497feadf19aeea8b4fab2a8e113b7b65021744212f2b357a3907e2e1df74b4ea253875123d689 > >>>> > >> > org.apache.commons_commons-rng-1.6.spdx.json=9742c043487e1e49fa2dd824370bef005148af757c0f8626188edf4d39deafe115e4caa10b015892e9d32c1fe1b911d4b4119cd33e59d5107900d8c1ad145112 > >>>> > >>>> Signatures may be validated on a system supporting a bash Unix shell > by > >>>> executing: > >>>> svn co https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/ > >>>> cd 1.6-RC1 > >>>> chmod +x ./signature-validator.sh > >>>> for m in client-api core simple sampling bom; do > >>>> ./signature-validator.sh > >>>> > >> > https://repository.apache.org/content/repositories/orgapachecommons-1754/org/apache/commons/commons-rng-${m}/1.6/ > >> ; > >>>> done > >>>> > >>>> The source code contains examples that are not part of the public API. > >>>> These examples contain Java 11 modules and are enabled using a profile > >> (see > >>>> below). > >>>> > >>>> Note: Testing randomness using statistical thresholds results in > >> failures > >>>> at a given probability. The 'maven-surefire-plugin' is configured to > >> re-run > >>>> tests that fail, and pass the build if they succeed within the > allotted > >>>> number of reruns (the test will be marked as 'flaky' in the report). > >>>> > >>>> I have tested this with 'mvn clean install' and 'mvn clean package > site > >>>> site:stage -Pcommons-rng-examples' using: > >>>> > >>>> Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) > >>>> Maven home: /Users/ah403/software/apache-maven-3 > >>>> Java version: 11.0.23, vendor: Eclipse Adoptium, runtime: > >>>> /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home > >>>> Default locale: en_GB, platform encoding: UTF-8 > >>>> OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac" > >>>> > >>>> Details of changes since 1.5 are in the release notes: > >>>> > >>>> > >> > https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/RELEASE-NOTES.txt > >>>> > >>>> > >> > https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/site/changes-report.html > >>>> > >>>> Site: > >>>> > >> https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/index.html > >>>> (note some *relative* links are broken and the 1.6 directories are > >> not > >>>> yet created - these will be OK once the site is deployed.) > >>>> > >>>> JApiCmp Report (compared to 1.5): > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-client-api/japicmp.html > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-core/japicmp.html > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-simple/japicmp.html > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-sampling/japicmp.html > >>>> > >>>> RevApi Report (compared to 1.5): > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-client-api/revapi-report.html > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-core/revapi-report.html > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-simple/revapi-report.html > >>>> > >>>> > >> > https://home.apache.org/~aherbert/commons-rng-1.6-RC1-site/commons-rng-sampling/revapi-report.html > >>>> > >>>> RAT Report: > >>>> > >>>> > >> > https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/site/rat-report.html > >>>> > >>>> KEYS: > >>>> https://downloads.apache.org/commons/KEYS > >>>> > >>>> Please review the release candidate and vote. > >>>> This vote will close no sooner than 72 hours from now. > >>>> > >>>> [ ] +1 Release these artifacts > >>>> [ ] +0 OK, but... > >>>> [ ] -0 OK, but really should fix... > >>>> [ ] -1 I oppose this release because... > >>>> > >>>> Thank you, > >>>> > >>>> Alex Herbert, > >>>> Release Manager (using key BC87A3FD0A54480F0BADBEBD21939FF0CA2A6567) > >>>> > >>>> The following is intended as a helper and refresher for reviewers. > >>>> > >>>> Validating a release candidate > >>>> ============================== > >>>> > >>>> These guidelines are NOT complete. > >>>> > >>>> Requirements: Git, Java, Maven. > >>>> > >>>> You can validate a release from a release candidate (RC) tag as > >> follows. > >>>> > >>>> 1a) Clone and checkout the RC tag > >>>> > >>>> git clone https://gitbox.apache.org/repos/asf/commons-rng.git > --branch > >>>> commons-rng-1.6-RC1 commons-rng-1.6-RC1 > >>>> cd commons-rng-1.6-RC1 > >>>> > >>>> 1b) Download and unpack the source archive from: > >>>> > >>>> https://dist.apache.org/repos/dist/dev/commons/rng/1.6-RC1/source > >>>> > >>>> 2) Check Apache licenses > >>>> > >>>> This step is not required if the site includes a RAT report page which > >> you > >>>> then must check. > >>>> > >>>> mvn apache-rat:check > >>>> > >>>> 3) Check binary compatibility > >>>> > >>>> Newer components use JApiCmp with the japicmp Maven Profile: > >>>> > >>>> This step is not required if the site includes a JApiCmp report page > >> which > >>>> you then must check. > >>>> > >>>> mvn install -DskipTests -P japicmp japicmp:cmp > >>>> > >>>> 4) Build the package > >>>> > >>>> mvn -V clean package > >>>> > >>>> You can record the Maven and Java version produced by -V in your VOTE > >> reply. > >>>> To gather OS information from a command line: > >>>> Windows: ver > >>>> Linux: uname -a > >>>> > >>>> 5) Build the site for a multi-module project > >>>> > >>>> mvn site > >>>> mvn site:stage > >>>> Check the site reports in: > >>>> - Windows: target\site\index.html > >>>> - Linux: target/site/index.html > >>>> > >>>> Note that the project reports are created for each module. > >>>> Modules can be accessed using the 'Project Modules' link under > >>>> the 'Project Information' menu (see <path-to-site>/modules.html). > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>> For additional commands, e-mail: dev-h...@commons.apache.org > >>> > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >> For additional commands, e-mail: dev-h...@commons.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
