David Christensen writes:
On 2/12/24 08:30, Linux-Fan wrote:David Christensen writes:On 2/11/24 02:26, Linux-Fan wrote:I wrote a program to automatically generate random bytes in multiple threads: https://masysma.net/32/big4.xhtmlWhat algorithm did you implement?I copied the algorithm from here: https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtmlThat Java code uses locks, which implies it uses global state and cannot be run multi-threaded (?). (E.g. one process with one JVM.)
Indeed, the example code uses locks which is bad from a performance point of view. That is why _my_ implementation works without this fine-grained locking and instead ensures that each thread uses its own instance as to avoid the lock. IOW: I copied the algorithm but of course adjusted the code to my use case.
My version basically runs as follows:
* Create one queue of ByteBuffers
* Create multiple threads
* Each thread runs their own RNG instance
* Upon finishing the creation of a buffer, it enqueues
the resulting ByteBuffer into the queue (this is the only
part where multiple threads access concurrently)
* The main thread dequeues from the queue and writes the
buffers to the output file
Is it possible to obtain parallel operation on an SMP machine with multiple virtual processors? (Other than multiple OS processes with one PRNG on one JVM each?)
Even the locked random could be instantiated multiple times (each instance gets their own lock) and this could still be faster than running just one of it. However, since the computation itself is fast, I suppose the performance hit from managing the locks could be significant. Multiple OS processes would also work, but is pretty uncommon in Java land AFAIR.
I found it during the development of another application where I needed a lot of random data for simulation purposes :)My implementation code is here: https://github.com/m7a/bo-big/blob/master/latest/Big4.java
See the end of that file to compare with the “Numerical Recipes” RNG linked further above to observe the difference wrt. locking :)
If I were to do it again today, I'd probably switch to any of these PRNGS: * https://burtleburtle.net/bob/rand/smallprng.html * https://www.pcg-random.org/Hard core. I'll let the experts figure it out; and then I will use their libraries and programs.
IIRC one of the findings of PCG was that the default RNGs of many programming languages and environments are surprisingly bad. I only arrived at using a non-default implementation after facing some issues with the Java integrated ThreadLocalRandom ”back then” :)
It may indeed be worth pointing out (as Jeffrey Walton already mentioned in another subthread) that these RNGs discussed here are _not_ cryptographic RNGs. I think for disk testing purposes it is OK to use fast non- cryptographic RNGs, but other applications may have higher demands on their RNGs.
HTH Linux-Fan öö [...]
pgpLYMPAsjGtj.pgp
Description: PGP signature
