dweiss commented on PR #16746: URL: https://github.com/apache/lucene/pull/16746#issuecomment-5926178078
Interesting. I also toyed with LLM-based benchmarking and improvements to various compression algorithms recently. The results do depend on which CPU you ran your training on (because the results of low-level trial and error optimizations tend to be different). Take a look at this forked code - it almost doesn't use java's built-ins for memory copying at all, instead relying on varhandles. This turned out to be the fastest path for typical compressed inputs (which didn't contain long repeated subsequences so block-copying wasn't as efficient). https://github.com/dweiss/aircompressor/blob/master-jdk25/src/main/java/io/airlift/compress/v3/lz4/Lz4RawDecompressor.java This is full lz4, not Lucene's simplified lz4 but I'd be curious as to how it fares against this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
