bvolpato opened a new pull request, #13668: URL: https://github.com/apache/iceberg/pull/13668
In one of our Trino -> Iceberg use case, which relies on equality deletes, we observed that a lot of time and allocations are being spent in `BaseFile.equalityFieldIds()` (~34% of our overall allocs). Checking the implementation, it seems that the current implementation based on streams has to copy and box the data, which is very inefficient. <img width="1310" height="802" alt="image" src="https://github.com/user-attachments/assets/b82fbd37-80fa-4bdd-9a33-81e8acf91504"; /> There is already prior art (https://github.com/apache/iceberg/pull/8336) to use Guava 0-copy wrappers for longs / split offsets in the same class, I'm doing that same change. Drafted a quick JMH to show the difference, and it clearly makes a huge difference: ``` Benchmark (arraySize) Mode Cnt Score Error Units IntListBenchmark.guavaImplementation 10 ss 10 1266.800 ± 199.548 ns/op IntListBenchmark.guavaImplementation 1000 ss 10 1387.200 ± 531.909 ns/op IntListBenchmark.guavaImplementation 100000 ss 10 1262.500 ± 188.043 ns/op IntListBenchmark.guavaImplementation 1000000 ss 10 1600.000 ± 1509.810 ns/op IntListBenchmark.streamImplementation 10 ss 10 8883.500 ± 1858.339 ns/op IntListBenchmark.streamImplementation 1000 ss 10 46229.000 ± 25265.952 ns/op IntListBenchmark.streamImplementation 100000 ss 10 736904.100 ± 299912.894 ns/op IntListBenchmark.streamImplementation 1000000 ss 10 7321966.800 ± 7146920.166 ns/op ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
