ConeyLiu commented on PR #7791: URL: https://github.com/apache/iceberg/pull/7791#issuecomment-1585492658
@pvary Thanks for pointing that out. This is really needed and found the potential problems. Previously, we use `java.util.WeakHashMap` to replace the `MapMaker().weakKeys().makeMap()`, while the key equally checking is really slow compared with identity. The result is as follows: ``` // java.util.WeakHashMap Benchmark Mode Cnt Score Error Units ManifestReadBenchmark.readManifestFile ss 5 12.896 ± 1.155 s/op // MapMaker().weakKeys().makeMap() Benchmark Mode Cnt Score Error Units ManifestReadBenchmark.readManifestFile ss 5 6.180 ± 0.067 s/op ``` Because for each avro record, we need to call the [DecoderResolver.resolveAndRead](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java#L72). So I keep the top map with `MapMaker().weakKeys().makeMap()` while the inner map changed to `java.util.WeakHashMap`. Here is the benchmarks: ``` // Master Benchmark Mode Cnt Score Error Units ManifestReadBenchmark.readManifestFile ss 5 12.896 ± 1.155 s/op // This change Benchmark Mode Cnt Score Error Units ManifestReadBenchmark.readManifestFile ss 5 6.190 ± 0.069 s/op ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
