bziobrowski opened a new pull request, #15130: URL: https://github.com/apache/pinot/pull/15130
PR changes how data is loaded into and kept in pre-loaded dimension tables: - lookup table is changed from generic HashMap to Object2ObjectOpenCustomHashMap - map re-sizings are limited by counting total number of documents prior to inserting into lookup table - lookup key and values are now Object tables, without being wrapped in PrimaryKey and GenericRow objects - key columns are not repeated in values table With BenchmarkDimensionTableOverhead I observed a 2.5 times memory usage reduction for the generated test data set and about 30% speedup in lookup table initialization. Benchmark results: master branch: ``` Benchmark (_numRows) (_numSegments) (_scenario) Mode Cnt Score Error Units benchmarkFastTableManager 3000000 1 EXP(0.001) avgt 5 3324.506 ± 842.324 ms/op benchmarkFastTableManager:gc.alloc.rate 3000000 1 EXP(0.001) avgt 5 468.456 ± 57.758 MB/sec benchmarkFastTableManager:gc.alloc.rate.norm 3000000 1 EXP(0.001) avgt 5 6431606028.800 ± 87270.360 B/op benchmarkFastTableManager:gc.count 3000000 1 EXP(0.001) avgt 5 152.000 counts benchmarkFastTableManager:gc.time 3000000 1 EXP(0.001) avgt 5 4869.000 ms ``` this branch: ``` Benchmark (_numRows) (_numSegments) (_scenario) Mode Cnt Score Error Units benchmarkFastTableManager 3000000 1 EXP(0.001) avgt 5 2345.304 ± 276.870 ms/op benchmarkFastTableManager:gc.alloc.rate 3000000 1 EXP(0.001) avgt 5 414.226 ± 28.436 MB/sec benchmarkFastTableManager:gc.alloc.rate.norm 3000000 1 EXP(0.001) avgt 5 5063652691.200 ± 53550.849 B/op benchmarkFastTableManager:gc.count 3000000 1 EXP(0.001) avgt 5 139.000 counts benchmarkFastTableManager:gc.time 3000000 1 EXP(0.001) avgt 5 2365.000 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
