[GitHub] [pinot] richardstartin opened a new pull request #7931: don't use mmap for compression except for huge chunks

[GitBox]

richardstartin opened a new pull request #7931:
URL: https://github.com/apache/pinot/pull/7931


   Following on from #7930, it's possible to speed up building of V4 raw 
indexes slightly (~10%) by using a compression buffer and only using memory 
mapping for huge chunks. The size of the compression is fixed because the size 
of the chunk buffer itself is fixed, so the potential for very large 
compression buffers which exists with V3 and V4 does not apply here.
   
   master
   ```
   Benchmark                                  (_chunkCompressionType)      
(_distribution)  (_maxChunkSize)  (_records)  Mode  Cnt           Score     
Error  Units
   BenchmarkRawForwardIndexWriter.writeV4                         LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5        9857.203 ± 
167.084  ms/op
   BenchmarkRawForwardIndexWriter.writeV4:b                       LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5  7781048545.000     
           #
   BenchmarkRawForwardIndexWriter.writeV4:kb                      LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5     7598680.000     
           #
   BenchmarkRawForwardIndexWriter.writeV4:mb                      LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5        7420.000     
           #
   BenchmarkRawForwardIndexWriter.writeV4                         LZ4           
EXP(0.001)          1048576      100000    ss    5        1802.487 ± 112.871  
ms/op
   BenchmarkRawForwardIndexWriter.writeV4:b                       LZ4           
EXP(0.001)          1048576      100000    ss    5  1416542875.000              
  #
   BenchmarkRawForwardIndexWriter.writeV4:kb                      LZ4           
EXP(0.001)          1048576      100000    ss    5     1383340.000              
  #
   BenchmarkRawForwardIndexWriter.writeV4:mb                      LZ4           
EXP(0.001)          1048576      100000    ss    5        1350.000              
  #
   ```
   
   branch
   ```
   Benchmark                                  (_chunkCompressionType)      
(_distribution)  (_maxChunkSize)  (_records)  Mode  Cnt           Score     
Error  Units
   BenchmarkRawForwardIndexWriter.writeV4                         LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5        8788.638 ± 
256.238  ms/op
   BenchmarkRawForwardIndexWriter.writeV4:b                       LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5  7781048545.000     
           #
   BenchmarkRawForwardIndexWriter.writeV4:kb                      LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5     7598680.000     
           #
   BenchmarkRawForwardIndexWriter.writeV4:mb                      LZ4  
UNIFORM(1000,10000)          1048576      100000    ss    5        7420.000     
           #
   BenchmarkRawForwardIndexWriter.writeV4                         LZ4           
EXP(0.001)          1048576      100000    ss    5        1625.317 ±  48.117  
ms/op
   BenchmarkRawForwardIndexWriter.writeV4:b                       LZ4           
EXP(0.001)          1048576      100000    ss    5  1416542875.000              
  #
   BenchmarkRawForwardIndexWriter.writeV4:kb                      LZ4           
EXP(0.001)          1048576      100000    ss    5     1383340.000              
  #
   BenchmarkRawForwardIndexWriter.writeV4:mb                      LZ4           
EXP(0.001)          1048576      100000    ss    5        1350.000              
  #
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org