[GitHub] [pinot] itschrispeck opened a new pull request, #11354: add additional lucene index configs

[via GitHub]

itschrispeck opened a new pull request, #11354:
URL: https://github.com/apache/pinot/pull/11354

   Adds two configs for Lucene backed text index, both can be used to optimize 
for index writing time
   1. luceneUseCompoundFile: [compound file 
format](https://lucene.apache.org/core/8_1_1/core/org/apache/lucene/index/IndexWriterConfig.html#setUseCompoundFile-boolean-)
   2. luceneMaxBufferSizeMB: [max buffer 
size](https://lucene.apache.org/core/8_1_1/core/org/apache/lucene/index/IndexWriterConfig.html#setRAMBufferSizeMB-double-)
   
   Internal testing showed tuning these can improve realtime -> offline segment 
build time by ~30%. Some sample settings are shown below, collected by creating 
identical (minus lucene configs) tables in the same cluster. Build time is 
visualized via ingestion delay metrics: 
   
   
![image](https://github.com/apache/pinot/assets/27231838/cb089d6c-ae41-488e-b142-88ad9aaf4d1a)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org