gortiz commented on PR #10372: URL: https://github.com/apache/pinot/pull/10372#issuecomment-1453113881
I know you are good, but I wouldn't request you to review the performance impact on an external project without showing you the code. But I'm sure you can review these changes and try to imagine the impact on Apache Pinot. The code is mostly the same but: 1. We now create two OptionaInt objects. 2. The buffer is stored as a local variable instead of a final attribute. 3. `ScanBasedDocIdIterator.applyAnd(ImmutableRoaringBitmap)` is implemented as a call to the new overloaded method that receives an iterator. I would say that 2 should have a positive but negligible impact and given the length of `ScanBasedDocIdIterator.applyAnd(ImmutableRoaringBitmap)` it should always be inlined. About 1, it may have a very low impact, but we can find a way to do not create them if we think it is problematic. About the StarTree optimization: our plan is not to provide a better general window size and keep it in the proprietary code but to use optimize an inefficient case we only have in our proprietary plugins. The default 256 value may be great to improve the CPU cache utilization when the data is in main memory and as said, we don't want to change that. But when data has to be fetched using the network, the cost network cost is larger than anything else. In that specific case, larger blocks reduce the number of network calls and significantly increase the performance. Ideally we would like to continue applying windows of size 256 in the calculation part but fetch (and cache) larger windows, but that would require to communicate the planning layer with the storage layer. That dependency totally make sense, but it would need to do a larger refactor, so we are happy with this intermediate solution right now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
