iverase opened a new pull request, #13563: URL: https://github.com/apache/lucene/pull/13563
Currently the DocValues skipper index collects the stats every 4096 documents that allow implementors to used them to decide if they want to process those documents or they can be skipped. The idea of adding levels is to be able to skip several of those block (called intervals in the code) in one step by collecting the stats of the blocks and adding them to the index. This implementation collects the stats of 8 of those intervals. For the first level, the first interval gets a new level with the stats from the next 8 intervals, then the 9th interval gets another level with the stats from the following 8 intervals and so on. For the second level, the first interval gets a new level with the stats from the next 8 level1 intervals, which is the same as the stats from the first 64 level 0 intervals and so on. I run some basic experiments and I must say I did not see much change on the performance, still it feels the right thing to do, therefore I opened this PR. relates https://github.com/apache/lucene/issues/11432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
