gf2121 commented on PR #12661: URL: https://github.com/apache/lucene/pull/12661#issuecomment-1766958252
An idea comes to me that maybe we do not really need to do combine all these `BytesRef`s to a single `BytesRef`, we can just build a `DataInput` over these `BytesRef`s to read. Luckily, only the encoded MSB VLong could probably be splitted into several arcs in FST while floor data, which is more sensitive to reading performance, is guaranteed stored continuously in single arc. With this optimization, i'm seeing PKLookup almost gets back to original speed: > Baseline: **before** https://github.com/apache/lucene/pull/12631; > Candidate: this patch; > Luceneutil Random seed = 0 ``` TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value Fuzzy2 39.46 (1.3%) 37.93 (1.0%) -3.9% ( -6% - -1%) 0.000 Respell 33.92 (2.3%) 33.04 (1.4%) -2.6% ( -6% - 1%) 0.000 Wildcard 60.14 (2.6%) 58.73 (2.3%) -2.3% ( -7% - 2%) 0.004 Fuzzy1 54.73 (1.5%) 53.62 (1.2%) -2.0% ( -4% - 0%) 0.000 PKLookup 111.96 (2.3%) 111.45 (2.3%) -0.5% ( -5% - 4%) 0.556 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
