sgup432 opened a new issue, #16792: URL: https://github.com/apache/lucene/issues/16792
### Description As of today, when one index a double field(prices, coordinates etc) as doc values, lucene often stores it essentially uncompressed ie around 8 bytes per value. That's because doubles are converted to a long via `NumericUtils.doubleToSortableLong` before storage, so sorting and range queries work as plain long comparisons. So the numeric doc values encoders (GCD, FOR, etc.) don't work well on them, the decimal structure of the values is lost in the IEEE-754 bit pattern, so there's little left for these integer techniques to exploit. While reading about ALP (Adaptive Lossless floating-Point, SIGMOD 2024), seems like it uses a clever technique. For a block of values(1024), it picks an exponent e and encodes each digit as = `round (value * 10^e)`, so 90.11 becomes 9011, then apply FOR(Frame of reference) and other techniques, and bit pack smaller integers. Decode is just `value = digit / 10^e`. Any rare high-precision outlier is recorded as an exception (position, original) and patched back on read, so it's bit-exact lossless regardless of the data. For columns that genuinely are high precision (scientific doubles like 23.718281828459045 etc) the `×10^e` trick can't produce small integers, so there's a second scheme which is tried i.e. ALP_RD: split each value's bits into a high part (low cardinality -> dictionary-encoded) and a low part (random -> stored raw). It's a smaller win (~1.1–1.2×) but still helps. And its adaptive per block like how we also classify our blocks into sparse/dense in lucene. I did some experimentation and saw up to 3x savings in storage for doubles with few decimal places, like prices with 2 decimal places (e.g. 437.29), going from ~64 to ~20 bits per value. I think if we have to do this, best way would be put into this sandbox module as a experimental feature, by defining custom ALP codecs. We can probably use our existing docValuesSkipper built on top of this and reuse it on the search side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
