peterxcli opened a new pull request, #5869: URL: https://github.com/apache/datafusion-comet/pull/5869
## Which issue does this PR close? Related to #5485. **Depends on #5859.** Review the [decoding-only diff](https://github.com/peterxcli/datafusion-comet/compare/091eb0020...2e65a02cb) (two commits, five files). This PR targets `main`, so GitHub also shows the parent commits until #5859 merges; it should be rebased afterward. ## Rationale for this change After #5859 removes intermediate row conversion, compressed numeric cache decoding remains expensive. Bypassing compression reduced six-long executor CPU from 494 to 172 ms, identifying the compression stream as the main remaining cost. Spark already uses delta encoding for these numeric values. ## What changes are included in this PR? Store suitable plain long columns as deltas before compression and restore their values in the shared cache reader. Keep validity bits and logical statistics unchanged. Require the delta stream to be over 25% smaller, skip a second compression for irregular full-width longs, and reuse codec factories per partition. ## How are these changes tested? Cache, row-reader, and Kryo suites pass on Spark 3.4.3, 3.5.9, and 4.1.3, including nulls, wrapping long arithmetic, native/Spark readers, projections, and recaching. Splitting the PR preserves the tested source exactly at `2e65a02cb`. ### Benchmark Spark 4.1.3, Java 17.0.20, AWS AMD EPYC 7282, `local[1]`, 6 GiB heap, 5M rows, 16 partitions, `MEMORY_ONLY`, LZ4. Mixed has three longs and three strings; numeric has six sequential/modulo-derived longs. All consuming operators run on Spark. Medians of 40 reads per cell across two fresh JVMs, after five warmups per query. Cache creation and planning are excluded. Parent, Spark, and child results come from separate rounds on the same host; these are descriptive comparisons across those runs. Parent code matches `091eb0020`; child is `2e65a02cb`. Answers match across all nine cases. | Schema | Columns read | Spark (ms) | #5859 (ms) | With this PR (ms) | | --- | --- | ---: | ---: | ---: | | Mixed | count(*) | 165.80 | 137.15 | 130.42 | | Mixed | 1 long | 189.16 | 188.63 | 143.66 | | Mixed | 1 string | 629.31 | 357.66 | 344.14 | | Mixed | 3 columns | 1125.97 | 657.88 | 616.77 | | Mixed | 6 columns | 2086.11 | 1100.20 | 1155.90 | | Numeric | count(*) | 123.58 | 86.79 | 91.54 | | Numeric | 1 long | 175.36 | 166.47 | 127.89 | | Numeric | 3 columns | 300.04 | 299.86 | 211.16 | | Numeric | 6 columns | 460.73 | 551.01 | 342.98 |  Whiskers show JVM median ranges. Mixed six-column child medians varied from 986–1,336 ms, and its pooled median was higher than the parent. Count-only and string-only differences do not establish benefits from long-delta decoding. These synthetic values favor delta encoding; native-consumer performance and broader data distributions remain unmeasured. Repeated cache builds increased from **1.62 to 2.26 s (+39%)** for numeric data and **1.49 to 1.61 s (+9%)** for random longs versus #5859. Numeric payload shrank from **107.7 to 2.1 MB**; random payload remained **241.3 MB**. Build figures use six measurements after two warmups per JVM, with reversed case order. Writing needs a temporary Arrow vector and can hold both compressed representations, so low-reuse caches may lose overall. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
