andygrove opened a new issue, #7069: URL: https://github.com/apache/datafusion-comet/issues/7069
### What is the problem the feature request solves? Shuffle decode runs on the task thread, serialized with downstream compute and with the next block's fetch. A reduce task alternates fetch, decompress, decode, then the rest of the native plan, with no overlap between them. In `native/core/src/execution/operators/shuffle_scan.rs` each batch is decoded inside the stream's poll (`decode_shuffle_batch`), and the JNI calls that pull the next block (`CometShuffleBlockIterator`) block the same thread. ### Describe the potential solution Investigate a bounded decode-ahead queue inside native: a worker decodes the next block (or next few) while the plan consumes the current batch. Open questions that make this the riskiest item in #5905: - Errors must reach the task thread so `on_decode_failure` still has a `JNIEnv` to report corrupt blocks. - The JVM iterator is not thread-safe, so block fetching probably has to stay on the task thread and only decode moves. - Memory accounting for the queued batches (`MemoryPool` reservation per queued batch). - Whether it helps at all when the reduce task is already CPU bound on 1 core per task. Start with a measurement: profile reduce tasks of a few TPC-DS queries and report decode time as a share of task time. If decode is under 10 percent, close this. ### Additional context Finding R4 in #5905. Related: #5686 (coalesce small shuffle-read batches before native sort), #6804 (ShuffleScan metrics missing under AQE, needed to measure this). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
