andygrove opened a new issue, #7068: URL: https://github.com/apache/datafusion-comet/issues/7068
### What is the problem the feature request solves? When a stage reads a Comet shuffle and ends in `ColumnarToRow`, every batch crosses the FFI boundary twice before it becomes rows: 1. Native code decodes the block and exports N columns through the Arrow C data interface (`NativeBatchDecoderIterator`, `NativeUtil.getNextBatch`). 2. The JVM imports them into Arrow Java vectors. 3. `CometNativeColumnarToRowExec` (`NativeColumnarToRowConverter`) exports the same N columns back to native code. 4. Native code re-imports them and builds `UnsafeRow`s. Each step recurses per child column for nested types, and the fixed per-batch cost is what makes small batches slow (see #5112 for the flat-schema numbers: 3.7x slower than the JVM converter at 8192 rows, 15.7x at 32 rows). ### Describe the potential solution - Add a fused decode-plus-C2R JNI entry for stages whose shuffle read feeds a row-producing converter, so the decoded batch never leaves native memory. - Fallback improvement if fusing is too invasive: export one struct-typed array per batch (one FFI struct instead of N) into a reused `VectorSchemaRoot`, and skip the per-column schema export on every batch. ### Additional context Finding R3 in #5905. Overlaps with #5112 (native C2R small batches), #6807 (the same JVM leg when the consumer is a native operator without AQE) and #6855 (nested-type conversion epic). A fix for #6807 that lets native consumers read blocks directly will not help this case, because the consumer here is the C2R converter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
