andygrove opened a new issue, #6565: URL: https://github.com/apache/datafusion-comet/issues/6565
## What / Why `CometSparkToColumnarExec` converts Spark rows and Spark column vectors to Arrow so that native operators can consume them. The conversion goes through `ArrowWriter`, which `CometLocalTableScanExec` and the Comet cache serializer also use. I measured it one column type at a time on `main` at `93d1189be`, with 8192-row batches. Several common cases are far slower than they need to be: - Decimals cost 30-160 ns per row, 30 to 100 times the cost of an int. Arrow's `BigDecimal` setter allocates a `BigInteger` and two byte arrays per value, on top of the `Decimal` and `BigDecimal` that Spark's getters build. - The bulk copy for fixed-width columns (#5442) is skipped when a batch holds a single null, and for every dictionary-encoded column. Spark's Parquet reader leaves low-cardinality columns dictionary-encoded. - Strings cost 13-16 ns per row, written one value at a time through Arrow's `setSafe`. A dictionary-encoded string also allocates a copy of its bytes on every row, because Parquet's dictionary returns a fresh array for each decode. - Arrays, maps and structs are written one element at a time: about 90 ns per row for `array<string>` and 200 ns for `map<string,string>`, with five entries on average. - Spark's vectorized Parquet reader produces 4096-row batches, and each becomes one Arrow batch, so native operators get half of Comet's batch size. The cache produces 10000-row batches, which become alternating Arrow batches of 8192 and 1808 rows. End to end, conversion took 549 ms of a 1088 ms TPC-H Q1-style query over Spark's Parquet reader. ## Work - [ ] Write Spark columns to Arrow in bulk for decimals, nullable and dictionary-encoded fixed-width columns, booleans, strings, arrays, maps and structs, and fill each Arrow batch across Spark batches. PR to follow. - [ ] Reallocating and zeroing every buffer for each batch (`allocateNew`) is about half the remaining cost of the fixed-width columnar path once the above lands. - [ ] Row input still writes strings and nested values one at a time. The failure semantics of the row-path string writer are pinned by the tests from #6281. - [ ] `CometSparkToColumnarExec.isTypeSupported` still declines arrays other than `ARRAY<STRING>` and maps other than `MAP<STRING,STRING>`, so those columns never reach the conversion and the operators above them stay on Spark. In the #6036 review, dropping that gate passed every probe query. - [ ] For row input, the `conversionTime` metric includes the time the child spends producing rows, so it overstates the conversion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
