sunchao opened a new pull request, #5368: URL: https://github.com/apache/datafusion-comet/pull/5368
## Why are the changes needed? Closes #4383. The Spark 4.x Comet Python runner currently copies every source Arrow column into an intermediate vector tree before serializing each batch for `mapInArrow` and `mapInPandas`. This extra per-batch allocation and memory copy is avoidable: the existing Comet vectors already have the Arrow physical layout required by the Python worker. ## What changes were proposed in this PR? - Serialize existing Comet `FieldVector` buffers directly into Arrow IPC record batches while retaining the existing schema-only stream root. - Allocate only the validity bitmap for the non-null struct that wraps the input columns, and retain/release borrowed native-backed Arrow buffers correctly on success and failure. - Preserve nested struct/list/map layouts, nullability, variadic-buffer metadata, empty batches, Spark 3.5 fallback, and incompatible large-variable-width fallback behavior. - Add focused JVM ownership/layout/error-cleanup coverage and multi-batch PySpark worker regressions; register the new Scala suite in both Linux and macOS CI manifests. - Update the Python UDF documentation and make the benchmark build profile match its documented PySpark version. ## How was this PR tested? - `JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn -B -Pspark-4.0 -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite test`: full reactor succeeded; all 5 focused Arrow regression tests and 23 additional JVM unit tests passed. - Spark 4.1 focused Arrow regression suite: 5 tests passed; Spark 3.5 compatibility compilation succeeded. - Real PySpark 4.0 accelerated/fallback worker regressions, including nested structs/lists/maps, nulls, and six changing source batches of sizes `[7, 7, 7, 7, 7, 2]`. - `python3 dev/ci/check-suites.py`: all 178 Linux/macOS suite registrations passed. - Maven ScalaStyle and Spotless checks passed; `ruff check spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py` passed. - `make -n release PROFILES='-Pspark-4.0 -Pscala-2.13'` confirmed benchmark build/profile alignment. - Local wide-schema benchmark measured approximately 1.24x faster `mapInArrow` and 1.27x faster `mapInPandas` (workload-dependent). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
