sunchao commented on PR #5368: URL: https://github.com/apache/datafusion-comet/pull/5368#issuecomment-5456745987
Thanks for the detailed review and the additional probes. Addressed the follow-ups in 1de4491c. I removed the ineffective Scala end-to-end test. CometMapInBatchSuite now keeps its five plan-rule checks, while actual worker execution is covered by test_pyarrow_udf.py, which exercises the accelerated and fallback plans and checks their results. All 83 Python cases passed locally on Spark 4.0.4, including source-batch turnover and chained UDFs. The three inline requests are addressed as described in their threads. I also added your isolated JVM measurements to the PR description with attribution, explicitly separated from the earlier Comet-versus-Spark end-to-end numbers and marked as not independently reproduced in this update. Validation: 12 focused JVM tests passed (7 Arrow serialization tests and 5 plan-rule tests), the Spark 4.0 Maven reactor packaged successfully, ScalaStyle and Spotless passed, and all 178 suite-registration checks passed. The FFI tests also passed with no pre-existing Maven temporary directory. The local Spark 4.1 rerun was blocked by dependency-mirror timeouts before the Spark-module compilation/tests; the existing CI matrix covers both Spark profiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
