andygrove commented on PR #6940: URL: https://github.com/apache/datafusion-comet/pull/6940#issuecomment-6328269466
Thanks @viirya, @comphead and @parthchandra. The two new commits address the review: `50efcb6ec` covers the code, docs and tests, and `81f116636` fixes the job commit test. Each thread has a reply. On validation, as promised: - `ParquetWriteWithMagicCommitterSuite`, as committed, ran against MinIO on a Linux Docker host. Spark 4.1 passed all 9 tests. Spark 3.5 passed 8, and the in-memory tracking test was canceled, since Hadoop 3.3.4 does not have it. - I also ran the manual `Spark SQL Writer Tests` workflow for [3.5](https://github.com/andygrove/datafusion-comet/actions/runs/39685994837) and [4.1](https://github.com/andygrove/datafusion-comet/actions/runs/39685998046) at `50efcb6ec`. Both passed, with 377 and 429 tests, but they don't tell us much. As #6749 describes, that workflow never turns on the native writer, because it doesn't set the operator's `allowIncompatible` key. Neither run's log shows a native write. So for now the native write path is covered by the Comet suites and the MinIO run above, and fixing the workflow is #6749. @parthchandra, I added the `run-all-spark-profiles` label and filed #7150 for a CI job that runs the MinIO suites, starting with this one. Two changes go beyond the threads. Write failures now fail the task with `TASK_WRITE_FAILED`, as Spark's writer does, which applies to every native write. And the planning gate now admits subclasses of `PathOutputCommitProtocol`. The description is updated for both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
