sam-1112 commented on code in PR #6119:
URL: https://github.com/apache/datafusion-comet/pull/6119#discussion_r4110101814
##########
spark/src/main/scala/org/apache/spark/sql/comet/IcebergCommitExec.scala:
##########
@@ -106,6 +108,7 @@ case class IcebergCommitExec(
try {
messages.foreach(batchWrite.onDataWriterCommit)
+ IcebergCommitExec.runPreCommitHook(messages)
Review Comment:
Thanks, that fixture fits this case better. I removed `withPreCommitHook`
from `IcebergCommitExec` and switched `native acceleration: a driver commit
failure aborts all completed task files` to the same serializable overwrite as
`a commit-time failure surfaces as the same exception as Spark's own write`:
`validate-from-snapshot-id` is an older snapshot, so commit validation fails
after the native tasks have written their files. The test still checks that the
data directory matches the manifests afterwards, and it passes with
`abortAfter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
