jordepic opened a new issue, #5618: URL: https://github.com/apache/datafusion-comet/issues/5618
## What When the native Iceberg write path (#5361) fails partway through a task, data files already finalized by that task attempt are left in the table's data location. They are invisible to readers (files resolve only through committed manifests) and are reclaimed by `remove_orphan_files`, but iceberg-java cleans them up synchronously: `SparkWrite`'s writers run inside `Utils.tryWithSafeFinallyAndFailureCallbacks`, and `DataWriter.abort()` calls `SparkCleanupUtil.deleteTaskFiles(io, ...)`. On spot-heavy or preemption-prone clusters the orphan cost accumulates silently on the native path. This gap is documented in the "Failure handling" section of `iceberg-writes.md`; this issue tracks closing it. ## Suggested shape `TaskContext.addTaskFailureListener` in `CometIcebergWriteExec.doExecute`, mirroring what iceberg-java's writer abort does: - For failures after the native writer has returned (manifest decode, JVM metrics rebuild, `TaskCommit` construction), the JVM already holds the decoded `DataFile`s — delete their paths through the table `FileIO` that is already in the task closure. - For failures inside the native write itself, the JVM never learns the written paths. Closing that half needs the native operator to report the paths of finalized files alongside the error (or a JNI call to fetch them from the failed writer), since iceberg-rust's writers have no abort/Drop cleanup of their own. Cleanup must be best-effort and must not mask the original task failure. ## Context - PR: #5361 (third of three for #4322) - The failure-handling documentation and this trade-off were discussed in review: https://github.com/apache/datafusion-comet/pull/5361#pullrequestreview-5080241910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
