unikdahal opened a new issue, #6240:
URL: https://github.com/apache/datafusion-comet/issues/6240
### What is the problem the feature request solves?
Comet has native Iceberg scans and native Iceberg data-file writes, but
Iceberg merge-on-read `DELETE`, `UPDATE`, and `MERGE` still fall back at the
row-level write path tracked in #5122.
For Iceberg MOR, `MergeRowsExec` produces rows carrying the operation
together with Iceberg metadata such as `_file`, `_pos`, `_spec_id`, and
`_partition`. These eventually reach Iceberg's `WriteDeltaExec` /
position-delta writer, which currently executes on the JVM.
This leaves the final executor-side portion of an otherwise native row-level
operation outside Comet.
### Describe the potential solution
Add a Comet Iceberg delta-write path corresponding to Iceberg's
`WriteDeltaExec`:
```text
MergeRowsExec / CometMergeRowsExec
|
| operation + row metadata
v
CometIcebergDeltaWriteExec
|
+-- data rows -----> iceberg-rust data writer
|
+-- delete rows ---> iceberg-rust position-delete writer
|
v
Iceberg WriterCommitMessage
|
v
existing JVM-side Iceberg BatchWrite commit
```
The native operator would:
- consume the row-operation output from `MergeRowsExec`
- write new/replacement data files using the existing Comet Iceberg write
infrastructure
- write V2 position-delete files using `_file` + `_pos`
- preserve `_spec_id` / `_partition` for partition and spec evolution
- support Iceberg's partition/file delete granularity
- return the data/delete file metadata expected by Iceberg's existing commit
path
Commit planning, snapshot validation and the final table commit should
remain with Iceberg/Spark rather than being reimplemented in Comet.
### iceberg-rust dependency
This requires some reusable primitives in iceberg-rust that are currently
missing/incomplete for the Comet path, particularly:
- position-delete writing for unordered executor input
- loading existing file-scoped position deletes when a delta write needs to
replace/carry them forward
- producing the required Iceberg `DataFile` metadata for delete files
These should be implemented in iceberg-rust and consumed by Comet rather
than duplicated inside the Comet writer.
### Additional context
### Related
- \#5122 — row-level `MERGE` / `UPDATE` / `DELETE` acceleration
- \#5121 — DataSource V2/native Iceberg writes
- \#5649 — production-quality native Iceberg writes
- \#3756 — Iceberg feature matrix
This issue would specifically track the **Iceberg merge-on-read delta-write
portion** of #5122.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
