zncleon opened a new issue, #16138:
URL: https://github.com/apache/iceberg/issues/16138
### Feature Request / Improvement
# Problem
The `RemoveDanglingDeleteFiles` interface is defined in `iceberg-api` but
the only
implementation is `RemoveDanglingDeletesSparkAction` (Spark-only). Flink
users have
no equivalent, which causes equality delete files to accumulate indefinitely
in
tables managed by the Flink maintenance API.
### Root cause
`RewriteDataFiles` in the Flink maintenance pipeline does not remove
dangling delete
files after compaction. There are two existing cleanup paths in the Spark
implementation:
**Path A — `ManifestFilterManager` (unreliable)**
During a `RewriteFiles` commit, `ManifestFilterManager` removes delete files
whose
sequence number is less than the global minimum data file sequence number
across the
entire table. In practice, this condition is rarely satisfied: data files
that have
already reached the target size are never rewritten, keeping their original
low
sequence numbers and permanently anchoring the global minimum below the
eq-delete
sequence numbers.
**Path B — `remove-dangling-deletes=true` (Spark-only, added in 1.8)**
`RewriteDataFilesSparkAction` offers a `remove-dangling-deletes` option that
invokes
`RemoveDanglingDeletesSparkAction` after compaction. It correctly computes
`min(data_file.sequence_number)` per partition using a Spark DataFrame join
on the
`ENTRIES` metadata table, then calls
`table.newRewrite().deleteFile(danglingDelete).commit()`.
This is the correct fix for Path A's shortcomings.
The Flink maintenance API has neither path working reliably.
### Observed impact
In a partitioned V2 table written by `IcebergSink` (equality deletes):
- After multiple rounds of `RewriteDataFiles`, data files that have already
reached
the target size are excluded from rewrite groups and retain their original
low
sequence numbers.
- These files anchor the global `minDataSequenceNumber`, preventing Path A
from
ever removing the eq-delete files.
- `removedEqualityDeleteFiles` is always `null` in every commit summary.
- Equality delete file count grows monotonically with each write, never
decreasing.
The only workaround is to temporarily set `rewriteAll(true)`, which forces
all data
files — including large ones — into rewrite groups, draining the global
minimum and
allowing Path A to fire. This is wasteful and should not be the normal
operating mode.
### Summary
Is there an existing mechanism in the Flink maintenance API that handles
this case,
or would it make sense to add `RemoveDanglingDeleteFiles` support here?
### Query engine
Flink
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
