jlkvanloon opened a new issue, #15981:
URL: https://github.com/apache/iceberg/issues/15981
### Feature Request / Improvement
### What
Add support for committing a `rewriteManifests` operation to a named branch
via `toBranch(String branch)`.
### Why
**Manifest clustering for scan performance**
In tables with many partitions, manifest files that span multiple partitions
cannot be pruned during planning even after the manifest list has been read.
Rewriting manifests clustered by partition (`.clusterBy(DataFile::partition)`)
significantly reduces the metadata read overhead for single-partition scans.
**Write-Audit-Publish (WAP) workflow**
Our ingestion pipeline follows a WAP pattern: data is written to a branch,
verified, and then merged into `main`. Before merging, we want to compact and
cluster the branch's manifests so that `main` remains efficient after the
merge. Without branch support in `rewriteManifests`, this step is not possible
without merging unoptimised metadata first.
### Current behaviour
`RewriteManifests` extends `SnapshotUpdate`, which provides a default
`toBranch` implementation that always throws:
```java
default ThisT toBranch(String branch) {
throw new UnsupportedOperationException(
String.format(
"Cannot commit to branch %s: %s does not support branch commits",
branch, this.getClass().getName()));
}
```
`BaseRewriteManifests` does not override this method, so calling
`.toBranch(...)` throws `UnsupportedOperationException` at runtime.
There is a second, related issue: the `apply` method in
`BaseRewriteManifests` reads manifests from `base.currentSnapshot()` (always
the `main` snapshot) rather than from the snapshot parameter passed in by
`SnapshotProducer`, which represents the tip of the target branch. Even if
`toBranch` were unblocked, the rewrite would operate on the wrong snapshot.
### Expected behaviour
- `table.rewriteManifests().toBranch("branch").commit()` rewrites the
manifests of the named branch without affecting `main`.
- If the branch does not yet exist, it is created (consistent with the
behaviour of other `SnapshotUpdate` operations such as `AppendFiles`).
- Passing `null` or a tag name as the branch argument throws
`IllegalArgumentException` (same validation as the rest of the API).
### Query engine
Other
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [x] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
