regadas opened a new issue, #18996:
URL: https://github.com/apache/iceberg/issues/18996

   ### Feature Request / Improvement
   
   `Snapshot.addedDataFiles(FileIO)`, `removedDataFiles(FileIO)`, 
`addedDeleteFiles(FileIO)`, and `removedDeleteFiles(FileIO)` were deprecated in 
#15241 in favor of `SnapshotChanges`. However, `SnapshotChanges` eagerly 
materializes both added and removed file lists in memory with full column 
statistics and requires a `Table` instance via 
`SnapshotChanges.builderFor(Table)`.
   
   Services that inspect newly committed snapshots from `TableMetadata` / 
`Snapshot` (such as catalog commit hooks, audit, or lineage listeners) often 
need to stream only `ADDED` or `DELETED` files with column projection (e.g. 
`partition` or `file_path` only) without decoding column-stats maps or 
materializing opposite-status entries.
   
   `ManifestFiles.read(manifest, io, specsById)` and 
`ManifestReader.select(...)` are already public, but 
`ManifestReader.iterator()` only exposes `liveEntries()` (`ADDED` + 
`EXISTING`), while `ManifestReader.entries()` and `ManifestEntry` are 
package-private. As a result, external callers cannot filter rewritten 
manifests containing mixed `ADDED`, `EXISTING`, and `DELETED` entries without 
split-package shims in `org.apache.iceberg` (and in-core 
`CherryPickOperation.validateReplacedPartitions` similarly hand-rolls this 
filter).
   
   Exposing `addedFiles()` and `deletedFiles()` returning 
`CloseableIterable<F>` on `ManifestReader<F>` (with defensive copies honoring 
`dropStats(columns)`, matching `iterator()`) allows streaming status-filtered 
manifest reads with column projection while keeping `ManifestEntry` 
package-private.
   
   ### Query engine
   
   Other
   
   ### Willingness to contribute
   
   - [x] I can contribute this improvement/feature independently
   - [ ] I would be willing to contribute this improvement/feature with 
guidance from the Iceberg community
   - [ ] I cannot contribute this improvement/feature at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to