amogh-jahagirdar commented on code in PR #13556:
URL: https://github.com/apache/iceberg/pull/13556#discussion_r2205986062
##########
core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java:
##########
@@ -281,17 +282,21 @@ private static Iterable<Long> toIds(Iterable<Snapshot>
snapshots) {
return Iterables.transform(snapshots, Snapshot::snapshotId);
}
- public static List<DataFile> newFiles(
+ public static CloseableIterable<DataFile> newFiles(
Long baseSnapshotId, long latestSnapshotId, Function<Long, Snapshot>
lookup, FileIO io) {
- List<DataFile> newFiles = Lists.newArrayList();
+ List<Snapshot> snapshots = Lists.newArrayList();
+ ParallelIterable<DataFile> newFilesIterable =
Review Comment:
Nit: Don't think we need `iterable` in the variable name, we can still
probably just keep the name `newFiles`, consumers processing it will anyways
know it's an iterable
##########
core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java:
##########
@@ -281,17 +282,21 @@ private static Iterable<Long> toIds(Iterable<Snapshot>
snapshots) {
return Iterables.transform(snapshots, Snapshot::snapshotId);
}
- public static List<DataFile> newFiles(
+ public static CloseableIterable<DataFile> newFiles(
Review Comment:
I think we should go through a deprecation cycle for this rather than
upfront changing it?
I do agree though that we should keep the API minimal and the right return
type generally for this is an iterable of some sort rather than always eagerly
collecting every Data File entry in the range into a list, because that can
still be a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
