Shekharrajak commented on PR #16224:
URL: https://github.com/apache/iceberg/pull/16224#issuecomment-4639752379
Thanks @szehon-ho for checking - updated the code changes to use it in
CherryPickOperation
we improved the cherrypick path by using the new streaming
SnapshotChanges.readAddedDataFiles() API instead of the cached/materialized
addedDataFiles() list.
improvements:
- Lets cherrypick process added files lazily through CloseableIterable and
Avoids eagerly loading all added data files into memory.
- Reduces memory pressure for large snapshots or multi-snapshot changes.
- Aligns cherrypick with the new SnapshotChanges streaming design.
- Keeps existing behaviour the same while making the implementation more
scalable.
I see there is CI check failure due to some docker connectivity - it must be
fine if we re-run the CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
