paveon opened a new pull request, #1133: URL: https://github.com/apache/iceberg-go/pull/1133
Fixes [#1132](https://github.com/apache/iceberg-go/issues/1132) ### Summary - Deduplicate manifest reads in `getReferencedFiles` using a two-pass approach: collect unique manifests from manifest lists (pass 1), then read entries in parallel via `errgroup.SetLimit` (pass 2) - Run S3 walk and referenced-file collection concurrently via errgroup - Fix `time.Now()` being evaluated per-file inside walk callback instead of once before the walk Reduces `getReferencedFiles` I/O from O(snapshots × manifests-per-snapshot) to O(unique manifests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
