fallintoplace opened a new pull request, #2364: URL: https://github.com/apache/iceberg-go/pull/2364
## What Emit partition metadata in batches of up to 4,096 rows. ## Why A table with many partitions can create a very large Arrow batch. ## Implementation - Encode batches as the reader advances. - Keep sorted aggregate pointers instead of copying each aggregate. - Return encoding and cancellation errors through the reader. ## Benchmark Encoding only, 65,536 partitions, Apple M1 Pro, Go 1.26.3, median of 3 runs: - **Largest batch:** 65,536 → 4,096 rows (**16x smaller**). - **Time:** 50.48 ms → 51.19 ms. - **Allocated:** 65.70 MB → 66.90 MB per operation. Aggregate discovery remains eager. This bounds Arrow batch size, not total memory. `GOMAXPROCS=1 go test ./table -run '^$' -bench '^BenchmarkInspectPartitionBatchEncoding$' -benchmem -benchtime=200ms -count=3 -cpu=1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
