fallintoplace opened a new pull request, #2269: URL: https://github.com/apache/iceberg-go/pull/2269
## What Append local Parquet split tasks directly to the final scan-task slice. ## Why Local scan finalization allocates a temporary task slice for each splittable file, then copies those tasks into the final result. Large scans create thousands of short-lived buffers. ## Implementation - Use existing validated row-group offsets and append split tasks directly. - Keep task ordering, split boundaries, and metadata. - Add a benchmark for split-heavy, mixed, and unsplittable ranges. ## Benchmark **Synthetic split-finalization benchmark** with prebuilt mock data files. Linux/amd64, AMD EPYC 9V74, Go 1.23.2, GOMAXPROCS=1; medians of five 200 ms runs. | Workload | Time before → after | Allocs/op before → after | Bytes/op before → after | | --- | --- | --- | --- | | 1,024 files, 4 groups, 100% split | 196 → 78 µs (-60.0%) | 1,025 → 1 | Not recorded | | 8,192 files, 4 groups, 100% split | 2.186 → 1.028 ms (-53.0%) | 8,193 → 1 | Not recorded | | 8,192 files, 32 groups, 100% split | 42.853 → 31.976 ms (-25.4%) | 8,203 → 11 | 132.19 → 94.19 MiB | | 8,192 files, 32 groups, 50% split | 23.277 → 15.010 ms (-35.5%) | 4,108 → 8 | 62.59 → 43.57 MiB | The 0%-split case remains at 0 B/op and 0 allocs/op on both versions. These measurements isolate finalization. They do not include manifest reads, file decoding, or a full scan. The PR adds a benchmark for the real finalizer: ```sh GOMAXPROCS=1 go test ./table -run '^$' -bench '^BenchmarkFinalizeTaskRangeDirectParquetSplits$' -benchmem -benchtime=200ms -count=5 ``` Copy the benchmark file unchanged to the base revision to collect comparable before numbers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
