fallintoplace opened a new pull request, #2253: URL: https://github.com/apache/iceberg-go/pull/2253
## What Append remote Parquet split tasks to one result slice across files. ## Why The planner allocated and copied a temporary slice for each splittable file. ## Implementation - Validate offsets before appending output. - Reuse one result slice and preserve task order and metadata. - Keep the single-file split API and unchanged-batch path. ## Benchmark Synthetic split-planner benchmark with prebuilt inputs. Median of three 220 ms runs on Linux/amd64, Go 1.23.2, AMD EPYC 9V74, with `GOMAXPROCS=1`. These are helper-level results, not whole-scan throughput. | Files | Row groups | Splittable | Time before → after | Speedup | B/op before → after | allocs/op before → after | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 1,024 | 4 | 100% | 287.7 → 187.3 µs | 1.54x | 1.23 → 0.64 MB | 1,027 → 3 | | 1,024 | 32 | 100% | 2.288 → 1.519 ms | 1.51x | 8.32 → 5.57 MB | 1,033 → 9 | | 1,024 | 128 | 100% | 9.114 → 6.742 ms | 1.35x | 31.29 → 21.60 MB | 1,038 → 14 | | 8,192 | 4 | 100% | 3.734 → 2.326 ms | 1.61x | 11.37 → 6.65 MB | 8,196 → 4 | | 8,192 | 32 | 100% | 21.254 → 12.833 ms | 1.66x | 61.94 → 39.92 MB | 8,202 → 10 | | 8,192 | 128 | 100% | 88.931 → 81.376 ms | 1.09x | 290.20 → 212.93 MB | 8,209 → 17 | | 8,192 | 32 | 50% | 11.732 → 6.969 ms | 1.68x | 34.60 → 23.58 MB | 4,108 → 8 | | 8,192 | 32 | 0% | 94.9 → 90.7 µs | 1.05x | 0 → 0 | 0 → 0 | The PR adds `BenchmarkSplitRemoteScanTasksBatch` with Iceberg `DataFile` inputs: ```sh GOMAXPROCS=1 go test ./table -run '^$' -bench '^BenchmarkSplitRemoteScanTasksBatch$' -benchmem -benchtime=350ms -count=5 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
