fallintoplace opened a new pull request, #2149: URL: https://github.com/apache/iceberg-go/pull/2149
**What** - Parallelize scan-task finalization after manifest planning. **Why** - Residual evaluation, delete accounting and Parquet splitting currently run serially for every planned file. **Implementation** - Process contiguous task ranges with bounded workers and local metrics. - Keep small plans serial. Preserve task order and the first task-order error. - Share the residual evaluator cache safely. Reuse the input task slice for unsplit plans. - Add correctness checks and a scaling benchmark. **Benchmark** Finalization only, Apple M1 Pro, darwin/arm64, Go 1.26.3, `GOMAXPROCS=8`. Median of 3 runs, 300ms each. 100K files for plain/residual cases; 20K files with 32 offsets each for splitting. Columns show ms/op by worker count. | Case | 1 worker | 2 workers | 4 workers | 8 workers | | --- | ---: | ---: | ---: | ---: | | plain | 3.93 | 2.07 | 1.50 | 1.02 | | identity_residual | 20.07 | 12.53 | 11.22 | 14.16 | | split_offsets | 22.52 | 23.11 | 21.44 | 17.69 | This is worker scaling within the new implementation, not a full `PlanFiles` before/after comparison. ```sh go test ./table -run '^$' -bench '^BenchmarkFinalizePlannedTasks$' -benchmem -benchtime=300ms -count=3 -cpu=8 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
