fallintoplace opened a new pull request, #2146: URL: https://github.com/apache/iceberg-go/pull/2146
**What** - Reuse bitmap position-delete indexes across scan splits. **Why** - Split tasks for one data file currently rebuild the same position hash set. Large delete sets repeat this work for every split. **Implementation** - Keep lazy delete-file reads. Cache per-target Roaring bitmaps. - Union applicable delete files once per data path and delete-file set. - Share immutable indexes across split tasks. Keep row-position tracking, `Take` filtering and DV precedence. - Keep the existing transaction delete path. **Benchmark** Apple M1 Pro, darwin/arm64, Go 1.26.3. Median of 3 runs, 300ms each, `-cpu=1`. 100,000 consecutive positions in one in-memory Parquet delete file. Includes delete decoding and index construction for all splits; excludes data-file scanning and row filtering. | Splits | Hash sets ms/op | Shared bitmap ms/op | B/op before → after | Allocs/op before → after | | --- | ---: | ---: | ---: | ---: | | 1 | 8.73 | 7.91 | 15,137,237 → 12,886,688 | 877 → 684 | | 4 | 14.16 | 7.87 | 21,965,778 → 12,790,773 | 1,653 → 707 | | 16 | 37.00 | 7.97 | 50,494,199 → 12,679,164 | 4,755 → 802 | At 16 splits: **4.6x faster setup**, **75% fewer allocated bytes**. ```sh go test ./table -run '^$' -bench '^BenchmarkPositionDeleteBitmapReuseAcrossSplits$' -benchmem -benchtime=300ms -count=3 -cpu=1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
