hcrosse opened a new pull request, #2442: URL: https://github.com/apache/iceberg-go/pull/2442
Equality-delete matching repeats the same work for rows that share a dictionary value. I added a path that matches each dictionary entry once, then checks each row's index. ## What Single string or binary keys use this path when the dictionary is no larger than the batch. The existing key matcher runs once per dictionary entry and builds a bitmap of matching entries. Each row then looks up its index in that bitmap. If no entry matches and no null can match, the batch is returned unchanged without visiting rows. Larger dictionaries and multi-column keys keep the per-row path. `makeColEncoder` and the delete-file reader are unchanged. ## Why https://github.com/apache/iceberg-go/pull/2427 made dictionary keys match correctly through the per-row encoder. Data files reach that path today when their embedded `ARROW:schema` records a dictionary type, as it does when PyIceberg writes pyarrow dictionary arrays. The scan option proposed in https://github.com/apache/iceberg-go/issues/2410 would make dictionary columns common. I compared main with this change on a Linux amd64 VM with 8 AMD EPYC vCPUs, Go 1.27.1 and arrow-go v18.8.0, over six interleaved rounds. Microbenchmarks used `-cpu=1` and a 1,000-entry dictionary: - Matching cases took 23% to 70% less time across 1,000, 100,000 and 1 million rows. Bytes allocated rose by at most 0.5%. - No-match cases took 71% to 73% less time at 1,000 rows and about 99.7% to 99.97% less at larger batches, because they skip the rows. Bytes allocated fell to 48 per operation. - In an uncommitted scan of a synthetic PyIceberg-written table with 2 million rows, the 131,072-row batches each carried the whole row group's dictionary. With a 4,096-entry key column, the scan fell from 267.6 to 222.0 ms, a 17% reduction. A 200,000-entry column is larger than the batch, so it kept the per-row path and showed no significant change. Memory use and row counts were unchanged. ## Testing The added tests cover no matches, null indices without a null entry, empty dictionaries with all-null indices, sliced batches, and a dictionary larger than the batch. Each test uses a checked allocator and asserts whether the dictionary path ran. Existing dictionary tests cover null entries and other value and index types. An uncommitted randomized comparison against the per-row path found no differences across 600 scenarios. To reproduce the microbenchmarks: `go test ./table/ -run '^$' -bench '^BenchmarkProcessEqualityDeletesSingleStringDictionary$' -benchmem -cpu=1 -benchtime=300ms -count=6`. On Linux amd64 with Go 1.27.1, `gofmt`, `make lint`, `go test ./table/...`, `go test -race ./table/` and `make test` passed. I did not test remote storage or files from other writers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
