fallintoplace opened a new pull request, #1913: URL: https://github.com/apache/iceberg-go/pull/1913
## What changed - **Classify manifest entries before taking `manifestEntries.mu`.** - Keep only the bulk appends to the shared slices inside the lock. - Reuse homogeneous batches and use exact-size buckets for mixed batches. - Preserve the existing error behavior for unknown file content. ## Why Large manifest batches were doing `DataFile()` and content checks while holding the shared mutex. This made concurrent manifest collection wait on work that does not touch shared state. ## Benchmark Command: ```text go test ./table -run '^$' -bench '^BenchmarkManifestEntryCollection/content=(data|deletes)/manifests=(8|64)/concurrency=(1|4|16)/entries=10000$' -benchtime=1s -count=3 ``` Apple M1 Pro, Go 1.26.3. Median of 3 runs, 8 manifests with 10,000 entries each: | Workload | Concurrency | Before | After | | --- | ---: | ---: | ---: | | Data | 4 | 1.952 ms/op | 1.403 ms/op | | Data | 16 | 1.959 ms/op | 1.357 ms/op | | Mixed deletes | 4 | 1.810 ms/op | 1.636 ms/op | | Mixed deletes | 16 | 1.924 ms/op | 1.772 ms/op | The 64-manifest cases stayed roughly flat at concurrency 16 while the lock covered less work. Data batches also dropped from 54 to 34 allocs/op in the 8-manifest cases. ## Tests - `go test ./table -count=1` - `go test -race ./table -run 'TestManifestEntries|TestIsDeletionVector' -count=1` - `go vet ./table` - `go test ./... -run '^$' -count=1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
