laskoviymishka opened a new issue, #834: URL: https://github.com/apache/iceberg-go/issues/834
### Feature Request / Improvement Parent: #829 (v2 spec completion) Bloom filter table properties are defined but never applied. The Arrow Parquet library fully supports bloom filters on both sides — we just need to wire them in. ### Write side (not wired) Properties already defined in `table/properties.go` and `table/internal/parquet_files.go`: - `write.parquet.bloom-filter-max-bytes` (default 1MB) - `write.parquet.bloom-filter-enabled.column.<name>` But `GetWriteProperties()` in `parquet_files.go` never reads them. The underlying Arrow Parquet writer supports: - `parquet.WithBloomFilterEnabled(bool)` — global default - `parquet.WithBloomFilterEnabledFor(path, bool)` — per-column - `parquet.WithMaxBloomFilterBytes(n)` — max size Just need to parse the properties and append to `writerProps` in `GetWriteProperties`. ### Read side (not implemented) Row group filtering currently uses column statistics only (`newParquetRowGroupStatsEvaluator` in `table/evaluators.go`, wired in `table/arrow_scanner.go`). After stats say "possible", a bloom filter check could eliminate additional row groups for `EqualTo` and `In` predicates. Arrow Parquet reader provides: - `file.Reader.GetBloomFilterReader()` → `BloomFilterReader` - `RowGroupBloomFilterReader.GetColumnBloomFilter(i)` → `BloomFilter` with `CheckHash(uint64) bool` Integration point: `wrapPqArrowReader.GetRecords` in `table/internal/parquet_files.go` — after the existing `testRg` stats pass, add a bloom filter pass over surviving row groups. ### Impact Bloom filters are most valuable for high-cardinality columns used in equality filters (e.g. `user_id = X`). For tables with many row groups and selective equality predicates, this can dramatically reduce I/O by skipping row groups that column stats alone can't eliminate (when min/max range is wide but the actual value is absent). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
