jverhoeks commented on issue #2376: URL: https://github.com/apache/iceberg-rust/issues/2376#issuecomment-4337066868
Closing the loop on this issue with hard data and a sharper API ask. **What we shipped (downstream fork):** a `DynamicPredicate` trait in `iceberg::expr` plus `TableScanBuilder::with_dynamic_predicate(...)` that wires runtime filters into the reader's existing row-group / page-index / row-filter pruning paths. Sample once per `FileScanTask`, intersect with the static predicate, no other reader changes. Net **-12.4% on TPC-H SF10 (22/22 match, 20.3 s saved)** in the iceberg-datafusion integration. **What we couldn't fix without upstream support:** caching the *bound* predicate across tasks. Five different cache designs (Mutex+ptr_eq, OnceLock, IN-list cap=4096, IN-list cap=200, per-filter `Mutex<CachedBound>`) all regressed because of one unsolvable problem: in multi-join scans, `DynamicFilterPhysicalExpr` instances seal at staggered times, and there's no signal we can cheaply observe to know when the inner expression is finalized. Every cache either (a) populates with a partial predicate, (b) never hits because `current()`'s `remap_children` rebuilds the inner Arc, or (c) pays Mutex contention on every per-task call. **Microbench data** (committed in our fork, runnable upstream): `Predicate::bind(schema)` is O(N) at **61 ms for N=580K** on a hash-join probe-set runtime filter, dominating the 19 ms conversion cost. With ~88 lineitem files at SF10 and parallelism 12, that's ~560 ms wall-clock per scan that we cannot amortize without an upstream signal. **Concrete API ask:** an `is_sealed()` (or monotonic `generation()` counter) on `DynamicFilterPhysicalExpr` so consumers can populate a cache exactly once after the build side completes. With that, the bound-predicate cache becomes lock-free read-only after the first sealed observation. We're happy to draft a PR if there's interest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
