mohitgurav20 commented on PR #25811: URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6580775624
Thanks @haohuaijin, that's a very fair point regarding network RTT on remote object stores. In practice, most queries only have 1 (or at most 2) Bloom filter columns in the predicate, so the sequential latency penalty rarely shows up. But for queries with multiple filter columns where none prune the row group, serializing `get_range` requests would definitely add latency over high-RTT storage like S3. A couple of ways we could handle this: 1. **Concurrent prefetch with lazy evaluation:** We can dispatch the async filter fetches concurrently (or batch the byte ranges if using `get_ranges`), but evaluate them in memory as they complete so downstream row groups or subsequent checks can still short-circuit. 2. **Threshold / Heuristic:** If there's only 1 filter column (or if the reader is backed by local disk), evaluate sequentially. If there are multiple filter columns on a remote `ObjectStore`, fetch candidate filters concurrently before evaluating. What do you think makes the most sense here? Happy to benchmark the concurrent prefetch approach against sequential fetches on multi-filter queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
