adriangb opened a new issue, #24708: URL: https://github.com/apache/datafusion/issues/24708
Follow-up from #24526. Today an `IN` list longer than `datafusion.execution.parquet.max_in_list_size` falls through to the unhandled-predicate hook, so it contributes no container pruning at all. (Other predicates and literal/containment pruning, including Bloom filters, still apply.) A much weaker form is available at close to no cost: collapse the list to its extrema and test interval overlap, roughly `vmin <= col_max AND col_min <= vmax`. Constant-size predicate, a couple of comparisons per container, and finding the extrema is one O(N) pass with no sort. It also wouldn't need a custom `PhysicalExpr` or the string-only restriction, so it could apply to any orderable type. The tradeoff is that it can only rule out containers lying entirely outside the hull — it can never prune a gap, which is exactly the precision the compact form exists to preserve. How much it's worth probably depends on how clustered the requested values are relative to the data's ordering: for a set spread across the whole domain it prunes nothing, while for clustered or time-correlated identifiers (ULIDs came up in review) it may capture much of the available win. Open questions: - Is this worth having as a tier below the cap's current give-up behaviour? - If so, what policy? It changes the current eligibility contract, and should presumably still respect `max_in_list_size = 0` as a full opt-out. - Does it also make sense as the fallback for types a compact path doesn't cover? It doesn't help `NOT IN` — an overlapping interval says nothing about whether every row is excluded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
