adriangb commented on PR #21931: URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4364567871
Yes agreed. My thought is that this a good default for now that is less brittle to things messing with partitioning. And down the line once we've got a good story for range partitioning we can use https://github.com/apache/datafusion/pull/21900 for hash partitioned cases (assuming performance looks good) and some other system for range partitioning. @Dandandan I looked into https://github.com/apache/datafusion/pull/21900 but immediately realized the point is that would make the partition routing faster, but part of the goal here is to not have any partition routing because it introduces brittleness and is slower than just probing more hash tables (although https://github.com/apache/datafusion/pull/21900 could flip the performance story, we'd have to confirm). In any case - we can always merge this now and follow up with a version of `MultiMapLookupExpr` that uses https://github.com/apache/datafusion/pull/21900 if perf looks good? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
