jayshrivastava commented on issue #23436: URL: https://github.com/apache/datafusion/issues/23436#issuecomment-6201207256
Ah I just saw https://github.com/apache/datafusion/pull/25279. That PR implements support for hive partitioning, detecting that files are Hive partitioned and automatically giving the data source a `Range` partitioning property. That's a reasonable change but I think we should keep this issue open since it's tracking a more generalized problem. Notably **Trino supports the following**: - Can we support a non-range style of partitioning? For example, my data might be partitioned without any presumed sort order like `p1 = {A, C, E}, p2 = {B, D}`, so `Range` doesn't apply - Not assuming a file-backed data source. The data source might be a wrapper on top of an external service that exposes partitions (notably datafusion's hash function cannot be pushed down to another service / language). - Dynamic filtering works without specifying the partitioning expression / hash function* \*caveat - Filters are unioned across partitions within a machine. However, there's still a distinct filter per machine. I think it would be good to support the above for feature parity with Trino. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
