jayshrivastava commented on issue #23436: URL: https://github.com/apache/datafusion/issues/23436#issuecomment-6279475675
Interestingly, supporting Range partitioning in Datafusion-Distributed is proving be challenging (or at least, "not seamless"). - https://github.com/datafusion-contrib/datafusion-distributed/pull/782 - https://github.com/datafusion-contrib/datafusion-distributed/issues/628#issuecomment-6278601164 - https://github.com/apache/datafusion/issues/26509 This is raising some interesting questions - If the perfect trait-abstraction existed, I imagine we would want to add a bunch of distribution-specific stuff to that trait - Range partitioning is very different from hash. The modulo-arithmetic for `Hash` let's us support it pretty transparently in DFD in a pretty elegant way. `Range` is fundamentally different with sorted split points. Distributing the data and implementing shuffles on `Range` partitioned data requires different algorithms. If we add `KeyPartitioning` into the mix, we may want a different shuffle implementation again. That's all to say - I wonder if an `Enum` actually is the right way to capture this difference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
