goutamadwant commented on issue #24712: URL: https://github.com/apache/datafusion/issues/24712#issuecomment-5658949686
> Ok, sorry for the delay! > > To ensure that scaling `RangePartitioning` is exercised on `main` and improve behavior related to PR [#24600](https://github.com/apache/datafusion/pull/24600), we should integrate the feature from this ticket into `EnsureRequirements` when we land it. > > Currently, when a child operator is already partitioned with `Partitioning::Range` that satisfies `Distribution::KeyPartitioned`, `EnsureRequirements` replaces it with `Partitioning::Hash` whenever `target_partitions > child_partitions`: > > https://github.com/apache/datafusion/blob/8642154323c3800948f495af9ba2ee1539399105/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L1574-L1595 > > Because `Distribution::KeyPartitioned::create_partitioning` [unconditionally generates `Partitioning::Hash`](https://github.com/apache/datafusion/blob/8642154323c3800948f495af9ba2ee1539399105/datafusion/physical-expr/src/partitioning.rs#L766-L777), any preexisting range partitioning is destroyed before `enforce_distribution_relationships` can align child partitions. > > To fix this and exercise the new scaling logic: > > 1. In `EnsureRequirements`, check if the existing satisfying `Partitioning` supports scaling before falling back to `Distribution::create_partitioning`. > 2. When scaling `RangePartitioning` is possible, use the scaled range partitioning. > 3. When scaling is not possible, preserve the satisfying range partitioning rather than overriding it with `Partitioning::Hash`, avoiding unnecessary 2-sided hash shuffles. > > [@goutamadwant](https://github.com/goutamadwant): If you have time to do this on [#24766](https://github.com/apache/datafusion/pull/24766), that would be awesome: otherwise, I may tackle it this week. @stuhood Implemented in #24766. Can you add the api change label for the FFI/protobuf changes? Thanks. Let me know if you have any suggestions/comments. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
