jayzhan211 commented on code in PR #25787:
URL: https://github.com/apache/datafusion/pull/25787#discussion_r4218757869
##########
datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:
##########
@@ -1210,6 +1210,19 @@ fn enforce_distribution_relationships(
}
};
+ // Reject a reference when another child in the co-partitioned group
Review Comment:
When this fires, the larger child is shuffled in both plans (it is
unsatisfied), so rejecting the reference never saves a shuffle; it only adds a
shuffle of the reference. With strict `>`, that also happens when the sizes are
close: at 10M vs 11M rows, 21M rows get shuffled instead of 11M. It also
reverses #24766's native-reference rule, and
`range_fallback_preserves_target_parallelism` now asserts the opposite of what
it was added to guard. Could we reject only when the reference is small,
reusing `JoinSelection`'s `CollectLeft` thresholds
(`hash_join_single_partition_threshold{,_rows}`)? The issue's 100-row case
would still fall back to hash, inputs of similar size would keep a single
shuffle, and #24766's test could keep its original assertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
