dwsmith1983 commented on code in PR #7126: URL: https://github.com/apache/datafusion-comet/pull/7126#discussion_r4332342998
########## docs/source/user-guide/latest/compatibility/operators.md: ########## @@ -129,12 +129,15 @@ express falls back to Spark and records a fallback reason (see [Understanding Comet Plans](../understanding-comet-plans.md)), because Comet cannot tell whether an operator above the join relies on the order. -Where the order Spark reports might not hold, Comet declares nothing and an outer join can return -its unmatched rows out of order. The output of a nested loop join -([#6898](https://github.com/apache/datafusion-comet/issues/6898)) is one such case, and so is a -cache or checkpoint that one wrote. Comet also declares nothing when it cannot follow the -order back to its source, as with an RDD checkpoint, and over some caches written by an outer hash -join whose order does hold ([#6922](https://github.com/apache/datafusion-comet/issues/6922)). +Where Comet cannot check the order Spark reports, it declares nothing. If that order reaches the +outer join through an input from the JVM, such as an RDD checkpoint or some caches written by an +outer hash join whose own probe input was not a scan +([#6922](https://github.com/apache/datafusion-comet/issues/6922)), or through a nested +loop join, the join falls back to Spark, which keeps the rows in the order they arrive. The output +of a nested loop join can already be out of order when it reaches the join +([#6898](https://github.com/apache/datafusion-comet/issues/6898)). A join over other native +operators in the same plan does not fall back for its order, since DataFusion keeps the order it +can see there. Review Comment: Reworked in 01fdf012a. The paragraph now leads with when the join falls back, notes that the checkpoint example needs `spark.comet.convert.rdd.enabled=true`, and then gives the exception: an order that comes only from another native hash join in the same plan keeps the join native. I narrowed that from "native operators" because a nested loop join in the same plan still makes it fall back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
