dwsmith1983 commented on PR #6835: URL: https://github.com/apache/datafusion-comet/pull/6835#issuecomment-6127689659
> Would you be OK with this one landing first? #6810 has changed shape since then, and the new version is going up shortly. The native sort needed about four times the pool under `fair_unified` (I'll put the numbers on #6810), so #6810 now falls back to Spark for a NOT IN whose streamed side reports an ordering. It adds no proto field and touches nothing in the planner, so field 11 stays with this PR, and the only overlap is the guard in `CometHashJoin.doConvert`. I'd still keep #6810 first, because of the P1 above. That NOT IN reads a sorted cache, and #6810 sends exactly that case to Spark. The second cache then holds Spark's ordered rows, and the order declared here is true. If this PR lands first, it needs its own check at that cache boundary. #6810 is a small change now, so it shouldn't hold this one up for long. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
