Xuanwo commented on PR #23738: URL: https://github.com/apache/datafusion/pull/23738#issuecomment-5088954478
Thanks @2010YOUY01, this is really helpful! > An alternative is the broadcast-based ASOF join, described below. I tend to think it's better to implement the broadcast-based algorithm in the first version, here are the reasons: I initially chose the repartitioned merge to keep the join state bounded, but your point about equality-free joins collapsing to a single partition is convincing. Let’s start with the broadcast-based ASOF join. We can keep the repartition-based algorithm as a follow-up for large right inputs or well-distributed equality keys, then use benchmarks to guide strategy selection. > Question: Are the current split PRs independently mergeable? 🤔 In other words, would merging the first one leave the main branch in a consistent and functional state, or are the PRs split primarily to make the implementation easier to read and review? The stack PRs are intended to be mergeable in dependency order, with each step leaving `main` consistent. I’ll rework the physical PR first, update the dependent PRs afterward, and add the top-level design comment you suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
