alamb commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5541561672
Sorry for the delay @jayzhan211 -- I have been out this week at VLDB. I view this project (spilling hash join) as an important strategic asset for DataFusion (in that it unlocks more users / use cases). Along with the work with @Dandandan in https://github.com/apache/datafusion/pull/24456 to get a real JoinGraph API in place, I think this would address long standing gaps in DataFusion's core. I will try and comment on your most recent post, but I will admit I am finding it hard to follow all the details. I think this is the current plan: > Proposed next steps (one reviewable PR each) But I am not familiar enough with the current code details to be able to evaluate it clearly. What would help me (not required), would be a higher level description (ideally with diagrams) of what our current hash join implementation does (e.g. show inputs, how partitions are related, CollectLeft, etc) , and then the summary (at the same high level) of what the proposed implementaiton would look like I get the sense there may be multiple intermediate "milestones" in the sense of new behavior visible to users that were not visible before that might be good to spell out So something like * Milestone 1: (no user visible behavior) refactoring / infrastructure preparation * Milestone 2: Partitioned spilling (Partitioned joins that used to generate OOM will now complete) * Milestone 3: CollectLeft spilling (joins marked with CollectLeft will now complete) I am not quite sure if there are interemediate steps too (e.g. some state where a PartitionedJoin might spill the build side, but still require that the probe side all fit in memory?) I will try and provide more detail feedback about your most recent response probably tomorrow. I am out of time for today sadly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
