viirya commented on code in PR #21833:
URL: https://github.com/apache/datafusion/pull/21833#discussion_r3172496633
##########
datafusion/physical-plan/src/joins/nested_loop_join.rs:
##########
@@ -654,13 +656,8 @@ impl ExecutionPlan for NestedLoopJoinExec {
let probe_side_data = self.right.execute(partition,
Arc::clone(&context))?;
// Determine if OOM fallback to memory-limited mode is possible.
- // Conditions:
- // 1. Disk manager supports temp files (needed for right-side spilling)
- // 2. Join type does not require tracking right-side matched state
- // across multiple left chunks (RIGHT/FULL joins not yet supported)
- let spill_state = if
context.runtime_env().disk_manager.tmp_files_enabled()
- && !need_produce_right_in_final(self.join_type)
- {
+ // Condition: disk manager supports temp files (needed for spilling)
+ let spill_state = if
context.runtime_env().disk_manager.tmp_files_enabled() {
Review Comment:
Good catch — you're right that the fallback path doesn't yet coordinate
left-bitmap state across right partitions. I've added a guard that disables
fallback for FULL JOIN when right_partition_count > 1, falling back to standard
OOM behavior in that case. This preserves correctness for the case you flagged.
A note: I noticed the same pre-existing concern applies to LEFT/LEFT
SEMI/LEFT ANTI/LEFT MARK in the multi-partition fallback path, which was
introduced in Phase 1 of this work (#21448) — not new to this PR. We should fix
it. Since it requires a larger refactor (sharing the per-chunk JoinLeftData and
probe-thread counter across partitions), I'll address it in a follow-up PR
rather than expanding this one's scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
