sunchao commented on code in PR #25693:
URL: https://github.com/apache/datafusion/pull/25693#discussion_r4197013550
##########
datafusion/physical-plan/src/joins/hash_join/stream.rs:
##########
@@ -1251,11 +1251,7 @@ fn null_aware_skip_probe_batch(
// `on[0]` is the `NOT IN` value key for both modes.
let probe_key_column = &state.values[0];
let is_anti = matches!(mode, NullAwareMode::LeftAnti { .. });
- let probe_has_null = if is_anti {
- probe_key_column.logical_null_count() > 0
- } else {
- probe_key_column.null_count() > 0
- };
+ let probe_has_null = probe_key_column.logical_null_count() > 0;
Review Comment:
[P2] Record probe logical nulls once per input batch
`process_probe_batch` calls this on every resumed lookup chunk, not just
when a probe batch is first fetched. With dictionary values containing a NULL,
Arrow 60's `logical_null_count()` walks every key in the whole probe batch,
even if that NULL dictionary entry is unused. This change therefore adds
another full probe-key scan for every lookup chunk.
I reproduced the slowdown with one build value 1 and one native probe batch
of 524,288 dictionary keys all 0, dictionary values [1, NULL], and join batch
size 8192. Both pre-PR base 2306a4b7 and head 4d50daac return exactly one TRUE
mark. Median unoptimized operator time over three runs increased from 1.610s to
2.348s; the plain Int32 control stayed about 0.057s. Compiler, locked Arrow 60
dependencies, and fixture were identical, with separate target directories.
This oversized-batch case is supported through native/custom execution plans;
normal DataSourceExec scans split batches, so these are not ordinary SQL
performance claims.
Please compute and record the probe summary only on the first chunk
(`state.offset == (0, None)`), as the adjacent correlated bookkeeping already
does. Keep the anti-join's shared NULL-hint check outside that guard so another
partition can still trigger early exit between chunks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
