viirya commented on code in PR #25693:
URL: https://github.com/apache/datafusion/pull/25693#discussion_r4197974260
##########
datafusion/physical-plan/src/joins/utils.rs:
##########
@@ -1391,12 +1391,19 @@ pub(crate) fn build_null_aware_left_mark_column(
probe_side_has_null: bool,
probe_side_non_empty: bool,
) -> ArrayRef {
+ let build_key_nulls = build_key_column.logical_nulls();
Review Comment:
Thanks, confirmed. Slicing the outer dictionary retains its full child
dictionaries, so chunk-local logical_nulls still rescanned nested values. Fixed
in 3bc2308ef by computing build-key logical nulls once during build collection,
retaining only a non-empty bitmap, accounting for newly materialized buffer
memory, and indexing the cached bitmap by the original build row during final
emission. The 524,288-row nested-dictionary diagnostic dropped from about 561ms
to 177ms locally. I also added nested-dictionary correctness coverage across
five batch sizes; all 109 null-aware tests and the complete 2,409-test
datafusion-physical-plan all-features suite pass.
##########
datafusion/physical-plan/src/joins/hash_join/stream.rs:
##########
@@ -1251,11 +1251,7 @@ fn null_aware_skip_probe_batch(
// `on[0]` is the `NOT IN` value key for both modes.
let probe_key_column = &state.values[0];
let is_anti = matches!(mode, NullAwareMode::LeftAnti { .. });
- let probe_has_null = if is_anti {
- probe_key_column.logical_null_count() > 0
- } else {
- probe_key_column.null_count() > 0
- };
+ let probe_has_null = probe_key_column.logical_null_count() > 0;
Review Comment:
Thanks, confirmed. Fixed in 3bc2308ef by computing and recording the probe
logical-null summary only when state.offset is the initial (0, None). The
shared NULL-hint check for LeftAnti remains outside that guard, so another
partition can still stop resumed lookup early. The 524,288-row oversized-probe
diagnostic dropped from about 1.29s to 910ms with identical result assertions.
All 109 null-aware tests and the complete datafusion-physical-plan all-features
suite pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
