sunchao commented on code in PR #25693:
URL: https://github.com/apache/datafusion/pull/25693#discussion_r4197011498
##########
datafusion/physical-plan/src/joins/utils.rs:
##########
@@ -1391,12 +1391,19 @@ pub(crate) fn build_null_aware_left_mark_column(
probe_side_has_null: bool,
probe_side_non_empty: bool,
) -> ArrayRef {
+ let build_key_nulls = build_key_column.logical_nulls();
Review Comment:
Thanks, @viirya. I confirmed that 4d50daac fixes the original
flat-dictionary case: the same 524,288-row diagnostic improved from a median
1.152s on d3ccef05 to 0.209s on the new head.
One part of this P2 remains for nested dictionary build keys at [the new
slice/logical_nulls
call](https://github.com/apache/datafusion/blob/4d50daacc6373e83c85dec742c1713cb9e339c45/datafusion/physical-plan/src/joins/hash_join/stream.rs#L1355).
Slicing a DictionaryArray only slices its keys and retains all its values.
When those values are themselves a dictionary, Arrow 60 recursively
materializes the entire inner dictionary's logical-null bitmap on every output
chunk. The outer slice therefore does not bound that inner scan.
I verified a nonempty nested-dictionary join executes and produces the
expected TRUE/NULL/FALSE marks. To isolate final emission, I then used an empty
probe and Dictionary<Int32, Dictionary<Int32, Int32>> build keys: outer keys
0..N, N inner keys all 0, leaf values [1, NULL, 4], batch size 8192. Both
pre-PR base 2306a4b7 and current head produce exactly N non-null FALSE marks.
At 524,288 rows, medians were 0.234s on base versus 0.954s on head; at
1,048,576 rows, 0.471s versus 3.307s. These are three-run unoptimized
native-operator diagnostics with identical compiler/locked dependencies and
separate target directories, not release SQL benchmarks.
Could we compute logical nulls once on entry to final emission and slice
that bitmap for each chunk, accounting for any retained allocation? That avoids
repeatedly traversing nested dictionary values as well as the flat keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
