jayzhan211 commented on code in PR #26793:
URL: https://github.com/apache/datafusion/pull/26793#discussion_r4442789258
##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -3217,28 +3241,39 @@ fn build_in_memory(
// Arc is used instead of Box to allow sharing with
SharedBuildAccumulator for hash map pushdown
let mut hashmap = new_join_hashmap(num_rows, &mut reservation,
metrics)?;
+ // Every slice shares everything beneath a dictionary or list view,
+ // at any depth of a key, which `create_hashes` hashes on each call: a
+ // slice at least as long as the longest of those arrays keeps that
+ // cost linear in the build rows. No slice is longer than the build
side.
+ let chunk_rows = left_values
+ .iter()
+ .map(|values| unsliced_hash_len(&values.to_data()))
+ .fold(max_batch_rows, usize::max)
+ .min(num_rows);
Review Comment:
@dwsmith1983 thanks for the update. When each batch has its own dictionary,
every slice re-hashes the whole merged dictionary, so dictionary keys build
slower than on base: 5.2 → 7.3 ms for 256 batches × 8192 rows with 32 values
per batch. Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
