peterxcli commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5464193377
@alamb Thanks, I think I understand the proposal better now. This is essentially an adaptive hybrid radix hash join: the plan remains Partitioned, while each output partition can dynamically transition from today’s buffered build path into bucketed spilling. One possible implementation sequence: 1. On memory pressure, hash the buffered build rows once and route them into bucket-owned RecordBatch chunks using radix bits. Use the existing memory reservation and SpillManager infrastructure rather than introducing a generic spillable hash map. 2. Destage victim buckets to disk, storing their rows and precomputed hashes. Reserve explicit headroom for concatenating the resident working set and constructing its exact JoinHashMap; block spilling alone does not solve the hash-table allocation ceiling. 3. Build one ordinary JoinHashMap over the resident working set. Initially, spill all probe rows routed to nonresident buckets. Then process each spilled build/probe pair: restore the build rows, construct another ordinary map, replay the probe rows, and delete the pair. 4. If a restored bucket still does not fit, repartition it using the next radix-bit window. For unsplittable skew such as all-equal keys, use chunked build plus probe replay. 5. Complete the join-type-specific behavior—visited bitmaps, unmatched-row emission, and null-aware semantics. 6. Add per-bucket Bloom filters afterward, only if sparse-match benchmarks show that probe spill I/O dominates. Blooms can reduce probe I/O, but they do not reduce build spilling or hash-table memory. This borrows DuckDB’s radix partitioning, bucket ownership, exact-map reconstruction, and partition-at-a-time replay, but applies them only after spilling happened. A narrow block-storage primitive might eventually be reusable by aggregation, but I would treat that as an extraction opportunity rather than making [#24704](https://github.com/apache/datafusion/issues/24704) a dependency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
