kosiew opened a new pull request, #24744: URL: https://github.com/apache/datafusion/pull/24744
## Which issue does this PR close? * Part of #23393 ## Rationale for this change `OneSideHashJoiner::size()` can overcount memory by adding inline fields that are already included in the joiner descriptor, while its previous `HashSet<usize>` estimate only accounted for element payload capacity and omitted hashbrown control/metadata allocation. This change defines the component-level accounting contract more explicitly so the joiner descriptor is counted once and directly owned container allocations are included using their actual allocation-aware helpers where available. ## What changes are included in this PR? * Document the `OneSideHashJoiner::size()` contract as including the joiner descriptor, directly owned container allocations, and input arrays, while excluding allocations referenced by shared `on` expressions. * Remove double counting of inline fields already covered by `size_of_val(self)`. * Account for the `on` vector's owned backing allocation using its capacity. * Adjust `PruningJoinHashMap` accounting so its descriptor is not counted twice. * Replace the `visited_rows` payload-only capacity calculation with `allocation_size()`, which accounts for the hashbrown allocation layout, including control bytes. * Add regression coverage for empty and grown `visited_rows` states and for the owned `on`, hash map, and hashes-buffer allocations. * No transformer composition or shared-record-batch accounting is changed. ## Are these changes tested? Yes. This PR adds the focused unit test: `one_side_hash_joiner_size_counts_descriptor_and_hash_set_allocation_once` The test checks the explicit expected size formula for an empty joiner, verifies `visited_rows` allocation accounting across capacity growth, and verifies that owned container allocation changes are reflected without double-counting descriptors. ## Are there any user-facing changes? No public API or query-result behavior changes are introduced. This changes internal memory-size accounting for `OneSideHashJoiner`, making its reported size more consistent with the documented ownership contract. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
