kosiew opened a new pull request, #24257: URL: https://github.com/apache/datafusion/pull/24257
## Which issue does this PR close? * Part of #23393 ## Rationale for this change `ArrowBytesViewMap::size()` currently undercounts retained heap memory by accounting for used lengths rather than allocated capacities in some of its owned storage. In particular, it does not account for unused capacity in `views`, the backing allocation of `completed`, or unused capacity retained by completed Arrow buffers. Since `size()` is used for memory accounting, it should report the heap allocations owned by the map while continuing to exclude `self` and external input-array buffers. ## What changes are included in this PR? * Updates `ArrowBytesViewMap::size()` to account for the allocated capacity of `views`. * Accounts for the backing allocation of the `completed` vector and the retained capacity of each completed `Buffer`. * Uses allocation-based accounting for the in-progress buffer. * Clarifies that `size()` excludes both `self` and input-array buffers. * Adds `DFHeapSize` support for `u128`, allowing `Vec<u128>` capacity to be accounted for through the existing heap-size infrastructure. * Adds tests covering `u128` and `Vec<u128>` heap-size accounting and retained-capacity accounting in `ArrowBytesViewMap`. ## Are these changes tested? Yes. The patch adds: * `test_vec_u128`, which verifies that a `Vec<u128>` is accounted for using its allocated capacity. * An additional `u128` assertion in the existing zero-heap-size test. * `test_size_counts_retained_buffer_capacities`, which verifies that `ArrowBytesViewMap::size()` accounts for unused `views` capacity, the `completed` vector allocation, and retained completed-buffer capacity. It also verifies that reinserting duplicate values does not report additional owned storage. ## Are there any user-facing changes? No direct user-facing or API behavior changes. This corrects internal memory accounting for `ArrowBytesViewMap`. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
