ywskycn opened a new pull request, #24684: URL: https://github.com/apache/datafusion/pull/24684
## Which issue does this PR close? - N/A; performance follow-up for Spark map construction. ## Rationale for this change `map_from_entries` / `map_from_arrays` can do unnecessary work when deduplicating map keys: - Rows without a `LAST_WIN` overwrite still materialize values through `take()`, copying value buffers unnecessarily. - A very wide map row can leave a large retained key lookup table, making later small rows repeatedly clear oversized capacity. This adds avoidable CPU, allocation, and memory overhead for common and skewed map workloads. ## What changes are included in this PR? - Reuse the original value buffers through `filter()` when no `LAST_WIN` overwrite occurs. - Only use `take()` after an actual duplicate key overwrite changes value ordering. - Reuse normal-sized key lookup tables, but shrink capacity retained from disproportionately large rows. - Add a unit test verifying distinct-key rows reuse the original value payload buffer. ## Are these changes tested? - `cargo fmt --all --check` - Added unit coverage for value buffer reuse. - Local targeted test execution was blocked by the configured dependency mirror missing locked `blake3 1.8.7`; CI should run the full suite. ## Are there any user-facing changes? No semantic or API changes. This only reduces map construction overhead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
