neilconway opened a new pull request, #25502:
URL: https://github.com/apache/datafusion/pull/25502
## Which issue does this PR close?
- N/A
## Rationale for this change
Many UDFs used the raw size of an input array's values buffer to reserve
capacity in a builder or similar data structure. This can result in
significantly over-allocating space for sliced array inputs. We use the new
offset span helpers added in #25481.
`array_element` had a similar but slightly distinct allocation mistake: it
reserved space for every child element, but it only returns a single value per
input row.
Along the way, refactor `array_add` to use a shared array math kernel.
## What changes are included in this PR?
* Update ~31 UDFs to size buffer allocations using visible input ranges, not
raw input size
* Add unit tests checking that allocated capacity for various UDFs matches
visible input size
* Reuse the shared array-math kernel for `array_add` and consolidate test
fixtures
The following UDFs are improved:
- `array_add`
- `array_append`
- `array_compact`
- `array_concat`
- `array_element`
- `array_normalize`
- `array_pop_back`
- `array_pop_front`
- `array_prepend`
- `array_remove`
- `array_remove_all`
- `array_remove_n`
- `array_replace`
- `array_replace_all`
- `array_replace_n`
- `array_reverse` (List/LargeList inputs; ListView is more complex)
- `array_scale`
- `array_slice` (List/LargeList inputs; ListView is more complex)
- `array_subtract`
- `concat` (DataFusion)
- `concat_ws` (DataFusion)
- `initcap`
- `string_to_array`
- `translate`
- `concat` (Spark)
- `shuffle` (Spark)
- `slice` (Spark)
- `try_url_decode` (Spark)
- `unhex` (Spark)
- `url_decode` (Spark)
- `url_encode` (Spark)
## What is the testing strategy for this PR?
Existing tests pass. Added new unit tests to check that the behavior of
various UDFs doesn't change for sliced vs. unsliced inputs; also checked that
the allocated capacity for the output array reflects the visible span of the
input, not the raw input buffer size.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
