viirya opened a new pull request, #25516:
URL: https://github.com/apache/datafusion/pull/25516
## Which issue does this PR close?
N/A
## Rationale for this change
Spark map functions accept `LargeList` inputs whose offsets are represented
as
`i64`. The map construction path converts these offsets to `i32`, but the
existing unchecked cast silently wraps oversized values.
For example, an offset of `2147483648` becomes `-2147483648`. Downstream code
can then interpret the wrapped value as an invalid child-array index instead
of
reporting the unsupported input.
## What changes are included in this PR?
- Replace the unchecked `i64` to `i32` conversion with a checked conversion.
- Return an actionable execution error when an offset cannot be represented
as
`i32`.
- Add a regression test using a sliced `LargeList` whose first offset exceeds
`i32::MAX`.
## What is the testing strategy for this PR?
Added `large_list_offsets_do_not_truncate`.
Ablation testing confirmed that the test fails with the old implementation,
which returns the truncated offsets `[-2147483648, -2147483647]`, and passes
with the checked conversion.
Also verified:
- `cargo fmt --all -- --check`
- `git diff --check`
The full test suite is covered by CI.
## Are there any user-facing changes?
Yes. Oversized `LargeList` offsets now produce a clear execution error
instead
of being silently truncated. There are no API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
