adriangb opened a new issue, #24347: URL: https://github.com/apache/datafusion/issues/24347
### Describe the bug `array_append`, `array_prepend`, `array_replace`, `array_replace_n` and `array_replace_all` have exactly the defect that #24341 describes for `array_slice`: `return_type` promises the input list type verbatim (inner field name, nullability and metadata included), but the kernel rebuilds the output's inner field from scratch with `Field::new_list_field(..., true)`. Promise sites: - `array_append` — [`concat.rs:107-114`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/concat.rs#L107-L114) (`Ok(array_type.clone())`) - `array_prepend` — [`concat.rs:189-196`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/concat.rs#L189-L196) - `array_replace` / `array_replace_n` / `array_replace_all` — [`replace.rs:121`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/replace.rs#L121-L123), [`:220`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/replace.rs#L220-L222), [`:329`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/replace.rs#L329-L331) (`Ok(args[0].clone())`) Payload sites: - `general_append_and_prepend` — [`concat.rs:568`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/concat.rs#L568) - `general_replace` — [`replace.rs:505`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/replace.rs#L505) - `general_replace_with_scalar` — [`replace.rs:601`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/functions-nested/src/replace.rs#L601) As in #24341, this trips the return-type assertion from #17515 on debug builds, and on release builds silently yields a batch whose inner field name/nullability disagrees with the schema the planner recorded. `array_concat` is **not** affected — it computes a fresh return type via `type_union_resolution` rather than cloning `arg_types[0]`, so promise and payload agree. I also checked `array_remove`, `array_distinct`, `array_union`, `array_intersect`, `array_sort` and `array_resize`: all fine. ### To Reproduce **Inner field name** — via the Spark `array` function, whose list field is named `element` ([`spark_array.rs`](https://github.com/apache/datafusion/blob/f17158738b74afbe71b009f197e0c84272062615/datafusion/spark/src/function/array/spark_array.rs#L32)): ```sql SELECT array_append(array(1, 2), 3); SELECT array_prepend(0, array(1, 2)); SELECT array_replace(array(1, 2, 3), 2, 9); SELECT array_replace_n(array(1, 2, 2), 2, 9, 1); SELECT array_replace_all(array(1, 2, 2), 2, 9); ``` ``` Internal error: Assertion failed: result_data_type == *expected_type: Function 'array_append' returned value of type 'List(Int64)' while the following type was promised at planning time and expected: 'List(Int64, field: 'element')'. ``` **Non-nullable inner field:** ```sql select array_append(arrow_cast(make_array(1, 2), 'List(non-null Int32)'), 3); select array_replace(arrow_cast(make_array(1, 2), 'List(non-null Int32)'), 2, 9); ``` ``` Internal error: Assertion failed: result_data_type == *expected_type: Function 'array_append' returned value of type 'List(Int64)' while the following type was promised at planning time and expected: 'List(non-null Int64)'. ``` ### Expected behavior The returned list's inner field matches the promised return type — with the caveat below. ### Additional context **The field-name symptom is a 55.0.0 regression, same origin as #24341.** I bisected it: at [`5b22857036`](https://github.com/apache/datafusion/commit/5b22857036) ("fix: avoid extraneous casts for equivalent nested types", #20945, merged 2026-06-04) all five queries above fail; at its parent `467d2c3db8` all five succeed and return `List(Int64)`. That commit is in `55.0.0-rc2` and not in `54.1.0`. Before it, signature coercion inserted a cast normalizing `List(element)` → `List(item)`, which made promise and payload agree by accident. The non-nullable symptom is **not** a regression — it reproduces identically at `467d2c3db8`. **These are not a pure copy of the #24345 fix.** For `array_slice` it is sufficient to thread the input's `FieldRef` through, because slicing can only ever remove elements. For append/prepend/replace the new element can itself be null, so the promised type is wrong at the source: ```sql select array_append(arrow_cast(make_array(1, 2), 'List(non-null Int32)'), NULL); select array_replace(arrow_cast(make_array(1, 2), 'List(non-null Int32)'), 2, NULL); ``` Both promise `List(non-null Int32)` while the result genuinely contains a NULL element. Simply cloning the input field would produce an array arrow rejects with `Non-nullable field of ListArray cannot contain nulls`. So the fix needs `return_type` (or better, `return_field_from_args`) to carry the input field's name and metadata through while widening `nullable` when the appended/replacement argument is nullable, and the kernel to use that same field. Found while reviewing #24345, which fixes the `array_slice` half of this in #24341. Worth deciding whether these should also land before 55.0.0 final, since they regressed in the same commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
