timsaucer opened a new issue, #24341: URL: https://github.com/apache/datafusion/issues/24341
### Describe the bug `ArraySlice::return_type` promises `arg_types[0].clone()`, i.e. the input list type verbatim — inner field name, nullability and metadata included. But the kernel rebuilds the output's inner field from scratch in `general_array_slice` ([`extract.rs:685`](https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/extract.rs#L685)): ```rust Ok(Arc::new(GenericListArray::<O>::try_new( Arc::new(Field::new_list_field(array.value_type(), true)), ... ``` That drops the field name (any name → `item`), drops inner metadata, and forces `nullable: true`. Whenever the input list's inner field is anything other than `Field("item", T, nullable: true)`, the returned array disagrees with the promised type. On debug builds this trips the return-type assertion added in #17515; on release builds it silently produces a batch whose field name/nullability disagrees with the schema the planner recorded. This is the same defect fixed for `array_sort` in #19948 (issue #19947). The list-view path in this very function already handles it correctly — `general_list_view_array_slice` threads the input field through with `ListView(field) | LargeListView(field) => Arc::clone(field)` ([`extract.rs:705`](https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/extract.rs#L705)) — so the two paths inside `array_slice` disagree. ### To Reproduce Non-nullable inner field: ```sql select array_slice(arrow_cast(make_array(1, 3, 5, -5), 'List(non-null Int32)'), 2, 3); ``` ``` DataFusion error: Internal error: Assertion failed: result_data_type == *expected_type: Function 'array_slice' returned value of type 'List(Int32)' while the following type was promised at planning time and expected: 'List(non-null Int32)'. ``` Inner field name, via the Spark `slice` function (whose `SparkSlice::return_field_from_args` also copies the input field and delegates to `array_slice_udf()`), where `datafusion-spark`'s `array` names its list field `element`: ``` slice(array(1, 2, 3, 4), 2, 2) ``` ``` Internal error: Assertion failed: result_data_type == *expected_type: Function 'array_slice' returned value of type 'List(Int64)' while the following type was promised at planning time and expected: 'List(Int64, field: 'element')'. ``` Both reproduce on `55.0.0-rc2` (`209fd9406`). ### Expected behavior `array_slice` returns a list whose inner field matches the promised return type — the input list's field carried through unchanged, as `array_sort` does since #19948 and as the list-view path in `array_slice` already does. For the Spark `slice` case that also means the Spark element name `element` survives the slice, matching pyspark. ### Additional context Not a regression in `array_slice` itself — the kernel has always rebuilt the field. What changed is that #20945 ("avoid extraneous casts for equivalent nested types") stopped normalizing list arg types during signature coercion, so field-name differences now reach the function instead of being erased by an inserted cast. Before #20945, `slice(array(...), ...)` had a cast rewriting `List(element)` to `List(item)`, which made promise and payload agree by accident. The non-nullable case (`List(non-null Int32)`) fails regardless of #20945. Fix looks like #19948: thread the input list's `FieldRef` into `GenericListArray::try_new` instead of constructing `Field::new_list_field(array.value_type(), true)`. That likely also addresses the inner-metadata loss described in #21982 for this function. Would be good to land before 55.0.0 final — noticed while updating [datafusion-python](https://github.com/apache/datafusion-python) to `55.0.0-rc2`, where the Spark `slice` case forced a local workaround. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
