friendlymatthew commented on code in PR #9544: URL: https://github.com/apache/arrow-rs/pull/9544#discussion_r2976100612
########## arrow-cast/src/cast/union.rs: ########## @@ -0,0 +1,470 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Cast support for union arrays. + +use crate::cast::can_cast_types; +use crate::cast_with_options; +use arrow_array::{Array, ArrayRef, UnionArray, new_null_array}; +use arrow_schema::{ArrowError, DataType, FieldRef, UnionFields}; +use arrow_select::union_extract::union_extract; + +use super::CastOptions; + +// this is used during variant selection to prefer a "close" type over a distant cast +// for example: when targeting Utf8View, a Utf8 variant is preferred over Int32 despite both being castable +fn same_type_family(a: &DataType, b: &DataType) -> bool { + use DataType::*; + matches!( + (a, b), + (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View) + | ( + Binary | LargeBinary | BinaryView, + Binary | LargeBinary | BinaryView + ) + | (Int8 | Int16 | Int32 | Int64, Int8 | Int16 | Int32 | Int64) + | ( + UInt8 | UInt16 | UInt32 | UInt64, + UInt8 | UInt16 | UInt32 | UInt64 + ) + | (Float16 | Float32 | Float64, Float16 | Float32 | Float64) + ) +} Review Comment: I wonder if this is something that would be useful as a `DataType` api... Equivalence classes let you answer "are these the same logical data" without caring about the physical repr ########## arrow-cast/src/cast/union.rs: ########## @@ -0,0 +1,470 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Cast support for union arrays. + +use crate::cast::can_cast_types; +use crate::cast_with_options; +use arrow_array::{Array, ArrayRef, UnionArray, new_null_array}; +use arrow_schema::{ArrowError, DataType, FieldRef, UnionFields}; +use arrow_select::union_extract::union_extract; + +use super::CastOptions; + +// this is used during variant selection to prefer a "close" type over a distant cast +// for example: when targeting Utf8View, a Utf8 variant is preferred over Int32 despite both being castable +fn same_type_family(a: &DataType, b: &DataType) -> bool { + use DataType::*; + matches!( + (a, b), + (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View) + | ( + Binary | LargeBinary | BinaryView, + Binary | LargeBinary | BinaryView + ) + | (Int8 | Int16 | Int32 | Int64, Int8 | Int16 | Int32 | Int64) + | ( + UInt8 | UInt16 | UInt32 | UInt64, + UInt8 | UInt16 | UInt32 | UInt64 + ) + | (Float16 | Float32 | Float64, Float16 | Float32 | Float64) + ) +} + +// variant selection heuristic — 3 passes with decreasing specificity: +// +// first pass: field type == target type +// second pass: field and target are in the same equivalence class +// (e.g., Utf8 and Utf8View are both strings) +// third pass: field can be cast to target +// note: this is the most permissive and may lose information +// also, the matching logic is greedy so it will pick the first 'castable' variant +// +// each pass picks the first matching variant by type_id order. Review Comment: the main heuristic! I think pass 2 is necessary because `can_cast_types` is permissive enough that without it, `Union(Int32, Utf8)` cast to a `Utf8View` would pick `Int32` over `Utf8`... Though we can fix this by defining the `Utf8` variant first. But I wonder if we should leave it up to the user... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
