CurtHagenlocher opened a new issue, #500:
URL: https://github.com/apache/arrow-dotnet/issues/500
### Describe the bug, including details regarding any error messages,
version, and platform.
`ArrayDataConcatenator` checks only each input's `TypeId` (and buffer count)
before joining struct arrays. It then joins the children **by position** and
labels the result with the **first** input's `StructType`. Struct inputs whose
types differ are therefore joined anyway:
- **Field names differ** (`{a}` + `{b}`): joined, and the result is typed
`{a}`.
- **Field order differs** (`{a, b}` + `{b, a}`): the second input's `b`
values land under `a` and vice versa. The data is silently wrong.
- **Nullability differs** (`{a NOT NULL}` + `{a nullable}`): the result
claims `a` is non-nullable, though the second input can hold nulls there.
- **Field count differs:** `Visit(StructType)` iterates the first type's
`Fields` and calls `SelectSlicedChildren(i)` on every input. A second input
with more children has the extra ones dropped, and one with fewer would fail on
the missing child index rather than with a clear message.
From reading `src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs` on `main`
(bb2f7ed):
```csharp
public void Visit(StructType type)
{
CheckData(type, 1); // EnsureDataType(type.TypeId) + buffer count only
...
for (int i = 0; i < type.Fields.Count; i++)
children.Add(Concatenate(SelectSlicedChildren(i), _allocator));
Result = new ArrayData(type, ...); // the first input's type
}
```
The same applies to structs nested in lists, maps and other structs, and
likely to unions, whose arm follows the same pattern.
Leaf types are fine: timestamps with different units or time zones, and
decimals with different scales, are already refused by
`CheckAndResolveFixedWidthType`.
Repro sketch:
```csharp
static StructArray S(params string[] names)
{
var type = new StructType(names.Select(n => new Field(n,
Int32Type.Default, true)).ToList());
var children = names.Select((_, i) => (IArrowArray)new
Int32Array.Builder().Append(i + 1).Build()).ToArray();
return new StructArray(type, 1, children, ArrowBuffer.Empty, 0);
}
// Expected: an error, since the types differ. Actual: a {a, b} array whose
second row is a = 1, b = 2,
// though that input had a = 2, b = 1.
var joined = ArrowArrayConcatenator.Concatenate([S("a", "b"), S("b", "a")]);
```
The swapped-order case was observed through a downstream library's
concatenation, which joins struct children the same way on Apache.Arrow 23.0.0.
The upstream behaviour above is from reading the source; the snippet itself has
not been run against `main`.
A possible fix: compare the inputs' struct types field by field (name,
nullability, and child type recursively) before joining, and throw on a
mismatch, as is already done for timestamp units and decimal scales.
### Component(s)
C#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
