CurtHagenlocher opened a new issue, #499: URL: https://github.com/apache/arrow-dotnet/issues/499
### Describe the bug, including details regarding any error messages, version, and platform. `ShreddedVariantArrayBuilder` declares every shredded field group nullable: each field of an object's `typed_value` struct, and an array's `element` group. When an object lacks one of the shredded fields, it marks that by setting the **group itself** to null (`BuildElementGroupArray`: `if (r.IsMissing) validity.Append(false)`). The Parquet [VariantShredding spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md) has no null group. Each shredded object field is a `required group` holding `optional value` and `optional typed_value`, and so is an array's `element`. A field missing from a present object is spelled as a non-null group with **both** `value` and `typed_value` null. A Parquet writer that maps Arrow nullability to repetition therefore writes an `OPTIONAL` group with nulls at the group's definition level. Spark 4.1.2 (with delta-spark 4.3.1) refuses such a file as soon as one object row lacks a shredded field: ``` [MALFORMED_VARIANT] Variant binary is malformed. Please check the data source is valid. ``` Files where every object row has every shredded field read fine, which makes this easy to miss in tests. ### Repro `Apache.Arrow.Operations` 23.0.0. Current `main` has the same code (`BuildObjectTyped`, `BuildArrayTyped`, `BuildElementGroupArray`): ```csharp var schema = ShredSchema.ForObject(new Dictionary<string, ShredSchema> { ["a"] = ShredSchema.Primitive(ShredType.Int8), }); var values = new List<VariantValue> { VariantValue.FromObject(new Dictionary<string, VariantValue> { ["a"] = VariantValue.FromInt8(1) }), VariantValue.FromObject(new Dictionary<string, VariantValue> { ["b"] = VariantValue.FromInt8(1) }), // lacks "a" }; var (metadata, rows) = VariantShredder.Shred(values, schema); VariantArray array = ShreddedVariantArrayBuilder.Build(schema, metadata, rows); var typed = (StructArray)array.TypedValueArray; var aField = ((StructType)typed.Data.DataType).Fields[0]; Console.WriteLine(aField.IsNullable); // True -- expected False Console.WriteLine(typed.Fields[0].IsNull(1)); // True -- expected False, with value and typed_value both null ``` ### Expected - Each object field's `Field` and the list's `element` `Field` are `nullable: false`. - A missing field is a valid group slot with `value` and `typed_value` both null. The children already hold nulls there, so the only change is to stop nulling the group and stop declaring it nullable. - Any reader-side code that treats a null group as "missing" also has to accept both-children-null. A spec-conforming file from Spark or DuckDB is already shaped that way. ### Workaround [clast-project/engineered-wood](https://github.com/clast-project/engineered-wood) rewrites the builder's output: it pushes each group's nulls into its two children and drops the group's validity and nullability, recursing through nested objects and list elements. With that rewrite, Spark 4.1.2, DuckDB 2.0 and pyarrow all read the files correctly (clast-project/engineered-wood#679, clast-project/engineered-wood#683). ### Component(s) C# -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
