CurtHagenlocher opened a new issue, #398:
URL: https://github.com/apache/arrow-dotnet/issues/398
### Describe the enhancement requested
Every entry point in `Apache.Arrow.Operations.Shredding` works on values,
never on a column with validity, so there is no way to shred a nullable variant
column:
```csharp
ShredSchema ShredSchemaInferer.Infer(IEnumerable<VariantValue> values,
ShredOptions options = null)
(byte[], IReadOnlyList<ShredResult>)
VariantShredder.Shred(IEnumerable<VariantValue> values, ShredSchema schema)
VariantArray ShreddedVariantArrayBuilder.Build(ShredSchema schema, byte[]
metadata, IReadOnlyList<ShredResult> rows, MemoryAllocator allocator = null)
```
`Build` produces a `VariantArray` whose storage struct carries no validity
buffer, so every row is valid. A caller shredding a column that contains SQL
NULLs has to pass a placeholder value for each null row and then repair the
result afterwards.
This matters because the two things are not interchangeable. The parquet
VariantShredding spec gives each encoding a distinct meaning: a SQL-NULL row is
the optional group itself being null (Arrow: the storage struct's validity),
while a present value holding a variant JSON null is `value` = basic type 0 /
physical type 0. Collapsing them changes what `IS NULL` means for every
consumer of the column, and the round trip stops being lossless.
### Repro
`Apache.Arrow.Operations` 23.0.0 (the only published version), .NET 8:
```csharp
using Apache.Arrow;
using Apache.Arrow.Operations.Shredding;
using Apache.Arrow.Operations.VariantJson;
using Apache.Arrow.Scalars.Variant;
static VariantValue Obj(int a) => VariantValue.FromObject(
new Dictionary<string, VariantValue> { ["a"] = VariantValue.FromInt32(a)
});
// Three rows whose MIDDLE row is meant to be SQL NULL. Nothing in the
pipeline takes a mask,
// so the best a caller can do is put a placeholder there.
var values = new List<VariantValue> { Obj(1), VariantValue.Null, Obj(3) };
var schema = new ShredSchemaInferer().Infer(values, ShredOptions.Default);
var (metadata, rows) = VariantShredder.Shred(values, schema);
VariantArray array = ShreddedVariantArrayBuilder.Build(schema, metadata,
rows);
Console.WriteLine($"storage NullCount = {array.StorageArray.NullCount}");
for (int i = 0; i < array.Length; i++)
Console.WriteLine($" row {i}: IsNull={array.IsNull(i),-5}
logical={VariantJsonWriter.ToJson(array.GetLogicalVariantValue(i), false)}");
```
```
storage NullCount = 0
row 0: IsNull=False logical={"a":1}
row 1: IsNull=False logical=null <-- wanted a NULL ROW, got a present
JSON null
row 2: IsNull=False logical={"a":3}
```
### Current workaround
Rebuild the storage struct with a validity bitmap and re-wrap it, which
reaches past the public shredding API into `ArrayData`:
```csharp
var storage = array.StorageArray.Data;
var validity = new ArrowBuffer.BitmapBuilder(array.Length);
validity.Append(true); validity.Append(false); validity.Append(true);
var patched = new VariantArray(array.VariantType,
ArrowArrayFactory.BuildArray(
new ArrayData(storage.DataType, storage.Length, nullCount: 1,
storage.Offset,
new[] { validity.Build() }, storage.Children,
storage.Dictionary)));
// row 1: IsNull=True
```
Two sharp edges in it: the bitmap is built from bit 0 while the `ArrayData`
keeps `storage.Offset`, so it is only correct for an unsliced array; and the
placeholder still travels through `VariantShredder.Shred`, so it has to be a
value the shredder accepts for the inferred schema.
### Suggested API
An overload that carries validity through, e.g.
```csharp
VariantArray ShreddedVariantArrayBuilder.Build(
ShredSchema schema, byte[] metadata, IReadOnlyList<ShredResult> rows,
ReadOnlySpan<bool> isNull, MemoryAllocator allocator = null);
```
or a validity `ArrowBuffer` + `nullCount` pair if that fits the surrounding
style better. Null rows would also be excluded from `ShredSchemaInferer.Infer`,
which today has to be done by the caller filtering the sequence.
### Component(s)
C#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
