CurtHagenlocher opened a new issue, #399:
URL: https://github.com/apache/arrow-dotnet/issues/399
### Describe the enhancement requested
`Apache.Arrow.Operations.Shredding` implements the parquet VariantShredding
spec one value at a time, but a shredded variant is a COLUMN-level layout.
There is no entry point in either direction that takes or returns a
`VariantArray`:
- **read**:
`VariantArrayShreddingExtensions.GetLogicalVariantValue(VariantArray, int)`
resolves one row's typed_value + residual, and `VariantUnshredder.Reconstruct`
works on a single `ShredResult`. Nothing turns a shredded `VariantArray` into
its canonical unshredded equivalent.
- **write**: `ShredSchemaInferer.Infer` / `VariantShredder.Shred` /
`ShreddedVariantArrayBuilder.Build` take and return values and `ShredResult`s.
Nothing takes a `VariantArray`.
So every consumer that holds a column writes the same two loops. Ours, from
a parquet implementation that reassembles on read and shreds on write:
```csharp
public static VariantArray Reassemble(VariantArray array)
{
if (!array.IsShredded) return array;
var builder = new VariantArray.Builder();
for (int i = 0; i < array.Length; i++)
{
if (array.IsNull(i)) { builder.AppendNull(); continue; }
builder.Append(array.GetLogicalVariantValue(i));
}
return builder.Build(allocator: null);
}
```
and, on the way in, a loop that decodes every row to `VariantValue` (plus a
null mask) purely to hand the sequence to `Infer`/`Shred`.
Neither loop encodes any judgement — they are the obvious implementation,
which is the argument for having them once, upstream, next to the code whose
invariants they depend on. As it stands each consumer re-derives details like
"read the LOGICAL value so an already-shredded input re-shreds rather than
losing its typed columns", and gets to find out the hard way that
`VariantArray` is not an `Apache.Arrow.Array`.
### Suggested API
```csharp
// read
public static VariantArray Reassemble(this VariantArray array);
// write
public static VariantArray Shred(this VariantArray array, ShredSchema
schema);
public static bool TryShred(this VariantArray array, ShredOptions options,
out VariantArray shredded);
```
Splitting inference from shredding matters for the write side, and is why
`Shred(array, schema)` is listed separately from `TryShred`: a parquet file has
ONE schema while a writer sees one batch at a time, so a caller must infer once
over a representative batch and shred every later batch into that same layout.
An API that only offered infer-and-shred-together would quietly produce row
groups whose layouts disagree.
These also compose with #398 — an array-level entry point has somewhere
natural to carry validity, which the value-level one does not.
Happy to open a PR for any of this if the shape sounds right.
### Component(s)
C#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
