andygrove commented on PR #5932: URL: https://github.com/apache/datafusion-comet/pull/5932#issuecomment-5876680489
This is a light fully automated review since there are so many PRs open. The physical side of the check at `native/core/src/parquet/schema_adapter.rs:549` is the reader's Arrow field, and the doc on `is_variant_marked` (line 497) treats that marker as the file's VARIANT annotation. As far as I can tell that only holds when the file has no embedded `ARROW:schema`. When it has one, `convert_field` in parquet 59.3's `arrow/schema/complex.rs` copies each field's metadata from the hint and never derives the extension from the Parquet logical type. For a plain-struct request `projects_variant` is false, so `parquet_exec.rs:198` keeps the hint. A concrete case is a file written by arrow-rs `ArrowWriter` without the non-default `variant_experimental` feature, from a column built with `parquet-variant-compute`'s `VariantArray::field`. `logical_type_for_struct` returns `None` without that feature, so the group has no annotation but the hint carries `arrow.parquet.variant`. Spark 4.1 infers and reads that column as `struct<metadata binary, value binary>`, and mai n reads it the same way, but with this change the read fails with `_LEGACY_ERROR_TEMP_3071`. The reverse holds too: an annotated group whose hint lacks the marker still returns the storage bytes. Could the physical side come from the Parquet schema instead, for example by reconciling the hint's Variant markers with the file's annotations in `EagerPageIndexReaderFactory`, which already rewrites the hint for Variant projections? A test that writes an unannotated group with a hand-built `ARROW:schema` carrying the marker would pin this down. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
