peterxcli opened a new issue, #5551: URL: https://github.com/apache/datafusion-comet/issues/5551
### Problem After the schema, normalization, and physical-reader layers are ready, Comet still needs a deliberately narrow JVM admission path for direct top-level Variant projection and Spark-compatible behavior for missing columns and unsupported consumers. Parent: #5546 Depends on: #5550 ### Solution - Admit only direct top-level `VariantType` fields in ordinary native Parquet scans. - Build the native data schema from Spark's pruned required schema so unread Variant roots remain removable. - Preserve existence-default value/index pairing and transport a Variant default as a scan-only constant `[value, metadata]` struct. - Apply a default only when the physical column is absent. - Export the marked Field and reuse `CometStructVector` so inherited `ColumnVector.getVariant` works. - Keep nested Variant, pushed `VariantStruct` (#5519), strict malformed reads, expressions, writes, C2R, shuffle/spill, Python operators, encryption, and Iceberg on explicit Spark fallback where applicable. - Document the supported surface and remaining fallbacks. ### Tests - `SELECT v` and `SELECT id, v, tail` remain native and match Spark. - Objects, arrays, scalars, JSON null, SQL NULL, nullable parents, and entirely-null columns round-trip. - Spark sees logical `VariantType` with exact `[value, metadata]` Binary children. - A missing Variant column receives its existence default without shifting later defaults. - Unread Variant pruning from #5377 remains native. - Unsupported consumers above have focused fallback assertions. - Spark 4.0/4.1 profiles pass and Spark 3.5 still compiles unchanged. Completing this issue closes the ordinary-Parquet phase of #4295. Iceberg and the broader shredded optimization scope in #3983 remain separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
