peterxcli opened a new issue, #5548: URL: https://github.com/apache/datafusion-comet/issues/5548
### Problem Comet cannot represent Spark `VariantType` in its protobuf, construct a logically marked Arrow Field for it, or recover the logical type from that Field on the JVM. Parent: #5546 Depends on: #5547 ### Solution - Append `VARIANT` to `DataTypeId` without renumbering existing values. - Serialize Spark `VariantType` through Spark-version shims; Spark 3.x remains inert. - Map it natively to physical `Struct<value: Binary, metadata: Binary>`. - Attach `ARROW:extension:name=arrow.parquet.variant` to the outer Field. - Make `Utils.fromArrowField` recognize only that explicit marker. - Reuse `CometStructVector`; do not add a new vector class. - Keep general expression, operator, and scan admission gates closed. This task transports logical identity only. It must not yet admit Variant Parquet scans. ### Tests - Protobuf and Arrow Field mapping use `[value, metadata]`. - Only the explicit extension marker becomes Spark `VariantType`. - An unmarked identical Struct remains `StructType`. - General native Variant expressions remain unsupported. - Spark 3.x compiles and behaves unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
