sunchao opened a new pull request, #5377: URL: https://github.com/apache/datafusion-comet/pull/5377
## Why are the changes needed? Spark 4 VARIANT columns currently prevent native Parquet and Iceberg scans even when queries do not project those columns. Some native Parquet scans can also fail while serializing an unprojected VARIANT field. Refs #4295. ## What changes were proposed in this PR? Allow native scans when VARIANT columns are safely excluded from the requested projection. For Parquet, prune unsupported VARIANT-bearing fields from the serialized data schema and correctly rebase projected, partition, and metadata column indices. For Iceberg, identify projected fields by stable field IDs, including historical snapshots after column renames. Continue falling back to Spark for projected VARIANT fields, unsupported nested projections, empty or metadata-only projections, and unsupported equality-delete keys. ## How was this PR tested? - Spark 4.0 Iceberg VARIANT regression tests, including renamed historical snapshots: three tests passed. - Spark 4.0 Parquet VARIANT SQL regression: passed. - Spark 3.5 Iceberg historical-snapshot compatibility regression: passed. - Spotless, Scalastyle, and git diff --check passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
