rich7420 opened a new pull request, #7085: URL: https://github.com/apache/datafusion-comet/pull/7085
## Which issue does this PR close? Closes #7030. ## Rationale for this change When all requested nested leaves are absent from a Parquet file, Comet can keep containers non-null where Spark returns NULL, changing `IS NULL` and `count` results. ## What changes are included in this PR? - Propagate field presence through structs, arrays and maps, matching Spark's version-dependent nullness rules without treating present all-NULL fields as missing. - Ignore the Spark 4.1-only legacy setting on older versions. - Preserve file parent nullness in native Iceberg adaptation on every Spark version. For name-mapped files, containers that previously became NULL can change `IS NULL` and `count` results; leaf values remain tracked by #6790. - Add native and Spark regressions and document the behavior. ## How are these changes tested? - Both new native regressions fail before their fixes. Targeted reader tests pass on Spark 3.4, 3.5, 4.0, 4.1 and 4.2, with native scan assertions. - Native Parquet: 307 passed, 1 ignored, excluding an S3 refresh test that failed in the full run but passed alone. Iceberg adapter: 8 passed. Clippy, formatting and source RAT pass. - Broad Spark SQL and Iceberg JVM CI remain pending; required before Ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
