DengYiping opened a new pull request, #18447: URL: https://github.com/apache/iceberg/pull/18447
When a Parquet Variant group contains metadata but omits its payload, the vectorized reader returns SQL NULL while the row reader returns Variant null (`00`). Use required metadata to determine presence and null counts, and normalize missing payloads to `00`, following the [Parquet specification](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding). This fixes Spark 4.0, 4.1, and 4.2 while preserving ordinary SQL NULL. Test plan: - Reproduced four failures using actual Parquet files before fixing. - Cover missing payloads, explicit Variant null, and SQL NULL with required/optional groups and dictionary encoding enabled/disabled; verify payload bytes and null counts. - All 24 focused tests pass across the three Spark versions, including existing Variant projection tests. Java formatting and main/test Checkstyle checks pass. --- **AI Disclosure** - Model: GPT-6 - Platform/Tool: Codex - Human Oversight: human reproduced the problem & review the changes - Prompt Summary: Reproduce and surgically fix missing Variant payload handling. The vector changes and regression tests were generated by Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
