nssalian opened a new pull request, #18851: URL: https://github.com/apache/iceberg/pull/18851
Fixes #18601. Part of #18669. ### Changes Spark decides at planning time whether a scan is vectorized, using the shred table property, metrics mode and variant lower bounds. When a file is shredded but none of those say so (for example, shredded arrays, which write no variant bounds, with shredding enabled through the session conf), the vectorized reader fails with `Unsupported variant: shredded typed_value`. This follows #18434: `VectorizedReaderBuilder` now checks each file's variant group, and a shredded group is read by the existing row-based `VariantReaderBuilder` into the same metadata/value vectors the unshredded path produces. Nested shredded variants without field ids are skipped, as before. Planning is unchanged. #18434's `projectingShreddedVariantUnsupported` test is renamed to `projectedShreddedVariantUsesRowBasedReader` and now asserts the new reader. ### Test Plan - arrow `TestVectorizedReaderBuilder`: a projected shredded variant gets the row-based reader, an unshredded variant keeps the vectorized reader, and a shredded list element without a field id is skipped. - arrow `TestVectorizedArrowReader`: values and nulls stay correct when vectors are reused across batches. - Spark 4.0/4.1/4.2 `TestSparkVariantRead`: vectorized reads return the same metadata and value as the row reader, also with 2-row batches that split a shredded file and end in a partial batch, and the scan is columnar. The new tests fail on main without this change with the exception above. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed and ran tests locally - Prompt Summary: Route projected shredded variants to the row-based variant reader inside the vectorized reader builder, with arrow and Spark tests. The reader, the builder routing and the tests are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
