nssalian opened a new pull request, #17736: URL: https://github.com/apache/iceberg/pull/17736
## Rationale for this change In my testing, I found that Vectorized Parquet reads throw `UnsupportedOperationException` when they hit a shredded variant column (`typed_value` subtree). `SparkBatch` tried to route shredded files away from the vectorized reader using the manifest variant bounds and the `write.parquet.shred-variants` property, but neither is sound. Bounds are optional (a file written with metrics `none`/`counts` carries none), and the property reflects writer intent, not how existing files were written. Shredding can also be enabled through the session conf or a write option with the table property left false. A shredded file can therefore reach the vectorized reader and crash, and it cannot be detected soundly at plan time, since shredding is a per-file Parquet-footer fact that is not recorded in the manifest. After checking all possible options, this seemed like the easiest to do to avoid any edge cases. Follow up section has more. ## Changes Route variant columns to the row reader in the batch path for now. `SparkBatch.supportsParquetBatchReads` now returns false for any variant column, and the unsound bounds and property/metrics checks are removed. A query that projects a variant column reads through the row reader, which reconstructs both shredded and unshredded variants correctly. Queries that do not project a variant column are unaffected and still vectorize. ## Testing Added `testReadShreddedViaSessionConfWithoutTableProperty` (v4.0 and v4.1). It enables shredding through the session conf with the table property left false and metrics disabled, then reads under vectorization and asserts the variant values are correct. On the previous routing the shredded file reached the vectorized reader and threw `UnsupportedOperationException`, so the test failed; with this change it reads through the row reader and passes. It guards against the crash returning: any routing that lets a shredded file reach the vectorized reader again fails the read, and the test. ## Follow up This turns off vectorized reads for variant columns, including unshredded ones, since plan time cannot tell them apart. The follow-up is to write the vectorized reader to reconstruct shredded variants, which lets variant vectorize again for both shredded and unshredded data in a subsequent release. This is a more involved work that will take a while so disabling this for now until that can go in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
