nssalian opened a new issue, #18669: URL: https://github.com/apache/iceberg/issues/18669
### Feature Request / Improvement Vectorized Parquet reads in Spark support unshredded VARIANT columns ([#16292](https://github.com/apache/iceberg/pull/16292)), but not shredded ones. Spark decides at planning time whether to read a scan vectorized. Whether a variant is shredded is only recorded in each data file's Parquet footer, so planning has to infer it from the table property, the column's metrics mode, and variant lower bounds. When that inference misses a shredded file, the vectorized read fails with `UnsupportedOperationException: Unsupported variant: shredded typed_value ...`. Reported failures: - [#18430](https://github.com/apache/iceberg/issues/18430): a shredded variant is in the file but not projected. - [#18601](https://github.com/apache/iceberg/issues/18601): a projected shredded variant has no variant bounds, for example when shredding was enabled through the session conf, or for shredded arrays. Planning also sends any scan that projects a variant column to the row reader when the table sets the shred property or the column uses `none`/`counts` metrics, even if none of the files are shredded. This issue tracks the work needed for Spark to read shredded variants vectorized without failing, and without relying on planning-time inference. Will update this issue as the PRs get added. Open: - [ ] [#18434](https://github.com/apache/iceberg/pull/18434) Skip unprojected shredded variants in vectorized reads ([#18430](https://github.com/apache/iceberg/issues/18430)) - [ ] [#18447](https://github.com/apache/iceberg/pull/18447) Return Variant null for missing Parquet payloads - [ ] [#18121](https://github.com/apache/iceberg/pull/18121) Vectorized Parquet reads for struct type (needed for shredded objects - will split this) - [ ] Read projected shredded variants in vectorized scans ([#18601](https://github.com/apache/iceberg/issues/18601)), PR to follow - [ ] Remove the planning-time shredding checks in `SparkBatch`, PR to follow - [ ] Vectorized reconstruction of shredded variants (scalars, arrays, objects), PRs to follow Related: - [#18392](https://github.com/apache/iceberg/pull/18392) Incorrect file skipping for filters on VARIANT fields - [#18518](https://github.com/apache/iceberg/pull/18518) Variant extract filters in table scan planning - [#10392](https://github.com/apache/iceberg/issues/10392) Variant data type support ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
