rajarshisarkar opened a new pull request, #18459: URL: https://github.com/apache/iceberg/pull/18459
### Problem In `SparkScan.java`, when Spark CBO is enabled (`spark.sql.cbo.enabled=true` and `readConf.reportColumnStats()=true`), column statistics are read from Puffin files. 1. If a column has been dropped from the table schema after statistics were computed, `table.schema().findColumnName(entry.getKey())` returns `null`, causing Spark's `FieldReference.column(colName)` to throw `IllegalArgumentException: The column name cannot be null` and crash query planning. 2. For non-field or table-level blobs where `fields()` is empty, `metadata.fields().get(0)` throws `IndexOutOfBoundsException`. ### Solution 1. Filter out `BlobMetadata` with empty `fields()` before grouping by field ID. 2. Gracefully skip statistics entries for columns that no longer exist in the table schema. 3. Import `StandardBlobTypes` to replace inline FQCN. 4. Add regression unit tests in `TestSparkScan` across all Spark versions (3.5, 4.0, 4.1, 4.2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
