rajarshisarkar opened a new pull request, #18655: URL: https://github.com/apache/iceberg/pull/18655
Backport of #18459 to Spark 4.0. When Puffin statistics files are used for CBO, `SparkScan.estimateStatistics` maps blob metadata field IDs to column names using `table.schema().findColumnName(fieldId)`. If columns were dropped in a later snapshot while the statistics file belongs to the older snapshot, `table.schema().findColumnName` returns `null`, causing an NPE or unexpected error. Furthermore, blob metadata with empty fields (e.g. table-level blobs) caused index errors. This change: * Filters out empty blob fields before grouping (`filter(metadata -> !metadata.fields().isEmpty())`). * Retrieves the snapshot schema (`SnapshotUtil.schemaFor(table, snapshot.snapshotId())`) to resolve field IDs accurately against the snapshot's state when stats were generated. * Adds a unit test verifying statistics mapping after column drop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
