chenjian2664 opened a new pull request, #18808: URL: https://github.com/apache/iceberg/pull/18808
Backports #18148 to Spark 4.1. On Spark 4.1, filtering a changelog table on `_change_type`, `_change_ordinal` or `_commit_snapshot_id`, such as `SELECT * FROM t.changes WHERE _change_type = 'INSERT'`, fails with: ``` ValidationException: Cannot find field '_change_ordinal' in struct ``` The predicate is pushed into the Iceberg scan and bound against the table schema, which doesn't contain these generated columns. This affects anyone filtering CDC changelog output by change type, ordinal or commit snapshot on Spark 4.1. Predicates on these columns are now left for Spark to evaluate after the scan, matching Spark 4.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
