shivadarshan-devadiga opened a new pull request, #18992: URL: https://github.com/apache/iceberg/pull/18992
Closes #18949. [SPARK-59572](https://github.com/apache/spark/pull/58851) makes a `PartitionPredicate` that fails to evaluate during planning fail the query, instead of being reported as a partition match after the predicate has been fully pushed down. This adds `partitionPredicateEvaluationFailureIsPropagated` to `TestFilterPushDown` in `spark/v4.2`. It partitions a table by a string column, filters with a UDF that throws `NumberFormatException` for one partition value, and checks that planning the query fails with `FAILED_EXECUTE_UDF`, caused by that `NumberFormatException`. The test is the regression from the Spark 4.2.1 verification draft #18294 by @manuzhang, kept as a permanent test. It is added to Spark 4.2 only: `PartitionPredicate` pushdown is new in Spark 4.2, and only `spark/v4.2` tests it (`partitionPredicatePushdownWithUDF`). Testing: - The new test passes in both planning modes on Spark 4.2.1, the version on main. The rest of `TestFilterPushDown` passes too. - With `spark42` set to 4.2.0, which doesn't have SPARK-59572, the new test fails in both planning modes: planning doesn't throw, so the failing predicate is treated as a match. - `spotlessCheck` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
