singhpk234 commented on issue #9518: URL: https://github.com/apache/iceberg/issues/9518#issuecomment-1909005110
ok so looks like a bug in spark 3.5 introduced via : https://github.com/apache/spark/commit/53df45650af1e48e01e392caed6c1f83c2e9e9f1#diff-b2edb35eb7c49f1f3c1fd074927328ac103e9a2cd058e6183e4cb7604ada6dd4 essentially : ``` object PushablePredicate { def unapply(e: Expression): Option[Predicate] = new V2ExpressionBuilder(e, true).build().map { v => assert(v.isInstanceOf[Predicate]) v.asInstanceOf[Predicate] } } ``` which is not true as the expression we get from `V2ExpressionBuilder` is not of Predicate type but of GeneralScalarExpression at this time we could have easily returned None so that this is not pushed down or handled the extractor in such a way that this works with `GeneralScalarExpression` but an assert was added which doesn't seems correct to me. I think the fix needs to be done in spark rather than iceberg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
