salimpadela opened a new issue, #11789:
URL: https://github.com/apache/iceberg/issues/11789
### Query engine
how to pass predicate in where clause for rewrite_data_files using Pyspark?
If it matters, i am using AWS Glue to execute this job.
### Question
I cant seem to figure out what is wrong with the way I am passing where
clause predicate in rewrite_data_files.
`
spark.sql("CALL
glue_catalog.system.rewrite_data_files(table=>'my-awesome-table', where =>
"col1 IN ('CT') AND col2 IN (5) AND year(CAST(col3 as DATE)) IN (1990)",
strategy => 'binpack', options => map('min-input-files', '2'))")
Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 3550;
IllegalArgumentException: Cannot translate Spark expression: ((col1#50421 INSET
CT AND col2#50422 INSET 10) AND year(cast(col3#50424 as date)) INSET 1990) to
data source filter
`
If i don't pass `AND year(CAST(col3 as DATE)) IN (1990)` in the where
clause, it works fine.
what am I missing here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
