kevinjqliu commented on PR #3540: URL: https://github.com/apache/iceberg-rust/pull/3540#issuecomment-6433482644
thanks for digging into this! the fix works, but this feels like a workaround to me 😅 I think the root cause is that predicates are bound to the table schema but evaluated against file schema batches. `try_cast_literal` narrows the literal to the file type, and a failed cast silently becomes null. this PR casts the column back up in each comparison, but only for numeric types, so the lossy literal cast is still there for everything else. i think we should cast the filter batch to table types once in `get_row_filter`, using the same rule as `RecordBatchTransformer`. then `try_cast_literal` can error instead of returning null. another option is `ArrowReaderOptions::with_schema` so parquet decodes directly into table types. not sure it supports `int` -> `long` though Let me know what you think! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
