abnobdoss commented on PR #3420: URL: https://github.com/apache/iceberg-python/pull/3420#issuecomment-6514199462
Hi @azwanzuharimi, thanks for testing this! I agree yours covers all three paths, this one was intentionally scoped to just the initial scan. That scan is the step that decides which files get read, so this PR keeps its filter simple on purpose: per-column min/max bounds, which always prune by file stats no matter how many keys there are, and the exact matching happens after. A flat In doesn't have that property. Over 200 values it skips stats-based pruning (IN_PREDICATE_LIMIT), so once every key column has over 200 unique keys, which is pretty low for a real upsert, you end up scanning every file in the matched partitions or the whole table. So I'd rather not close this yet. ANDing these bounds onto the Ins keeps pruning without making the filter deep. If #4027 implements these bounds specifically for the file filtering, I'm happy to close this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
