smaheshwar-pltr commented on PR #14373:
URL: https://github.com/apache/iceberg/pull/14373#issuecomment-3614944636
> Can Iceberg table metadata be used to prune these expressions when passed
down? We don't want to add expressions that can't be used. Filtering individual
rows or values is for engines.
Thanks for taking a look, @rdblue.
I believe so - Spark filters like `F.col('url').endswith('fr')` /
`F.col('url').contains('fr')` when your Iceberg table is partitioned on a `url`
column are surprisingly slow, because Iceberg partitioning isn't used to filter
out unnecessary files during planning, prior to this PR. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
