rustyconover commented on PR #8204: URL: https://github.com/apache/iceberg/pull/8204#issuecomment-1666912399
Hello @Fokko, Sometimes, applications need to verify the presence of files based on specific filter expressions. For instance, if you're interested in checking for file membership in Iceberg, particularly for files with a `date > 2023-01-01`, utilizing the scan planner can be quite beneficial. This approach allows you to avoid loading unnecessary manifest files, enhancing efficiency. Although working with manifest data provides valuable insights, it lacks the capability to filter using expressions. This is where the scan planner shines, enabling you to efficiently narrow down your results. In my experience, I frequently encounter Iceberg tables containing an extensive number of data files, often exceeding 80,000. Therefore, any functionality that aids in filtering is immensely valuable. Rusty -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
