MrDerecho opened a new issue, #2130: URL: https://github.com/apache/iceberg-python/issues/2130
### Feature Request / Improvement I use pyiceberg add_files to perform enterprise-grade ETL loading and backfilling of iceberg tables- on occasion, there will be a duplicate file, I load so many files that I can't use the duplicate check due to performance constraints (500-2000 files at a time). Note: AWS Athena doesn't have an issue with duplicate loads, however upstream Snowflake external tables and Databricks Delta uniform tables do. Often time the error messages provided have the S3 URI regarding the impacted file- I just need a process to surgically de-dupe the file in question from the metadata in-situ, v.s. reverting to an older snapshot (which may be 100+ since). The process should be able to both check for duplicate entries in the manifest lists, and/or take a known duplicate file as an input. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
