ragnard opened a new issue, #18263: URL: https://github.com/apache/iceberg/issues/18263
### Apache Iceberg version 1.11.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 With a REST catalog that vends S3 credentials per table, client-side deletes silently delete nothing, or leave the last few hundred files behind. `DROP TABLE ... PURGE` drops the table but its files stay in the bucket; expire_snapshots` leaves expired files behind. The failures are only logged as warnings. Reads and writes are not affected. `S3FileIO` keeps one S3 client per vended credential prefix (the table's location) and a root client, which has no credentials when they are vended. `deleteFiles` batches keys per bucket (250 by default). Full batches are sent with the table's client, but the final partial batch is sent with the client for `s3://<bucket>`, which matches no credential prefix and so gets the root client. That batch fails. A delete of fewer than 250 files is a single partial batch, so nothing is deleted. A related problem: because keys are grouped by bucket rather than by credential prefix, a full batch can mix keys from several prefixes (e.g. two tables in one bucket) and is sent with just one of their clients. This is the same issue reported for `GCSFileIO` in #16480. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
