djouallah commented on PR #3361:
URL: https://github.com/apache/iceberg-python/pull/3361#issuecomment-6072278482
We're running this in production now: a build of `4adf237` in the daily
maintenance of 25 Iceberg tables written by DuckDB to the OneLake (Microsoft
Fabric) Iceberg REST catalog, right after `expire_snapshots`. The first pass
deleted 14,563 orphan files (~11.5 GB), with no failed deletes. Our tests
passed afterwards, and the next pass found only a handful of files that had
just crossed the 3-day line. Thank you!
A few notes from running it on ADLS:
1. **The fsspec listing drops the ADLS account.** adlfs lists
`abfss://<container>/...` while the metadata references
`abfss://<container>@<account-host>/...`. As a result, every table hit the
prefix-mismatch `ValidationException`. The default `ERROR` mode did exactly the
right thing there. We worked around it with `equal_authorities({container:
f"{container}@{host}"})`. #3974's `list_prefix` already puts the netloc back
for `_ADLS_SCHEMES`, so rebasing this PR on #3974 instead of carrying its own
copy would fix it. An ADLS case in the tests would catch it.
2. **`compare_to_file_list` skips the hidden-path filter.** It returns
before `_is_hidden`, so an inventory that includes `_delta_log/` or `.crc`
files would delete them. It also reports a size of 0 for every file, so
`total_bytes` is wrong on that path.
3. **Deletes run one call per file.** 6,995 files took about 7 minutes on
ADLS. A bulk path would help a lot when there is a backlog, for example
`fs.rm(list)` in fsspec, the counterpart of Java's `SupportsBulkOperations`.
Happy to test a new revision against the same tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
