Zevrap-81 commented on code in PR #1958:
URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r3094045024
##########
pyiceberg/table/maintenance.py:
##########
@@ -27,12 +33,105 @@
from pyiceberg.table.update.snapshot import ExpireSnapshots
+@dataclass(kw_only=True)
+class RemoveOrphansResult:
+ deleted_files: Set[str]
+ failed_deletions: Set[str]
+
+
class MaintenanceTable:
tbl: Table
def __init__(self, tbl: Table) -> None:
self.tbl = tbl
+ def _orphaned_files(self, location: str, older_than: Optional[timedelta] =
None) -> Set[str]:
+ """Get all files which are not referenced in any metadata files of an
Iceberg table and can thus be considered "orphaned".
+
+ Args:
+ location: The location to check for orphaned files.
+ older_than: The time period to check for orphaned files. Defaults
to 3 days.
+
+ Returns:
+ A set of orphaned file paths.
+ """
+ from pyiceberg.io.fsspec import FsspecFileIO
+
+ if older_than is None:
+ older_than = timedelta(0)
+ as_of = datetime.now(timezone.utc) - older_than
+
+ if isinstance(self.tbl.io, FsspecFileIO):
+ logger.info("Determined io to be FsspecFileIO.")
+ uri = urlparse(location)
+ fs = self.tbl.io.get_fs(uri.scheme)
+ all_files = [f for f in fs.glob(f"{location}/**") if fs.isfile(f)
and fs.modified(f) < as_of]
Review Comment:
In my testing the files have scheme prefix missing and consequently this
line should be
```
all_files = [{uri.scheme}://{f} for f in
fs.glob(f"{location}/**") if fs.isfile(f) and fs.modified(f) < as_of]
```
Also is there a reason to use `fs.glob` instead of `fs.find`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
