Zevrap-81 commented on code in PR #3124:
URL: https://github.com/apache/iceberg-python/pull/3124#discussion_r3110373554
##########
pyiceberg/table/maintenance.py:
##########
@@ -43,3 +43,31 @@ def expire_snapshots(self) -> ExpireSnapshots:
from pyiceberg.table.update.snapshot import ExpireSnapshots
return ExpireSnapshots(transaction=Transaction(self.tbl,
autocommit=True))
+
+ def compact(self) -> None:
+ """Compact the table's data and delete files by reading and
overwriting the entire table.
+
+ Note: This is a full-table compaction that leverages Arrow for
binpacking.
+ It currently reads the entire table into memory via `.to_arrow()`.
+
+ This reads all existing data into memory and writes it back out using
the
+ target file size settings (write.target-file-size-bytes), atomically
+ dropping the old files and replacing them with fewer, larger files.
+ """
+ # Read the current table state into memory
+ arrow_table = self.tbl.scan().to_arrow()
+
Review Comment:
Isn't this going to be a problem to load the entire table in memory? as the
tables are quite big. Is there a way to just do this per partition or some
other way that wouldn't cause memory issues? for example perform compaction on
last months data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
