qzyu999 commented on code in PR #3124:
URL: https://github.com/apache/iceberg-python/pull/3124#discussion_r3170474257
##########
pyiceberg/table/maintenance.py:
##########
@@ -43,3 +43,31 @@ def expire_snapshots(self) -> ExpireSnapshots:
from pyiceberg.table.update.snapshot import ExpireSnapshots
return ExpireSnapshots(transaction=Transaction(self.tbl,
autocommit=True))
+
+ def compact(self) -> None:
+ """Compact the table's data and delete files by reading and
overwriting the entire table.
+
+ Note: This is a full-table compaction that leverages Arrow for
binpacking.
+ It currently reads the entire table into memory via `.to_arrow()`.
+
+ This reads all existing data into memory and writes it back out using
the
+ target file size settings (write.target-file-size-bytes), atomically
+ dropping the old files and replacing them with fewer, larger files.
+ """
+ # Read the current table state into memory
+ arrow_table = self.tbl.scan().to_arrow()
+
Review Comment:
Hi @Zevrap-81, great callout, I initially built this with an MVP mindset to
prove functionality, but your point is completely valid. Currently we're
waiting on #3131 to complete before returning to this PR. However, here are
some alternatives. I think it may make more sense to simply go with a scalable
strategy from the onset rather than implement a new patch on top in the future.
To avoid loading all the data into memory, there are definitely some obvious
alternatives. You mentioned doing so on the previous month's data. I think some
logical alternatives in that same line of thinking are to do compaction only
after applying a row filter to limit the data to only the user's query.
Additionally, we can make sure compaction happens at the partition level, so
that any data loaded into memory is restricted to the partition size. However,
all these methods are still lacking and don't fully answer the question in a
scalable manner, as it's easily possible that a user has data exceeds local
memory constraints even after applying filtering and partitions.
The better solution I believe would be to align with the Java implementation
and the bin-packing strategy. This way, the scan will group the files into
manageable bins (e.g. 512MB or 1GB) before applying the compaction step. I will
look to go this route once #3131 is completed. Thanks for bringing up your
valid concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
