Fokko commented on PR #1742: URL: https://github.com/apache/iceberg-python/pull/1742#issuecomment-2692859499
Thanks @andormarkus for working on this, I've just commented on #1737, would love to hear your thoughts there as well. > Passing DataFile objects between processes requires serialization, which I've found challenging to implement properly. I tried jsonpickle and custom serialization methods but encountered significant issues. `DataFile`'s tend to be pretty big. Java solves this by writing the DataFiles out into a Manifest, and serializing the ManifestFile and sending it to the main process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
