kevinjqliu commented on PR #2205:
URL: https://github.com/apache/iceberg-python/pull/2205#issuecomment-3343839287
While I agree with the need to retry for the `add_files` function, i dont
think we should add specific retry logic for this function only. These are all
commit operations so it would be more maintainable to address commit retries in
general.
The use case you described regarding collecting file stats is useful. I do
think we need to improve our retry mechanism that is better than "starting
everything from scratch". However, I think we should have separation of
concerns; `add_files` shouldnt need to worry about caching file stats.
As a library, pyiceberg aims to provide all the building blocks. We let
users implement their own specific solution for these type of optimizations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
