CTTY commented on PR #2620: URL: https://github.com/apache/iceberg-rust/pull/2620#issuecomment-5418648118
Hi folks, thanks for the discussion here! I still want to push this forward, but unfortunately, due to limited bandwidth, the progress has been slow. Last week I had a offline discussion with @blackmwk and he proposed something on top of what's in this PR: https://github.com/blackmwk/iceberg-rust/pull/19 . I think moving the mutable states to `Transaction` level makes sense, as it would help remove unneeded lock in the current MSP, also cleaning up uncommitted data will be much easier. I'm still unclear how MSP will integrate this change, and will need to spend more time exploring it. Another thing worth mentioning is that the onging V4 work on the Java side will change how manifests are generated, filtered, and merged. It may be a bit early, but I believe a design that can survive thru V4 upgrade would greatly help us in the long term. Considering the complexity of even refactoring the transaction level logic, I think it's worth investing a bit more time make sure we don't have to rewrite the code when adopting V4 in the future. I'll update this PR as soon I have a clearer view of how it's going to look like in rust from end to end. Meanwhile, any thoughts about the transaction layer and how it could look like in rust would be appreciated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
