snazy commented on PR #10688: URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226834838
It seems there are lot of individual points that should be discussed. @BsoBird do you mind adding a point to the "Discussions" section on the [agenda for the next community sync on July 31st](https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit)? Let me comment on some of the points. TL;DR I think it's not an easy problem to solve. > balance this with the use of lockManager (which in itself is not a difficult task). Locking is one of the most difficult problems in distributed computing. (And actually in non-distributed computing, too.) Just very few things that come to my mind (the real list is much longer): * Evil users lock all your tables (-> authorization) * Retry/back-off -> piling up waiting requests -> threading issues * Crashed clients leave stuff locked * "all the things" about race conditions > ... a large number of users use GlueCatalog ... want to access both iceberg and non-iceberg tables in Glue. That's IMO a fair point. > 2.You might think that users could solve this problem by implementing rest-catalog. However, many users do not have the ability to implement rest-catalog.Also, rest-catalog is still evolving, and many of the specifications have not yet been finalised. I suspect, this wouldn't be an issue, if Glue would have a rest-catalog? > 3.Regarding the fact that all clients need to be involved in the locking in order for it to work. I think this is a problem that users need to solve themselves. Similar to my statement above, users should not be forced into implementing/providing a proper distributed locking mechanism. Also isolation levels play a role here. Some use cases are probably fine with "dirty reads", some with "read committed" and others require full "serializeable" guarantees. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
