hililiwei commented on PR #6614: URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1407592885
> That is a good question. It is a bit challenging. The easier model is share nothing across tasks (e.g. no global static conn pools). Let's say each TM as 8 subtask, each task need to open a client (and connection pool) talking to some external system. Each task is responsible for the ownership and lifecycle of the client. This ties the client/conn pool lifecycle to the Flink subtask lifecycle. +1 for this. > In this way, we need to access the table through the accessor of a table. tableLoader is maintained by the accessor. When the accessor is closed, we close the tableLoader to ensure that it is closed correctly and that the reference type in the table will not be unavailable because catalog is closed first. We should first standardize the behavior. The current approach, in my opinion, is not the best way to solve the problem. From the above discussion, it seems that we have the following two solutions: 1. Use catalog instead of `TableLoader`. It's a big change, but it's more in line with the current behavior of catalog. In addition, all resources are managed by the catalog. 2. Clone a new table. In Flip-27, Read-only table does not meet our requirements, so we can try to clone a new table in the table loader. It's independent of catalog. We need to manage it manually. 3. Clone a new `TableLoader` Pass in a cloned TableLoader to the ContinuousSplitPlannerImpl, and add a clone method to TableLoader interface. Did I miss anything? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
