stevenzwu commented on PR #6614:
URL: https://github.com/apache/iceberg/pull/6614#issuecomment-1399807014
> I think the problem is that the TableLoader is taking ownership of the
catalog and closing it. That seems incorrect to me.
Yes, that will be bigger discussion.
> Alternatively, we could try to get rid of tableLoader altogether and
switch to catalog, but this would be a compatibility breaking change.
We can deprecate the `TableLoader` and provide an alternative if we can come
up with a reasonable solution.
I am wondering if we can have a short-term solution for the immediate issue
#6455.
I think this Flink usage pattern of `TableLoader` is fine if the `Table` is
used as a read-only table.
```
if (table == null) {
try (TableLoader loader = tableLoader) {
loader.open();
this.table = tableLoader.loadTable();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
```
The problem for `ContinuousSplitPlannerImpl` of FLIP-27 `IcebergSource` is
that it uses the `Table` as non read-only, as it needs to refresh the table.
With catalog closed, the table is invalid for refresh. One solution is to pass
in a cloned `TableLoader` to the `ContinuousSplitPlannerImpl`, which can
assumes the ownership of the `TableLoader`. `ContinuousSplitPlannerImpl` also
has a `close` method and can call `TableLoader#close` method to release the
resources. If we can add a `clone` method to `TableLoader` interface, this
approach can work.
@xuzhiwen1255 I definitely think we should close this PR as this is not the
right way to fix the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
