mxm commented on code in PR #17437:
URL: https://github.com/apache/iceberg/pull/17437#discussion_r3710450832
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestTableSerializerCache.java:
##########
@@ -121,4 +127,25 @@ void testCacheSize() {
cache = new TableSerializerCache(CATALOG_EXTENSION.catalogLoader(), 1000);
assertThat(cache.maximumSize()).isEqualTo(1000);
}
+
+ @Test
+ void testClosesCatalogAfterSchemaLookup() throws Exception {
+ Table table =
+ CATALOG_EXTENSION
+ .catalogLoader()
+ .loadCatalog()
+ .createTable(TableIdentifier.of("table"), schema1);
+
+ Catalog catalog = mock(Catalog.class,
withSettings().extraInterfaces(Closeable.class));
+ when(catalog.loadTable(any(TableIdentifier.class))).thenReturn(table);
+ CatalogLoader catalogLoader = mock(CatalogLoader.class);
+ when(catalogLoader.loadCatalog()).thenReturn(catalog);
+ cache = new TableSerializerCache(catalogLoader, 10);
Review Comment:
Could we write this test without using Mockito?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java:
##########
@@ -120,9 +127,22 @@ private class SerializerInfo {
}
private void update() {
- Table table =
catalogLoader.loadCatalog().loadTable(TableIdentifier.parse(tableName));
- schemas = table.schemas();
- specs = table.specs();
+ // The serializer has no teardown hook, so the freshly loaded catalog is
closed here, after
+ // reading the table metadata, to avoid leaking one per cache miss.
+ Catalog catalog = catalogLoader.loadCatalog();
+ try {
+ Table table = catalog.loadTable(TableIdentifier.parse(tableName));
+ schemas = table.schemas();
+ specs = table.specs();
+ } finally {
+ if (catalog instanceof Closeable) {
+ try {
+ ((Closeable) catalog).close();
+ } catch (IOException e) {
+ LOG.warn("Failed to close catalog {}", catalog.name(), e);
+ }
+ }
Review Comment:
I think we want to move to a TaskManager-level catalog cache that we close
on job shutdown. Out of scope for this PR. I think @swapna267 is looking into
that already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
