yuqi1129 commented on issue #10100: URL: https://github.com/apache/iceberg/issues/10100#issuecomment-2041959506
@pvary Thank you for your kind reply. The reason we need to close the `HiveCatalog` and `HiveClientPool` is that we are working on a metadata management system https://github.com/datastrato/gravitino and Gravitino will manage several catalogs. Each Iceberg catalog will have the following operations: 1. Create an Iceberg catalog. In this step, Gravitino will use a `IsolatedClassloader` to load classes that belongs to Icebergs catalog 2. `LoadTable`, `LoadSchema`, and other operations under the classloader 3. Finally, we are going to drop the Iceberg catalog if it's no longer needed, so we need to close the catalog, close the classloader, and release associated resources. However, due to a daemon thread introduced by `CachedClientPool` (Implementation class of CachedClientPool), the classloader can't be GC and thus will leads to OOM eventually. ```java // Code block from CachedClientPool#init clientPoolCache = Caffeine.newBuilder() .expireAfterAccess(evictionInterval, TimeUnit.MILLISECONDS) .removalListener((ignored, value, cause) -> ((HiveClientPool) value).close()) .scheduler( // This daemon thread can't be explicitly closed. Scheduler.forScheduledExecutorService( ThreadPools.newScheduledPool("hive-metastore-cleaner", 1))) .build(); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
