Kingsley-working opened a new issue, #18377:
URL: https://github.com/apache/iceberg/issues/18377
### Apache Iceberg version
1.11.0 (also present on `main` today:
`core/src/main/java/org/apache/iceberg/rest/auth/AuthSessionCache.java` L54-58,
`OAuth2Manager.java` L216-217, `util/ThreadPools.java` L101-104)
### Query engine
Spark 4.1.1, as a long-lived Spark Connect server (many short-lived sessions
on one driver JVM), REST catalog with the default `oauth2` auth (Apache
Polaris).
### Please describe the bug 🐞
`OAuth2Manager.newSessionCache` creates its `AuthSessionCache` through the
public constructor, which builds a dedicated single-thread pool with
`ThreadPools.newExitingWorkerPool(name + "-auth-session-evict", 1)`. That goes
through Guava's `MoreExecutors.getExitingExecutorService`, which registers a
JVM shutdown hook (`Runtime.addShutdownHook(new Thread(...))`) that is never
removed. `AuthSessionCache.close()` shuts the executor down but the hook stays
registered.
On a long-lived JVM that creates many catalog instances (Spark creates one
catalog plugin, hence one `OAuth2Manager`, per Spark session) this has two
effects:
1. `java.lang.ApplicationShutdownHooks.hooks` grows by one unstarted
`Thread` per catalog instance for the life of the JVM.
2. Each hook `Thread` is constructed on the calling thread, so it copies
that thread's inheritable thread-locals. Spark's
`SparkSession.activeThreadSession` is an `InheritableThreadLocal`, so every
hook pins the Spark session that created the catalog, including sessions that
have since been closed, together with their catalogs, table caches and every
loaded `TableMetadata` (full snapshot history). Heap dump, Eclipse MAT, GC-root
path with weak/soft references excluded (it is the only path):
```
java.lang.ApplicationShutdownHooks.hooks
└ Thread
"DelayedShutdownHook-for-java.util.concurrent.ThreadPoolExecutor@…" (never
started)
└ inheritableThreadLocals → ThreadLocalMap → Entry.value
└ org.apache.spark.sql.classic.SparkSession (a CLOSED Spark Connect
session, ~36 MB retained)
```
Measured on a shared Spark Connect driver (4 GiB heap): 319 such hook
threads after ~3 h; `ApplicationShutdownHooks` retained 51% of the live heap;
releasing or idle-expiring sessions returned 0-9% of it; the heap left after
each full GC only grew until the driver ran out of memory, roughly daily.
The same pattern (a per-call exiting pool, hence a hook per call) exists in
`RewriteDataFilesSparkAction.rewriteService()` (`Rewrite-Service`: 296 hooks
after one `rewrite_data_files` sweep),
`RewritePositionDeleteFilesSparkAction.rewriteService()` and
`BaseProcedure.executorService(...)` when a pool size is passed.
#15031 covers the hooks of the *static* pools in `ThreadPools` (class-loader
leaks on hot reload). This report is about the per-instance and per-call pools,
whose hooks multiply with the number of catalogs and calls.
**To reproduce:** on one JVM, create N `OAuth2Manager` instances (for
example N Spark Connect sessions that each read from a REST catalog with
OAuth2), close them, force a full GC, then look at
`java.lang.ApplicationShutdownHooks.hooks` (N new entries) or a live heap
histogram (the N closed `SparkSession` objects are still reachable).
**Suggested fix:** per-instance and per-call caches should not use an
"exiting" pool. Options:
- (a) one shared static daemon evictor for all `AuthSessionCache` instances,
passed as a plain `Executor` so `close()` does not shut it down, with its
thread created with inheritable thread-locals disabled (`new Thread(null, r,
name, 0, false)`) so even that single thread pins nothing;
- (b) a plain daemon pool that `close()` shuts down, with no shutdown hook;
- (c) if a hook is kept, remove it in `close()` and construct the hook
thread with inherited thread-locals disabled.
We run (a) in production as an `OAuth2Manager` subclass loaded through
`rest.auth.type=<fqcn>` (overriding `newSessionCache` and using the
package-private `AuthSessionCache(Duration, Executor, Ticker)` constructor):
closed sessions are collected at the next GC and the heap floor returns.
### Willingness to contribute
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
