wangmingzhou1986 opened a new issue, #4539: URL: https://github.com/apache/flink-cdc/issues/4539
### Search before asking - [X] I searched in the [issues](https://github.com/apache/flink-cdc/issues) and found nothing similar. Closest prior art, none of it covering this: #3533 / #3534 (`[hotfix][runtime] Invalidate cache correctly to avoid classloader leakage`, a different cache), #4534 (`[FLINK-40623][cdc-base] Release chunk splitter JDBC connection once a table is split`, about a single splitter connection rather than the pool), #2512 (adds `hikari.*` options but does not touch pool lifecycle). ### Flink version Flink 1.20.5, Flink CDC 3.6.0, MySQL CDC source (MySQL 5.7.28). Standalone session cluster on Kubernetes, Temurin 17.0.19, TaskManager `-XX:MaxMetaspaceSize=768m`. ### Bug **HikariCP connection pools created by the CDC source are never closed, so their `housekeeper` threads outlive the job and permanently pin the job's `ChildFirstClassLoader`.** In a session cluster this is an unbounded Metaspace leak. `JdbcConnectionPools` is a static singleton holding the pools: ```java private static JdbcConnectionPools instance; private final Map<ConnectionPoolId, HikariDataSource> pools = new HashMap<>(); public void clear() throws IOException { synchronized (pools) { pools.values().stream().forEach(HikariDataSource::close); ... } } ``` `clear()` is the only thing that ever calls `HikariDataSource.close()`. Of the eight files in this repository that reference `JdbcConnectionPools`, **the only one that calls `clear()` is `flink-cdc-base/src/test/java/.../GenericConnectionPoolTest.java`**. There is no production caller — not in `JdbcDataSourceDialect`, `JdbcConnectionFactory`, `MySqlSnapshotSplitAssigner` or `MySqlBinlogSplitAssigner`. (The same pattern exists in both copies: `flink-cdc-base/...relational/connection/JdbcConnectionPools.java` and `flink-connector-mysql-cdc/...source/connection/JdbcConnectionPools.java`.) Each `HikariDataSource` starts a daemon thread named `<poolName> housekeeper`, where `poolName` is built as `connection-pool-<host>:<port>` in `JdbcConnectionPoolFactory` / `PooledDataSourceFactory`. Because nothing closes the pool, that thread runs for the lifetime of the TaskManager JVM. ### Evidence from a heap dump I dumped a TaskManager that had **zero allocated slots** — verified two ways: `allocatedSlots` was empty in the REST API, and a sweep of all 71 subtasks of all 7 running jobs found none on that host (other TaskManagers returned non-zero counts, so the sweep itself was working). Every thread below therefore belongs to a job that is **no longer scheduled on that TaskManager**. Thread-name histogram of that idle TaskManager (110 threads total): ``` connection-pool-<mysql-host>:3306 housekeeper x4 blc-keepalive-<mysql-host>:3306 x1 IOManager reader thread #1 x2 ... ``` Of the 19 live `ChildFirstClassLoader` instances in that JVM, **8 are retained by lingering threads**, and the biggest single group is these pools: | retaining thread | count | classloaders pinned | |---|---|---| | `connection-pool-<mysql-host>:3306 housekeeper` | 4 | **4** | | `java-sdk-progress-listener-callback-thread` | 1 | 1 | | `FileChannelManagerImpl-io` / `IOManagerAsync` shutdown hooks | 2 | 2 | | `Thread-11` | 1 | 1 | The retention path, taken from the dump: ``` java.lang.Thread ("connection-pool-<host>:3306 housekeeper") <- GC root (ROOT_THREAD_OBJECT) .inheritedAccessControlContext -> java.security.AccessControlContext .context -> java.security.ProtectionDomain[] [0] -> java.security.ProtectionDomain .classloader -> org.apache.flink.util.ChildFirstClassLoader <- pinned ``` A thread created while user-code frames are on the stack inherits an `AccessControlContext` whose `ProtectionDomain` references the job's classloader. So the thread does not merely leak itself — it pins the entire user classloader and everything it loaded. `blc-keepalive-<host>:3306` (the binlog client keepalive thread) also survives on the idle TaskManager. It did not show up on a retention path in this particular dump, but it is the same class of problem and probably worth fixing together. ### Impact In a long-running session cluster this is unbounded. Our TaskManagers climb to the 768 MB Metaspace cap and have to be rotated; only a TaskManager restart returns the memory. Measured across 3,008 samples on 48 TaskManager instances, several climb monotonically from ~5 % to 86–92 % and never recover. This is one of three retention mechanisms we found in the same dump; the others are Hadoop's `ReflectionUtils.CONSTRUCTOR_CACHE` and a JDK `SoftReference` cache, reported separately. The connection pools are the one that clearly belongs to Flink CDC. ### What to reproduce 1. Session cluster, MySQL CDC source job. 2. Let it run, then stop/cancel it (or let it be rescheduled to another TaskManager). 3. On a TaskManager that now holds **no** tasks: `jcmd <pid> Thread.print | grep housekeeper` — the pool housekeeper threads are still there. 4. `jcmd <pid> VM.classloader_stats` — the job's `ChildFirstClassLoader` is still live, and Metaspace is not returned. Note for step 3/4: the official Flink image ships a JRE, so `jcmd` has to be brought in; and HotSpot attach requires the JVM's own uid, so `su` to the `flink` user first rather than running as root. ### Suggested direction The pools need a lifecycle tied to the source. Options that come to mind, in rough order of containment: 1. Close the pool when the last reader/enumerator using a given `ConnectionPoolId` is closed (reference-count the `ConnectionPoolId`), rather than relying on a singleton that is never cleared. 2. Failing that, call `JdbcConnectionPools.getInstance(...).clear()` from the source reader / enumerator `close()` path — coarse, but it at least gives `clear()` a production caller. 3. Independently, creating the pool inside a `Thread` whose context is the user classloader is what turns "a leaked thread" into "a leaked classloader". Creating HikariCP's threads under a neutral context, or setting a thread factory that does not inherit the AccessControlContext, would limit the blast radius even if a pool is left open. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! I can also re-run the heap dump to verify a fix — the check is simply that the `connection-pool-* housekeeper` threads are gone from an idle TaskManager and that the corresponding `ChildFirstClassLoader` count drops. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
