[
https://issues.apache.org/jira/browse/HDFS-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101893#comment-18101893
]
ASF GitHub Bot commented on HDFS-17957:
---------------------------------------
joseluisll opened a new pull request, #8659:
URL: https://github.com/apache/hadoop/pull/8659
### Description of PR
Shuts down 67 leaked mini-cluster instances across 42 test files in
hadoop-hdfs-project, and fixes an assertion race in `TestFsVolumeList`.
A leaked cluster keeps its NameNode and DataNode threads, heap and ports
alive
under the remaining tests of the class, so one real failure turns into
several
bogus ones. The three categories are cluster never shut down (22), shut down
only on the happy path so it leaks precisely when the test fails (36), and
held
in a field with no teardown (9). The fixes are `try`-with-resources,
`shutdown()`
moved into `finally`, or an `@AfterEach`.
Two things a reviewer should look at:
`TestFsVolumeList` also gets an unrelated fix, without which it stays red.
Its
`testAddRplicaProcessorForAddingReplicaInMap` compares the configured pool
size
against `BlockPoolSlice#getAddReplicaForkPoolSize()`, which returned
`ForkJoinPool#getPoolSize()` — threads currently started, which the pool
grows
lazily — giving its `expected: <5> but was: <4>` failures. The
`@VisibleForTesting` getter now returns `getParallelism()`. This is the only
`src/main` change in the PR and the method has no production callers.
`TestNNWithQJM#testNewNamenodeTakesOverWriter` had its shutdown commented out
on purpose: the test leaves its NameNode fenced behind a second cluster
holding
the journal quorum, so closing the edit log terminates the JVM with "Could
not
sync enough journals to persistent storage". It is now shut down inside a
`try`/`catch` that tolerates that, so the rest of the class no longer runs
underneath a live fenced NameNode. This is the only judgement call in the
diff.
`TestFetchImage` is the one survey case deliberately left alone — same fenced
shutdown problem, but with a single `@Test` there is no later test to
protect,
so adding teardown would only break a class that passes today.
The remaining survey candidates are itemised in the JIRA: 4 need no fix by
design, 4 were scan false positives, and 11 are outside hadoop-hdfs-project
and
tracked separately.
### How was this patch tested?
Every touched test class was run on Ubuntu 24.04 with JDK
`17.0.19+10-1-24.04.2-Ubuntu`, the same JDK and OS as the precommit agents:
| Module | Classes | Tests | Result |
|---|---|---|---|
| hadoop-hdfs | 36 | 367 | green |
| hadoop-hdfs-rbf | 4 | 22 | green |
| hadoop-hdfs-nfs | 1 | 1 | green |
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id?
- [ ] Object storage: N/A
- [x] If adding new dependencies … — no new dependencies
- [x] If applicable, have you updated the `LICENSE`… — N/A
### AI Tooling
Contains content generated by Claude Code.
- [x] The PR includes the phrase "Contains content generated by Claude Code"
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> Cleanup of leaked mini-cluster instances in tests
> -------------------------------------------------
>
> Key: HDFS-17957
> URL: https://issues.apache.org/jira/browse/HDFS-17957
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Reporter: Jose Luis López
> Priority: Minor
> Original Estimate: 8m
> Remaining Estimate: 8m
>
> While analyzing HDFS-17951, I found that many tests leak mini-cluster
> instances (MiniDFSCluster, MiniQJMHACluster, MiniJournalCluster,
> MiniRouterDFSCluster, MiniYARNCluster, MiniMRYarnCluster and variants).
> A static scan of the test sources — vetted by hand to remove false positives
> such as clusters owned by harnesses or torn down through wrapper objects —
> found 86 cases in ~60 files, in three categories:
> A) 27 tests create a local cluster and never shut it down on any path.
> B) 38 tests shut the cluster down only on the happy path (no finally /
> try-with-resources), so the cluster leaks exactly when the test fails.
> C) 21 classes hold a cluster in a field with no teardown in the class or
> its superclasses.
> Impact: leaks stay within the class, but a leaked cluster keeps CI resources
> under the remaining tests, and one failure can cascade into bogus failures of
> later tests in the class.
>
> There is no impact on production, only improvement for CI testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
