[
https://issues.apache.org/jira/browse/HDFS-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López updated HDFS-17957:
-----------------------------------
Description:
Tests in hadoop-hdfs-project leak mini-cluster instances. A leaked cluster
holds NameNode and DataNode threads, heap and ports open for the rest of the
class, so one real failure produces several bogus ones.
A static scan of the test sources found 82 real cases repo-wide. This issue
covers the 67 in hadoop-hdfs-project, in three categories:
A) 22 never shut the cluster down on any path.
B) 36 shut it down only on the happy path, so it leaks exactly when the
test fails.
C) 9 hold it in a field with no teardown in the class or its superclasses.
Fixes are try-with-resources, shutdown() in finally, or @AfterEach. The 15
cases outside this scope are itemised in the reconciliation attached as a
comment: 4 need no fix by design, 11 are outside hadoop-hdfs-project and get
their own JIRAs.
Two further defects found while running these tests:
TestFsVolumeList also needs an assertion fix, without which it stays red.
testAddRplicaProcessorForAddingReplicaInMap compares the configured pool size
against BlockPoolSlice#getAddReplicaForkPoolSize(), which returns the lazily
grown ForkJoinPool#getPoolSize() - the source of its "expected: <5> but was:
<4>" failures. The @VisibleForTesting getter now returns getParallelism(). It
has no production callers.
Seven timeout budgets are mis-calibrated. Four @Timeout values are written in
milliseconds as if seconds (30000, 30000, 300000, 100000). Three are shorter
than the waits they wrap. hadoop-hdfs and hadoop-hdfs-rbf also gain
junit.jupiter.execution.timeout.default (600s and 1800s), so a hang in a
method with no explicit @Timeout fails as a named per-method timeout instead
of a surefire fork kill that discards the whole class's results.
Test-only apart from the BlockPoolSlice getter. Follow-up once this merges:
remove TestFsVolumeList from .github/gha-tests/exclude-tests.txt (line 56).
was:
While analyzing HDFS-17951, I found that many tests leak mini-cluster instances
(MiniDFSCluster, MiniQJMHACluster, MiniJournalCluster, MiniRouterDFSCluster,
MiniYARNCluster, MiniMRYarnCluster and variants).
A static scan of the test sources — vetted by hand to remove false positives
such as clusters owned by harnesses or torn down through wrapper objects —
found 86 cases in ~60 files, in three categories:
A) 27 tests create a local cluster and never shut it down on any path.
B) 38 tests shut the cluster down only on the happy path (no finally /
try-with-resources), so the cluster leaks exactly when the test fails.
C) 21 classes hold a cluster in a field with no teardown in the class or its
superclasses.
Impact: leaks stay within the class, but a leaked cluster keeps CI resources
under the remaining tests, and one failure can cascade into bogus failures of
later tests in the class.
There is no impact on production, only improvement for CI testing.
Remaining Estimate: 8h (was: 8m)
Original Estimate: 8h (was: 8m)
> Cleanup of leaked mini-cluster instances in tests
> -------------------------------------------------
>
> Key: HDFS-17957
> URL: https://issues.apache.org/jira/browse/HDFS-17957
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Reporter: Jose Luis López
> Priority: Minor
> Labels: pull-request-available
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> Tests in hadoop-hdfs-project leak mini-cluster instances. A leaked cluster
> holds NameNode and DataNode threads, heap and ports open for the rest of the
> class, so one real failure produces several bogus ones.
>
> A static scan of the test sources found 82 real cases repo-wide. This issue
> covers the 67 in hadoop-hdfs-project, in three categories:
>
> A) 22 never shut the cluster down on any path.
> B) 36 shut it down only on the happy path, so it leaks exactly when the
> test fails.
> C) 9 hold it in a field with no teardown in the class or its superclasses.
>
> Fixes are try-with-resources, shutdown() in finally, or @AfterEach. The 15
> cases outside this scope are itemised in the reconciliation attached as a
> comment: 4 need no fix by design, 11 are outside hadoop-hdfs-project and get
> their own JIRAs.
>
> Two further defects found while running these tests:
>
> TestFsVolumeList also needs an assertion fix, without which it stays red.
> testAddRplicaProcessorForAddingReplicaInMap compares the configured pool size
> against BlockPoolSlice#getAddReplicaForkPoolSize(), which returns the lazily
> grown ForkJoinPool#getPoolSize() - the source of its "expected: <5> but was:
> <4>" failures. The @VisibleForTesting getter now returns getParallelism(). It
> has no production callers.
>
> Seven timeout budgets are mis-calibrated. Four @Timeout values are written in
> milliseconds as if seconds (30000, 30000, 300000, 100000). Three are shorter
> than the waits they wrap. hadoop-hdfs and hadoop-hdfs-rbf also gain
> junit.jupiter.execution.timeout.default (600s and 1800s), so a hang in a
> method with no explicit @Timeout fails as a named per-method timeout instead
> of a surefire fork kill that discards the whole class's results.
>
> Test-only apart from the BlockPoolSlice getter. Follow-up once this merges:
> remove TestFsVolumeList from .github/gha-tests/exclude-tests.txt (line 56).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
