[
https://issues.apache.org/jira/browse/HDFS-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063820#comment-18063820
]
ASF GitHub Bot commented on HDFS-17824:
---------------------------------------
deepujain opened a new pull request, #8309:
URL: https://github.com/apache/hadoop/pull/8309
### Summary
When `dfs.datanode.directoryscan.threads` is set to 0 or a negative value,
the DataNode throws `IllegalArgumentException` from
`Executors.newFixedThreadPool(threads)` during DirectoryScanner initialization
and fails to start. This change validates the config and uses the default (1)
when the value is non-positive, with a warning log, so the DataNode starts and
the operator sees a clear message.
### Change
- **DirectoryScanner**: After reading `dfs.datanode.directoryscan.threads`,
if the value is ≤ 0, log a warning (parameter name, invalid value, default) and
set threads to `DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT` (1) before creating
the thread pool. Matches the existing pattern used for
`reconcile.blocks.batch.size` and `reconcile.blocks.batch.interval` in the same
class.
- **TestDirectoryScanner**: New test
`testInvalidDirectoryScanThreadsUsesDefault()` — set `directoryscan.threads` to
0, build a minimal cluster, create and start DirectoryScanner; verifies no
exception and scanner runs (HDFS-17824).
### JIRA
Fixes HDFS-17824
> DataNode fails to start when dfs.datanode.directoryscan.threads is 0 or
> negative
> --------------------------------------------------------------------------------
>
> Key: HDFS-17824
> URL: https://issues.apache.org/jira/browse/HDFS-17824
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.8.5
> Reporter: AMC-team
> Assignee: AMC-team
> Priority: Major
> Attachments: HDFS-17824.000.patch
>
>
> When {{dfs.datanode.directoryscan.threads}} is configured to {{0}} or a
> negative value, the DataNode initialization path constructs the directory
> scanner with {{{}Executors.newFixedThreadPool(threads){}}}. Because the
> thread count is non-positive, the JDK throws
> {{{}java.lang.IllegalArgumentException{}}}, and the DataNode fails to start.
> The code currently does not validate this configuration before creating the
> thread pool.
> *Steps to Reproduce*
> 1. Set the properties above in {{{}hdfs-site.xml{}}}.
> 2. Start or restart the DataNode.
> 3. Observe the following stack trace and process exit:
> {code:java}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1314)
> at
> java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1237)
> at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:151)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.<init>(DirectoryScanner.java:*
> )
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initDirectoryScanner(DataNode.java:*
> )
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1543)
> ...
> {code}
> *Actual Result*
> DataNode startup aborts with IllegalArgumentException during DirectoryScanner
> initialization; the node never becomes available.
> *Expected Result*
> Non-positive values of {{dfs.datanode.directoryscan.threads}} are handled
> gracefully—either rejected early with a clear configuration error naming the
> key and offending value, or automatically clamped to a safe default (≥1) with
> a {{{}WARN{}}}, rather than crashing the process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
