joseluisll opened a new pull request, #8637:
URL: https://github.com/apache/hadoop/pull/8637
### Description of PR
Documents **42 configuration properties** in `hdfs-default.xml` that are read
by production code but were previously discoverable only by reading the
source. Every property listed below is live and non-deprecated: each read
site was verified, none appears in any `DeprecationDelta` table, and none
carries a deprecated constant. This answers the question asked on HDFS-11039
in 2018 — *which* properties should be added.
**No production code is changed.** The diff touches exactly two files:
`hdfs-default.xml` (+42 `<property>` entries, 1 rename of a deprecated
property name to its current one) and `TestHdfsConfigFields.java` (skip-list
bookkeeping only, explained below).
#### The 42 properties
| Property | Default | Rationale |
| --- | --- | --- |
| `dfs.datanode.synconclose` | `false` | Controls fsync of block files on
close; read by `BlockReceiver`. Was skip-listed under `// Fully deprecated
properties?` — disproved. |
| `dfs.datanode.non.local.lazy.persist` | `false` | Permits LAZY_PERSIST
writes from non-local clients; read by `DNConf`. Was skip-listed under the same
stale comment — disproved. |
| `dfs.namenode.tolerate.heartbeat.multiplier` | `4` | Heartbeat-staleness
tolerance when deleting excess replicas; read by `BlockPlacementPolicyDefault`.
Same stale skip entry — disproved. |
| `dfs.namenode.replqueue.threshold-pct` | *(empty; falls back to
`dfs.namenode.safemode.threshold-pct`)* | Threshold for initializing
replication queues; read by `BlockManagerSafeMode`. Same stale skip entry —
disproved. |
| `dfs.ha.log-roll.rpc.timeout` | `20000` | RPC timeout for the Standby's
edit-log-roll requests; read by `EditLogTailer`. Was skip-listed as `// Removed
by HDFS-6440` — the property is still read; the claim is incorrect. |
| `dfs.ha.tail-edits.max-txns-per-lock` | `9223372036854775807` | Bounds
transactions applied per namesystem write-lock hold on the Standby; declared in
`EditLogTailer`, outside any `*ConfigKeys` class. |
| `dfs.client.read.shortcircuit.metrics.sampling.percentage` | `0` |
Sampling percentage for short-circuit read latency metrics; 0 disables.
Declared in `HdfsClientConfigKeys.Read.ShortCircuit`, which the comparison test
does not reflect over. |
| `dfs.webhdfs.oauth2.credential` | *(empty)* | OAuth2 credential for
`CredentialBasedAccessTokenProvider`. Siblings were documented; this constant
lives outside `HdfsClientConfigKeys`. |
| `dfs.webhdfs.oauth2.refresh.token` | *(empty)* | OAuth2 refresh token for
`ConfRefreshTokenBasedAccessTokenProvider`. Same situation. |
| `dfs.webhdfs.oauth2.refresh.token.expires.ms.since.epoch` | *(empty)* |
Expiry of the OAuth2 refresh token. Same situation. |
| `nfs.http.port` | `50079` | Fallback HTTP port for the NFS gateway web UI;
effective port comes from `nfs.http.address`. |
| `nfs.http.address` | `0.0.0.0:50079` | HTTP bind address of the NFS
gateway web UI. |
| `nfs.https.port` | `50579` | Fallback HTTPS port; effective port comes
from `nfs.https.address`. |
| `nfs.https.address` | `0.0.0.0:50579` | HTTPS bind address of the NFS
gateway web UI. |
| `nfs.file.dump` | `true` | Enables dumping out-of-order NFS writes to the
local dump directory. |
| `nfs.dtmax` | `65536` | Maximum readdir transfer size of the gateway. |
| `nfs.export.point` | `/` | The HDFS path exported by the gateway. |
| `nfs.max.open.files` | `256` | Maximum concurrently open file streams in
the gateway. |
| `nfs.stream.timeout` | `600000` | Inactivity timeout for NFS write streams
(ms). |
| `nfs.large.file.upload` | `true` | Enables the gateway's large-file upload
handling. |
| `nfs.aix.compatibility.mode.enabled` | `false` | Enables workarounds for
AIX NFS clients. |
| `nfs.metrics.percentiles.intervals` | *(empty)* | Rollover intervals for
gateway latency percentile metrics. |
| `nfs.superuser` | *(empty)* | NFS client user granted HDFS superuser
access through the gateway. |
| `nfs.registration.port` | `40` | Privileged port used when registering
with the portmapper. |
| `nfs.udp.client.portmap.timeout.millis` | `500` | Timeout for portmap RPC
connections. |
| `nfs.port.monitoring.disabled` | `true` | Whether clients may connect from
unprivileged ports. Renamed from the deprecated `nfs.allow.insecure.ports` per
`NfsConfiguration`'s `DeprecationDelta`. |
| `dfs.datatransfer.server.fixedBlackList.file` |
`/etc/hadoop/fixedBlackList` | Fixed server-side deny list for
`BlackListBasedTrustedChannelResolver`. |
| `dfs.datatransfer.server.variableBlackList.enable` | `false` | Enables the
reloadable server-side deny list. |
| `dfs.datatransfer.server.variableBlackList.file` | `/etc/hadoop/blackList`
| Reloadable server-side deny-list file. |
| `dfs.datatransfer.server.variableBlackList.cache.secs` | `3600` | Reload
interval of the server-side deny list. |
| `dfs.datatransfer.client.fixedBlackList.file` | *(empty; falls back to the
server value)* | Fixed client-side deny list. |
| `dfs.datatransfer.client.variableBlackList.enable` | `false` | Enables the
reloadable client-side deny list. |
| `dfs.datatransfer.client.variableBlackList.file` | *(empty; falls back to
the server value)* | Reloadable client-side deny-list file. |
| `dfs.datatransfer.client.variableBlackList.cache.secs` | `3600` | Reload
interval of the client-side deny list. |
| `dfs.datatransfer.server.fixedwhitelist.file` |
`/etc/hadoop/fixedwhitelist` | Fixed server-side allow list for
`WhitelistBasedTrustedChannelResolver`. |
| `dfs.datatransfer.server.variablewhitelist.enable` | `false` | Enables the
reloadable server-side allow list. |
| `dfs.datatransfer.server.variablewhitelist.file` | `/etc/hadoop/whitelist`
| Reloadable server-side allow-list file. |
| `dfs.datatransfer.server.variablewhitelist.cache.secs` | `3600` | Reload
interval of the server-side allow list. |
| `dfs.datatransfer.client.fixedwhitelist.file` | *(empty; falls back to the
server value)* | Fixed client-side allow list. |
| `dfs.datatransfer.client.variablewhitelist.enable` | `false` | Enables the
reloadable client-side allow list. |
| `dfs.datatransfer.client.variablewhitelist.file` | *(empty; falls back to
the server value)* | Reloadable client-side allow-list file. |
| `dfs.datatransfer.client.variablewhitelist.cache.secs` | `3600` | Reload
interval of the client-side allow list. |
#### Why TestHdfsConfigFields is in the diff
`TestHdfsConfigFields` enforces, in both directions, that `hdfs-default.xml`
and the reflected `*ConfigKeys` classes agree, modulo its skip lists.
Documenting these properties therefore requires skip-list bookkeeping — the
same idiom as the existing `nfs` prefix entry, routinely updated by past
patches (HDFS-15870, HDFS-15514, HDFS-15559):
* **Removed** the five stale skip entries (`dfs.datanode.synconclose`,
`dfs.datanode.non.local.lazy.persist`,
`dfs.namenode.tolerate.heartbeat.multiplier`,
`dfs.namenode.replqueue.threshold-pct`, `dfs.ha.log-roll.rpc.timeout`), so
those constants are again enforced against the xml and the gap cannot
silently reopen for them.
* **Added** skip entries for `dfs.datatransfer.{server,client}.*`,
`dfs.ha.tail-edits.max-txns-per-lock`, `dfs.webhdfs.oauth2.*` and
`dfs.client.read.shortcircuit.metrics.sampling.percentage`, whose constants
live in classes the test does not reflect over. The `nfs.*` entries are
covered by the existing `nfs` prefix entry; `NfsConfigKeys` itself cannot
be
reflected, as `hadoop-hdfs-nfs` depends on `hadoop-hdfs` and can never be
on
this test's classpath.
* **Corrected** the comments on the two entries that legitimately remain
skip-listed (`dfs.corruptfilesreturned.max`, `dfs.metrics.session-id`).
A follow-up can strengthen enforcement by promoting the declaring classes
into `configurationClasses` where module dependencies allow.
<details>
<summary><b>Method</b> — how the list was produced and verified</summary>
The list was produced by static analysis of the entire `hadoop-hdfs-project`
source tree, cross-validated by two independently implemented scanners (one
Java, one Python), and verified against Hadoop's own machinery:
1. **Forward extraction** — every `String` config-key constant in every class
(not just `*ConfigKeys` registries), resolving multi-line declarations and
`PREFIX + "suffix"` concatenation, including cross-class references.
2. **Inverse sweep** — every `Configuration` accessor call site
(`get*`, `getTimeDuration`, `getPropsWithPrefix`, …) resolved back to a
key, catching literal-only keys that have no constant at all.
3. **Deprecation cross-check** — `DeprecationDelta` tables
(`HdfsConfiguration`, `NfsConfiguration`),
`HdfsClientConfigKeys.DeprecatedKeys`,
and `@Deprecated` annotations. A `@Deprecated` *constant* was not treated
as a deprecated *property* (most such constants merely moved classes).
4. **Skip-list audit** — every `Test*ConfigFields` skip entry re-checked
against the code. Reasons stating intent ("not intended for users",
"purposely hidden") were respected; reasons stating checkable facts
("deprecated", "removed") were verified.
5. **Oracle validation** — before trusting any finding, the extraction was
required to reproduce the green
`TestHdfsConfigFields`/`TestRBFConfigFields`
results with a zero diff; any discrepancy was treated as a scanner bug.
6. **Live verification** — `HdfsConfiguration` was instantiated and its
merged
property set diffed against the xml through the real
`Configuration.loadResource` pipeline, including `${...}` resolution.
**Descriptions policy.** Descriptions are derived from the declaring
constant's Javadoc, surrounding code comments, or direct inspection of the
read site — never invented. Secrets and unset-by-default properties use an
empty `<value>`, matching file convention.
</details>
<details>
<summary><b>Properties deliberately not documented</b> — exclusions and
their justification</summary>
The audit identified further undocumented keys that are deliberately excluded
from this patch. Each exclusion is justified by explicit source-level intent,
by deprecation status, or by demonstrated absence of effect:
| Excluded | Justification |
| --- | --- |
| `dfs.namenode.snapshot.deletion.ordered`,
`dfs.namenode.snapshot.deletion.ordered.gc.period.ms`,
`dfs.namenode.snapshot.trashroot.enabled` | The declaring sources
(`SnapshotManager`, `FSNamesystem`) designate these keys as private
configuration. Documenting them would contradict recorded upstream intent. |
| `dfs.ha.tail-edits.qjm.rpc.max-txns` | Annotated in `QuorumJournalManager`
as not publicly exposed. |
| `dfs.datanode.xceiver.stop.timeout.millis` | Annotated in `DFSConfigKeys`
as hidden and intentionally undocumented. |
| `dfs.datanode.startup`, `dfs.namenode.startup` | Recorded in the skip list
as not intended for users; they carry the startup option internally and are not
operator-settable configuration. |
| `ignore.secure.ports.for.testing`,
`dfs.datanode.enable.fileio.fault.injection`,
`dfs.datanode.duplicate.replica.deletion` | Test-only instrumentation hooks
with no operational function. |
| `dfs.datanode.min.supported.namenode.version`,
`dfs.namenode.min.supported.datanode.version` | Internal version-compatibility
floors; not operator-tunable in any supported scenario. |
| `dfs.corruptfilesreturned.max` | The value is read but never consulted by
any code path. Documenting an ineffective key would misinform operators. The
skip-list comment has been corrected to record this. |
| `dfs.metrics.session-id` | Deprecated alias, registered in
`HdfsClientConfigKeys.DeprecatedKeys`. The skip-list comment has been corrected
to record this. |
| `dfs.namenode.audit.log.async` | Obsolete. The sole read site exists to
warn that the key is no longer in use. |
| `dfs.federation.router.*` (4 keys) | Owned by `hdfs-rbf-default.xml`; a
separate JIRA will be filed. |
</details>
Contains content generated by Anthropic Claude Code.
### How was this patch tested?
* `mvn test -Dtest=TestHdfsConfigFields` on the rebased branch against
current trunk (JDK 17): 4/4 pass, BUILD SUCCESS.
* XML well-formedness and duplicate-name check: clean. (Three pre-existing
duplicate `dfs.journalnode.kerberos.*` entries were noticed upstream;
untouched here as out of scope.)
* Live round-trip: instantiated `HdfsConfiguration` and diffed its merged
property set against the xml through the real resource-loading pipeline;
every non-empty documented value matches the code default, including
`${...}` resolution.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation? *(N/A — no
object-storage changes)*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)? *(N/A — no new
dependencies)*
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? *(N/A)*
### AI Tooling
This contribution was prepared with the assistance of Anthropic Claude Code:
property discovery, classification and description drafting were performed
with its assistance and were reviewed, verified against the source, and
approved by the contributor.
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
