[
https://issues.apache.org/jira/browse/HDFS-17974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111763#comment-18111763
]
ASF GitHub Bot commented on HDFS-17974:
---------------------------------------
rdhabalia opened a new pull request, #8715:
URL: https://github.com/apache/hadoop/pull/8715
### Description of PR
Multi-tenant HDFS clusters often need to pin a subset of data, identified by
its HDFS path, to a dedicated pool of DataNodes. This prevents noisy or
isolation-sensitive tenants from sharing read/write capacity with the rest of
the cluster.
The default `BlockPlacementPolicy` has no notion of path-to-DataNode
affinity, so operators typically resort to separate clusters or brittle
rack-based workarounds.
### Approach
Introduce a pluggable `DatanodeAffinityManager` abstraction, resolved via
reflection from:
```properties
dfs.datanode.affinity.manager.classname
```
An affinity group maps:
* **Source path regex** → **DataNode hostname regex**
On refresh, the manager resolves each DataNode hostname regex against the
live cluster and builds a restricted `NetworkTopology` for each group
containing only that group's eligible DataNodes.
### Components
* **`DatanodeManager`**
* Instantiates the configured affinity manager.
* Removes affinity DataNodes from the default `NetworkTopology` as they
register, ensuring the default placement policy cannot select them.
* Prunes DataNodes from affinity structures when they are decommissioned
or removed.
* Re-triggers an affinity refresh on `dfsadmin -refreshNodes`.
* **`BlockManager`**
* Maintains one `BlockPlacementPolicy` per affinity group.
* Each policy is backed by the group's restricted topology.
* Routes initial block placement (`chooseTarget4NewBlock`) through the
matching group's policy.
* Routes mid-write pipeline recovery (`chooseTarget4AdditionalDatanode`)
through the matching group's policy.
* Ensures replacement nodes for failed pipeline members remain within the
isolated pool.
* Uses the default placement policy when no affinity group matches.
* **`FileDatanodeAffinityManager`**
* Built-in implementation that loads affinity groups from a JSON file.
* Configuration:
```properties
dfs.datanode.affinity.file.path
```
* Reloads the configuration on `dfsadmin -refreshNodes`.
* **`hdfs fsck`**
* Supports:
```bash
hdfs fsck <path> -favored-nodes
```
* Prints the affinity-resolved DataNodes for a path as a dry run.
* Does not require the path to exist.
## Isolation vs. Availability
If an affinity group cannot place all requested replicas because of:
* an under-provisioned group,
* all group DataNodes being unavailable, or
* a `datanodesRegex` matching no nodes,
the default behavior falls back to the shared pool so the write can still
succeed. Spillover is logged at `WARN`.
Strict isolation can be enabled with:
```properties
dfs.namenode.affinity.strict.isolation.enabled=true
```
In strict mode, initial placement fails closed unless **all requested
replicas** can be placed inside the affinity group.
This is important because placing only `minReplication` replicas in-group
would allow the redundancy monitor to repair the remaining replicas onto
shared-pool DataNodes, leaking data outside the isolated group.
## Scope and Limitations
Affinity currently governs:
* Initial block placement.
* In-pipeline recovery.
Background replication and EC reconstruction still use the default placement
policy.
Making these paths affinity-aware would require a reverse block-to-path
lookup inside a lock-sensitive hot loop and is intentionally left as follow-up
work. This is documented in `BlockManager`.
### DataNode Regex Matching
DataNode regex patterns are matched using `find()` against each node's:
```text
hostname:port
```
Therefore, a fully anchored pattern must account for the trailing port.
For example:
```regex
^dn-tenant-a[0-9]+\.example\.com(:\d+)?$
```
## Configuration
| Property | Description
| Default |
| ----------------------------------------------
> HDFS DataNode Affinity for Tenant Isolation
> -------------------------------------------
>
> Key: HDFS-17974
> URL: https://issues.apache.org/jira/browse/HDFS-17974
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Reporter: Rajan Dhabalia
> Priority: Major
>
> h2. Summary
> Introduce a pluggable, regex-based DataNode affinity mechanism that maps HDFS
> paths to dedicated DataNode pools.
> This enables *tenant/dataset-level storage and I/O isolation* without
> requiring separate HDFS clusters or rack-based workarounds.
> h2. Motivation
> The current HDFS block placement policy has no native path-based mechanism to
> restrict data to a specific DataNode pool. This can cause noisy-neighbor
> interference for isolation-sensitive workloads.
> The feature provides:
> * Path-to-DataNode-pool mapping
> * Tenant/dataset I/O isolation
> * More predictable performance
> * Independent capacity planning
> * Runtime configuration updates without NameNode restart
> h2. Design
> Introduce a pluggable `DatanodeAffinityManager` configured through:
> {code:java}
> dfs.datanode.affinity.manager.classname
> {code}
> An affinity rule maps:
> {code:java}
> HDFS path regex -> DataNode hostname regex
> {code}
> Example:
> {code:java}
> /data/tenantA/.* -> dn-tenantA-.*
> /data/tenantB/.* -> dn-tenantB-.*
> {code}
> The manager resolves the hostname regex against registered DataNodes and
> builds a restricted `NetworkTopology` containing only eligible DataNodes for
> each affinity group.
> h2. NameNode Integration
> *DatanodeManager*
> * Identifies DataNodes belonging to affinity pools during registration.
> * Removes affinity-only DataNodes from the default placement topology.
> * Prevents non-affinity workloads from using dedicated DataNodes.
> * Refreshes affinity state through `hdfs dfsadmin -refreshNodes`.
> *BlockManager*
> For each block placement request:
> # Match the source path against configured affinity groups.
> # Use the group's `BlockPlacementPolicy` if matched.
> # Select targets from the group's restricted topology.
> # Fall back to the default placement policy when no group matches.
> This avoids large exclusion lists on the placement hot path.
> h2. Pluggable Implementation
> `DatanodeAffinityManager` is an abstraction that allows different affinity
> sources without changing block-placement logic.
> Built-in implementation:
> {code:java}
> FileDatanodeAffinityManager
> {code}
> It loads affinity rules from a JSON configuration file and reloads them
> through `dfsadmin -refreshNodes`.
> h2. Configuration
> ||Property||Default||Description||
> |`dfs.datanode.affinity.manager.classname`|Empty|`DatanodeAffinityManager`
> implementation. Empty disables the feature.|
> |`dfs.datanode.affinity.file.path`|Empty|JSON affinity configuration used by
> `FileDatanodeAffinityManager`.|
> Example:
> {code:java}
> /data/tenantA/.* -> dn-tenantA-.*
> /data/tenantB/.* -> dn-tenantB-.*
> {code}
> h2. Operational Visibility
> Add:
> {code:java}
> hdfs fsck <path> -favored-nodes
> {code}
> to display the DataNodes resolved for a path.
> Example:
> {code:java}
> hdfs fsck /data/tenantA -favored-nodes
> {code}
> This provides a dry-run mechanism to validate affinity configuration before
> enabling it.
> h2. Runtime Refresh
> Affinity configuration and DataNode membership can be updated using:
> {code:java}
> hdfs dfsadmin -refreshNodes
> {code}
> No NameNode restart is required.
> Supported changes include:
> * Adding/removing DataNodes from an affinity pool
> * Updating path-to-pool mappings
> * Changing hostname matching rules
> h2. Benefits
> * *Tenant Isolation:* Affinity-enabled data is placed only on its dedicated
> DataNode pool.
> * *Predictable Performance:* Reduces noisy-neighbor impact and isolates I/O
> capacity.
> * *Default Pool Protection:* Dedicated DataNodes are excluded from normal
> placement.
> * *Efficient Placement:* Restricted topology limits placement to the
> relevant pool.
> * *Runtime Configuration:* Changes take effect without NameNode restart.
> h2. Expected Impact
> ||Dimension||Expected Impact||
> |Cross-tenant interference|Eliminated within the isolated DataNode pool|
> |Tail latency|Reduced when contention exists|
> |Placement scope|Limited to affinity pool|
> |Default-pool throughput|No expected regression|
> |Configuration changes|Runtime refresh|
> The primary goal is {*}isolation and performance predictability{*}, not raw
> throughput improvement.
> h2. Implementation
> Key changes:
> * `DatanodeAffinityManager` abstraction
> * `FileDatanodeAffinityManager`
> * Path-regex to DataNode-hostname-regex mapping
> * Per-group restricted `NetworkTopology`
> * `DatanodeManager` topology integration
> * Per-group `BlockPlacementPolicy` in `BlockManager`
> * Affinity-aware `chooseTarget4NewBlock`
> * Runtime refresh via `dfsadmin -refreshNodes`
> * `hdfs fsck -favored-nodes` validation
> h2. Backward Compatibility
> * Disabled by default.
> * Existing block placement behavior is unchanged when affinity is not
> configured.
> * Non-matching paths continue to use the default placement policy.
> * No HDFS client changes are required.
> * Can be enabled selectively for specific tenants or datasets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
