[
https://issues.apache.org/jira/browse/HDFS-17962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yue Wang updated HDFS-17962:
----------------------------
Description:
NNTop stores a RollingWindow for each (metric, user) pair in a
ConcurrentHashMap.
In RollingWindowManager#getTopUsersForMetric(), expired user windows are
removed when getSum(time) returns 0. Previously, the cleanup used an iterator
to remove the map entry, while RollingWindowManager#recordMetric() obtained a
RollingWindow from the map and incremented it outside of any per-key atomic
operation.
This allows the following race:
1. A recordMetric() thread obtains an existing RollingWindow for a user.
2. A snapshot thread observes that the window sum is 0 and removes the
user-to-window mapping.
3. The recordMetric() thread increments the RollingWindow it obtained earlier.
4. The increment is applied to an object that is no longer in the map, so the
metric is lost from subsequent NNTop snapshots.
> NNTop may lose user metrics during expired window cleanup
> ---------------------------------------------------------
>
> Key: HDFS-17962
> URL: https://issues.apache.org/jira/browse/HDFS-17962
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Yue Wang
> Assignee: Yue Wang
> Priority: Major
>
> NNTop stores a RollingWindow for each (metric, user) pair in a
> ConcurrentHashMap.
> In RollingWindowManager#getTopUsersForMetric(), expired user windows are
> removed when getSum(time) returns 0. Previously, the cleanup used an iterator
> to remove the map entry, while RollingWindowManager#recordMetric() obtained a
> RollingWindow from the map and incremented it outside of any per-key atomic
> operation.
> This allows the following race:
> 1. A recordMetric() thread obtains an existing RollingWindow for a user.
> 2. A snapshot thread observes that the window sum is 0 and removes the
> user-to-window mapping.
> 3. The recordMetric() thread increments the RollingWindow it obtained earlier.
> 4. The increment is applied to an object that is no longer in the map, so the
> metric is lost from subsequent NNTop snapshots.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
