[
https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241805#comment-15241805
]
Colin Patrick McCabe commented on HADOOP-12975:
-----------------------------------------------
bq. But a percentage is chosen as it makes the jitter scale with anyone who
changes du periods. If it's a set number then someone with a refresh period of
days won't get any benefit from the jitter.
Hmm. It seems like a fixed amount of jitter still provides a benefit, even to
someone with a longer refresh interval. Let's say my refresh period is 7 days.
At the end of that, I would still appreciate having my DU processes launch at
slightly different times on the 7th day, rather than all launching at once.
My concern with varying based on a percentage is that there will be enormous
variations in how long different volumes go between DU operations, when longer
refresh intervals are in use. Like if I have a 7 day period and one volume
refreshes after 3.5 days, and the other ways for the full 7 days, that's quite
a variation. Similarly, if our period is short -- like 1 hour-- having some
datanodes refresh after only 30 minutes seems unwelcome. That's why I
suggested a fixed jitter amount, to be configured by the sysadmin.
I don't feel very strongly about this, though, so if you want to make it
percentage-based, that's fine too. As long as its configurable and the
defaults are reasonable.
> Add jitter to CachingGetSpaceUsed's thread
> ------------------------------------------
>
> Key: HADOOP-12975
> URL: https://issues.apache.org/jira/browse/HADOOP-12975
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: 2.9.0
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch,
> HADOOP-12975v2.patch
>
>
> Running DU across lots of disks is very expensive and running all of the
> processes at the same time creates a noticeable IO spike. We should add some
> jitter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
