[
https://issues.apache.org/jira/browse/HADOOP-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263993#comment-13263993
]
Tom White commented on HADOOP-8328:
-----------------------------------
Task's updateCounters() method uses FileSystem.getAllStatistics() to get all
the statistics for all filesystems. However, when there are two Statistics
objects for the 'file' scheme, it will use the first one, whichever that
happens to be (it is unstable). MapReduce uses the RawLocalFileSystem to read
and write local files (e.g. when merging), so if the Statistics object for
LocalFileSystem happens to come first out of the Map, then the counters will be
too low.
The quick fix would be to change LocalFileSystem's initialize method to create
a single Statistics object - for RawLocalFileSystem - to bring the behaviour in
line with branch-1 (i.e. pre-HADOOP-8013). There might be a better longer-term
fix though.
> Duplicate FileSystem Statistics object for 'file' scheme
> --------------------------------------------------------
>
> Key: HADOOP-8328
> URL: https://issues.apache.org/jira/browse/HADOOP-8328
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Tom White
>
> Because of a change in HADOOP-8013, there are duplicate Statistics objects in
> FileSystem's statistics table: one for LocalFileSystem and one for
> RawLocalFileSystem. This causes MapReduce local file system counters to be
> incorrect some of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
