[
https://issues.apache.org/jira/browse/HADOOP-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734257#comment-13734257
]
Sangjin Lee commented on HADOOP-9854:
-------------------------------------
One easy way to recreate this problem is as follows. Do
{code}
hadoop jar some_jar -D mapred.reduce.tasks=100 ...
{code}
where my class uses ToolRunner to do its task:
{code}
public static void main(String[] args) {
ToolRunner.run(new Configuration(), new MyClass(), args);
}
{code}
ToolRunner will start using GenericOptionsParser to parse the -D overrides, and
it will call Configuration.set() to set the overrides. However, in this case
the JobConf class is still not initialized. Therefore, the
"mapred.reduce.tasks" key is not registered as deprecated. As a result, the
Configuration will have "mapred.reduce.tasks" set to 100, but
"mapreduce.job.reduces" (the new key) still set to 1 (the default).
The configuration will be left in this inconsistent state until the JobConf
class is initialized and a get() call for either of these keys is made.
In our particular scenario, a map was created out of this Configuration using
Configuration.iterator(), and eventually caused it to lose the overridden value.
I think this inconsistent state has many ways to cause havoc, and I believe
this is a major bug.
> Configuration.set() may be called before all the deprecated keys are
> registered, causing inconsistent state
> -----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9854
> URL: https://issues.apache.org/jira/browse/HADOOP-9854
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.5-alpha
> Reporter: Sangjin Lee
>
> Currently deprecated keys are registered at various times. Some are
> registered when the Configuration class itself is initialized, but the vast
> majority are registered when the JobConf class is initialized.
> Therefore, it is entirely possible (and does happen) that Configuration.set()
> is called for a key before its deprecation mapping is registered, thus
> leaving the internal state of Configuration in an inconsistent state.
> We actually had this problem occur in real life, causing the set value not to
> be recognized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
