[
https://issues.apache.org/jira/browse/HADOOP-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775613#comment-13775613
]
Sangjin Lee commented on HADOOP-9854:
-------------------------------------
[~acmurthy], the sequence of events is a little involved. The specific
combination is scalding and cascading. The problem starts when the user sets an
old key. Let's say the user set the old key to 100 (where the default is 1).
Scalding runs the task with the ToolRunner but uses Configuration. In its tool,
scalding converts the configuration items from the Configuration into a
Map<String,String> and passes it to cascading. In cascading, using a Map as
opposed to a Configuration or JobConf is a standard mode of operation. When
scalding creates the Map, it uses Configuration.iterator() to get the values
out of the Configuration. So up until this point, the deprecated keys from
JobConf have not been registered (because JobConf was not initialized), and
thus the Map now contains inconsistent values for any map-reduce-related config
params. At this point, the Map has 100 under the old key, and 1 under the new
key.
Once cascading receives this Map, it eventually constructs a JobConf instance
using the Map. This triggers all the deprecated keys for JobConf to be
registered. And when values from Map are set onto JobConf via set(), the new
value from the Map wins (which is still the default value, not the
user-overridden value), thus wiping out the user override. The JobConf instance
now has 1 for both the new and the old keys.
> Configuration.set() may be called before all the deprecated keys are
> registered, causing inconsistent state
> -----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9854
> URL: https://issues.apache.org/jira/browse/HADOOP-9854
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.5-alpha
> Reporter: Sangjin Lee
> Priority: Blocker
>
> Currently deprecated keys are registered at various times. Some are
> registered when the Configuration class itself is initialized, but the vast
> majority are registered when the JobConf class is initialized.
> Therefore, it is entirely possible (and does happen) that Configuration.set()
> is called for a key before its deprecation mapping is registered, thus
> leaving the internal state of Configuration in an inconsistent state.
> We actually had this problem occur in real life, causing the set value not to
> be recognized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
