[
https://issues.apache.org/jira/browse/HADOOP-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775618#comment-13775618
]
Sangjin Lee commented on HADOOP-9854:
-------------------------------------
[~jlowe] Actually that's precisely what we ended up doing with scalding to
address this issue:
https://github.com/twitter/scalding/commit/c8d963496cbe3e327359a893ee737cd42367a4a4
And I also cannot think of a clean solution that can prevent this type of
problems, or I would have suggested a patch.
But I feel some discomfort in this in the sense that there is nothing that
stops people from setting map-reduce specific properties using Configuration,
and yet there is this window of opportunity between when Configuration is
initialized and JobConf is initialized where such an action of setting a
property will leave the internal state of Configuration in an inconsistent
state.
Since we fixed this on the scalding side, we're proceeding OK now. However, it
would be great if we could at least clarify this somewhere (javadoc or wiki
or?) that if you're going to use map-reduce properties you must use JobConf
instead of vanilla Configuration when setting the properties.
Thoughts?
> Configuration.set() may be called before all the deprecated keys are
> registered, causing inconsistent state
> -----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9854
> URL: https://issues.apache.org/jira/browse/HADOOP-9854
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.5-alpha
> Reporter: Sangjin Lee
> Priority: Blocker
>
> Currently deprecated keys are registered at various times. Some are
> registered when the Configuration class itself is initialized, but the vast
> majority are registered when the JobConf class is initialized.
> Therefore, it is entirely possible (and does happen) that Configuration.set()
> is called for a key before its deprecation mapping is registered, thus
> leaving the internal state of Configuration in an inconsistent state.
> We actually had this problem occur in real life, causing the set value not to
> be recognized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
