Andrew Sherman created HADOOP-14701:
---------------------------------------
Summary: Configuration can log misleading warnings about an
attempt to override final parameter
Key: HADOOP-14701
URL: https://issues.apache.org/jira/browse/HADOOP-14701
Project: Hadoop Common
Issue Type: Bug
Components: conf
Reporter: Andrew Sherman
The class Configuration (in hadoop-common-project) is used to hold
configuration properties.
The method loadResources() is used to load properties from a list of default
resources into a parameter called “properties” (which happens to always be the
field called “properties”).
This is done by calling the method loadResource() which is used to load
properties from a single resource into a parameter called “properties” (which
happens to always be the field called “properties”).
This is done by calling loadProperty() which is used to load a single property
into a parameter called “properties” (which is sometimes, but not always, the
field called “properties”).
In loadProperty() there is some logic for dealing with properties that are
marked as final.
(1) a final property is not set into the “properties” parameter.
(2) a warning is printed for a final property if the value of the property with
the same name in the “properties” parameter is not equal to the new value, and
the property name is in the finalParameters field.
When loadResource() is reading properties from a stream it creates a temporary
Properties object that it populates by calling loadProperty(). This is the case
where loadProperty() has a parameter called “properties” which is not the
same as the field called “properties”).
When loadProperty() is called on a final property then because of (1) it is not
set into the “properties” parameter. But sometimes the property name is already
in the finalParameters field and now (2) the warning is triggered because the
property is not present in the “properties” parameter (although it IS present
in the “properties” field). The warning is misleading and confusing.
h3. Possible Fix
One way of looking at the problem is that loadProperty is using the
finalParameters field in combination with the “properties” parameter. Change
the code to only generate a warning for a final property if the value of the
property with the same name in the “properties” field is not equal to the new
value, and the property name is in the finalParameters field.
I’ve made a patch containing this tiny change.
Note that I have done very minimal testing of this fix
h3. Example warning:
2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration:
[HiveServer2-Background-Pool: Thread-77]:
hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
