[
https://issues.apache.org/jira/browse/HADOOP-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855142#comment-16855142
]
Erik Krogen commented on HADOOP-16345:
--------------------------------------
Within the constructor for {{FairCallQueue}}, we have:
{code}
// Make this the active source of metrics
MetricsProxy mp = MetricsProxy.getInstance(ns);
mp.setDelegate(this);
{code}
After HADOOP-15481, the constructor for {{MetricsProxy}} looks like:
{code}
private MetricsProxy(String namespace) {
this.namespace = namespace;
MBeans.register(namespace, "FairCallQueue", this);
final String name = namespace + ".FairCallQueue";
DefaultMetricsSystem.instance().register(name, name, this);
}
{code}
Immediately upon creation of the metrics proxy, it is registered with the
metrics system, so its {{getQueueSizes()}} and {{getOverflowedCalls()}} methods
can be called at any time. Currently, those methods assume that {{delegate}} is
non-null, but this is only true after {{setDelegate()}} has been called for the
first time. Since this occurs after registration, there is a window of time
where the metrics system may attempt to read one of the metrics methods, but
the delegate has not yet been set, causing an NPE.
I have attached a simple v000 patch which solves this issue. I don't see any
easy way to test it beyond adding some timing injection logic between the 2nd
and 3rd line in the first code snippet, but that seems overkill for this
situation. I am open to other ideas.
cc [~vagarychen] who helped commit HADOOP-15481.
> Potential NPE when instantiating FairCallQueue metrics
> ------------------------------------------------------
>
> Key: HADOOP-16345
> URL: https://issues.apache.org/jira/browse/HADOOP-16345
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
> Affects Versions: 3.0.4, 3.3.0, 3.2.1, 3.1.3
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Major
> Attachments: HADOOP-16345.000.patch
>
>
> [~ahussein] initially reported the following NullPointerException issue on
> HADOOP-16266:
> {code}
> 2019-05-31 14:32:10,417 ERROR [main] impl.MetricsSourceAdapter
> (MetricsSourceAdapter.java:getMetrics(202)) - Error getting metrics from
> source ipc.8020.FairCallQueue
> java.lang.NullPointerException
> at
> org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getQueueSizes(FairCallQueue.java:382)
> at
> org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getMetrics(FairCallQueue.java:410)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:100)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:73)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
> at
> org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.<init>(FairCallQueue.java:361)
> at
> org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getInstance(FairCallQueue.java:368)
> at org.apache.hadoop.ipc.FairCallQueue.<init>(FairCallQueue.java:114)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.CallQueueManager.createCallQueueInstance(CallQueueManager.java:149)
> at org.apache.hadoop.ipc.CallQueueManager.<init>(CallQueueManager.java:79)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:3059)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:1039)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:427)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:347)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:467)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:803)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:709)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:960)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:933)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1699)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1766)
> 2019-05-31 14:32:10,418 INFO [main] ipc.CallQueueManager
> (CallQueueManager.java:<init>(84)) - Using callQueue: class
> org.apache.hadoop.ipc.FairCallQueue, queueCapacity: 1000, scheduler: class
> org.apache.hadoop.ipc.DecayRpcScheduler, ipcBackoff: true.
> {code}
> It would appera that when HADOOP-15481 added support for FCQ stats to the
> metrics2 system, it opens up the potential for the {{MetricsProxy}} to be
> used _before_ the delegate has been set, causing an NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
