The first one: NoSuchMethodException isn't something to worry about, it is just irrelevant noise, HADOOP-16453 fixed it in 3.3.0 The second one is due to the rpc response length only, if you can patch it with HADOOP-17914 and try, it can give you the actual response length as well in the exception message. The RPC call seems to getNodeReports() from the trace, I think that might return a huge response if the cluster size is big...
-Ayush On Thu, 16 Jun 2022 at 08:17, Leon Xu <[email protected]> wrote: > Thanks for the quick response. Overall I think 128M is pretty big and > don't expect that to be the issue. > There seems to be two errors: > *First one is* > java.lang.NoSuchMethodException: > org.apache.hadoop.ipc.RpcException.<init>(java.lang.String) > at java.lang.Class.getConstructor0(Class.java:3349) ~[?:?] > at java.lang.Class.getConstructor(Class.java:2151) ~[?:?] > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:830) > [hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > [hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) > [hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.ipc.Client.call(Client.java:1437) > [hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.ipc.Client.call(Client.java:1347) > [hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > [hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > [hadoop-common-3.1.0.jar!/:?] > at com.sun.proxy.$Proxy188.getClusterNodes(Unknown Source) [?:?] > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:319) > [hadoop-yarn-common-3.1.0.jar!/:?] > at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown > Source) ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > [hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > [hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > [hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > [hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) > [hadoop-common-3.1.0.jar!/:?] > at com.sun.proxy.$Proxy189.getClusterNodes(Unknown Source) [?:?] > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:614) > [hadoop-yarn-client-3.1.0.jar!/:?] > at > org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:44) > [flink-yarn_2.12-1.12.7.jar!/:1.12.7] > at > org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:321) > [flink-yarn_2.12-1.12.7.jar!/:1.12.7] > at > org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:534) > [flink-yarn_2.12-1.12.7.jar!/:1.12.7] > at > org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:458) > [flink-yarn_2.12-1.12.7.jar!/:1.12.7] > at > com.attentivemobile.eventplatformregistration.flinkappmode.YarnServiceImpl.deployFlinkCluster(YarnServiceImpl.java:91) > [classes!/:?] > at > com.attentivemobile.eventplatformregistration.flinkappmode.FlinkServiceImpl.startJob(FlinkServiceImpl.java:79) > [classes!/:?] > at > com.attentivemobile.eventplatformregistration.flinkappmode.api.FlinkDeploymentController.startJob(FlinkDeploymentController.java:29) > [classes!/:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) > [tomcat-embed-core-9.0.60.jar!/:4.0.1] > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) > [spring-webmvc-5.3.18.jar!/:5.3.18] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) > [tomcat-embed-core-9.0.60.jar!/:4.0.1] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) > [tomcat-embed-websocket-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > datadog.trace.instrumentation.springweb.HandlerMappingResourceNameFilter.doFilterInternal(HandlerMappingResourceNameFilter.java:49) > [?:?] > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) > [spring-boot-actuator-2.6.6.jar!/:2.6.6] > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > [spring-web-5.3.18.jar!/:5.3.18] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) > [tomcat-embed-core-9.0.60.jar!/:?] > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > [tomcat-embed-core-9.0.60.jar!/:?] > at java.lang.Thread.run(Thread.java:829) [?:?] > > > *And the second one is* > Caused by: java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data > length; Host Details : local host is: > "registration-service-7c45cb48d8-wl4gm/100.127.6.161"; destination host > is: "yarn-dev.xyzmobile.com":8032; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:816) > ~[hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) > ~[hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.ipc.Client.call(Client.java:1437) > ~[hadoop-common-3.1.0.jar!/:?] > at org.apache.hadoop.ipc.Client.call(Client.java:1347) > ~[hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > ~[hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > ~[hadoop-common-3.1.0.jar!/:?] > at com.sun.proxy.$Proxy188.getClusterNodes(Unknown Source) ~[?:?] > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:319) > ~[hadoop-yarn-common-3.1.0.jar!/:?] > at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown > Source) ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > ~[hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > ~[hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > ~[hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > ~[hadoop-common-3.1.0.jar!/:?] > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) > ~[hadoop-common-3.1.0.jar!/:?] > at com.sun.proxy.$Proxy189.getClusterNodes(Unknown Source) ~[?:?] > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:614) > ~[hadoop-yarn-client-3.1.0.jar!/:?] > at > org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:44) > ~[flink-yarn_2.12-1.12.7.jar!/:1.12.7] > ...... > > Thanks > Leon > > On Wed, Jun 15, 2022 at 7:34 PM Ayush Saxena <[email protected]> wrote: > >> You didn't paste the entire trace or details about what operation failed, >> not very familiar with ALB, seems AWS specific, those guys should be having >> some support forum or so, can try your luck there.... >> Just going by the exception message. if you have configured everything >> properly and the server is indeed sending a genuine big response, so you >> can try increase the max response limit via config: >> >> ipc.maximum.response.length >> >> The default is 128 * 1024 * 1024 >> >> -Ayush >> >> >> On Thu, 16 Jun 2022 at 07:35, Leon Xu <[email protected]> wrote: >> >>> Hi Hadoop/Yarn Users, >>> >>> I am trying to set up AWS ALB(application load balancer) for YARN >>> resource managers. I am wondering if anyone has experience on that? >>> I am able to connect my yarn client to the YARN RM instance directly. >>> But after I set up the ALB and try to connect through the ALB, I am getting >>> this error: >>> >>> *java.io.IOException: Failed on local exception: >>> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data >>> length;* >>> >>> Wondering if anyone has ideas >>> >>> Thanks >>> Leon >>> >>> >>>
