Thanks. I tried setting "ipc.maximum.response.length" to 256 MB but still getting the same error. For HADOOP-17914, is there a patched hadoop library that I can try out? Our cluster is only like 50 nodes, which shouldn't be a problem.
Thanks Leon On Wed, Jun 15, 2022 at 8:20 PM Ayush Saxena <[email protected]> wrote: > The first one: NoSuchMethodException isn't something to worry about, it is > just irrelevant noise, HADOOP-16453 fixed it in 3.3.0 > The second one is due to the rpc response length only, if you can patch it > with HADOOP-17914 and try, it can give you the actual response length as > well in the exception message. > The RPC call seems to getNodeReports() from the trace, I think that might > return a huge response if the cluster size is big... > > -Ayush > > On Thu, 16 Jun 2022 at 08:17, Leon Xu <[email protected]> wrote: > >> Thanks for the quick response. Overall I think 128M is pretty big and >> don't expect that to be the issue. >> There seems to be two errors: >> *First one is* >> java.lang.NoSuchMethodException: >> org.apache.hadoop.ipc.RpcException.<init>(java.lang.String) >> at java.lang.Class.getConstructor0(Class.java:3349) ~[?:?] >> at java.lang.Class.getConstructor(Class.java:2151) ~[?:?] >> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:830) >> [hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) >> [hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) >> [hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.ipc.Client.call(Client.java:1437) >> [hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.ipc.Client.call(Client.java:1347) >> [hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) >> [hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) >> [hadoop-common-3.1.0.jar!/:?] >> at com.sun.proxy.$Proxy188.getClusterNodes(Unknown Source) [?:?] >> at >> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:319) >> [hadoop-yarn-common-3.1.0.jar!/:?] >> at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown >> Source) ~[?:?] >> at >> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:?] >> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) >> [hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) >> [hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) >> [hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) >> [hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) >> [hadoop-common-3.1.0.jar!/:?] >> at com.sun.proxy.$Proxy189.getClusterNodes(Unknown Source) [?:?] >> at >> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:614) >> [hadoop-yarn-client-3.1.0.jar!/:?] >> at >> org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:44) >> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >> at >> org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:321) >> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >> at >> org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:534) >> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >> at >> org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:458) >> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >> at >> com.attentivemobile.eventplatformregistration.flinkappmode.YarnServiceImpl.deployFlinkCluster(YarnServiceImpl.java:91) >> [classes!/:?] >> at >> com.attentivemobile.eventplatformregistration.flinkappmode.FlinkServiceImpl.startJob(FlinkServiceImpl.java:79) >> [classes!/:?] >> at >> com.attentivemobile.eventplatformregistration.flinkappmode.api.FlinkDeploymentController.startJob(FlinkDeploymentController.java:29) >> [classes!/:?] >> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[?:?] >> at >> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> ~[?:?] >> at >> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:?] >> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] >> at >> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) >> [tomcat-embed-core-9.0.60.jar!/:4.0.1] >> at >> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) >> [spring-webmvc-5.3.18.jar!/:5.3.18] >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) >> [tomcat-embed-core-9.0.60.jar!/:4.0.1] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) >> [tomcat-embed-websocket-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> datadog.trace.instrumentation.springweb.HandlerMappingResourceNameFilter.doFilterInternal(HandlerMappingResourceNameFilter.java:49) >> [?:?] >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) >> [spring-boot-actuator-2.6.6.jar!/:2.6.6] >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> [spring-web-5.3.18.jar!/:5.3.18] >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at >> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >> [tomcat-embed-core-9.0.60.jar!/:?] >> at java.lang.Thread.run(Thread.java:829) [?:?] >> >> >> *And the second one is* >> Caused by: java.io.IOException: Failed on local exception: >> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data >> length; Host Details : local host is: >> "registration-service-7c45cb48d8-wl4gm/100.127.6.161"; destination host >> is: "yarn-dev.xyzmobile.com":8032; >> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:816) >> ~[hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) >> ~[hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.ipc.Client.call(Client.java:1437) >> ~[hadoop-common-3.1.0.jar!/:?] >> at org.apache.hadoop.ipc.Client.call(Client.java:1347) >> ~[hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) >> ~[hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) >> ~[hadoop-common-3.1.0.jar!/:?] >> at com.sun.proxy.$Proxy188.getClusterNodes(Unknown Source) ~[?:?] >> at >> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:319) >> ~[hadoop-yarn-common-3.1.0.jar!/:?] >> at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown >> Source) ~[?:?] >> at >> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:?] >> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) >> ~[hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) >> ~[hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) >> ~[hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) >> ~[hadoop-common-3.1.0.jar!/:?] >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) >> ~[hadoop-common-3.1.0.jar!/:?] >> at com.sun.proxy.$Proxy189.getClusterNodes(Unknown Source) ~[?:?] >> at >> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:614) >> ~[hadoop-yarn-client-3.1.0.jar!/:?] >> at >> org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:44) >> ~[flink-yarn_2.12-1.12.7.jar!/:1.12.7] >> ...... >> >> Thanks >> Leon >> >> On Wed, Jun 15, 2022 at 7:34 PM Ayush Saxena <[email protected]> wrote: >> >>> You didn't paste the entire trace or details about what operation >>> failed, not very familiar with ALB, seems AWS specific, those guys should >>> be having some support forum or so, can try your luck there.... >>> Just going by the exception message. if you have configured everything >>> properly and the server is indeed sending a genuine big response, so you >>> can try increase the max response limit via config: >>> >>> ipc.maximum.response.length >>> >>> The default is 128 * 1024 * 1024 >>> >>> -Ayush >>> >>> >>> On Thu, 16 Jun 2022 at 07:35, Leon Xu <[email protected]> wrote: >>> >>>> Hi Hadoop/Yarn Users, >>>> >>>> I am trying to set up AWS ALB(application load balancer) for YARN >>>> resource managers. I am wondering if anyone has experience on that? >>>> I am able to connect my yarn client to the YARN RM instance directly. >>>> But after I set up the ALB and try to connect through the ALB, I am getting >>>> this error: >>>> >>>> *java.io.IOException: Failed on local exception: >>>> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data >>>> length;* >>>> >>>> Wondering if anyone has ideas >>>> >>>> Thanks >>>> Leon >>>> >>>> >>>>
