Nopes, that hasn’t been released yet Sent from my iPhone
> On 16-Jun-2022, at 9:25 AM, Leon Xu <[email protected]> wrote: > > > Thanks. I tried setting "ipc.maximum.response.length" to 256 MB but still > getting the same error. > For HADOOP-17914, is there a patched hadoop library that I can try out? Our > cluster is only like 50 nodes, which shouldn't be a problem. > > Thanks > Leon > >> On Wed, Jun 15, 2022 at 8:20 PM Ayush Saxena <[email protected]> wrote: >> The first one: NoSuchMethodException isn't something to worry about, it is >> just irrelevant noise, HADOOP-16453 fixed it in 3.3.0 >> The second one is due to the rpc response length only, if you can patch it >> with HADOOP-17914 and try, it can give you the actual response length as >> well in the exception message. >> The RPC call seems to getNodeReports() from the trace, I think that might >> return a huge response if the cluster size is big... >> >> -Ayush >> >>> On Thu, 16 Jun 2022 at 08:17, Leon Xu <[email protected]> wrote: >>> Thanks for the quick response. Overall I think 128M is pretty big and don't >>> expect that to be the issue. >>> There seems to be two errors: >>> First one is >>> java.lang.NoSuchMethodException: >>> org.apache.hadoop.ipc.RpcException.<init>(java.lang.String) >>> at java.lang.Class.getConstructor0(Class.java:3349) ~[?:?] >>> at java.lang.Class.getConstructor(Class.java:2151) ~[?:?] >>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:830) >>> [hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) >>> [hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) >>> [hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.ipc.Client.call(Client.java:1437) >>> [hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.ipc.Client.call(Client.java:1347) >>> [hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) >>> [hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) >>> [hadoop-common-3.1.0.jar!/:?] >>> at com.sun.proxy.$Proxy188.getClusterNodes(Unknown Source) [?:?] >>> at >>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:319) >>> [hadoop-yarn-common-3.1.0.jar!/:?] >>> at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) >>> ~[?:?] >>> at >>> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> ~[?:?] >>> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) >>> [hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) >>> [hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) >>> [hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) >>> [hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) >>> [hadoop-common-3.1.0.jar!/:?] >>> at com.sun.proxy.$Proxy189.getClusterNodes(Unknown Source) [?:?] >>> at >>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:614) >>> [hadoop-yarn-client-3.1.0.jar!/:?] >>> at >>> org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:44) >>> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >>> at >>> org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:321) >>> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >>> at >>> org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:534) >>> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >>> at >>> org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:458) >>> [flink-yarn_2.12-1.12.7.jar!/:1.12.7] >>> at >>> com.attentivemobile.eventplatformregistration.flinkappmode.YarnServiceImpl.deployFlinkCluster(YarnServiceImpl.java:91) >>> [classes!/:?] >>> at >>> com.attentivemobile.eventplatformregistration.flinkappmode.FlinkServiceImpl.startJob(FlinkServiceImpl.java:79) >>> [classes!/:?] >>> at >>> com.attentivemobile.eventplatformregistration.flinkappmode.api.FlinkDeploymentController.startJob(FlinkDeploymentController.java:29) >>> [classes!/:?] >>> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> ~[?:?] >>> at >>> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> ~[?:?] >>> at >>> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> ~[?:?] >>> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] >>> at >>> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) >>> [tomcat-embed-core-9.0.60.jar!/:4.0.1] >>> at >>> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) >>> [spring-webmvc-5.3.18.jar!/:5.3.18] >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) >>> [tomcat-embed-core-9.0.60.jar!/:4.0.1] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) >>> [tomcat-embed-websocket-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> datadog.trace.instrumentation.springweb.HandlerMappingResourceNameFilter.doFilterInternal(HandlerMappingResourceNameFilter.java:49) >>> [?:?] >>> at >>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) >>> [spring-boot-actuator-2.6.6.jar!/:2.6.6] >>> at >>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >>> [spring-web-5.3.18.jar!/:5.3.18] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at >>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >>> [tomcat-embed-core-9.0.60.jar!/:?] >>> at java.lang.Thread.run(Thread.java:829) [?:?] >>> >>> >>> And the second one is >>> Caused by: java.io.IOException: Failed on local exception: >>> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data >>> length; Host Details : local host is: >>> "registration-service-7c45cb48d8-wl4gm/100.127.6.161"; destination host is: >>> "yarn-dev.xyzmobile.com":8032; >>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:816) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.ipc.Client.call(Client.java:1437) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at org.apache.hadoop.ipc.Client.call(Client.java:1347) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at com.sun.proxy.$Proxy188.getClusterNodes(Unknown Source) ~[?:?] >>> at >>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:319) >>> ~[hadoop-yarn-common-3.1.0.jar!/:?] >>> at jdk.internal.reflect.GeneratedMethodAccessor141.invoke(Unknown Source) >>> ~[?:?] >>> at >>> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> ~[?:?] >>> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) >>> ~[hadoop-common-3.1.0.jar!/:?] >>> at com.sun.proxy.$Proxy189.getClusterNodes(Unknown Source) ~[?:?] >>> at >>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:614) >>> ~[hadoop-yarn-client-3.1.0.jar!/:?] >>> at >>> org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:44) >>> ~[flink-yarn_2.12-1.12.7.jar!/:1.12.7] >>> ...... >>> >>> Thanks >>> Leon >>> >>>> On Wed, Jun 15, 2022 at 7:34 PM Ayush Saxena <[email protected]> wrote: >>>> You didn't paste the entire trace or details about what operation failed, >>>> not very familiar with ALB, seems AWS specific, those guys should be >>>> having some support forum or so, can try your luck there.... >>>> Just going by the exception message. if you have configured everything >>>> properly and the server is indeed sending a genuine big response, so you >>>> can try increase the max response limit via config: >>>> ipc.maximum.response.length >>>> The default is 128 * 1024 * 1024 >>>> -Ayush >>>> >>>>> On Thu, 16 Jun 2022 at 07:35, Leon Xu <[email protected]> wrote: >>>>> Hi Hadoop/Yarn Users, >>>>> >>>>> I am trying to set up AWS ALB(application load balancer) for YARN >>>>> resource managers. I am wondering if anyone has experience on that? >>>>> I am able to connect my yarn client to the YARN RM instance directly. But >>>>> after I set up the ALB and try to connect through the ALB, I am getting >>>>> this error: >>>>> >>>>> java.io.IOException: Failed on local exception: >>>>> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data >>>>> length; >>>>> >>>>> Wondering if anyone has ideas >>>>> >>>>> Thanks >>>>> Leon >>>>>
