Thanks Gaurav. For my case, I called the HDFS API to write the reducer result into HDFS directly, not using Spark.
2016-10-17 23:24 GMT+08:00 Gaurav Kumar <[email protected]>: > Hi, > > Please also check for coalesced RDD. I encountered the same error while > writing a coalesced rdd/df to HDFS. If this is the case, please use > repartition instead. > > Sent from OnePlus 3 > > Thanks & Regards, > Gaurav Kumar > > On Oct 17, 2016 11:22 AM, "Zhang Jianfeng" <[email protected]> wrote: > > Thanks Rakesh for your kind help. Actually during the job only one > reducer result file (for example part-r-2) had this error, other reducers > worked well. > > Best Regards, > Jian Feng > > 2016-10-17 11:49 GMT+08:00 Rakesh Radhakrishnan <[email protected]>: > >> Hi Jian Feng, >> >> Could you please check your code and see any possibilities of >> simultaneous access to the same file. Mostly this situation happens when >> multiple clients tries to access the same file. >> >> Code Reference:- https://github.com/apache/hadoop/blob/branch-2.2 >> /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/ha >> doop/hdfs/server/namenode/FSNamesystem.java#L2737 >> >> Best Regards, >> Rakesh >> Intel >> >> On Mon, Oct 17, 2016 at 7:16 AM, Zhang Jianfeng <[email protected]> >> wrote: >> >>> Hi , >>> >>> I hit an wired error. On our hadoop cluster (2.2.0), occasionally a >>> LeaseExpiredException is thrown. >>> >>> The stacktrace is as below: >>> >>> >>> *org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): >>> No lease on /user/biadmin/analytic‐root/SX5XPWPPDPQH/.executions/.at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2737)* >>> >>> *at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2801)* >>> >>> *at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2783)* >>> >>> *at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.com >>> <http://namenode.NameNodeRpcServer.com>plete(NameNodeRpcServer.java:611)* >>> >>> *at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:428)* >>> >>> *at >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59586)* >>> >>> *at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)* >>> >>> *at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)* >>> >>> *at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)* >>> >>> *at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)* >>> >>> *at >>> java.security.AccessController.doPrivileged(AccessController.java:310)* >>> >>> *at javax.security.auth.Subject.do >>> <http://javax.security.auth.Subject.do>As(Subject.java:573)* >>> >>> *at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)* >>> >>> *at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)* >>> >>> *at org.apache.hadoop.ipc.Client.call(Client.java:1347)* >>> >>> *at org.apache.hadoop.ipc.Client.call(Client.java:1300)* >>> >>> *at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)* >>> >>> *at $Proxy7.complete(Unknown Source)* >>> >>> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* >>> >>> *at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)* >>> >>> *at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)* >>> >>> at java.lang.reflect.Method.invoke(Method.java:611) >>> >>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >>> od(RetryInvocationHandler.java:186) >>> >>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >>> ryInvocationHandler.java:102) >>> >>> at $Proxy7.complete(Unknown Source) >>> >>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran >>> slatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:371) >>> >>> at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutpu >>> tStream.java:1894) >>> >>> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream >>> .java:1881) >>> >>> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close( >>> FSDataOutputStream.java:71) >>> >>> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputSt >>> ream.java:104) >>> >>> at java.io.FilterOutputStream.close(FilterOutputStream.java:154) >>> >>> Any help will be appreciated! >>> >>> -- >>> Best Regards, >>> Jian Feng >>> >> >> > > > -- > Best Regards, > Jian Feng > > > -- Best Regards, Jian Feng
