Thanks Ishan, I've raised a JIRA for it.
On 11 January 2016 at 20:17, Ishan Chattopadhyaya wrote:
> Not sure how reliably renewals are taken care of in the context of
> kerberized HDFS, but here's my 10-15 minute analysis.
> Seems to me that the auto renewal thread is not spawned [0]. This relies on
> kinit.
> Not sure if having a login configuration with renewTGT is sufficient (which
> seems to be passed in by default, unless there's a jaas config being
> explicitly passed in with renewTGT=false). As per the last comments from
> Devraj & Owen [1] kinit based logins have worked more reliably.
>
> If you can rule out any setup issues, I suggest you file a JIRA and someone
> who has worked on the HdfsDirectoryFactory would be able to suggest better.
> Thanks,
> Ishan
>
> [0] -
>
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-common/2.7.1/org/apache/hadoop/security/UserGroupInformation.java#UserGroupInformation.spawnAutoRenewalThreadForUserCreds%28%29
>
> [1] - https://issues.apache.org/jira/browse/HADOOP-6656
>
> On Fri, Jan 8, 2016 at 10:21 PM, Andrew Bumstead <
> andrew.bumst...@bigdatapartnership.com> wrote:
>
> > Hello,
> >
> > I have Solr Cloud configured to stores its index files on a Kerberized
> HDFS
> > (I followed documentation at
> > https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS),
> > and
> > have been able to index some documents with the files being written to
> the
> > HDFS as expected. However, it appears that some time after starting, Solr
> > is unable to connect to HDFS as it no longer has a valid Kerberos TGT.
> The
> > time-frame of this occurring is consistent with my default Kerberos
> ticket
> > lifetime of 24 hours, so it appears as though Solr is not renewing its
> > Kerberos ticket upon expiry. A restart of Solr resolves the issue again
> for
> > 24 hours.
> >
> > Is there any configuration I can add to make Solr automatically renew its
> > ticket or is this an issue with Solr?
> >
> > The following is the stack trace I am getting in Solr.
> >
> > java.io.IOException: Failed on local exception: java.io.IOException:
> > Couldn't setup connection for solr/
> sandbox.hortonworks@hortonworks.com
> > to sandbox.hortonworks.com/10.0.2.15:8020; Host Details : local host
> is: "
> > sandbox.hortonworks.com/10.0.2.15"; destination host is: "
> > sandbox.hortonworks.com":8020;
> > at
> org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1472)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> > at
> >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> > at com.sun.proxy.$Proxy10.renewLease(Unknown Source)
> > at
> >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> > at com.sun.proxy.$Proxy11.renewLease(Unknown Source)
> > at
> org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:879)
> > at
> org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
> > at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
> > at
> > org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> > at
> org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.io.IOException: Couldn't setup connection for solr/
> > sandbox.hortonworks@hortonworks.com to
> > sandbox.hortonworks.com/10.0.2.15:8020
> > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:672)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >
