Hi Akash,
In general "GSSException: No valid credentials provided" means you don’t
have valid Kerberos credentials. I'm suspecting some issues related to
spnego, could you please revisit all of your kerb related configurations,
probably you can start from the below configuration. Please share
*-site.xml configurations of JN and NNs. Also, please check any unexpected
exceptions in KDC server logs.
I've filtered out "REQUEST /getJournal on org.mortbay.jetty.HttpConnection"
in your "qjm.log" log file and I could see this has came immediately after
your restart, few has succeeded and few others failed with this exception.
2016-08-19 10:34:14,345 DEBUG org.mortbay.log: RESPONSE /getJournal 401
2016-08-19 10:34:14,374 DEBUG org.mortbay.log: RESPONSE /getJournal 403
2016-08-19 10:34:14,382 DEBUG org.mortbay.log: RESPONSE /getJournal 401
2016-08-19 10:34:14,398 DEBUG org.mortbay.log: RESPONSE /getJournal 403
2016-08-19 10:34:49,679 DEBUG org.mortbay.log: RESPONSE /getJournal 401
<property>
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>
<value></value>
<description>
The server principal used by the JournalNode HTTP Server for
SPNEGO authentication when Kerberos security is enabled. This is
typically set to HTTP/[email protected]. The SPNEGO server principal
begins with the prefix HTTP/ by convention.
If the value is '*', the web server will attempt to login with
every principal specified in the keytab file
dfs.web.authentication.kerberos.keytab.
For most deployments this can be set to
${dfs.web.authentication.kerberos.principal}
i.e use the value of dfs.web.authentication.kerberos.principal.
</description>
</property>
Rakesh,
Intel
On Fri, Aug 19, 2016 at 4:15 PM, Akash Mishra <[email protected]>
wrote:
> Hi *,
>
> I am trying to run Hadoop cluster [ 2.7.1] in Secure mode. In my cluster
> Namenode is failing while restart with
>
> 2016-08-19 10:34:49,754 DEBUG org.apache.hadoop.security.
> authentication.client.KerberosAuthenticator: Using fallback authenticator
> sequence.
> 2016-08-19 10:34:49,774 DEBUG org.apache.hadoop.security.UserGroupInformation:
> PrivilegedActionException as:hdfs/[email protected]
> (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.
> authentication.client.AuthenticationException: Authentication failed,
> status: 403, message: GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos credentails)
> 2016-08-19 10:34:49,775 ERROR
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream:
> caught exception initializing http://hadoopdev1:8480/
> getJournal?jid=hadoopdev&segmentTxId=2275460&storageInfo=-63%3A1455401088%
> 3A1444912570574%3ACID-f748dfef-c174-4d19-8d18-43b74552c8e6
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, status: 403, message: GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos
> credentails)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1657)
> at org.apache.hadoop.security.SecurityUtil.doAsUser(
> SecurityUtil.java:448)
> at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(
> SecurityUtil.java:442)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:
> 455)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream.init(EditLogFileInputStream.java:141)
> at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.
> nextOpImpl(EditLogFileInputStream.java:192)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> readOp(EditLogInputStream.java:85)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> skipUntil(EditLogInputStream.java:151)
> at org.apache.hadoop.hdfs.server.namenode.
> RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> readOp(EditLogInputStream.java:85)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> skipUntil(EditLogInputStream.java:151)
> at org.apache.hadoop.hdfs.server.namenode.
> RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> readOp(EditLogInputStream.java:85)
>
>
> I am using MIT 5 Kerberos. I am able to successfully kinit using keytab
> file. I have DEBUG log enabled and attaching log from Namenode [nn.log]
> and one of QJM [ qjm.log]
>
>
>
> Thanks.
>
>
>
>
> --
>
> Regards,
> Akash Mishra.
>
>
> "It's not our abilities that make us, but our decisions."--Albus Dumbledore
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
