Issue: repeated request

[Luca Rea]

Hi,
we have scripts in crontab that fork parallel jobs, these ones call "kinit" and run commands that sometimes fail returning the error below:


15/05/14 06:28:35 ERROR security.UserGroupInformation: PriviledgedActionException as:clabhdfs cause:GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
2015-05-14 06:28:35 SEVERE  org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:311)
	at com.contactlab.connection.ConnectToMetaStore.executePrepararation(ConnectToMetaStore.java:169)
	at com.contactlab.connection.ConnectToMetaStore.main(ConnectToMetaStore.java:67)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:281)
	at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:297)
	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
	at org.apache.hive.service.cli.thrift.TCLIService$Client.send_GetOperationStatus(TCLIService.java:403)
	at org.apache.hive.service.cli.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:395)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:280)
	... 2 more
Caused by: org.apache.http.client.ClientProtocolException
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827)
	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:235)
	... 7 more
Caused by: org.apache.http.HttpException: Unknown exception in doAs
	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.process(HttpKerberosRequestInterceptor.java:69)
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
	at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	... 9 more
Caused by: java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1203)
	at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:67)
	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.process(HttpKerberosRequestInterceptor.java:63)
	... 13 more
Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	... 15 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:111)
	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:79)
	... 18 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:299)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
	... 22 more



In krb5kdc.log of the kdc server I see the following line:

May 14 13:13:15 <hostname> krb5kdc[23330](info): DISPATCH: repeated (retransmitted?) request from <IP of server running scripts> during request processing, dropping repeated request
krb5kdc: The KDC should discard this request - while dispatching (udp)


How can I avoid the issue? Any idea or suggestion?


Regards,
Luca Rea