krb5.conf implementation question

Fri Apr 10 17:40:35 EDT 2020

Hi all,

I'm not developing an application using the MIT Kerberos libraries, but am implementing a third-party application that uses the libraries.  So, I hope it's ok to be emailing this list.  I'm hitting intermittent authentication failures and looking for guidance on how they might be avoided.

The environment is using Active Directory 2012 R2 as the KDC.  The krb5.conf file is configured with a list of individual KDC machines in the realms section.  It appears that the libraries use the first KDC in the list as long as it is responding.  The problem happens when the first KDC in the list is rebooted.  There is a short window where the KDC will respond with KDC_ERR_C_PRINCIPAL_UNKNOWN or KDC_ERR_S_PRINCIPAL_UNKNOWN (probably depending on type of request) as it is shutting down.  Microsoft has an article about this type of behavior for 2008 SP2 https://support.microsoft.com/en-us/help/982801/a-domain-controller-returns-the-no-such-user-0xc0000064-status-code-or, but we're seeing the same symptoms with 2008 R2, 2012 R2 and 2019.

When we enable KRB5_TRACE we see up to 4 request attempt being made.  I don't know if the multiple tries are initiated by the MIT code or by the application code.  In many cases the 4 tries will happen within the window that the AD server is responding with the error.  Usually all 4 attempts will be sent to the same KDC within about a half-second window.  This is in a short enough timeframe that we can get 4 errors from the KDC as it shuts down.

Here are the questions I had around this:

  *   If the MIT code is making the 4 request attempts, is there any way (krb5.conf configuration, env variables, etc.) that we could force each retry to use a different KDC entry in the krb5.conf.  This would move the retries away from the server that is rebooting and the first of those should get a good response.
  *   As mentioned above we are listing out the individual KDC machines, would it be better to set up the krb5.conf in a different way, perhaps using DNS to find the KDCs?
  *   I saw a mention in one email about setting master_kdc, that suggested if there is an error a subsequent request might be sent to the master_kdc.  However the documentation says this only happens on an invalid password.  Is that the case or is it worth setting master_kdc?  We don't do that currently.

Thanks for any help with this.  I'm pasting the KRB5_TRACE output from the application client, in case that helps demonstrate the problem.  In this case there were two KDCs and the first one in the krb5.conf was rebooted.  We never switched to the second one and the authentication failed after sending 4 requests within a second to the rebooting KDC.

Thanks,

Kieran.

[81339] 1574879653.119817: Getting credentials user at EXAMPLE.COM -> *** SPN ***@EXAMPLE.COM using ccache FILE:/tmp/user.ccache
[81339] 1574879653.120309: Retrieving user at EXAMPLE.COM -> *** SPN ***@EXAMPLE.COM from FILE:/tmp/user.ccache with result: 0/Success
[81339] 1574879653.120799: Retrieving user at EXAMPLE.COM -> krbtgt/EXAMPLE.COM at EXAMPLE.COM from FILE:/tmp/user.ccache with result: 0/Success
[81339] 1574879653.120825: Get cred via TGT krbtgt/EXAMPLE.COM at EXAMPLE.COM after requesting krbtgt/EXAMPLE.COM at EXAMPLE.COM (canonicalize off)
[81339] 1574879653.120886: Generated subkey for TGS request: aes256-cts/2A2B
[81339] 1574879653.120983: etypes requested in TGS request: aes256-cts
[81339] 1574879653.121286: Encoding request body and padata into FAST request
[81339] 1574879653.121423: Sending request (1831 bytes) to EXAMPLE.COM
[81339] 1574879653.121515: Resolving hostname win2012-01.EXAMPLE.COM
[81339] 1574879653.122060: Initiating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.131869: Sending TCP request to stream 192.168.240.27:88
[81339] 1574879653.135706: Received answer (357 bytes) from stream 192.168.240.27:88
[81339] 1574879653.135729: Terminating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.135843: Response was not from master KDC
[81339] 1574879653.135872: Decoding FAST response
[81339] 1574879653.136000: Decoding FAST response
[81339] 1574879653.136060: Got cred; -1765328378/Client not found in Kerberos database
[81339] 1574879653.136125: Get cred via TGT krbtgt/EXAMPLE.COM at EXAMPLE.COM after requesting krbtgt/EXAMPLE.COM at EXAMPLE.COM (canonicalize off)
[81339] 1574879653.136150: Generated subkey for TGS request: aes256-cts/5D5B
[81339] 1574879653.136187: etypes requested in TGS request: aes256-cts
[81339] 1574879653.136253: Encoding request body and padata into FAST request
[81339] 1574879653.136313: Sending request (1831 bytes) to EXAMPLE.COM
[81339] 1574879653.136324: Resolving hostname win2012-01.EXAMPLE.COM
[81339] 1574879653.136909: Initiating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.139206: Sending TCP request to stream 192.168.240.27:88
[81339] 1574879653.142052: Received answer (357 bytes) from stream 192.168.240.27:88
[81339] 1574879653.142068: Terminating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.142299: Response was not from master KDC
[81339] 1574879653.142367: Decoding FAST response
[81339] 1574879653.142772: Decoding FAST response
[81339] 1574879653.142869: Got cred; -1765328378/Client not found in Kerberos database
[81339] 1574879653.143135: Creating authenticator for user at EXAMPLE.COM -> *** SPN ***@EXAMPLE.COM, seqnum 826984930, subkey aes256-cts/2647, session key aes256-cts/A880
[81339] 1574879653.269412: Read AP-REP, time 1574879653.143176, subkey aes256-cts/71A7, seqnum 749727685
[81339] 1574879653.455353: Getting credentials user at EXAMPLE.COM -> *** SPN ***@EXAMPLE.COM using ccache FILE:/tmp/user.ccache
[81339] 1574879653.455624: Retrieving user at EXAMPLE.COM -> *** SPN ***@EXAMPLE.COM from FILE:/tmp/user.ccache with result: 0/Success
[81339] 1574879653.456055: Retrieving user at EXAMPLE.COM -> krbtgt/EXAMPLE.COM at EXAMPLE.COM from FILE:/tmp/user.ccache with result: 0/Success
[81339] 1574879653.456079: Get cred via TGT krbtgt/EXAMPLE.COM at EXAMPLE.COM after requesting krbtgt/EXAMPLE.COM at EXAMPLE.COM (canonicalize off)
[81339] 1574879653.456119: Generated subkey for TGS request: aes256-cts/9C28
[81339] 1574879653.456154: etypes requested in TGS request: aes256-cts
[81339] 1574879653.456225: Encoding request body and padata into FAST request
[81339] 1574879653.456279: Sending request (1831 bytes) to EXAMPLE.COM
[81339] 1574879653.456301: Resolving hostname win2012-01.EXAMPLE.COM
[81339] 1574879653.456560: Initiating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.457220: Sending TCP request to stream 192.168.240.27:88
[81339] 1574879653.458621: Received answer (357 bytes) from stream 192.168.240.27:88
[81339] 1574879653.458641: Terminating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.458764: Response was not from master KDC
[81339] 1574879653.459100: Decoding FAST response
[81339] 1574879653.459205: Decoding FAST response
[81339] 1574879653.459243: Got cred; -1765328378/Client not found in Kerberos database
[81339] 1574879653.459252: Get cred via TGT krbtgt/EXAMPLE.COM at EXAMPLE.COM after requesting krbtgt/EXAMPLE.COM at EXAMPLE.COM (canonicalize off)
[81339] 1574879653.459266: Generated subkey for TGS request: aes256-cts/65FB
[81339] 1574879653.459292: etypes requested in TGS request: aes256-cts
[81339] 1574879653.459378: Encoding request body and padata into FAST request
[81339] 1574879653.459456: Sending request (1831 bytes) to EXAMPLE.COM
[81339] 1574879653.459468: Resolving hostname win2012-01.EXAMPLE.COM
[81339] 1574879653.459726: Initiating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.460139: Sending TCP request to stream 192.168.240.27:88
[81339] 1574879653.461270: Received answer (357 bytes) from stream 192.168.240.27:88
[81339] 1574879653.461290: Terminating TCP connection to stream 192.168.240.27:88
[81339] 1574879653.461410: Response was not from master KDC
[81339] 1574879653.461426: Decoding FAST response
[81339] 1574879653.461461: Decoding FAST response
[81339] 1574879653.461487: Got cred; -1765328378/Client not found in Kerberos database
[81339] 1574879653.461607: Creating authenticator for user at EXAMPLE.COM -> *** SPN ***@EXAMPLE.COM, seqnum 1054877856, subkey aes256-cts/863F, session key aes256-cts/A880
[81339] 1574879653.535150: Read AP-REP, time 1574879653.461618, subkey aes256-cts/8C75, seqnum 337205966