help with OTP

Wed Apr 26 11:12:15 EDT 2023

Hi Ken, Greg, and BuzzSaw,

On Tue, Apr 25, 2023 at 7:02 PM Ken Hornstein <kenh at cmf.nrl.navy.mil> wrote:
>
> >Making progress... but still need some pointers.
> >[...]
>
> Remember when I said setting up PKINIT is about as much fun as getting a
> punch in the face from John Cena?  Well, you're about to discover what
> I mean by that.

Ha. Yup. Isn't that all IT though?

> First, there's about 500x ways for PKINIT to go wrong, and when it does
> go wrong 99% of the time you fall back to a password so it's hard to
> figure out exactly what failed.  I work with a large PKINIT deployment
> that uses smartcards on the client side, so I feel I can speak with
> some authority here.  But, some pointers to get you going.
>
> - You can use the KRB5_TRACE environment variable (on both the client
>   and server) to figure out if PKINIT was even attempted.  Do something
>   like:
>
>   env KRB5_TRACE=/dev/stdout kinit [... kinit arguments ...]

Great hint. Thank you!

KRB5_TRACE=/dev/stdout kinit -n -c /tmp/somecache
[1180611] 1682514855.418738: Getting initial credentials for
WELLKNOWN/ANONYMOUS at MYDOMAIN.COM
[1180611] 1682514855.418739: Error loading plugin module pkinit:
2/unable to load plugin
[/usr/lib/x86_64-linux-gnu/krb5/plugins/preauth/pkinit.so]:
/usr/lib/x86_64-linux-gnu/krb5/plugins/preauth/pkinit.so: cannot open
shared object file: No such file or directory

Whoops. Looks like I need:

sudo apt install krb5-pkinit

I installed that on both the client and KDC system.

>   That should at least tell you if PKINIT is attempted and if it is
>   being attempted why it failed (but it will produce a lot so it requires
>   some experience to determine the useful bit you need).
>
> - If you are generating the KDC certificate yourself and you do all of
>   the right magic (as specified in the MIT documentation) to put the
>   realm in the certificate you should not need this:
>
> >> YOURREALM = {
> >>   pkinit_kdc_hostname = yourkdc.fqdn
> >> }
>
> - Did you put the right stuff to trust the KDC certificate on the client?
>   I did not see that.  The PKINIT documentation does mention that you
>   need a pkinit_anchors entry on the client (at a minimum, you may need
>   others).

Again, great hint.

I read the docs from:

https://web.mit.edu/kerberos/www/krb5-latest/doc/admin/pkinit.html

It says:

---<cut>---
If any clients will authenticate using regular (as opposed to
anonymous) PKINIT, the KDC must also have filesystem access to the CA
certificate (cacert.pem), and the following configuration (with the
appropriate pathname):

pkinit_anchors = FILE:/var/lib/krb5kdc/cacert.pem
---<cut>---

Since I am currently only interested in anonymous auth, I thought I
could skip that directive. But alas:

KRB5_TRACE=/dev/stdout /usr/sbin/krb5kdc -n -P /var/run/krb5-kdc.pid
[362890] 1682516397.385786: Retrieving K/M at MYDOMAIN.COM from
FILE:/etc/krb5kdc/stash (vno 0, enctype 0) with result: 0/Success
[362890] 1682516397.385787: PKINIT server initializing realm MYDOMAIN.COM
[362890] 1682516397.385788: PKINIT server initialization failed for
realm MYDOMAIN.COM: 22/No pkinit_anchors supplied for realm
MYDOMAIN.COM
[362890] 1682516397.385791: Retrieving K/M at MYDOMAIN.COM from
FILE:/etc/krb5kdc/stash (vno 0, enctype 0) with result: 0/Success
krb5kdc: starting...

I added:

pkinit_anchors  = FILE:/etc/krb5kdc/cacert.pem

KRB5_TRACE=/dev/stdout /usr/sbin/krb5kdc -n -P /var/run/krb5-kdc.pid
[363318] 1682516577.731993: Retrieving K/M at MYDOMAIN.COM from
FILE:/etc/krb5kdc/stash (vno 0, enctype 0) with result: 0/Success
[363318] 1682516577.731994: PKINIT server initializing realm MYDOMAIN.COM
[363318] 1682516577.731995: PKINIT loading CA certs and CRLs from FILE
[363318] 1682516577.731998: Retrieving K/M at MYDOMAIN.COM from
FILE:/etc/krb5kdc/stash (vno 0, enctype 0) with result: 0/Success
krb5kdc: starting...

So, that looks good.

I'm able to anonymous auth:

$ kinit -n -c /tmp/somecache

but I still get a kerberos principal password as opposed to a RADIUS one...

$ kinit -T /tmp/somecache
Password for bob at MYDOMAIN.COM:
[type in kerberos password]
$

Lots of debug stuff follows. I've read through it, but don't see
anything that jumps out as to why I'm not getting the OTP prompt:

kadmin.local:  get_principal bob
Principal: bob at MYDOMAIN.COM
Expiration date: [never]
Last password change: Mon Dec 10 14:18:53 CST 2018
Password expiration date: [never]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Apr 25 10:13:24 CDT 2023 (root/admin at MYDOMAIN.COM)
Last successful authentication: Wed Apr 26 08:55:41 CDT 2023
Last failed authentication: Wed Apr 26 08:55:16 CDT 2023
Failed password attempts: 0
Number of keys: 2
Key: vno 1, aes256-cts-hmac-sha1-96
Key: vno 1, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: [none]

kadmin.local:  get_strings bob
otp: [{type:MyRemoteTokenType ,username:bob}]

anonymous auth:

$ KRB5_TRACE=/dev/stdout kinit -n -c /tmp/somecache
[1185075] 1682519339.169989: Getting initial credentials for
WELLKNOWN/ANONYMOUS at MYDOMAIN.COM
[1185075] 1682519339.169991: Sending unauthenticated request
[1185075] 1682519339.169992: Sending request (194 bytes) to MYDOMAIN.COM
[1185075] 1682519339.169993: Resolving hostname auth-test.mydomain.com
[1185075] 1682519339.169994: Sending initial UDP request to dgram fc00::1:88
[1185075] 1682519339.169995: Received answer (323 bytes) from dgram fc00::1:88
[1185075] 1682519339.169996: Sending DNS URI query for _kerberos.MYDOMAIN.COM.
[1185075] 1682519339.169997: No URI records found
[1185075] 1682519339.169998: Sending DNS SRV query for
_kerberos-master._udp.MYDOMAIN.COM.
[1185075] 1682519339.169999: Sending DNS SRV query for
_kerberos-master._tcp.MYDOMAIN.COM.
[1185075] 1682519339.170000: No SRV records found
[1185075] 1682519339.170001: Response was not from primary KDC
[1185075] 1682519339.170002: Received error from KDC:
-1765328359/Additional pre-authentication required
[1185075] 1682519339.170005: Preauthenticating using KDC method data
[1185075] 1682519339.170006: Processing preauth types: PA-PK-AS-REQ
(16), PA-FX-FAST (136), PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147),
PA-ENC-TIMESTAMP (2), PA_AS_FRESHNESS (150), PA-FX-COOKIE (133)
[1185075] 1682519339.170007: Selected etype info: etype aes256-cts,
salt "MYDOMAIN.COMWELLKNOWNANONYMOUS", params ""
[1185075] 1682519339.170008: Received cookie: MIT
[1185075] 1682519339.170009: Preauth module pkinit (147) (info)
returned: 0/Success
[1185075] 1682519339.170010: PKINIT client received freshness token from KDC
[1185075] 1682519339.170011: Preauth module pkinit (150) (info)
returned: 0/Success
[1185075] 1682519339.170012: PKINIT loading CA certs and CRLs from
FILE /etc/krb5/cacert.pem
[1185075] 1682519339.170013: PKINIT client computed kdc-req-body
checksum 14/0CF6EE90E51246F7626115CA71D38D89C16D42AB
[1185075] 1682519339.170015: PKINIT client making DH request
[1185075] 1682519339.170016: Preauth module pkinit (16) (real)
returned: 0/Success
[1185075] 1682519339.170017: Produced preauth for next request:
PA-FX-COOKIE (133), PA-PK-AS-REQ (16)
[1185075] 1682519339.170018: Sending request (1680 bytes) to MYDOMAIN.COM
[1185075] 1682519339.170019: Resolving hostname auth-test.mydomain.com
[1185075] 1682519339.170020: Initiating TCP connection to stream fc00::1:88
[1185075] 1682519339.170021: Sending TCP request to stream fc00::1:88
[1185075] 1682519339.170022: Received answer (2955 bytes) from stream fc00::1:88
[1185075] 1682519339.170023: Terminating TCP connection to stream fc00::1:88
[1185075] 1682519339.170024: Sending DNS URI query for _kerberos.MYDOMAIN.COM.
[1185075] 1682519339.170025: No URI records found
[1185075] 1682519339.170026: Sending DNS SRV query for
_kerberos-master._udp.MYDOMAIN.COM.
[1185075] 1682519339.170027: Sending DNS SRV query for
_kerberos-master._tcp.MYDOMAIN.COM.
[1185075] 1682519339.170028: No SRV records found
[1185075] 1682519339.170029: Response was not from primary KDC
[1185075] 1682519339.170030: Processing preauth types: PA-PK-AS-REP
(17), PA-PKINIT-KX (147)
[1185075] 1682519339.170031: Preauth module pkinit (147) (info)
returned: 0/Success
[1185075] 1682519339.170032: PKINIT client verified DH reply
[1185075] 1682519339.170033: PKINIT client config accepts KDC dNSName
SAN auth-test.mydomain.com
[1185075] 1682519339.170034: PKINIT client found 1 SANs (1 princs, 0
UPNs, 0 DNS names) in certificate
/C=US/ST=State/L=Location/O=Organization/OU=Unit/CN=auth-test.mydomain.com
[1185075] 1682519339.170035: PKINIT client found id-pkinit-san in KDC
cert: krbtgt/MYDOMAIN.COM at MYDOMAIN.COM
[1185075] 1682519339.170036: PKINIT client matched KDC principal
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM against id-pkinit-san; no EKU check
required
[1185075] 1682519339.170037: PKINIT client used KDF 2B06010502030602
to compute reply key aes256-cts/71E2
[1185075] 1682519339.170038: Preauth module pkinit (17) (real)
returned: 0/Success
[1185075] 1682519339.170039: Produced preauth for next request: (empty)
[1185075] 1682519339.170040: AS key determined by preauth: aes256-cts/71E2
[1185075] 1682519339.170041: Decrypted AS reply; session key is: aes256-cts/0D0B
[1185075] 1682519339.170042: FAST negotiation: available
[1185075] 1682519339.170043: Resolving unique ccache of type MEMORY
[1185075] 1682519339.170044: Initializing MEMORY:IfWp4iF with default
princ WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS
[1185075] 1682519339.170045: Storing config in MEMORY:IfWp4iF for
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM: fast_avail: yes
[1185075] 1682519339.170046: Storing
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krb5_ccache_conf_data/fast_avail/krbtgt\/MYDOMAIN.COM\@MYDOMAIN.COM at X-CACHECONF:
in MEMORY:IfWp4iF
[1185075] 1682519339.170047: Storing config in MEMORY:IfWp4iF for
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM: pa_type: 16
[1185075] 1682519339.170048: Storing
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krb5_ccache_conf_data/pa_type/krbtgt\/MYDOMAIN.COM\@MYDOMAIN.COM at X-CACHECONF:
in MEMORY:IfWp4iF
[1185075] 1682519339.170049: Storing config in MEMORY:IfWp4iF for :
start_realm: MYDOMAIN.COM
[1185075] 1682519339.170050: Storing
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krb5_ccache_conf_data/start_realm at X-CACHECONF: in MEMORY:IfWp4iF
[1185075] 1682519339.170051: Storing
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM in MEMORY:IfWp4iF
[1185075] 1682519339.170052: Moving ccache MEMORY:IfWp4iF to FILE:/tmp/somecache
[1185075] 1682519339.170053: Destroying ccache MEMORY:IfWp4iF

and the attempt to auth via OTP:

$ KRB5_TRACE=/dev/stdout kinit -T /tmp/somecache
[1185088] 1682519355.427393: Getting initial credentials for bob at MYDOMAIN.COM
[1185088] 1682519355.427394: FAST armor ccache: /tmp/somecache
[1185088] 1682519355.427395: Retrieving
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krb5_ccache_conf_data/fast_avail/krbtgt\/MYDOMAIN.COM\@MYDOMAIN.COM at X-CACHECONF:
from FILE:/tmp/somecache with result: 0/Success
[1185088] 1682519355.427396: Read config in FILE:/tmp/somecache for
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM: fast_avail: yes
[1185088] 1682519355.427397: Using FAST due to armor ccache negotiation result
[1185088] 1682519355.427398: Getting credentials
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM using ccache FILE:/tmp/somecache
[1185088] 1682519355.427399: Retrieving
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krb5_ccache_conf_data/start_realm at X-CACHECONF: from
FILE:/tmp/somecache with result: 0/Success
[1185088] 1682519355.427400: Read config in FILE:/tmp/somecache for :
start_realm: MYDOMAIN.COM
[1185088] 1682519355.427401: Retrieving
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM from FILE:/tmp/somecache with result:
0/Success
[1185088] 1682519355.427402: Armor ccache sesion key: aes256-cts/0D0B
[1185088] 1682519355.427403: Creating authenticator for
WELLKNOWN/ANONYMOUS at WELLKNOWN:ANONYMOUS ->
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM, seqnum 0, subkey aes256-cts/E842,
session key aes256-cts/0D0B
[1185088] 1682519355.427405: FAST armor key: aes256-cts/B8EF
[1185088] 1682519355.427407: Sending unauthenticated request
[1185088] 1682519355.427408: Encoding request body and padata into FAST request
[1185088] 1682519355.427409: Sending request (968 bytes) to MYDOMAIN.COM
[1185088] 1682519355.427410: Resolving hostname auth-test.mydomain.com
[1185088] 1682519355.427411: Sending initial UDP request to dgram fc00::1:88
[1185088] 1682519355.427412: Received answer (551 bytes) from dgram fc00::1:88
[1185088] 1682519355.427413: Sending DNS URI query for _kerberos.MYDOMAIN.COM.
[1185088] 1682519355.427414: No URI records found
[1185088] 1682519355.427415: Sending DNS SRV query for
_kerberos-master._udp.MYDOMAIN.COM.
[1185088] 1682519355.427416: Sending DNS SRV query for
_kerberos-master._tcp.MYDOMAIN.COM.
[1185088] 1682519355.427417: No SRV records found
[1185088] 1682519355.427418: Response was not from primary KDC
[1185088] 1682519355.427419: Received error from KDC:
-1765328359/Additional pre-authentication required
[1185088] 1682519355.427420: Decoding FAST response
[1185088] 1682519355.427423: Preauthenticating using KDC method data
[1185088] 1682519355.427424: Processing preauth types: PA-PK-AS-REQ
(16), PA-FX-FAST (136), PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147),
PA-ENCRYPTED-CHALLENGE (138), PA_AS_FRESHNESS (150), PA-FX-COOKIE
(133), PA-FX-ERROR (137)
[1185088] 1682519355.427425: Selected etype info: etype aes256-cts,
salt "MYDOMAIN.COMbob", params ""
[1185088] 1682519355.427426: Received cookie: MIT
[1185088] 1682519355.427427: PKINIT client has no configured identity; giving up
[1185088] 1682519355.427428: Preauth module pkinit (147) (info)
returned: 0/Success
[1185088] 1682519355.427429: PKINIT client received freshness token from KDC
[1185088] 1682519355.427430: Preauth module pkinit (150) (info)
returned: 0/Success
[1185088] 1682519355.427431: PKINIT client has no configured identity; giving up
[1185088] 1682519355.427432: Preauth module pkinit (16) (real)
returned: 22/Invalid argument
Password for bob at MYDOMAIN.COM:
[1185088] 1682519365.538171: Preauth module encrypted_challenge (138)
(real) returned: 0/Success
[1185088] 1682519365.538172: Produced preauth for next request:
PA-FX-COOKIE (133), PA-ENCRYPTED-CHALLENGE (138)
[1185088] 1682519365.538173: Encoding request body and padata into FAST request
[1185088] 1682519365.538174: Sending request (1067 bytes) to MYDOMAIN.COM
[1185088] 1682519365.538175: Resolving hostname auth-test.mydomain.com
[1185088] 1682519365.538176: Sending initial UDP request to dgram fc00::1:88
[1185088] 1682519365.538177: Received answer (1051 bytes) from dgram fc00::1:88
[1185088] 1682519365.538178: Sending DNS URI query for _kerberos.MYDOMAIN.COM.
[1185088] 1682519365.538179: No URI records found
[1185088] 1682519365.538180: Sending DNS SRV query for
_kerberos-master._udp.MYDOMAIN.COM.
[1185088] 1682519365.538181: Sending DNS SRV query for
_kerberos-master._tcp.MYDOMAIN.COM.
[1185088] 1682519365.538182: No SRV records found
[1185088] 1682519365.538183: Response was not from primary KDC
[1185088] 1682519365.538184: Decoding FAST response
[1185088] 1682519365.538185: Processing preauth types:
PA-ENCRYPTED-CHALLENGE (138), PA-ETYPE-INFO2 (19)
[1185088] 1682519365.538186: Selected etype info: etype aes256-cts,
salt "MYDOMAIN.COMbob", params ""
[1185088] 1682519365.538187: Preauth module encrypted_challenge (138)
(real) returned: 0/Success
[1185088] 1682519365.538188: Produced preauth for next request: (empty)
[1185088] 1682519365.538189: AS key determined by preauth: aes256-cts/2CFC
[1185088] 1682519365.538190: FAST reply key: aes256-cts/DA2D
[1185088] 1682519365.538191: Decrypted AS reply; session key is: aes256-cts/E0C1
[1185088] 1682519365.538192: FAST negotiation: available
[1185088] 1682519365.538193: Resolving unique ccache of type MEMORY
[1185088] 1682519365.538194: Initializing MEMORY:jh1396e with default
princ bob at MYDOMAIN.COM
[1185088] 1682519365.538195: Storing config in MEMORY:jh1396e for
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM: fast_avail: yes
[1185088] 1682519365.538196: Storing bob at MYDOMAIN.COM ->
krb5_ccache_conf_data/fast_avail/krbtgt\/MYDOMAIN.COM\@MYDOMAIN.COM at X-CACHECONF:
in MEMORY:jh1396e
[1185088] 1682519365.538197: Storing config in MEMORY:jh1396e for
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM: pa_type: 138
[1185088] 1682519365.538198: Storing bob at MYDOMAIN.COM ->
krb5_ccache_conf_data/pa_type/krbtgt\/MYDOMAIN.COM\@MYDOMAIN.COM at X-CACHECONF:
in MEMORY:jh1396e
[1185088] 1682519365.538199: Storing bob at MYDOMAIN.COM ->
krbtgt/MYDOMAIN.COM at MYDOMAIN.COM in MEMORY:jh1396e
[1185088] 1682519365.538200: Moving ccache MEMORY:jh1396e to
FILE:/tmp/krb5cc_1000
[1185088] 1682519365.538201: Destroying ccache MEMORY:jh1396e

Thanks for any additional help. I appreciate it!

-m