OpenSSL in FIPS mode - MD5 hash in replay cache

[Tomas Kuthan]

Hi,

rcache uses an unkeyed MD5 hash of the authenticator to distinguish 
between different request with equal client principal, server principal 
and microsecond time. When OpenSSL crypto provider is used and 
underlying OpenSSL is run in FIPS mode, MD5 algorithm is disabled and 
gss_accept_sec_context() result in an abort in rcache processing:

root at x2270-brm-03:/var/tmp# ./a.out
fips_md.c(146): OpenSSL internal error, assertion failed: Digest update 
previous FIPS forbidden algorithm error ignored
Abort (core dumped)
root at x2270-brm-03:/var/tmp# mdb core
Loading modules: [ libc.so.1 libuutil.so.1 libnvpair.so.1 ld.so.1 ]
 > $C
feffdd08 libc_hwcap1.so.1`__lwp_sigqueue+0x15(1, 6, feffdd28, 78f42fe)
feffdd28 libc_hwcap1.so.1`raise+0x22(6, 0, feffdd78, 78c7fae)
feffdd78 libc_hwcap1.so.1`abort+0xe6(7dfdda8, 7dfc000, feffdda8,
feffdd88 libcrypto.so.1.0.0`fips_des_ede3_cfb64_encrypt(7c79ccc, 92,
feffdda8 libcrypto.so.1.0.0`bad_update+0x2e(feffde10, 80bc4b8, b4,
feffddc8 libcrypto.so.1.0.0`FIPS_digestupdate+0x2c(feffde10, 80bc4b8,
feffdde8 libcrypto.so.1.0.0`EVP_DigestUpdate+0x24(feffde10, 80bc4b8,
feffde48 libk5crypto.so.3.1`k5_md5_hash+0x137(feffdeb0, 1, feffdea0,
feffde68 libk5crypto.so.3.1`krb5int_unkeyed_checksum+0x1b(76ad4e0, 0,
feffded8 libk5crypto.so.3.1`krb5_k_make_checksum+0xd5(80bc2a8, 7, 0, 0,
feffdf18 libk5crypto.so.3.1`krb5_c_make_checksum+0x55(80bc2a8, 7, 0, 0,
feffdf78 libkrb5.so.3.3`krb5_rc_hash_message+0x34(80bc2a8, 80b0c10,
feffe078 libkrb5.so.3.3`rd_req_decoded_opt+0x945(80bc2a8, feffe124,
feffe0a8 libkrb5.so.3.3`krb5_rd_req_decoded+0x22(80bc2a8, feffe124,
feffe328 libgssapi_krb5.so.2.2`kg_accept_krb5+0x6ee(feffe4ac, 8063cc0,
feffe378 libgssapi_krb5.so.2.2`krb5_gss_accept_sec_context+0x87(feffe4ac
feffe408 libgssapi_krb5.so.2.2`gss_accept_sec_context+0x225(feffe4ac,
feffe4d8 main+0x2b6(1, feffe51c, feffe524)
feffe510 _start+0x46(1, feffe5e6, 0, feffe5ee, feffe5f2, feffe606)
 >

I understand, that using MD5 for this purpose is legitimate, because 
collision resistance is not required. Sadly FIPS 140-2 doesn't recognize 
non-cryptographic use of hash function and that makes advocating MD5 use 
difficult.

Would it be possible to use SHA-1 instead of MD5?

I was thinking about using a new extension identifier ("SHA1:" instead 
of "HASH:") and using CKSUMTYPE_NIST_SHA instead of CKSUMTYPE_RSA_MD5 in 
krb5_rc_hash_message.
This way if the an older and a newer implementations were sharing an 
rcache, they would not recognize each other extensions and would fall 
back to using old records with (client, server, timestamp, utime) 
tuples. This could result in some false positives replay detections, but 
would retain correctness otherwise.

If complete switch to SHA-1 is not possible, could we make the algorithm 
for rcache hashes configurable?

Thanks,
Tomas