OpenSSL in FIPS mode - MD5 hash in replay cache

Tomas Kuthan tomas.kuthan at
Fri Dec 18 07:56:36 EST 2015


rcache uses an unkeyed MD5 hash of the authenticator to distinguish 
between different request with equal client principal, server principal 
and microsecond time. When OpenSSL crypto provider is used and 
underlying OpenSSL is run in FIPS mode, MD5 algorithm is disabled and 
gss_accept_sec_context() result in an abort in rcache processing:

root at x2270-brm-03:/var/tmp# ./a.out
fips_md.c(146): OpenSSL internal error, assertion failed: Digest update 
previous FIPS forbidden algorithm error ignored
Abort (core dumped)
root at x2270-brm-03:/var/tmp# mdb core
Loading modules: [ ]
 > $C
feffdd08`__lwp_sigqueue+0x15(1, 6, feffdd28, 78f42fe)
feffdd28`raise+0x22(6, 0, feffdd78, 78c7fae)
feffdd78`abort+0xe6(7dfdda8, 7dfc000, feffdda8,
feffdd88`fips_des_ede3_cfb64_encrypt(7c79ccc, 92,
feffdda8`bad_update+0x2e(feffde10, 80bc4b8, b4,
feffddc8`FIPS_digestupdate+0x2c(feffde10, 80bc4b8,
feffdde8`EVP_DigestUpdate+0x24(feffde10, 80bc4b8,
feffde48`k5_md5_hash+0x137(feffdeb0, 1, feffdea0,
feffde68`krb5int_unkeyed_checksum+0x1b(76ad4e0, 0,
feffded8`krb5_k_make_checksum+0xd5(80bc2a8, 7, 0, 0,
feffdf18`krb5_c_make_checksum+0x55(80bc2a8, 7, 0, 0,
feffdf78`krb5_rc_hash_message+0x34(80bc2a8, 80b0c10,
feffe078`rd_req_decoded_opt+0x945(80bc2a8, feffe124,
feffe0a8`krb5_rd_req_decoded+0x22(80bc2a8, feffe124,
feffe328`kg_accept_krb5+0x6ee(feffe4ac, 8063cc0,
feffe4d8 main+0x2b6(1, feffe51c, feffe524)
feffe510 _start+0x46(1, feffe5e6, 0, feffe5ee, feffe5f2, feffe606)

I understand, that using MD5 for this purpose is legitimate, because 
collision resistance is not required. Sadly FIPS 140-2 doesn't recognize 
non-cryptographic use of hash function and that makes advocating MD5 use 

Would it be possible to use SHA-1 instead of MD5?

I was thinking about using a new extension identifier ("SHA1:" instead 
of "HASH:") and using CKSUMTYPE_NIST_SHA instead of CKSUMTYPE_RSA_MD5 in 
This way if the an older and a newer implementations were sharing an 
rcache, they would not recognize each other extensions and would fall 
back to using old records with (client, server, timestamp, utime) 
tuples. This could result in some false positives replay detections, but 
would retain correctness otherwise.

If complete switch to SHA-1 is not possible, could we make the algorithm 
for rcache hashes configurable?


More information about the krbdev mailing list