wrong checksum type for arcfour-hmac-md5

[Greg Hudson]

On Wed, 2010-09-15 at 13:35 -0400, Sam Hartman wrote:
> My current thinking on this is that the bug is in Samba.  Based on the
> evidence so far I don't support this change.  Having the mandatory
> checksum for an enctype be unkeyed is problematic for a number of things
> including FAST, PKINIT, the securID stuff I'm working on and the OTP
> preauth under last call in krb-wg.

It turns out Heimdal has special logic for choosing an authenticator
checksum type:

* For des-cbc-crc, it chooses md4, "to make DCE secd (and older MIT
kdc?) happy".

* For rc4-hmac, rc4-hmac-exp, des-cbc-md4, and des-cbc-md5, it chooses
md5, "to make MS kdc happy".

* For everything else it uses the enctype's associated keyed checksum
type, or unkeyed checksum type if there isn't one.  The Heimdal
associated checksum types do not always match the MIT mandatory
cksumtype; here is the chart:

Enctype	       MIT mandatory   	    Heimdal cksum  Heimdal keyed
-------	       -------------   	    -------------  -------------
des-cbc-crc    md5                  crc32          none
des-cbc-md4    md4                  md4            md4-des
des-cbc-md5    md5                  md5            md5-des
des3-cbc-sha1  hmac-sha1-des3       sha1           hmac-sha1-des3
rc4-hmac       hmac-md5             hmac-md5       none
rc4-hmac-exp   hmac-md5             (???)          (???)
aes128-cts     hmac-sha1-96-aes128  sha1           hmac-sha1-96-aes128
aes256-cts     hmac-sha1-96-aes256  sha1           hmac-sha1-96-aes256

(rc4-hmac-exp doesn't seem to have an entry in heimdal's
lib/krb5/crypto.c, even though the associated constant is still
referenced elsewhere.)

The resulting authenticator checksum chart is, I think:

Enctype        MIT                  Heimdal
-------        ---                  -------
des-cbc-crc    md5                  md4
des-cbc-md4    md4                  md5
des-cbc-md5    md5                  md5
des3-cbc-sha1  hmac-sha1-des3       hmac-sha1-des3
rc4-hmac       hmac-md5             md5
rc4-hmac-exp   hmac-md5             md5
aes128-cts     hmac-sha1-96-aes128  hmac-sha1-96-aes128
aes256-cts     hmac-sha1-96-aes256  hmac-sha1-96-aes256

Since the authenticator checksum doesn't need to be keyed, I don't
object in principal to messing with that specific choice for
interoperability.  I'd prefer a better understanding of the reasons why,
though; the bug presented so far only seems to affect malformed GSSAPI
token authenticators, and is as easy to fix in Samba as it is in MIT
krb5.