NFSsec/krb/AES interop issue with Solaris 10/11

[Will Fiveash]

A while back I discovered a bug in the Solaris 10 and versions of 11
wherein the implementation of Kerberos in the Solaris kernel was not
dealing with 16 byte input data properly when a AES enctype is in use.
The impact is that NFS sec=(krb5|krb5i|krb5p) is not generating a RFC
3961 compliant derived key (used to create the MIC) when using a AES
enctype session key.  I have recently putback the fix for this in
Solaris 11 and there will be a patch/update released for Solaris 10.

For those doing interop testing, one workaround is to rename the Solaris
Kerberos kernel module (do a "find /kernel /platform -name 'kmech_krb5'
-print" and rename any instances output) and reboot.  What will happen
is that Solaris will fall back to using user space Kerberos which is
doing the right thing.  NFSsec will work but will be slow as compared to
using the kernel module.  When the patch/update is release, rename the
renamed kmech_krb5(s) back to their original name before applying the
patch/update.

The bug can be viewed here:
http://bugs.opensolaris.org/view_bug.do?bug_id=6548599

Please follow-up to kerberos-discuss at opensolaris.org.
-- 
Will Fiveash
Sun Microsystems Inc.
Austin, TX, USA (TZ=CST6CDT)