[krbdev.mit.edu #6738] PKINIT DH exchange occasionally produces mismatch

[Greg Hudson via RT]

Approximately 1% of the time, a PKINIT Diffie-Hellman exchange (between 
a trunk client and a trunk KDC) arrives at a different result on the 
client and KDC.

One way of reproducing this bug is with t_anonpkinit.py in tests/.  If 
you run it with --shell-before=5 and then run the anonymous kinit 
command repeatedly in a loop, after roughly 100 iterations it will ask 
for a password.  (This behavior is a little unfortunate; if get_in_tkt.c 
fails to decrypt a response with a reply key determined by preauth, it 
silently falls back to gak_fct, due to some enctype issues related to 
SAM preauth.)

If you display the value of *client_key and *server_key just after the 
calls to DH_compute_key() in pkinit_crypto_openssl.c, in the successful 
case they will be identical, while in the failure case they differ in 
the last two bytes.  To me, this suggests something going wrong inside 
OpenSSL's crypto library; if the inputs were bad, the values would be 
much more different.  The problem has been observed with OpenSSL 0.9.8g-
4ubuntu3.9 and 0.9.8k-7ubuntu8.