[krbdev.mit.edu #6738] PKINIT DH exchange occasionally produces mismatch

[Greg Hudson via RT]

This bug happens when the DH result fits in fewer than DH_size() bytes.  
For example, if the result fits in 255 bytes, OpenSSL stores the result 
in the first 255 bytes of the buffer and returns 255, but we use all 256 
bytes of the buffer as the secret.

Some sample values:

Prime (p):
  FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024E088A67CC74
  020BBEA63B139B22514A08798E3404DDEF9519B3CD3A431B302B0A6DF25F1437
  4FE1356D6D51C245E485B576625E7EC6F44C42E9A637ED6B0BFF5CB6F406B7ED
  EE386BFB5A899FA5AE9F24117C4B1FE649286651ECE45B3DC2007CB8A163BF05
  98DA48361C55D39A69163FA8FD24CF5F83655D23DCA3AD961C62F356208552BB
  9ED529077096966D670C354E4ABC9804F1746C08CA18217C32905E462E36CE3B
  E39E772C180E86039B2783A2EC07A28FB5C55DF06F4C52C9DE2BCBF695581718
  3995497CEA956AE515D2261898FA051015728E5A8AACAA68FFFFFFFFFFFFFFFF
Generator (g):
  2
Client private key (x):
  7CBBD084EE01805E895D28182EB2DF68ED6E7A47BC70874F96D2F1B7AE82D99B
  4FB2E169F725878871B87333F6A829A9E5FC1D41F634C20290705AA5E77D740E
  9B251CED322C65C915CA81B4E0DD369F23AAB0D46245813234CFE04057848D9F
  D871CE9FAFC846E190A810C0C665CA088D3C42EC2FEA85E53CF7955963C4254B
  C4B2B5844E4B885DA80E2E153EEED751A47CB449D1538F7FFFA98F26A193F600
  F531C2CE204FBED233ED77B667FAA0D371CB00033201DF039D180137C8DB6FD1
  032B15A446C5C104189B082A66F6FE06007845C59B53F462FA9CB7D8AAB87C6B
  2FBE3C4EFBCDDA6D4F590BB37D97B650A2836694C5CA395D114C0C4AB9E40339
KDC private key (y):
  4B37A49520F728DF3D437AD128FAABC65876E8DDB5F3AF44CF4352A4C2DE5B44
  C6A1582A359DB2DFDB4BFAC3CF17C52174B28C822D01E55FBCBD5C507A8B5BC4
  81D32C807C624EF9ED45F2F3454217F49D0129CB8561A813C824CBB8FA542C11
  B3CE715215F9B2CF4836A267B400F3EF4C19555BC603222312459B65FB60206F
  2686D5C5826F46A183546CB1B5670E6EEB8A39C57FF59C037CBDB48D3622A653
  7A81CBECB9917E5FD38D1501AE8FF4ABBF88D76C50D25E54FCC2BF2B0A13EC00
  F30E5B6E640FD25944C7BE2EDABFC29EF76B534DB4FA14CAFF3755060F8FBDD8
  D41170334AEC37D6A80895F0431261EF6DF3974EC58D11ABAC5F8B5778E739CA

Client public key (g^x mod p):
  963CAD055FD1E7EE0B8965D813E0E15A3BFB03D5E3A33E6A08B341D168A555A6
  5A45F43B6531D5AE75A10245A2190082E501F6293565D3B5554F372AC702B854
  595FEB480AC516BD554FDA59483501FAE92827AC9DC0D7E589D879FA59DA1B09
  C3BF32576C5D8F02E18722FE7E81D24BF7666A2C9AF5E7EBC85FBC7B1426FDD6
  19EB1E19A8DA710025C8816635EA17985EE9435D9887188737BB732C5A9F830B
  E073A4BD4B4D3CB39BB3D139BA0FA139DFA28C8C34FB5122010A5C04052BAC7B
  F6960A3D70E584890C3830D2314AA4221C9B166E6AE617F4ABC305A42BECE8CD
  C481072D8CCEBB304EEB5BE25E1B04400C8FCABA7364B9FFF4732F23C037052F
KDC public key (g^y mod p):
  8C510AFD652BF0FBE3D23B040AB2AD95C496730EF81B1FA5E15E7B2FC4DBA326
  171A7A438098CC1B7BD9BD9F73C2214A325249286CB4C536B50CC061BD9EF76B
  BC3821A28CAC65E54FBEFAF6EF50A720CA25D50164D998167A6324618C738261
  D9CBAAB913B0307F51D3D3DE162F97C8CB94D7E35AACB2D6DEF7FF5A90FA1B11
  241B5E6959872ED239B14B42690C3A457BD299E730326938D2492C3A539F7EA9
  A919E5458EFA29F9AE4FD92077DDAD9A8EF81B89A0D77816A38F2D0B9E0A3457
  D93E1A0283CA55FAAAB6AB0E37A3AAC52A0AF503EA719A94F630D0F03E180679
  88B69354ECDB3E5B4C15055F8978967B1F21A38BD9FAC7716BCF0FC76343772E

Proper value of shared key (g^xy mod p):
  00D1CD51685C16CDA0A58E6285CDF72D6FFE5ADD321D8BF72F98BEC9F2CBFE4D
  06DEF9CE2C0A6ECA89E18DB5E0EA3FE22B27E4851D09C9A41C14483F216FA35B
  90DEB85EF61270C05B680E2707CA2741F7A1B6A02022AD1549D1E5D938B71353
  D0B217FBC3BB856E432B3A29D8D185C6AFA67CDA766CA1CA4EE785A2E5218550
  5585B7E3A84AE445B09B32C38F9F4EC95A51D463910298D105AC1A7A4B504548
  3CE2B185BA950442D8DE4B7E3E09B2BC946942C8DA8C475A92DDDF9A1C25B288
  437D6201E6E82116735D16F9391D3F823E599518D125B6A141FF84F3D8B468DD
  1A2F85D93F0A92F3FCF4630ECE22053745E089FE097742CAC17C67BCC24EE892
Client buffer contents after DH computation:
  D1CD51685C16CDA0A58E6285CDF72D6FFE5ADD321D8BF72F98BEC9F2CBFE4D06
  DEF9CE2C0A6ECA89E18DB5E0EA3FE22B27E4851D09C9A41C14483F216FA35B90
  DEB85EF61270C05B680E2707CA2741F7A1B6A02022AD1549D1E5D938B71353D0
  B217FBC3BB856E432B3A29D8D185C6AFA67CDA766CA1CA4EE785A2E521855055
  85B7E3A84AE445B09B32C38F9F4EC95A51D463910298D105AC1A7A4B5045483C
  E2B185BA950442D8DE4B7E3E09B2BC946942C8DA8C475A92DDDF9A1C25B28843
  7D6201E6E82116735D16F9391D3F823E599518D125B6A141FF84F3D8B468DD1A
  2F85D93F0A92F3FCF4630ECE22053745E089FE097742CAC17C67BCC24EE89278
KDC buffer contents after DH computation:
  D1CD51685C16CDA0A58E6285CDF72D6FFE5ADD321D8BF72F98BEC9F2CBFE4D06
  DEF9CE2C0A6ECA89E18DB5E0EA3FE22B27E4851D09C9A41C14483F216FA35B90
  DEB85EF61270C05B680E2707CA2741F7A1B6A02022AD1549D1E5D938B71353D0
  B217FBC3BB856E432B3A29D8D185C6AFA67CDA766CA1CA4EE785A2E521855055
  85B7E3A84AE445B09B32C38F9F4EC95A51D463910298D105AC1A7A4B5045483C
  E2B185BA950442D8DE4B7E3E09B2BC946942C8DA8C475A92DDDF9A1C25B28843
  7D6201E6E82116735D16F9391D3F823E599518D125B6A141FF84F3D8B468DD1A
  2F85D93F0A92F3FCF4630ECE22053745E089FE097742CAC17C67BCC24EE8924B

The fix is straightforward, although it's unfortunate that we have to 
account for this.