svn rev #24210: trunk/src/lib/rpc/

[ghudson@MIT.EDU]

http://src.mit.edu/fisheye/changelog/krb5/?cs=24210
Commit By: ghudson
Log Message:
ticket: 6753
subject: Fix XDR decoding of large values in xdr_u_int

Our ancient RPC value internally decodes 32-bit wire values into a
signed long, which is then casted to the appropriate type.
xdr_u_int() contains a check intended to catch wire values that don't
fit into a u_int on platforms with 16-ints, but on platforms with
64-bit longs it was failing on values of 2^31 or larger because the
sign-extended value appeared larger than UINT_MAX.  Fix the check by
casting the value to uint32_t before comparing.

This bug, in combination with a poor choice of types in
kadm_rpc_xdr.c's xdr_krb5_enctype(), prevented negative enctype values
from being transported properly in kadmin's change_password command
result.



Changed Files:
U   trunk/src/lib/rpc/xdr.c
Modified: trunk/src/lib/rpc/xdr.c
===================================================================
--- trunk/src/lib/rpc/xdr.c	2010-07-23 20:39:45 UTC (rev 24209)
+++ trunk/src/lib/rpc/xdr.c	2010-07-26 18:18:57 UTC (rev 24210)
@@ -145,7 +145,7 @@
 		if (!XDR_GETLONG(xdrs, (long *) &l))
 			return (FALSE);
 
-		if (l > UINT_MAX)
+		if ((uint32_t)l > UINT_MAX)
 			return (FALSE);
 
 		*up = (u_int) l;