[krbdev.mit.edu #1262] asn1 decoding of unsigned integers will eventually vause interoperability problems

[Ezra Peisach via RT]

I was thinking about changing the sequence numbers in the various
structures to 
unsigned to comply with the current draft specifications - when I cam
accross the
feature that asn1_decode_unsigned_integer will fail if passed as signed
integer with the high bit set.  This may arise from interoperability
with old clients sending signed integers.

I am assuming that the unsigned_integer encoder is correct - they look
right. (I may be wrong about this) - but I have a program that will test
encoding decoding.

I am also feeding the decoder the encoding as a signed integer and
comparing the results.

The following are the encodings of unsigned ints: (The dotest is a macro
- but you see the value
and the der encoding decoding test)

  dotest(0, "02 01 00");
  dotest(127, "02 01 7F");
  dotest(0x7fffffff, "02 04 7F FF FF FF");
  dotest(0xFFFF0000, "02 05 00 FF FF 00 00");
  dotest(0xFF7f0000, "02 05 00 FF 7F 00 00");
  dotest(0xFF800000, "02 05 00 FF 80 00 00");
  dotest(0x80000001, "02 05 00 80 00 00 01"); /* Unsigned encoding */

Now - if some of the values are encoded as signed integers - the decoder
fails.

For instance: 

 do_decode_test("02 04 FF 7F 00 00", 0xFF7f0000);
 do_decode_test("02 03 80 00 00", 0xFF800000);

Essentially - the test if the first octet of the encoding is signed.

I believe we should modify the code to allow a signed first bit and
operate in a compatibilty mode
while generating proper unsigned ints.

(As a separate note - I have discovered that heimdal encoding of
unsigned integers violates X.690
by outputing a first byte with all bits set along with the next with the
high bit set).