krb5-1.12.1 krb5kdc segfaulting on ARMv6 10-stable FreeBSD

Mon Feb 10 18:46:22 EST 2014

On Mon, 10 Feb 2014 17:23:10 -0500
Greg Hudson <ghudson at MIT.EDU> wrote:

> Just in case this turns out to be a security issue (unlikely, but
> always a risk with KDC crashes), I'm taking this to krbcore-security.
> 
> On 02/10/2014 04:19 PM, Christopher J. Ruwe wrote:
> > root at krb5ldap:~ # env KRB5_TRACE=/dev/stdout krb5kdc -n -p 88
> > [5231] 1392063323.707758: Retrieving K/M at HB22.CRUWE.DE from
> > FILE:/usr/local/var/krb5kdc/.k5.HB22.CRUWE.DE (vno 0, enctype 0)
> > with result: 0/Success krb5kdc: starting... Segmentation fault
> > (core dumped)
> 
> Can you repeat this test, but instead run the KDC as:
> 
>   gdb --args krb5kdc -n -p 88
> 
> When the seg fault happens, type "back" to get a backtrace, and send
> me the output.

This is interesting. When running in gdb nearly as in your command,
nothing happens. 

root at krb5ldap:~ # date && gdb --args krb5kdc -n -p 88
Mon Feb 10 22:43:25 UTC 2014
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under
certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details. This GDB was configured as "arm-marcel-freebsd"...(no
debugging symbols found)... (gdb) 

There is no output to the /usr/local/var/krb5kdc.log

I tried to do that differently on three terminals via ssh:

1. term

root at krb5ldap:~ # env KRB5_TRACE=/dev/stdout krb5kdc -n -p 88
[5468] 1392072327.447794: Retrieving K/M at HB22.CRUWE.DE from
FILE:/usr/local/var/krb5kdc/.k5.HB22.CRUWE.DE (vno 0, enctype 0) with
result: 0/Success krb5kdc: starting...

2. term

root at krb5ldap:~ # ps ax | grep krb
5468  0  I+      0:00.19 krb5kdc -n -p 88

root at krb5ldap:~ # gdb program 5468
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under
certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details. This GDB was configured as "arm-marcel-freebsd"...program: No
such file or directory.

Attaching to process 5468
/usr/home/cjr/media/src/freebsd-git/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1444:
internal-error: legacy_fetch_link_map_offsets called without legacy
link_map support enabled. A problem internal to GDB has been detected,
further debugging may prove unreliable. Quit this debugging session? (y
or n)  n

/usr/home/cjr/media/src/freebsd-git/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1444:
internal-error: legacy_fetch_link_map_offsets called without legacy
link_map support enabled. A problem internal to GDB has been detected,
further debugging may prove unreliable. Create a core file of GDB? (y
or n) n

/usr/home/cjr/media/src/freebsd-git/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1444:
internal-error: legacy_fetch_link_map_offsets called without legacy
link_map support enabled. A problem internal to GDB has been detected,
further debugging may prove unreliable. Quit this debugging session? (y
or n) n

/usr/home/cjr/media/src/freebsd-git/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1444:
internal-error: legacy_fetch_link_map_offsets called without legacy
link_map support enabled. A problem internal to GDB has been detected,
further debugging may prove unreliable. ---Type <return> to continue,
or q <return> to quit--- <return>

Create a core file of GDB? (y or n) n

0x20385fb8 in ?? ()
(gdb) 

3. term

root at krb5ldap:~ # time kinit admin/admin at HB22.CRUWE.DE
kinit: Cannot contact any KDC for realm 'HB22.CRUWE.DE' while getting
initial credentials 0.074u 0.083s 0:18.34 0.8%      25+167k 0+0io 0pf+0w

(I do not know that format. Weird.)

root at krb5ldap:~ # date && kinit admin/admin at HB22.CRUWE.DE ; date
Mon Feb 10 22:54:23 UTC 2014
kinit: Cannot contact any KDC for realm 'HB22.CRUWE.DE' while getting
initial credentials Mon Feb 10 22:54:41 UTC 2014

OK, 0:18.34 seems to be total time.

Note that all the time krb5kdc has not crashed and is silently sitting
in terminal 1. logfile is also unchanged.

2. term

(gdb) back
#0  0x20385fb8 in ?? ()

does not seem very useful to me. I quit, detaching on the way.

(gdb) quit 
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: , process 5468

2 sec later (felt, not measured), krb5kdc segfaults in terminal 1

root at krb5ldap:~ # env KRB5_TRACE=/dev/stdout krb5kdc -n -p 88
[5468] 1392072327.447794: Retrieving K/M at HB22.CRUWE.DE from
FILE:/usr/local/var/krb5kdc/.k5.HB22.CRUWE.DE (vno 0, enctype 0) with
result: 0/Success krb5kdc: starting... Segmentation fault (core dumped)

> > This is not observable via nmap, because krb5kdc does not listen as
> > specified.
> 
> We weren't sure what you meant by this.  You ran krb5kdc with "-p 88",
> which overrides the kdc_ports option in kdc.conf, and nmap shows it
> listening on port 88.
> 

I am sorry. I forgot one part of my initial process:

Initially, I ran

[cjr at dijkstra:~]$ nmap krb5ldap                   (02-11 00:00)

Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-11 00:00 CET
Nmap scan report for krb5ldap (192.168.178.3)
Host is up (0.0041s latency).
rDNS record for 192.168.178.3: krb5ldap.hb22.cruwe.de
Not shown: 997 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
464/tcp open  kpasswd5
749/tcp open  kerberos-adm

Nmap done: 1 IP address (1 host up) scanned in 7.85 seconds

Only then, I ran

[cjr at dijkstra:~]$ sudo nmap -sU -sT -p U:88,464,750,T:464,749,754
krb5ldap

Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-11 00:02 CET
Nmap scan report for krb5ldap (192.168.178.3)
Host is up (0.0065s latency).
rDNS record for 192.168.178.3: krb5ldap.hb22.cruwe.de
PORT    STATE         SERVICE
464/tcp open          kpasswd5
749/tcp open          kerberos-adm
754/tcp closed        krb_prop
88/udp  open|filtered kerberos-sec
464/udp open|filtered kpasswd5
750/udp closed        kerberos
MAC Address: B8:27:EB:07:73:60 (Raspberry Pi Foundation)

Nmap done: 1 IP address (1 host up) scanned in 1.56 seconds

>From man nmap(1), I gather that

  -sU (UDP scans) .

    [....]

    UDP scan is activated with the -sU option. It can be combined with
    a TCP scan type such as SYN scan (-sS) to check both protocols
    during the same run.

    UDP scan works by sending a UDP packet to every targeted port. For
    some common ports such as 53 and 161, a protocol-specific payload
    is sent, but for most ports the packet is empty..  The
    --data-length option can be used to send a fixed-length random
    payload to every port or (if you specify a value of 0) to disable
    payloads. If an ICMP port unreachable error (type 3, code 3) is
    returned, the port is closed. Other ICMP unreachable errors (type
    3, codes 1, 2, 9, 10, or 13) mark the port as filtered.
    Occasionally, a service will respond with a UDP packet, proving
    that it is open. If no response is received after retransmissions,
    the port is classified as open|filtered. This means that the port
    could be open, or perhaps packet filters are blocking the
    communication. Version detection (-sV) can be used to help
    differentiate the truly open ports from the filtered ones.

    A big challenge with UDP scanning is doing it quickly. Open and
    filtered ports rarely send any response, leaving Nmap to time out
    and then conduct retransmissions just in case the probe or response
    were lost. Closed ports are often an even bigger problem. They
    usually send back an ICMP port unreachable error. But unlike the
    RST packets sent by closed TCP ports in response to a SYN or
    connect scan, many hosts rate limit.  ICMP port unreachable
    messages by default. Linux and Solaris are particularly strict
    about this. For example, the Linux 2.4.20 kernel limits destination
    unreachable messages to one per second (in net/ipv4/icmp.c).

    [...]

I thought that the  "filtered" option signified a time-out. Perhaps
that was quick, but wrong thinking on part. If it does not error out,
but time out, obviously the port is bound and open. 

Rechecking, I get

[cjr at dijkstra:~]$ sudo nmap -sU -sT -sV -p U:88,464,750,T:464,749,754
krb5ldap

Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-11 00:07 CET
Nmap scan report for krb5ldap (192.168.178.3)
Host is up (0.0055s latency).
rDNS record for 192.168.178.3: krb5ldap.hb22.cruwe.de
PORT    STATE         SERVICE       VERSION
464/tcp open          kpasswd5?
749/tcp open          rpcbind
754/tcp closed        krb_prop
88/udp  open          kerberos-sec?
464/udp open|filtered kpasswd5
750/udp closed        kerberos
2 services unrecognized despite returning data. If you know the
service/version, please submit the following fingerprints at
http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
==============NEXT SERVICE FINGERPRINT (SUBMIT
INDIVIDUALLY)==============
SF-Port464-TCP:V=6.40%I=7%D=2/11%Time=52F95BC5%P=amd64-portbld-freebsd9.2%
SF:r(RPCCheck,60,"\0\0\0\\~Z0X\xa0\x03\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4
SF:\x11\x18\x0f20140210230749Z\xa5\x04\x02\x02 at Z\xa6\x03\x02\x01=\xa9\x0f\
SF:x1b\rHB22\.CRUWE\.DE\xaa\x1d0\x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x0
SF:6kadmin\x1b\x08changepw")%r(DNSVersionBindReq,61,"\0\0\0\]~\[0Y\xa0\x03
SF:\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4\x11\x18\x0f20140210230749Z\xa5\x05
SF:\x02\x03\0\x8b>\xa6\x03\x02\x01=\xa9\x0f\x1b\rHB22\.CRUWE\.DE\xaa\x1d0\
SF:x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x06kadmin\x1b\x08changepw")%r(He
SF:lp,61,"\0\0\0\]~\[0Y\xa0\x03\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4\x11\x1
SF:8\x0f20140210230754Z\xa5\x05\x02\x03\x01\0i\xa6\x03\x02\x01=\xa9\x0f\x1
SF:b\rHB22\.CRUWE\.DE\xaa\x1d0\x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x06k
SF:admin\x1b\x08changepw")%r(SSLSessionReq,61,"\0\0\0\]~\[0Y\xa0\x03\x02\x
SF:01\x05\xa1\x03\x02\x01\x1e\xa4\x11\x18\x0f20140210230754Z\xa5\x05\x02\x
SF:03\x01M\x94\xa6\x03\x02\x01=\xa9\x0f\x1b\rHB22\.CRUWE\.DE\xaa\x1d0\x1b\
SF:xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x06kadmin\x1b\x08changepw")%r(X11Pro
SF:be,61,"\0\0\0\]~\[0Y\xa0\x03\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4\x11\x1
SF:8\x0f20140210230754Z\xa5\x05\x02\x03\x027\xf7\xa6\x03\x02\x01=\xa9\x0f\
SF:x1b\rHB22\.CRUWE\.DE\xaa\x1d0\x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x0
SF:6kadmin\x1b\x08changepw")%r(FourOhFourRequest,61,"\0\0\0\]~\[0Y\xa0\x03
SF:\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4\x11\x18\x0f20140210230754Z\xa5\x05
SF:\x02\x03\x02\x866\xa6\x03\x02\x01=\xa9\x0f\x1b\rHB22\.CRUWE\.DE\xaa\x1d
SF:0\x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x06kadmin\x1b\x08changepw")%r(
SF:LPDString,61,"\0\0\0\]~\[0Y\xa0\x03\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4
SF:\x11\x18\x0f20140210230754Z\xa5\x05\x02\x03\x02\xd49\xa6\x03\x02\x01=\x
SF:a9\x0f\x1b\rHB22\.CRUWE\.DE\xaa\x1d0\x1b\xa0\x03\x02\x01\0\xa1\x140\x12
SF:\x1b\x06kadmin\x1b\x08changepw")%r(LDAPBindReq,61,"\0\0\0\]~\[0Y\xa0\x0
SF:3\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4\x11\x18\x0f20140210230754Z\xa5\x0
SF:5\x02\x03\x03\"e\xa6\x03\x02\x01=\xa9\x0f\x1b\rHB22\.CRUWE\.DE\xaa\x1d0
SF:\x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1b\x06kadmin\x1b\x08changepw")%r(S
SF:IPOptions,61,"\0\0\0\]~\[0Y\xa0\x03\x02\x01\x05\xa1\x03\x02\x01\x1e\xa4
SF:\x11\x18\x0f20140210230754Z\xa5\x05\x02\x03\x03pn\xa6\x03\x02\x01=\xa9\
SF:x0f\x1b\rHB22\.CRUWE\.DE\xaa\x1d0\x1b\xa0\x03\x02\x01\0\xa1\x140\x12\x1
SF:b\x06kadmin\x1b\x08changepw");
==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port88-UDP:V=6.40%I=7%D=2/11%Time=52F95BB4%P=amd64-portbld-freebsd9.2%r
SF:(Kerberos,6E,"~l0j\xa0\x03\x02\x01\x05\xa1\x03\x02\x01\x1e\xa2\x11\x18\
SF:x0f19860718214913Z\xa4\x11\x18\x0f20140210230727Z\xa5\x05\x02\x03\x0e\x
SF:99\xa3\xa6\x03\x02\x01\x06\xa9\x04\x1b\x02NM\xaa\x170\x15\xa0\x03\x02\x
SF:01\0\xa1\x0e0\x0c\x1b\x06krbtgt\x1b\x02NM\xab\r\x1b\x0bNULL_CLIENT");
MAC Address: B8:27:EB:07:73:60 (Raspberry Pi Foundation)

Service detection performed. Please report any incorrect results at
http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned
in 88.79 seconds

OK. 88 is open (it has to be, why should kinit try port 88 and then
trigger a segfault when 88 is closed? Obviously kinit is noticed,
resulting in the crash, so 88 needs to be open.)

The first line of the fingerprint seem like something is 
severely wrong, on ARMv6 running 10-stable, amd64-portbld-freebsd9.2
is awfully out of place. That is nmap reporting the machine it's
running on, on another machine I get

[cjr at mccarthy:~]$  sudo nmap -sU -sT -sV -p U:88,464,750,T:464,749,754
krb5ldap

Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-11 00:23 CET
Nmap scan report for krb5ldap (192.168.178.3)
Host is up (0.0034s latency).
rDNS record for 192.168.178.3: krb5ldap.hb22.cruwe.de
PORT    STATE         SERVICE       VERSION
464/tcp open          kpasswd5?
749/tcp open          rpcbind
754/tcp closed        krb_prop
88/udp  open          kerberos-sec?
464/udp open|filtered kpasswd5
750/udp closed        kerberos
2 services unrecognized despite returning data. If you know the
service/version, please submit the following fingerprints at
http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
==============NEXT SERVICE FINGERPRINT (SUBMIT
INDIVIDUALLY)==============
SF-Port464-TCP:V=6.40%I=7%D=2/11%Time=52F95F9D%P=amd64-portbld-freebsd10.0
[...]

file confirms this.

root at krb5ldap:~ # file /usr/local/sbin/krb5kdc 
/usr/local/sbin/krb5kdc: ELF 32-bit LSB executable, ARM, version 1
(SYSV), dynamically linked (uses shared libs), for FreeBSD 10.0
(1000702), stripped

BTW, I scanned nmap before agaist kerberos, which is a CNAME for
krb5ldap. Both yield the same result.

Anyways, thanks for your quick response. I am sorry that gdb did not
provide more data. 

Thanks again, cheers,
-- 
Christopher 
TZ:         GMT + 1h
GnuPG/GPG:  0xE8DE2C14

FreeBSD 9.2-STABLE #1 r256184: Thu Oct 10 19:12:54 CEST 2013
cjr at dijkstra.cruwe.de:/usr/obj/usr/home/cjr/media/src/freebsd/base/stable/9/sys/GEN_WDTRACE 

Punctuation matters:
"Lets eat Grandma." or "Lets eat, Grandma." - Punctuation saves lives.
"A panda eats shoots and leaves." or "A panda eats, shoots, and
leaves." - Punctuation teaches proper biology.

"With sufficient thrust, pigs fly just fine. However, this is not
necessarily a good idea. It is hard to be sure where they are going to
land, and it could be dangerous sitting under them as they fly
overhead." (RFC 1925)