Segmentation fault when trying to start kadmind
Joshua Schaeffer
jschaeffer0922 at gmail.com
Mon Jul 17 19:48:25 EDT 2017
On 07/17/2017 04:59 PM, Greg Hudson wrote:
> (Sent unicast.)
>
> Hm, our mailing list software seems to have removed all of the content
> from both of your messages, due to some incompatibility with the way
> they were formatted. Would it be possible to combine them and resend
> them as plain text? Unfortunately I no longer have a copy of the
> contents after moderating them through.
>
Sure, no problem, here they are. Let me know if there are still issues with getting my content. I sent this one in plaintext:
I ran the kdb5_util program under valgrind as well and saw this, thought I'd pass it along:
root at bllkrb501:~# valgrind kdb5_util stash -f /etc/krb5kdc/stash
==16389== Memcheck, a memory error detector
==16389== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==16389== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==16389== Command: kdb5_util stash -f /etc/krb5kdc/stash
==16389==
stat(/usr/local/lib/krb5/plugins/kdb/kldap): No such file or directory
get_plugin_data_sym(kdb_function_table)
==16389== Warning: invalid file descriptor -1 in syscall write()
kdb5_util: Cannot find master key record in database while getting master key list
kdb5_util: Warning: proceeding without master key list
Enter KDC database master key:
==16389== Invalid read of size 2
==16389== at 0x506DFA8: krb5_db_fetch_mkey (kdb5.c:1224)
==16389== by 0x406D56: kdb5_stash (kdb5_stash.c:110)
==16389== by 0x4048F1: main (kdb5_util.c:346)
==16389== Address 0x2 is not stack'd, malloc'd or (recently) free'd
==16389==
==16389==
==16389== Process terminating with default action of signal 11 (SIGSEGV)
[...]
NULL pointer?
On 07/17/2017 03:35 PM, Joshua Schaeffer wrote:
> TL;DR
> I'm getting a segmentation error when I run kdb5_util stash from a compiled version of 1.15.1:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff799afa8 in krb5_db_fetch_mkey (context=0x61eb80, mname=0x678a60, etype=18, fromkeyboard=1, twice=0,
> db_args=0x0, kvno=0x7fffffffe56c, salt=0x0, key=0x619c30 <master_keyblock>) at kdb5.c:1224
> 1224 *kvno = (krb5_kvno) master_entry->key_data->key_data_kvno;
>
> --------------------------------------------------------
>
> Hey all,
>
> I'm trying to figure out why I'm getting a segmentation fault when I try to start the krb5-admin-server service. I have a server running in an LXD container, which I think is causing the issue, but I'm not sure what the container doesn't have permissions/rights to that is causing this problem and I've searched all my log files far and wide and can't find any smoking gun. So here is what I've done:
>
> First I tried installing MIT Kerberos using the package management system which installs version 1.13.2. Then I setup my krb5.conf file and initialize my database:
>
> kdb5_ldap_util -D cn=admin,dc=appendata,dc=net create -subtrees 'ou=End Users,ou=People,dc=appendata,dc=net':'ou=Other Users,ou=People,dc=appendata,dc=net -r APPENDATA.NET -s -H ldaps://bllldap01.appendata.net
>
> This works without issue, so I proceed by stashing a few ldap user's passwords, create my kadm5.acl file and then I go and try to start kadmind:
>
> root at bllkrb501:~# systemctl start krb5-admin-server
> root at bllkrb501:~# systemctl status krb5-admin-server
> krb5-admin-server.service - Kerberos 5 Admin Server
> Loaded: loaded (/lib/systemd/system/krb5-admin-server.service; enabled; vendor preset: enabled)
> Drop-In: /lib/systemd/system/krb5-admin-server.service.d
> └─slapd-before-kdc.conf
> Active: failed (Result: core-dump) since Mon 2017-07-17 15:00:36 MDT; 6s ago
> Process: 3304 ExecStart=/usr/sbin/kadmind -nofork $DAEMON_ARGS (code=dumped, signal=SEGV)
> Main PID: 3304 (code=dumped, signal=SEGV)
>
> Jul 17 15:00:35 bllkrb501 systemd[1]: Started Kerberos 5 Admin Server.
> Jul 17 15:00:36 bllkrb501 systemd[1]: krb5-admin-server.service: Main process exited, code=dumped, status=11/SEGV
> Jul 17 15:00:36 bllkrb501 systemd[1]: krb5-admin-server.service: Unit entered failed state.
> Jul 17 15:00:36 bllkrb501 systemd[1]: krb5-admin-server.service: Failed with result 'core-dump'.
>
> And if I try to start kadmind manually:
>
> root at bllkrb501:~# kadmind -nofork
> Segmentation fault (core dumped)
>
> Here is an strace of the same command:
>
> [...]
> write(11, "\27\3\3\2Y\0\0\0\0\0\0\0\3ZMi\3049\2256\337\17y}\361\237\4Kv\f\347\233"..., 606) = 606
> poll([{fd=11, events=POLLIN|POLLPRI}], 1, 300000) = 1 ([{fd=11, revents=POLLIN}])
> read(11, "\27\3\3\0&", 5) = 5
> read(11, "\0\0\0\0\0\0\0\4\313(H\177\362\376\4\34\251\266T\23\5\ndj\327\311\304\30\177\31\26b"..., 38) = 38
> write(11, "\27\3\3\2[\0\0\0\0\0\0\0\4\244G3\341}F\35:\340\244\356\250\254T\365g\7\240r"..., 608) = 608
> poll([{fd=11, events=POLLIN|POLLPRI}], 1, 300000) = 1 ([{fd=11, revents=POLLIN}])
> read(11, "\27\3\3\0&", 5) = 5
> read(11, "\0\0\0\0\0\0\0\5\4\204S\v9\305v\217\324\r\316\313\207\2405\245\2749\242\356\341\361h\367"..., 38) = 38
> write(11, "\27\3\3\2g\0\0\0\0\0\0\0\5\34\306\243F\177zh\370s\352\230\206\243\215\345\3719\\_"..., 620) = 620
> poll([{fd=11, events=POLLIN|POLLPRI}], 1, 300000) = 1 ([{fd=11, revents=POLLIN}])
> read(11, "\27\3\3\2\27", 5) = 5
> read(11, "\0\0\0\0\0\0\0\6\f\332:\226l\34J\0\344v\304K\203\242\0\356[X~\225\347\253\37P"..., 535) = 535
> poll([{fd=11, events=POLLIN|POLLPRI}], 1, 299999) = 1 ([{fd=11, revents=POLLIN}])
> read(11, "\27\3\3\0&", 5) = 5
> read(11, "\0\0\0\0\0\0\0\7v\215\202\33\312\325\316xL4&\305i^\310\21,X\226\211\357\317\323\354"..., 38) = 38
> open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 23
> fstat(23, {st_mode=S_IFREG|0644, st_size=2453, ...}) = 0
> fstat(23, {st_mode=S_IFREG|0644, st_size=2453, ...}) = 0
> read(23, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 2560) = 2453
> lseek(23, -1559, SEEK_CUR) = 894
> read(23, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 2560) = 1559
> close(23) = 0
> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x4} ---
> +++ killed by SIGSEGV (core dumped) +++
> Segmentation fault (core dumped)
>
> I wasn't able to determine too much from this. To me it looks like the system is opening /etc/localtime and then the program crashes. Next I resorted to debugging the error and that is where I'm currently at. I downloaded the source code for 1.15.1, installed it, and reran through all my steps. I wasn't able to get to my step of trying to start kadmind, because it complains the stash file doesn't exist for the master key, which it doesn't, and I'm not sure why it isn't created when I issued my krb5_ldap_util command above, but I was able to still get a segmentation fault when I try to create the stash file:
>
> root at bllkrb501:~# kdb5_util stash
> stat(/usr/local/lib/krb5/plugins/kdb/kldap): No such file or directory
> get_plugin_data_sym(kdb_function_table)
> kdb5_util: Cannot find master key record in database while getting master key list
> kdb5_util: Warning: proceeding without master key list
> Enter KDC database master key:
> Segmentation fault (core dumped)
>
> root at bllkrb501:~# ls -l /usr/local/lib/krb5/plugins/kdb/
> total 407
> -rw-r--r-- 1 root root 366680 Jul 17 12:51 db2.so
> -rw-r--r-- 1 root root 21008 Jul 17 12:51 kldap.so
>
> I'm not sure why it is complaining about plugins/kdb/kldap not existing either. The shared object exists under that directory. Perhaps this is the problem. I compiled Kerberos with "--with-ldap". I've also run the same command through gdb and got the line it is failing at:
>
> root at bllkrb501:~# gdb kdb5_util
> [...]
> Reading symbols from kdb5_util...done.
> (gdb) run stash -f /etc/krb5kdc/stash
> Starting program: /usr/local/sbin/kdb5_util stash -f /etc/krb5kdc/stash
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> stat(/usr/local/lib/krb5/plugins/kdb/kldap): No such file or directory
> get_plugin_data_sym(kdb_function_table)
> kdb5_util: Cannot find master key record in database while getting master key list
> kdb5_util: Warning: proceeding without master key list
> Enter KDC database master key:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff799afa8 in krb5_db_fetch_mkey (context=0x61eb80, mname=0x678a60, etype=18, fromkeyboard=1, twice=0,
> db_args=0x0, kvno=0x7fffffffe56c, salt=0x0, key=0x619c30 <master_keyblock>) at kdb5.c:1224
> 1224 *kvno = (krb5_kvno) master_entry->key_data->key_data_kvno;
> (gdb) continue
> Continuing.
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> The program no longer exists.
> (gdb) quit
>
> I looked at the code and this is where it is actually failing:
>
> 1218 if (kvno != NULL && *kvno == IGNORE_VNO) {
> 1219 krb5_error_code rc;
> 1220 krb5_db_entry *master_entry;
>
> 1222 rc = krb5_db_get_principal(context, mname, 0, &master_entry);
> 1223 if (rc == 0) {
> 1224 *kvno = (krb5_kvno) master_entry->key_data->key_data_kvno;
> 1225 krb5_db_free_principal(context, master_entry);
> 1226 } else
> 1227 *kvno = 1;
> 1228 }
>
> I don't really know where to go from here. I don't know this code well enough to figure out why the segmentation error is occurring. Can anybody help me out? If you need additional information, I'd be happy to provide.
>
> Thanks,
> Joshua Schaeffer
More information about the Kerberos
mailing list