[krbdev.mit.edu #8744] Issues when rolling the master key online

Sun Sep 30 18:29:29 EDT 2018

Following the instructions as given when using incremental propagation
(https://web.mit.edu/kerberos/krb5-1.12/doc/admin/database.html#updating-the-master-key),
it seems that you can end up with the master KDC in a bad way AND a dead kpropd on the slave.

Also, there's another case where things can go wrong in a different way when the update log rolls over.
The full resync request gets raised but doesn't get fulfilled until daemon processes get restarted. kpropd doesn't crash in that case, though.
There may also be a bad result when the kdb principal encryption incremental update is bundled with the mkey puge. 
Let me know if you want the logs from those, too. 

Master: Solaris 11 server running MIT 1.15
Slave: Fedora 28 server running MIT 1.16.1 (provided with the distro)

Also tried this with both hosts being 1.13.2 (Solaris 10), 15.1 (RHEL 7) and later with both running 1.16.1 on the hosts as described above and achieved similar results.

Create new mkey, wait for slave update (kdb5_util add_mkey -s)
-------------------------------------------------------------------------------------
MASTER LOG
Sep 29 17:35:48 endless.foonon.com kadmind[18090](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=10; Outgoing SerialNo=11, success, client=kiprop/topper28.foonon.com at FOONON.COM, service=kiprop/endless.foonon.com at FOONON.COM, addr=192.168.1.224
SLAVE LOG
Sep 29 17:35:48 topper28 kpropd[27555]: Incremental updates: 1 updates / 7756 us

Use new mkey & update princs: kdb5_util use_mkey 2 ; kdb5_util update_princ_encryption (1132 principals)
-----------------------------------------------------------------------------------------------------------------------------------------------

# kdb5_util -d ./principal -sf ./.k5.FOONON.COM  list_mkeys
Master keys for Principal: K/M at FOONON.COM
KVNO: 2, Enctype: aes256-cts-hmac-sha1-96, Active on: Sat Sep 29 17:37:20 EDT 2018 *

MASTER LOG
Sep 29 17:37:48 endless.foonon.com kadmind[18090](Notice): Request: iprop_get_updates_1, UPDATE_ERROR; Incoming SerialNo=11; Outgoing SerialNo=N/A, Update log conversion error, client=kiprop/topper28.foonon.com at FOONON.COM, service=kiprop/endless.foonon.com at FOONON.COM, addr=192.168.1.224
Sep 29 17:37:48 endless.foonon.com kadmind[18090](info): closing down fd 21
SLAVE LOG
Sep 29 17:37:48 topper28 kpropd[27555]: get_updates, error returned from master KDC.
Sep 29 17:37:48 topper28 kpropd[27555]: ERROR returned by master KDC, bailing.
Sep 29 17:37:48 topper28 kpropd[27555]: /usr/sbin/kpropd: Operation not permitted do_iprop failed.

Purge old key: kdb5_util purge_mkeys
---------------------------------------------------
kadmin.local:  getprinc K/M
Principal: K/M at FOONON.COM
...
Last modified: Sat Sep 29 17:37:59 EDT 2018 (K/M at FOONON.COM)
...
Number of keys: 1
Key: vno 2, aes256-cts-hmac-sha1-96
MKey: vno 2
...

KDC MASTER LOG
Sep 29 17:39:49 endless.foonon.com krb5kdc[18065](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 192.168.1.200: DECRYPT_CLIENT_KEY: host/endless.foonon.com at FOONON.COM for krbtgt/FOONON.COM at FOONON.COM, Decrypt integrity check failed
Sep 29 17:43:08 endless.foonon.com krb5kdc[18065](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 192.168.1.200: DECRYPT_CLIENT_KEY: host/endless.foonon.com at FOONON.COM for krbtgt/FOONON.COM at FOONON.COM, Decrypt integrity check failed

Restart kadmind
----------------------
KDC MASTER LOG
Sep 29 17:44:36 endless.foonon.com krb5kdc[18065](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 192.168.1.200: DECRYPT_CLIENT_KEY: host/endless.foonon.com at FOONON.COM for krbtgt/FOONON.COM at FOONON.COM, Decrypt integrity check failed

Restart krb5kdc
---------------------
KDC MASTER LOG
Sep 29 17:45:00 endless.foonon.com krb5kdc[18166](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 192.168.1.200: ISSUE: authtime 1538257500, etypes {rep=18 tkt=18 ses=18}, host/endless.foonon.com at FOONON.COM for krbtgt/FOONON.COM at FOONON.COM

Restart kpropd
--------------------
MASTER LOG
Sep 29 17:47:53 endless.foonon.com kadmind[18160](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=11; Outgoing SerialNo=1145, success, client=kiprop/topper28.foonon.com at FOONON.COM, service=kiprop/endless.foonon.com at FOONON.COM, addr=192.168.1.224
SLAVE LOG
Sep 29 17:47:53 topper28 kpropd[27627]: Incremental updates: 1134 updates / 428790 us

Changes included in the incremental update: activating the new master key, the princ enc changes, purging the old K/M key

Normal operation resumes.