Help with replication

Sun Jul 17 22:36:51 EDT 2022

On 2022-07-16 22:19, Greg Hudson wrote:
> 
> Usually this is a hostname canonicalization issue.  You can set the
> environment variable KRB5_TRACE to a filename, start kpropd, and look 
> in
> the file to see what principal is being looked up.

Thanks Greg.  I should have remembered that.  It exposed the fact that 
the
kiprop/ principal for the host was missing.  I created the principal and 
added
it to /etc/krb5.keytab.  This moved the error, but I am still getting 
failures
to replicate.  Here is the debug log:

$ cat kpropd-debug.log
[27738] 1658108981.225608: Initializing MEMORY:kadm5_0 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108981.225609: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108981.225610: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658108981.225611: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658108981.225613: Sending unauthenticated request
[27738] 1658108981.225614: Sending request (244 bytes) to MYREALM.COM
[27738] 1658108981.225615: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658108981.225616: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658108981.225617: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658108981.225618: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658108981.225619: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658108981.225620: Sending TCP request to stream 172.25.5.49:88
[27738] 1658108981.225621: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658108981.225622: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658108981.225623: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658108981.225624: No URI records found
[27738] 1658108981.225625: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658108981.225626: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658108981.225627: No SRV records found
[27738] 1658108981.225628: Response was not from master KDC
[27738] 1658108981.225629: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658108981.225630: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108981.225631: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658108981.225632: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658108981.225634: Sending unauthenticated request
[27738] 1658108981.225635: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658108981.225636: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658108981.225637: No URI records found
[27738] 1658108981.225638: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658108981.225639: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658108981.225640: No SRV records found
[27738] 1658108981.225641: Destroying ccache MEMORY:kadm5_0
[27738] 1658108985.238223: Initializing MEMORY:kadm5_1 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108985.238224: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108985.238225: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658108985.238226: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658108985.238228: Sending unauthenticated request
[27738] 1658108985.238229: Sending request (244 bytes) to MYREALM.COM
[27738] 1658108985.238230: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658108985.238231: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658108985.238232: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658108985.238233: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658108985.238234: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658108985.238235: Sending TCP request to stream 172.25.5.49:88
[27738] 1658108985.238236: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658108985.238237: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658108985.238238: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658108985.238239: No URI records found
[27738] 1658108985.238240: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658108985.238241: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658108985.238242: No SRV records found
[27738] 1658108985.238243: Response was not from master KDC
[27738] 1658108985.238244: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658108985.238245: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108985.238246: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658108985.238247: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658108985.238249: Sending unauthenticated request
[27738] 1658108985.238250: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658108985.238251: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658108985.238252: No URI records found
[27738] 1658108985.238253: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658108985.238254: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658108985.238255: No SRV records found
[27738] 1658108985.238256: Destroying ccache MEMORY:kadm5_1
[27738] 1658108993.245551: Initializing MEMORY:kadm5_2 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108993.245552: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108993.245553: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658108993.245554: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658108993.245556: Sending unauthenticated request
[27738] 1658108993.245557: Sending request (244 bytes) to MYREALM.COM
[27738] 1658108993.245558: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658108993.245559: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658108993.245560: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658108993.245561: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658108993.245562: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658108993.245563: Sending TCP request to stream 172.25.5.49:88
[27738] 1658108993.245564: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658108993.245565: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658108993.245566: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658108993.245567: No URI records found
[27738] 1658108993.245568: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658108993.245569: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658108993.245570: No SRV records found
[27738] 1658108993.245571: Response was not from master KDC
[27738] 1658108993.245572: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658108993.245573: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658108993.245574: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658108993.245575: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658108993.245577: Sending unauthenticated request
[27738] 1658108993.245578: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658108993.245579: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658108993.245580: No URI records found
[27738] 1658108993.245581: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658108993.245582: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658108993.245583: No SRV records found
[27738] 1658108993.245584: Destroying ccache MEMORY:kadm5_2
[27738] 1658109009.252679: Initializing MEMORY:kadm5_3 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109009.252680: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109009.252681: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109009.252682: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109009.252684: Sending unauthenticated request
[27738] 1658109009.252685: Sending request (244 bytes) to MYREALM.COM
[27738] 1658109009.252686: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658109009.252687: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109009.252688: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658109009.252689: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109009.252690: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658109009.252691: Sending TCP request to stream 172.25.5.49:88
[27738] 1658109009.252692: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658109009.252693: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658109009.252694: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109009.252695: No URI records found
[27738] 1658109009.252696: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109009.252697: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109009.252698: No SRV records found
[27738] 1658109009.252699: Response was not from master KDC
[27738] 1658109009.252700: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658109009.252701: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109009.252702: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109009.252703: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109009.252705: Sending unauthenticated request
[27738] 1658109009.252706: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658109009.252707: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109009.252708: No URI records found
[27738] 1658109009.252709: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109009.252710: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109009.252711: No SRV records found
[27738] 1658109009.252712: Destroying ccache MEMORY:kadm5_3
[27738] 1658109041.259557: Initializing MEMORY:kadm5_4 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109041.259558: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109041.259559: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109041.259560: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109041.259562: Sending unauthenticated request
[27738] 1658109041.259563: Sending request (244 bytes) to MYREALM.COM
[27738] 1658109041.259564: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658109041.259565: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109041.259566: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658109041.259567: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109041.259568: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658109041.259569: Sending TCP request to stream 172.25.5.49:88
[27738] 1658109041.259570: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658109041.259571: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658109041.259572: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109041.259573: No URI records found
[27738] 1658109041.259574: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109041.259575: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109041.259576: No SRV records found
[27738] 1658109041.259577: Response was not from master KDC
[27738] 1658109041.259578: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658109041.259579: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109041.259580: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109041.259581: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109041.259583: Sending unauthenticated request
[27738] 1658109041.259584: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658109041.259585: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109041.259586: No URI records found
[27738] 1658109041.259587: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109041.259588: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109041.259589: No SRV records found
[27738] 1658109041.259590: Destroying ccache MEMORY:kadm5_4
[27738] 1658109105.265034: Initializing MEMORY:kadm5_5 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109105.265035: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109105.265036: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109105.265037: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109105.265039: Sending unauthenticated request
[27738] 1658109105.265040: Sending request (244 bytes) to MYREALM.COM
[27738] 1658109105.265041: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658109105.265042: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109105.265043: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658109105.265044: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109105.265045: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658109105.265046: Sending TCP request to stream 172.25.5.49:88
[27738] 1658109105.265047: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658109105.265048: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658109105.265049: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109105.265050: No URI records found
[27738] 1658109105.265051: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109105.265052: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109105.265053: No SRV records found
[27738] 1658109105.265054: Response was not from master KDC
[27738] 1658109105.265055: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658109105.265056: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109105.265057: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109105.265058: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109105.265060: Sending unauthenticated request
[27738] 1658109105.265061: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658109105.265062: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109105.265063: No URI records found
[27738] 1658109105.265064: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109105.265065: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109105.265066: No SRV records found
[27738] 1658109105.265067: Destroying ccache MEMORY:kadm5_5
[27738] 1658109233.272502: Initializing MEMORY:kadm5_6 with default 
princ kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109233.272503: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109233.272504: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109233.272505: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109233.272507: Sending unauthenticated request
[27738] 1658109233.272508: Sending request (244 bytes) to MYREALM.COM
[27738] 1658109233.272509: Resolving hostname corp-kdc-2.myrealm.com
[27738] 1658109233.272510: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109233.272511: Resolving hostname corp-kdc-1.myrealm.com
[27738] 1658109233.272512: Sending initial UDP request to dgram 
172.25.5.49:88
[27738] 1658109233.272513: Initiating TCP connection to stream 
172.25.5.49:88
[27738] 1658109233.272514: Sending TCP request to stream 172.25.5.49:88
[27738] 1658109233.272515: Received answer (210 bytes) from stream 
172.25.5.49:88
[27738] 1658109233.272516: Terminating TCP connection to stream 
172.25.5.49:88
[27738] 1658109233.272517: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109233.272518: No URI records found
[27738] 1658109233.272519: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109233.272520: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109233.272521: No SRV records found
[27738] 1658109233.272522: Response was not from master KDC
[27738] 1658109233.272523: Received error from KDC: -1765328377/Server 
not found in Kerberos database
[27738] 1658109233.272524: Getting initial credentials for 
kiprop/kdc-pdx-1.myrealm.com at MYREALM.COM
[27738] 1658109233.272525: Setting initial creds service to 
kiprop/corp-kdc-admin.myrealm.com
[27738] 1658109233.272526: Looked up etypes in keytab: aes256-cts, 
aes128-cts, rc4-hmac, des3-cbc-sha1
[27738] 1658109233.272528: Sending unauthenticated request
[27738] 1658109233.272529: Sending request (244 bytes) to MYREALM.COM 
(master)
[27738] 1658109233.272530: Sending DNS URI query for 
_kerberos.MYREALM.COM.
[27738] 1658109233.272531: No URI records found
[27738] 1658109233.272532: Sending DNS SRV query for 
_kerberos-master._udp.MYREALM.COM.
[27738] 1658109233.272533: Sending DNS SRV query for 
_kerberos-master._tcp.MYREALM.COM.
[27738] 1658109233.272534: No SRV records found
[27738] 1658109233.272535: Destroying ccache MEMORY:kadm5_6

This looks like the problem:

     Received error from KDC: -1765328377/Server not found in Kerberos 
database

This is telling me, I think, that the host principal does not exist.  
But, I can
getprinc in kadmin for both the master and the replica hosts.

Note, I was incorrect in the versions in my last note.  The master is 
indeed ubuntu 18.04,
but the replica that I build is 20.04 with krb5-kdc 1.17.  I thought 
that might be the
problem so I built another replica that is 18.04 with 1.16 with the same 
results.

I am not really sure what to look at it next.  I am considering creating 
a completely new
master, loading the current db in it, and then seeing if I can make 
replication work to
the new master.  I would go to Ubuntu 22.04 with krb5-kdc 1.18.  What 
would you recommend?

Thanks for you time,

Bill

-- 
Bill MacAllister <bill at ca-zephyr.org>

"Can't sing louder than the guns when I'm gone,
so I guess I'll have to do it while I'm here."
Phil Ochs