propagation of new service principal keys

[Jerry Shipman]

Hello,

I have seen a (very occasional / pretty minor) problem where:
- a service admin rekeys a service principal, and puts the new keytab on his server
- a user/client gets a service ticket for that service, but from a secondary KDC to which the new service key has not yet propagated
- when the user gives the service ticket to the service, he gets "-1765328154: Unknown code krb5 230", which is maybe KRB5_KT_KVNONOTFOUND (kvno mismatch).
- that problem goes away after a little while after the propagation runs.

I optimistically tried to fix this by adding a "master_kdc" line to the client's krb5.conf, but, maybe that only applies to TGTs and not to service tickets? (that makes sense, I think). It didn't seem to help.

What is the best way to avoid this problem?

I can think of a couple of things:
- service admin can put in a second/new keytab that has both keys, wait some length of time, then put in a third/new keytab that has just the new key. It's an extra step for the service admin, though?
- I can run the propagation more frequently, maybe. It still will have some chance to happen, just a smaller chance? (I have a NAT issue that has, at least so far, prevented me from getting incremental propagation to work.)
- does libkrb5 go through the KDCs in the listed order in krb5.conf? or does it pick a random one from the list? Maybe all I need to do is list the master first on the client? (I don't know why I didn't try that yet.) (It would be nice if the secondaries would take some of the load though, wouldn't it? we also have some geographically-far-apart regions, and maybe it's nice to prefer to go to the closer KDC, except that it happens to be a secondary?)

In practice, this almost never comes up. 
But, I wondered what the usual way to prevent this is?

Thanks a lot,
Jerry Shipman