Problems with kprop and incremental propagation - works but bugs found
Jeremy Hunt
jeremyh at optimation.com.au
Thu Nov 11 19:30:23 EST 2010
Hi Matej,
Aaah! Final dot in domainname is "nisplus"ese. I have been supporting nisplus for too long, I missed that. Perhaps you have too. My config has the client machines explicitly named, but in my environment with static machines I can do that, I presume yours has many machines dropping in and out.
Older versions of kerberos had database_name in a specific subsection for the kdc name in the realms section. So you should have it in the kdc1.my.domain and kdc2.my.domain subsections of the realms section, not the KRB.MY.REALM subsection. Alternatively you can define it in the KRB.MY.REALM subsection of the realms section of your kdc.conf file. In the latter case the location of the kdc.conf file is defined in the kdc section of your krb5.conf file.
What you are seeing is a default location set up in your kerberos build. That can be changed too if you want to rebuild kerberos.
But you seem to have solved your problems, document your findings for your successors, ... or you in a couple of years time :).
Regards,
Jeremy
On 12/11/2010 12:32 AM, Matej Zagiba wrote:
> [safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
>
>
> Hi Jeremy,
>
> thanx for shoving me right direction.
>
> First problem was solved by removing last dot in lines
> my.domain. = KRB.MY.REALM
> .my.domain. = KRB.MY.REALM
> in [domain_realm] section. It's strange, I clearly remember to read some advise to put the final dot there.
>
> Second problem is some kind of bug, I run
>
> strace kpropd -S -d
>
> when I expected inckemental propagation to happen, and at the end of output, I saw this:
>
> open("/etc/krb5kdc/principal", O_RDONLY) = -1 ENOENT (No such file or directory)
>
> Well, that's definetly strange, because in /etc/krb5conf/kdc.conf contains this:
>
>
> [kdcdefaults]
> kdc_ports = 88,750
> kdc_tcp_ports = 88,750
>
> [realms]
> KRB.MY.REALM = {
> database_name = /var/lib/krb5kdc/principal
> admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
> acl_file = /etc/krb5kdc/kadm5.acl
> key_stash_file = /etc/krb5kdc/stash
> max_life = 10h 0m 0s
> max_renewable_life = 7d 0h 0m 0s
> master_key_type = des3-hmac-sha1
> supported_enctypes = des3-hmac-sha1:normal arcfour-hmac:normal des-cbc-crc:normal aes256-cts:normal aes128-cts:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3
> default_principal_flags = +preauth
> }
>
> From strace I can see, that kpropd is looking at correct file - /var/lib/krb5kdc/principal.ulog when it checks for serial:
>
> stat("/var/lib/krb5kdc/principal.ulog", {st_mode=S_IFREG|0600, st_size=40, ...}) = 0
> open("/var/lib/krb5kdc/principal.ulog", O_RDWR) = 3
>
> when it comes to applying changes, it uses /etc/krb5kdc/principal. It fails, and kdb5_util invoked for full resync uses correct database.
>
> My workaround for now is to make symlinks of database files (principal, principal.ok, principal.kadm5.lock) to /etc/krb5kdc
> With this, it works, but I wonder, if it's just my misconfiguration, or kprop/kpropd have serious bug using standard kerberos configuration files.
>
>
> Matej
>
> On 11/11/2010 02:49 AM, Jeremy Hunt wrote:
>> Hi Matej,
>>
>> Hmm!
>>
>> Try explicitly setting:
>>
>> kdc1.my.domain = MY.DOMAIN
>> kdc2.my.domain = MY.DOMAIN
>>
>>
>> in the domain_realms section to fix the domain issue.
>>
>> On the other issue, you did restart the daemons on both kdc systems after changing the config didn't you? And you did change the config on both servers didn't you?
>>
>> The kproplog output is misleading after a full resync, I think that is a red herring. After it's first successful incremental update it reports the last matching log stamp. You want to get at least one incremental update to see that, because then you see both sides reporting the same or differing time stamps.
>>
>> I don't see why you think the log entries are strange, what do you see that you think merits consideration? Don't worry, you might be seeing something that I am not, just tell me what you think is strange. If you point it out I may agree, but at the moment it looks fine to me.
>>
>> When you have quiet periods does incremental propagation work? It should, and if it doesn't, then it is possible there are issues other than configuration issues. In a quiet period try getting a tcpdump of the traffic on both servers, make sure that there are not network or firewall issues blocking the dialogs.
>>
>> Is it true that you now have the the poll interval as 2 minutes and you have 550 entries in the logs? How long does a full resync take?
>>
>> You don't have kprop running as a process or out of inetd.conf on the root server (master) do you?
>>
>> Regards,
>>
>> Jeremy
>>
>> On 11/11/2010 11:33 AM, Matej Zagiba wrote:
>>> [safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
>>>
>>>
>>> Thanks for reply,
>>>
>>> according to strace kprop is reading /etc/krb5.conf
>>> setting default_realm as global parameter did not help.
>>>
>>> The other two settings (iprop_master_ulogsize = 2048, iprop_slave_poll = 30) shoud not have negative impact,
>>> according to documentation, default values are 1500 entries in log and 2 minutes poll intervall.
>>>
>>> I commented out those settings, restarted but no change. kproplog on master returns cca 550 entries,
>>> starting from 1, so it not the problem of too short logfile. What isstrange is this log sequence:
>>>
>>> Nov 10 10:49:09 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=210; Outgoing SerialNo=212, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 15002, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:50:45 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=213; Outgoing SerialNo=214, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>
>>> In master's kadmin logs, there are UPDATE_OK entries folowed by full_resync. After full_resync, slave's kproplog lookslike this:
>>>
>>> root at kdc2:~# kproplog
>>>
>>> Kerberos update log (/var/lib/krb5kdc/principal.ulog)
>>> Update log dump :
>>> Log version # : 1
>>> Log state : Stable
>>> Entry block size : 2048
>>> Number of entries : 0
>>> First serial # : None
>>> Last serial # : 548
>>> First time stamp : None
>>> Last time stamp : Thu Nov 11 01:00:40 2010
>>>
>>>
>>> But after UPDATE_OK event,it change to this:
>>> root at kdc2:~# kproplog
>>>
>>> Kerberos update log (/var/lib/krb5kdc/principal.ulog)
>>> Update log dump :
>>> Log version # : 1
>>> Log state : Stable
>>> Entry block size : 2048
>>> Number of entries : 0
>>> Last serial # : None
>>> Last time stamp : None
>>>
>>>
>>> So it's no wonder, next try triggers full resync.
>>> Any ideas?
>>>
>>> Matej
>>>
>>>
>>>
>>> On 11/10/2010 11:48 PM, Jeremy Hunt wrote:
>>>> Hi Matej,
>>>>
>>>> Try setting default_realm = KRB.MY.DOMAIN as a global parameter at the top of your krb5.conf file. That should fix problem one.
>>>>
>>>> Secondly, you only need iprop_enable and iprop_port to get the incremental propagation going.
>>>>
>>>> Your other two settings are nice to have tuning parameters. Until you get incremental proagation working you don't really know what they should be set to. I am guessing they are set too low and the propagation mechanism is spinning out trying to catch up.
>>>>
>>>> How incremental propagation works is that it compares the log files on all servers and propagates updates as it can while it can keep the two logs in a synchronised state. You appear to have your log size set too low and so I suspect you are truncating your driver file which sets the flag for full propagation. I am surprised you say that full propagation takes too long, but if so then it probably attempts a full propagation while it is busy. Because it is busy it fails the full propagation and throws away the replica's updates. Then it tries again next cycle. I bet it can do a full propagation in a quiet period.
>>>>
>>>> All of my iprop settings are in my kdc.conf file, but obviously your incremental propagation is attempting to work, so I learned something.
>>>>
>>>> Apart from that your configuration appears okay.
>>>>
>>>> After changing your configuration files, restart all you kerberos daemons.
>>>>
>>>> If kprop is still not picking up your domain, run strace or truss on it and see if it is reading the correct file.
>>>>
>>>> I hope that helps,
>>>>
>>>> Jeremy
>>>>
>>>> On 10/11/2010 9:04 PM, Matej Zagiba wrote:
>>>>> [safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have two problems with kprop/kpropd. At out site we are using (tying to use) two KDCs both version are 1.8.3 (1.8.3-dfsg-2 from debian repositories). One of them is production server with over 85k proncipals, second shoud be slave server.
>>>>> I followed install manualhttp://web.mit.edu/kerberos/krb5-1.8/krb5-1.8.3/doc/krb5-install.html#Install%20the%20Slave%20KDCs.
>>>>> Exact configuration details areat the end of post.
>>>>>
>>>>>
>>>>> First problem with kprop is, it=s not recognize defaut realm:
>>>>>
>>>>> root at kdc1:~# /usr/sbin/kprop -f /var/lib/krb5kdc/slave_datatrans kdc2.my.domain
>>>>> /usr/sbin/kprop: Cannot resolve network address for KDC in requested realm while getting initial ticket
>>>>>
>>>>> if I force realm with -r option, everything goes as expected:
>>>>>
>>>>> root at kdc1:~# time /usr/sbin/kdb5_util dump /var/lib/krb5kdc/slave_datatrans
>>>>> real 0m11.119s
>>>>> user 0m10.685s
>>>>> sys 0m0.404s
>>>>> root at kdc1:~# /usr/sbin/kprop.orig -r KRB.MY.DOMAIN -f /var/lib/krb5kdc/slave_datatrans kdc2.my.domain
>>>>> Database propagation to kdc2.my.domain: SUCCEEDED
>>>>>
>>>>> While in usual cron synchronization it is not a big deal, in incremental propagation it means that full resync never
>>>>> succeed. I wrote a little wrapper aroun kprobe, so full resync now works, but I wonder, if there is anything wrong in my configuration, or if it is bug.
>>>>>
>>>>>
>>>>> Second problem is that kpropd allways asks for full resync. In kadmin logs it looks like this:
>>>>> === start of kpropd on slave ===
>>>>> Nov 10 10:43:34 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:43:38 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:43:46 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:43:46 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 14944, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:44:51 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=208; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:45:21 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=208; Outgoing SerialNo=209, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:45:51 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:45:51 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 14968, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:46:57 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:47:27 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:47:57 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:48:27 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:48:57 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:49:01 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:49:09 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=210; Outgoing SerialNo=212, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 15002, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>> Nov 10 10:50:45 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=213; Outgoing SerialNo=214, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>>>
>>>>>
>>>>> Please help me solve this problem, because this way incrementall propagation has no meaning, and conventional use of kprop take too long.
>>>>>
>>>>> thanks
>>>>>
>>>>> Matej Zagiba
>>>>>
>>>>>
>>>>> configuration:
>>>>> /etc/krb5.conf (both master and slave):
>>>>>
>>>>> [libdefaults]
>>>>> default_realm = KRB.MY.DOMAIN
>>>>> kdc_timesync = 1
>>>>> ccache_type = 4
>>>>> forwardable = true
>>>>> proxiable = true
>>>>>
>>>>>
>>>>> [realms]
>>>>> KRB.MY.DOMAIN = {
>>>>> kdc = kdc1.my.domain
>>>>> kdc = kdc2.my.domain
>>>>> admin_server = kdc1.my.domain
>>>>> iprop_enable = true
>>>>> iprop_master_ulogsize = 2048
>>>>> iprop_slave_poll = 30
>>>>> iprop_port = 755
>>>>> }
>>>>>
>>>>> [domain_realm]
>>>>> .my.domain. = KRB.MY.DOMAIN
>>>>> my.domain. = KRB.MY.DOMAIN
>>>>>
>>>>> [logging]
>>>>> kdc = FILE:/var/log/kdc5.log
>>>>> admin_server = FILE:/var/log/kadm5.log
>>>>> default = FILE:/var/log/krb5.log
>>>>> ________________________________________________
>>>>> Kerberos mailing list Kerberos at mit.edu
>>>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>>>>>
> ________________________________________________
> Kerberos mailing list Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
--
"The whole modern world has divided itself into Conservatives and Progressives. The business of Progressives is to go on making mistakes. The business of the Conservatives is to prevent the mistakes from being corrected." -- G. K. Chesterton
More information about the Kerberos
mailing list