Problems with kprop and incremental propagation
Jeremy Hunt
jeremyh at optimation.com.au
Wed Nov 10 20:49:21 EST 2010
Hi Matej,
Hmm!
Try explicitly setting:
kdc1.my.domain = MY.DOMAIN
kdc2.my.domain = MY.DOMAIN
in the domain_realms section to fix the domain issue.
On the other issue, you did restart the daemons on both kdc systems after changing the config didn't you? And you did change the config on both servers didn't you?
The kproplog output is misleading after a full resync, I think that is a red herring. After it's first successful incremental update it reports the last matching log stamp. You want to get at least one incremental update to see that, because then you see both sides reporting the same or differing time stamps.
I don't see why you think the log entries are strange, what do you see that you think merits consideration? Don't worry, you might be seeing something that I am not, just tell me what you think is strange. If you point it out I may agree, but at the moment it looks fine to me.
When you have quiet periods does incremental propagation work? It should, and if it doesn't, then it is possible there are issues other than configuration issues. In a quiet period try getting a tcpdump of the traffic on both servers, make sure that there are not network or firewall issues blocking the dialogs.
Is it true that you now have the the poll interval as 2 minutes and you have 550 entries in the logs? How long does a full resync take?
You don't have kprop running as a process or out of inetd.conf on the root server (master) do you?
Regards,
Jeremy
On 11/11/2010 11:33 AM, Matej Zagiba wrote:
> [safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
>
>
> Thanks for reply,
>
> according to strace kprop is reading /etc/krb5.conf
> setting default_realm as global parameter did not help.
>
> The other two settings (iprop_master_ulogsize = 2048, iprop_slave_poll = 30) shoud not have negative impact,
> according to documentation, default values are 1500 entries in log and 2 minutes poll intervall.
>
> I commented out those settings, restarted but no change. kproplog on master returns cca 550 entries,
> starting from 1, so it not the problem of too short logfile. What isstrange is this log sequence:
>
> Nov 10 10:49:09 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=210; Outgoing SerialNo=212, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 15002, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
> Nov 10 10:50:45 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=213; Outgoing SerialNo=214, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>
> In master's kadmin logs, there are UPDATE_OK entries folowed by full_resync. After full_resync, slave's kproplog lookslike this:
>
> root at kdc2:~# kproplog
>
> Kerberos update log (/var/lib/krb5kdc/principal.ulog)
> Update log dump :
> Log version # : 1
> Log state : Stable
> Entry block size : 2048
> Number of entries : 0
> First serial # : None
> Last serial # : 548
> First time stamp : None
> Last time stamp : Thu Nov 11 01:00:40 2010
>
>
> But after UPDATE_OK event,it change to this:
> root at kdc2:~# kproplog
>
> Kerberos update log (/var/lib/krb5kdc/principal.ulog)
> Update log dump :
> Log version # : 1
> Log state : Stable
> Entry block size : 2048
> Number of entries : 0
> Last serial # : None
> Last time stamp : None
>
>
> So it's no wonder, next try triggers full resync.
> Any ideas?
>
> Matej
>
>
>
> On 11/10/2010 11:48 PM, Jeremy Hunt wrote:
>> Hi Matej,
>>
>> Try setting default_realm = KRB.MY.DOMAIN as a global parameter at the top of your krb5.conf file. That should fix problem one.
>>
>> Secondly, you only need iprop_enable and iprop_port to get the incremental propagation going.
>>
>> Your other two settings are nice to have tuning parameters. Until you get incremental proagation working you don't really know what they should be set to. I am guessing they are set too low and the propagation mechanism is spinning out trying to catch up.
>>
>> How incremental propagation works is that it compares the log files on all servers and propagates updates as it can while it can keep the two logs in a synchronised state. You appear to have your log size set too low and so I suspect you are truncating your driver file which sets the flag for full propagation. I am surprised you say that full propagation takes too long, but if so then it probably attempts a full propagation while it is busy. Because it is busy it fails the full propagation and throws away the replica's updates. Then it tries again next cycle. I bet it can do a full propagation in a quiet period.
>>
>> All of my iprop settings are in my kdc.conf file, but obviously your incremental propagation is attempting to work, so I learned something.
>>
>> Apart from that your configuration appears okay.
>>
>> After changing your configuration files, restart all you kerberos daemons.
>>
>> If kprop is still not picking up your domain, run strace or truss on it and see if it is reading the correct file.
>>
>> I hope that helps,
>>
>> Jeremy
>>
>> On 10/11/2010 9:04 PM, Matej Zagiba wrote:
>>> [safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
>>>
>>>
>>> Hello,
>>>
>>> I have two problems with kprop/kpropd. At out site we are using (tying to use) two KDCs both version are 1.8.3 (1.8.3-dfsg-2 from debian repositories). One of them is production server with over 85k proncipals, second shoud be slave server.
>>> I followed install manualhttp://web.mit.edu/kerberos/krb5-1.8/krb5-1.8.3/doc/krb5-install.html#Install%20the%20Slave%20KDCs.
>>> Exact configuration details areat the end of post.
>>>
>>>
>>> First problem with kprop is, it=s not recognize defaut realm:
>>>
>>> root at kdc1:~# /usr/sbin/kprop -f /var/lib/krb5kdc/slave_datatrans kdc2.my.domain
>>> /usr/sbin/kprop: Cannot resolve network address for KDC in requested realm while getting initial ticket
>>>
>>> if I force realm with -r option, everything goes as expected:
>>>
>>> root at kdc1:~# time /usr/sbin/kdb5_util dump /var/lib/krb5kdc/slave_datatrans
>>> real 0m11.119s
>>> user 0m10.685s
>>> sys 0m0.404s
>>> root at kdc1:~# /usr/sbin/kprop.orig -r KRB.MY.DOMAIN -f /var/lib/krb5kdc/slave_datatrans kdc2.my.domain
>>> Database propagation to kdc2.my.domain: SUCCEEDED
>>>
>>> While in usual cron synchronization it is not a big deal, in incremental propagation it means that full resync never
>>> succeed. I wrote a little wrapper aroun kprobe, so full resync now works, but I wonder, if there is anything wrong in my configuration, or if it is bug.
>>>
>>>
>>> Second problem is that kpropd allways asks for full resync. In kadmin logs it looks like this:
>>> === start of kpropd on slave ===
>>> Nov 10 10:43:34 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:43:38 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:43:46 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:43:46 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 14944, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:44:51 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=208; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:45:21 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=208; Outgoing SerialNo=209, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:45:51 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:45:51 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 14968, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:46:57 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:47:27 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:47:57 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:48:27 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_NIL; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:48:57 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:49:01 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_BUSY; Incoming SerialNo=210; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:49:09 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=210; Outgoing SerialNo=212, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:49:39 kdc1 kadmind[9394](Notice): Request: iprop_full_resync_1, spawned resync process 15002, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN, service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>> Nov 10 10:50:45 kdc1 kadmind[9394](Notice): Request: iprop_get_updates_1, UPDATE_OK; Incoming SerialNo=213; Outgoing SerialNo=214, success, client=kiprop/kdc2.my.domain at KRB.MY.DOMAIN,service=kiprop/kdc1.my.domain at KRB.MY.DOMAIN, addr=kdc2_ip
>>>
>>>
>>> Please help me solve this problem, because this way incrementall propagation has no meaning, and conventional use of kprop take too long.
>>>
>>> thanks
>>>
>>> Matej Zagiba
>>>
>>>
>>> configuration:
>>> /etc/krb5.conf (both master and slave):
>>>
>>> [libdefaults]
>>> default_realm = KRB.MY.DOMAIN
>>> kdc_timesync = 1
>>> ccache_type = 4
>>> forwardable = true
>>> proxiable = true
>>>
>>>
>>> [realms]
>>> KRB.MY.DOMAIN = {
>>> kdc = kdc1.my.domain
>>> kdc = kdc2.my.domain
>>> admin_server = kdc1.my.domain
>>> iprop_enable = true
>>> iprop_master_ulogsize = 2048
>>> iprop_slave_poll = 30
>>> iprop_port = 755
>>> }
>>>
>>> [domain_realm]
>>> .my.domain. = KRB.MY.DOMAIN
>>> my.domain. = KRB.MY.DOMAIN
>>>
>>> [logging]
>>> kdc = FILE:/var/log/kdc5.log
>>> admin_server = FILE:/var/log/kadm5.log
>>> default = FILE:/var/log/krb5.log
>>> ________________________________________________
>>> Kerberos mailing list Kerberos at mit.edu
>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>>>
More information about the Kerberos
mailing list