unsubscribe
Dao, Khanh (IS)
khanh.dao at ngc.com
Tue Aug 9 20:46:10 EDT 2011
unsubscribe
-----Original Message-----
From: kerberos-bounces at mit.edu [mailto:kerberos-bounces at mit.edu] On Behalf Of kerberos-request at mit.edu
Sent: Tuesday, August 09, 2011 12:40 PM
To: kerberos at mit.edu
Subject: EXT :Kerberos Digest, Vol 104, Issue 11
Send Kerberos mailing list submissions to
kerberos at mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.mit.edu/mailman/listinfo/kerberos
or, via email, send a message with subject or body 'help' to
kerberos-request at mit.edu
You can reach the person managing the list at
kerberos-owner at mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Kerberos digest..."
Today's Topics:
1. Re: Performance issues with krb5-1.9.1 (Jonathan Reams)
2. Re: Performance issues with krb5-1.9.1 (Jonathan Reams)
3. Problem after adding new realm in krb5.conf (Sanket Sangodkar)
4. Error while adding new realm to krb5.conf (Sangodkar, Sanket)
5. Re: Error while adding new realm to krb5.conf (Vipin Rathor)
6. Re: Performance issues with krb5-1.9.1 (Chris Hecker)
----------------------------------------------------------------------
Message: 1
Date: Mon, 8 Aug 2011 15:49:26 -0400
From: Jonathan Reams <jr3074 at columbia.edu>
Subject: Re: Performance issues with krb5-1.9.1
To: Greg Hudson <ghudson at mit.edu>
Cc: "kerberos at mit.edu" <kerberos at mit.edu>
Message-ID: <75985DF2-F49B-40FB-9473-E856F6EEB212 at columbia.edu>
Content-Type: text/plain; charset=us-ascii
Hi Greg,
I applied this patch and saw a great improvement on our test KDC. I should follow up and say that the performance degradation appeared to be compounded by clients resubmitting their requests after they timed out, so the KDC wasn't only handling new requests, it was trying to fulfill old requests that were being re-submitted. Below is the results of my little performance test with the patch applied to krb5-1.9.1.
[root at naiad ~]# for i in `seq 1 4`; do time for j in `seq 1 1000`; do kinit -k; kvno iprop_monitor | grep -v kvno; done; echo finished round $i; sleep 5; done
real 0m20.312s
user 0m4.513s
sys 0m13.690s
finished round 1
real 0m20.842s
user 0m4.512s
sys 0m13.896s
finished round 2
real 0m21.123s
user 0m4.605s
sys 0m13.787s
finished round 3
real 0m21.425s
user 0m4.508s
sys 0m13.642s
finished round 4
This is much better than the 56 seconds we saw without the patch. We'll roll this out to our secondary KDC and see how it goes this week. Thanks for resolving this so quickly.
Jonathan Reams
CUIT Systems Engineering
Columbia University
On Aug 8, 2011, at 2:21 PM, Greg Hudson wrote:
>
> I found a regression which would affect these tests, but I'm not sure it
> accounts for your global performance issues.
>
> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
> but due to a bug, it is actually opening and reading a replay cache for
> every TGS request, which is significantly less efficient than the 1.8
> behavior (using a replay cache which stays open for the lifetime of the
> KDC).
>
> In a test which runs in under five minutes, this regression produces
> visible O(n^2) performance characteristics. This would not necessarily
> account for performance degradation over hours, as the performance drag
> of the replay cache should become stable after five minutes. It's
> possible that the constant drag was enough to cause the KDC to fall
> behind on the request load, but it's also possible that there's a second
> problem which isn't so easily reproduced.
>
> I've attached a patch. Note that there is a second, in-memory
> "lookaside" cache with O(n^2) performance characteristics in the short
> term, which holds queries for up to two minutes. You may see a slight
> degradation in performance in test cases due to this. You can
> temporarily rebuild the kdc directory with "make clean;
> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
> performance tests.
------------------------------
Message: 2
Date: Tue, 9 Aug 2011 10:13:02 -0400
From: Jonathan Reams <jr3074 at columbia.edu>
Subject: Re: Performance issues with krb5-1.9.1
To: kerberos at mit.edu
Message-ID: <931614F1-F874-4249-BF79-367398654ED4 at columbia.edu>
Content-Type: text/plain; charset=us-ascii
Chris,
We didn't actually see any problems either until the KDC was under heavy load. The unpatched version of 1.9.1 was and still is running on our secondary KDC without issue, and we had been using 1.9.1 in testing and development for months without issue as well. During the period where we saw the performance degradation, the primary KDC handled 467000 distinct AS/TGS requests. Which means the KDC was handling roughly 43 requests per second (not counting lots of retransmits). That is typical of our primary production KDC's workload throughout the day, but we don't have any other KDC that gets that amount of traffic; by contrast, our secondary KDC gets a request once or twice a minute. So it would seem the performance problem only really comes into play when the KDC is under heavy load.
Jonathan
On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote:
>
> Just another data point: I'm not seeing this on my locally built (but
> not with the attached patch) 1.9.1:
>
> real 0m41.409s
> user 0m3.358s
> sys 0m3.683s
> finished round 1
>
> real 0m35.036s
> user 0m3.441s
> sys 0m3.658s
> finished round 2
>
> real 0m44.344s
> user 0m3.363s
> sys 0m3.728s
> finished round 3
>
> real 0m40.930s
> user 0m3.465s
> sys 0m3.973s
> finished round 4
>
> I had to reduce the number of inner iterations to 300 because my machine
> is slow. The variance in the above numbers is because there's a bunch
> of stuff running on this machine.
>
> Chris
>
> On 2011/08/08 11:21, Greg Hudson wrote:
>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote:
>>> I did some performance testing on our test KDC and was able to
>>> reproduce the performance issue with 1.9.1.
>>
>> I found a regression which would affect these tests, but I'm not sure it
>> accounts for your global performance issues.
>>
>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
>> but due to a bug, it is actually opening and reading a replay cache for
>> every TGS request, which is significantly less efficient than the 1.8
>> behavior (using a replay cache which stays open for the lifetime of the
>> KDC).
>>
>> In a test which runs in under five minutes, this regression produces
>> visible O(n^2) performance characteristics. This would not necessarily
>> account for performance degradation over hours, as the performance drag
>> of the replay cache should become stable after five minutes. It's
>> possible that the constant drag was enough to cause the KDC to fall
>> behind on the request load, but it's also possible that there's a second
>> problem which isn't so easily reproduced.
>>
>> I've attached a patch. Note that there is a second, in-memory
>> "lookaside" cache with O(n^2) performance characteristics in the short
>> term, which holds queries for up to two minutes. You may see a slight
>> degradation in performance in test cases due to this. You can
>> temporarily rebuild the kdc directory with "make clean;
>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
>> performance tests.
>>
>>
>>
>>
>> ________________________________________________
>> Kerberos mailing list Kerberos at mit.edu
>> https://mailman.mit.edu/mailman/listinfo/kerberos
> ________________________________________________
> Kerberos mailing list Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
------------------------------
Message: 3
Date: Tue, 9 Aug 2011 12:21:37 +0000 (UTC)
From: Sanket Sangodkar <sanket.sangodkar at atos.net>
Subject: Problem after adding new realm in krb5.conf
To: kerberos at mit.edu
Message-ID: <loom.20110809T140429-805 at post.gmane.org>
Content-Type: text/plain; charset=us-ascii
Hi,
We have configured SSO using Kerberos with Apache Http Server.
The realms name are defined in krb5.conf.
We needed to add one more domain to krb5.conf file -
We have inserted new realm as -
xxx.xxx.COM = {
kdc = <dns_name>
admin_server = <dns_name>
}
But after executing kinit command for respective domain xxx.xxx.COM it
specifies follwing error message -
kinit: Client not found in Kerberos database while getting initial credentials
We wanted to know whether only specifying new name under [realm] content will
add new domain/realm to Kerberos configuration.
Or are we missing any more configuration to add new realm ?
Regards,
Sanket
------------------------------
Message: 4
Date: Tue, 9 Aug 2011 17:57:25 +0530
From: "Sangodkar, Sanket" <Sanket.Sangodkar at atos.net>
Subject: Error while adding new realm to krb5.conf
To: <kerberos at mit.edu>
Message-ID:
<DC18C77D955E0447BCCBFBD690781B99F87BE6 at inicx002.in.int.atosorigin.com>
Content-Type: text/plain; charset="us-ascii"
Hi,
We have configured SSO using Kerberos with Apache Http Server.
The realms name are defined in krb5.conf.
We needed to add one more domain to krb5.conf file -
We have inserted new realm as -
xxx.xxx.COM = {
kdc = <dns_name>
admin_server = <dns_name>
}
But after executing kinit command for respective domain xxx.xxx.COM it
specifies following error message -
kinit: Client not found in Kerberos database while getting initial
credentials
We wanted to know whether only specifying new name under [realm] content
will
add new domain/realm to Kerberos configuration.
Or are we missing any more configuration to add new realm ?
Regards,
Sanket
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
------------------------------
Message: 5
Date: Wed, 10 Aug 2011 00:20:32 +0530
From: Vipin Rathor <v.rathor at gmail.com>
Subject: Re: Error while adding new realm to krb5.conf
To: "Sangodkar, Sanket" <Sanket.Sangodkar at atos.net>
Cc: kerberos at mit.edu
Message-ID:
<CAN-7Vpkoi+yoeyKWTcVGOcbO6W93rXewrtvLzYPqTsvB=GyrFw at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
> kinit: Client not found in Kerberos database while getting initial
> credentials
Are you sure that you are using correct username (for kinit) which
exist in the new realm?
Please make sure that:
1. you are specifying correct username + realm name for getting
ticket. (e.g. kinit user at NEWREALM.COM)
2. The new KDC & kadmin servers are reachable.
> Or are we missing any more configuration to add new realm ?
Can't think of any more configuration that is required.
On Tue, Aug 9, 2011 at 5:57 PM, Sangodkar, Sanket
<Sanket.Sangodkar at atos.net> wrote:
>
>
> Hi,
>
>
>
> We have configured SSO using Kerberos with Apache Http Server.
>
> The realms name are defined in krb5.conf.
>
> We needed to add one more domain to krb5.conf file -
>
>
>
> We have inserted new realm as -
>
> xxx.xxx.COM = {
>
> ? ? ? ? ? ? ? ?kdc = <dns_name>
>
> ? ? ? ? ? ? ? ?admin_server = <dns_name>
>
> ? ? ? ?}
>
>
>
> But after executing kinit command for respective domain xxx.xxx.COM it
>
> specifies following error message -
>
> kinit: Client not found in Kerberos database while getting initial
> credentials
>
>
>
> We wanted to know whether only specifying new name under [realm] content
> will
>
> add new domain/realm to Kerberos configuration.
>
> Or are we missing any more configuration to add new realm ?
>
>
>
> Regards,
>
> Sanket
>
>
>
>
>
>
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
> ________________________________________________
> Kerberos mailing list ? ? ? ? ? Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
--
-Rathor
------------------------------
Message: 6
Date: Tue, 09 Aug 2011 12:39:46 -0700
From: Chris Hecker <checker at d6.com>
Subject: Re: Performance issues with krb5-1.9.1
To: kerberos at mit.edu
Message-ID: <4E418D02.7070505 at d6.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Ah, yeah, my tests had krb5kdc at about 50% of one core (slapd was an
additional 15%), but it wasn't completely saturating the machine.
Glad the patch fixed it!
Chris
On 2011/08/09 07:13, Jonathan Reams wrote:
> Chris,
>
> We didn't actually see any problems either until the KDC was under heavy load. The unpatched version of 1.9.1 was and still is running on our secondary KDC without issue, and we had been using 1.9.1 in testing and development for months without issue as well. During the period where we saw the performance degradation, the primary KDC handled 467000 distinct AS/TGS requests. Which means the KDC was handling roughly 43 requests per second (not counting lots of retransmits). That is typical of our primary production KDC's workload throughout the day, but we don't have any other KDC that gets that amount of traffic; by contrast, our secondary KDC gets a request once or twice a minute. So it would seem the performance problem only really comes into play when the KDC is under heavy load.
>
> Jonathan
>
> On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote:
>
>>
>> Just another data point: I'm not seeing this on my locally built (but
>> not with the attached patch) 1.9.1:
>>
>> real 0m41.409s
>> user 0m3.358s
>> sys 0m3.683s
>> finished round 1
>>
>> real 0m35.036s
>> user 0m3.441s
>> sys 0m3.658s
>> finished round 2
>>
>> real 0m44.344s
>> user 0m3.363s
>> sys 0m3.728s
>> finished round 3
>>
>> real 0m40.930s
>> user 0m3.465s
>> sys 0m3.973s
>> finished round 4
>>
>> I had to reduce the number of inner iterations to 300 because my machine
>> is slow. The variance in the above numbers is because there's a bunch
>> of stuff running on this machine.
>>
>> Chris
>>
>> On 2011/08/08 11:21, Greg Hudson wrote:
>>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote:
>>>> I did some performance testing on our test KDC and was able to
>>>> reproduce the performance issue with 1.9.1.
>>>
>>> I found a regression which would affect these tests, but I'm not sure it
>>> accounts for your global performance issues.
>>>
>>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
>>> but due to a bug, it is actually opening and reading a replay cache for
>>> every TGS request, which is significantly less efficient than the 1.8
>>> behavior (using a replay cache which stays open for the lifetime of the
>>> KDC).
>>>
>>> In a test which runs in under five minutes, this regression produces
>>> visible O(n^2) performance characteristics. This would not necessarily
>>> account for performance degradation over hours, as the performance drag
>>> of the replay cache should become stable after five minutes. It's
>>> possible that the constant drag was enough to cause the KDC to fall
>>> behind on the request load, but it's also possible that there's a second
>>> problem which isn't so easily reproduced.
>>>
>>> I've attached a patch. Note that there is a second, in-memory
>>> "lookaside" cache with O(n^2) performance characteristics in the short
>>> term, which holds queries for up to two minutes. You may see a slight
>>> degradation in performance in test cases due to this. You can
>>> temporarily rebuild the kdc directory with "make clean;
>>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
>>> performance tests.
>>>
>>>
>>>
>>>
>>> ________________________________________________
>>> Kerberos mailing list Kerberos at mit.edu
>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>> ________________________________________________
>> Kerberos mailing list Kerberos at mit.edu
>> https://mailman.mit.edu/mailman/listinfo/kerberos
>>
>
>
> ________________________________________________
> Kerberos mailing list Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
------------------------------
_______________________________________________
Kerberos mailing list
Kerberos at mit.edu
https://mailman.mit.edu/mailman/listinfo/kerberos
End of Kerberos Digest, Vol 104, Issue 11
*****************************************
More information about the Kerberos
mailing list