unsubscribe

Dao, Khanh (IS) khanh.dao at ngc.com
Tue Aug 9 20:46:10 EDT 2011


unsubscribe

-----Original Message-----
From: kerberos-bounces at mit.edu [mailto:kerberos-bounces at mit.edu] On Behalf Of kerberos-request at mit.edu
Sent: Tuesday, August 09, 2011 12:40 PM
To: kerberos at mit.edu
Subject: EXT :Kerberos Digest, Vol 104, Issue 11

Send Kerberos mailing list submissions to
	kerberos at mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
	https://mailman.mit.edu/mailman/listinfo/kerberos
or, via email, send a message with subject or body 'help' to
	kerberos-request at mit.edu

You can reach the person managing the list at
	kerberos-owner at mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Kerberos digest..."


Today's Topics:

   1. Re: Performance issues with krb5-1.9.1 (Jonathan Reams)
   2. Re: Performance issues with krb5-1.9.1 (Jonathan Reams)
   3. Problem after adding new realm in krb5.conf (Sanket Sangodkar)
   4. Error while adding new realm to krb5.conf (Sangodkar, Sanket)
   5. Re: Error while adding new realm to krb5.conf (Vipin Rathor)
   6. Re: Performance issues with krb5-1.9.1 (Chris Hecker)


----------------------------------------------------------------------

Message: 1
Date: Mon, 8 Aug 2011 15:49:26 -0400
From: Jonathan Reams <jr3074 at columbia.edu>
Subject: Re: Performance issues with krb5-1.9.1
To: Greg Hudson <ghudson at mit.edu>
Cc: "kerberos at mit.edu" <kerberos at mit.edu>
Message-ID: <75985DF2-F49B-40FB-9473-E856F6EEB212 at columbia.edu>
Content-Type: text/plain; charset=us-ascii

Hi Greg,

I applied this patch and saw a great improvement on our test KDC. I should follow up and say that the performance degradation appeared to be compounded by clients resubmitting their requests after they timed out, so the KDC wasn't only handling new requests, it was trying to fulfill old requests that were being re-submitted. Below is the results of my little performance test with the patch applied to krb5-1.9.1.

[root at naiad ~]# for i in `seq 1 4`; do time for j in `seq 1 1000`; do kinit -k; kvno iprop_monitor | grep -v kvno; done; echo finished round $i; sleep 5; done

real	0m20.312s
user	0m4.513s
sys	0m13.690s
finished round 1

real	0m20.842s
user	0m4.512s
sys	0m13.896s
finished round 2

real	0m21.123s
user	0m4.605s
sys	0m13.787s
finished round 3

real	0m21.425s
user	0m4.508s
sys	0m13.642s
finished round 4

This is much better than the 56 seconds we saw without the patch. We'll roll this out to our secondary KDC and see how it goes this week. Thanks for resolving this so quickly.

Jonathan Reams
CUIT Systems Engineering
Columbia University

On Aug 8, 2011, at 2:21 PM, Greg Hudson wrote:

> 
> I found a regression which would affect these tests, but I'm not sure it
> accounts for your global performance issues.
> 
> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
> but due to a bug, it is actually opening and reading a replay cache for
> every TGS request, which is significantly less efficient than the 1.8
> behavior (using a replay cache which stays open for the lifetime of the
> KDC).
> 
> In a test which runs in under five minutes, this regression produces
> visible O(n^2) performance characteristics.  This would not necessarily
> account for performance degradation over hours, as the performance drag
> of the replay cache should become stable after five minutes.  It's
> possible that the constant drag was enough to cause the KDC to fall
> behind on the request load, but it's also possible that there's a second
> problem which isn't so easily reproduced.
> 
> I've attached a patch.  Note that there is a second, in-memory
> "lookaside" cache with O(n^2) performance characteristics in the short
> term, which holds queries for up to two minutes.  You may see a slight
> degradation in performance in test cases due to this.  You can
> temporarily rebuild the kdc directory with "make clean;
> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
> performance tests.




------------------------------

Message: 2
Date: Tue, 9 Aug 2011 10:13:02 -0400
From: Jonathan Reams <jr3074 at columbia.edu>
Subject: Re: Performance issues with krb5-1.9.1
To: kerberos at mit.edu
Message-ID: <931614F1-F874-4249-BF79-367398654ED4 at columbia.edu>
Content-Type: text/plain; charset=us-ascii

Chris,

We didn't actually see any problems either until the KDC was under heavy load. The unpatched version of 1.9.1 was and still is running on our secondary KDC without issue, and we had been using 1.9.1 in testing and development for months without issue as well. During the period where we saw the performance degradation, the primary KDC handled 467000 distinct AS/TGS requests. Which means the KDC was handling roughly 43 requests per second (not counting lots of retransmits). That is typical of our primary production KDC's workload throughout the day, but we don't have any other KDC that gets that amount of traffic; by contrast, our secondary KDC gets a request once or twice a minute. So it would seem the performance problem only really comes into play when the KDC is under heavy load. 

Jonathan

On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote:

> 
> Just another data point:  I'm not seeing this on my locally built (but 
> not with the attached patch) 1.9.1:
> 
> real    0m41.409s
> user    0m3.358s
> sys     0m3.683s
> finished round 1
> 
> real    0m35.036s
> user    0m3.441s
> sys     0m3.658s
> finished round 2
> 
> real    0m44.344s
> user    0m3.363s
> sys     0m3.728s
> finished round 3
> 
> real    0m40.930s
> user    0m3.465s
> sys     0m3.973s
> finished round 4
> 
> I had to reduce the number of inner iterations to 300 because my machine 
> is slow.  The variance in the above numbers is because there's a bunch 
> of stuff running on this machine.
> 
> Chris
> 
> On 2011/08/08 11:21, Greg Hudson wrote:
>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote:
>>> I did some performance testing on our test KDC and was able to
>>> reproduce the performance issue with 1.9.1.
>> 
>> I found a regression which would affect these tests, but I'm not sure it
>> accounts for your global performance issues.
>> 
>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
>> but due to a bug, it is actually opening and reading a replay cache for
>> every TGS request, which is significantly less efficient than the 1.8
>> behavior (using a replay cache which stays open for the lifetime of the
>> KDC).
>> 
>> In a test which runs in under five minutes, this regression produces
>> visible O(n^2) performance characteristics.  This would not necessarily
>> account for performance degradation over hours, as the performance drag
>> of the replay cache should become stable after five minutes.  It's
>> possible that the constant drag was enough to cause the KDC to fall
>> behind on the request load, but it's also possible that there's a second
>> problem which isn't so easily reproduced.
>> 
>> I've attached a patch.  Note that there is a second, in-memory
>> "lookaside" cache with O(n^2) performance characteristics in the short
>> term, which holds queries for up to two minutes.  You may see a slight
>> degradation in performance in test cases due to this.  You can
>> temporarily rebuild the kdc directory with "make clean;
>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
>> performance tests.
>> 
>> 
>> 
>> 
>> ________________________________________________
>> Kerberos mailing list           Kerberos at mit.edu
>> https://mailman.mit.edu/mailman/listinfo/kerberos
> ________________________________________________
> Kerberos mailing list           Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
> 




------------------------------

Message: 3
Date: Tue, 9 Aug 2011 12:21:37 +0000 (UTC)
From: Sanket Sangodkar <sanket.sangodkar at atos.net>
Subject: Problem after adding new realm in krb5.conf
To: kerberos at mit.edu
Message-ID: <loom.20110809T140429-805 at post.gmane.org>
Content-Type: text/plain; charset=us-ascii

Hi,

We have configured SSO using Kerberos with Apache Http Server.
The realms name are defined in krb5.conf.
We needed to add one more domain to krb5.conf file -

We have inserted new realm as -
 xxx.xxx.COM = {
                kdc = <dns_name>
                admin_server = <dns_name>
        }

But after executing kinit command for respective domain xxx.xxx.COM it 
specifies follwing error message -
kinit: Client not found in Kerberos database while getting initial credentials

We wanted to know whether only specifying new name under [realm] content will 
add new domain/realm to Kerberos configuration.
Or are we missing any more configuration to add new realm ?

Regards,
Sanket




------------------------------

Message: 4
Date: Tue, 9 Aug 2011 17:57:25 +0530
From: "Sangodkar, Sanket" <Sanket.Sangodkar at atos.net>
Subject: Error while adding new realm to krb5.conf
To: <kerberos at mit.edu>
Message-ID:
	<DC18C77D955E0447BCCBFBD690781B99F87BE6 at inicx002.in.int.atosorigin.com>
	
Content-Type: text/plain;	charset="us-ascii"

 

Hi,

 

We have configured SSO using Kerberos with Apache Http Server.

The realms name are defined in krb5.conf.

We needed to add one more domain to krb5.conf file -

 

We have inserted new realm as -

xxx.xxx.COM = {

                kdc = <dns_name>

                admin_server = <dns_name>

        }

 

But after executing kinit command for respective domain xxx.xxx.COM it 

specifies following error message -

kinit: Client not found in Kerberos database while getting initial
credentials

 

We wanted to know whether only specifying new name under [realm] content
will 

add new domain/realm to Kerberos configuration.

Or are we missing any more configuration to add new realm ?

 

Regards,

Sanket

 

 



This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.


------------------------------

Message: 5
Date: Wed, 10 Aug 2011 00:20:32 +0530
From: Vipin Rathor <v.rathor at gmail.com>
Subject: Re: Error while adding new realm to krb5.conf
To: "Sangodkar, Sanket" <Sanket.Sangodkar at atos.net>
Cc: kerberos at mit.edu
Message-ID:
	<CAN-7Vpkoi+yoeyKWTcVGOcbO6W93rXewrtvLzYPqTsvB=GyrFw at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

> kinit: Client not found in Kerberos database while getting initial
> credentials
Are you sure that you are using correct username (for kinit) which
exist in the new realm?
Please make sure that:
1. you are specifying correct username + realm name for getting
ticket. (e.g. kinit user at NEWREALM.COM)
2. The new KDC & kadmin servers are reachable.

> Or are we missing any more configuration to add new realm ?
Can't think of any more configuration that is required.

On Tue, Aug 9, 2011 at 5:57 PM, Sangodkar, Sanket
<Sanket.Sangodkar at atos.net> wrote:
>
>
> Hi,
>
>
>
> We have configured SSO using Kerberos with Apache Http Server.
>
> The realms name are defined in krb5.conf.
>
> We needed to add one more domain to krb5.conf file -
>
>
>
> We have inserted new realm as -
>
> xxx.xxx.COM = {
>
> ? ? ? ? ? ? ? ?kdc = <dns_name>
>
> ? ? ? ? ? ? ? ?admin_server = <dns_name>
>
> ? ? ? ?}
>
>
>
> But after executing kinit command for respective domain xxx.xxx.COM it
>
> specifies following error message -
>
> kinit: Client not found in Kerberos database while getting initial
> credentials
>
>
>
> We wanted to know whether only specifying new name under [realm] content
> will
>
> add new domain/realm to Kerberos configuration.
>
> Or are we missing any more configuration to add new realm ?
>
>
>
> Regards,
>
> Sanket
>
>
>
>
>
>
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
> ________________________________________________
> Kerberos mailing list ? ? ? ? ? Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>



-- 
-Rathor



------------------------------

Message: 6
Date: Tue, 09 Aug 2011 12:39:46 -0700
From: Chris Hecker <checker at d6.com>
Subject: Re: Performance issues with krb5-1.9.1
To: kerberos at mit.edu
Message-ID: <4E418D02.7070505 at d6.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed


Ah, yeah, my tests had krb5kdc at about 50% of one core (slapd was an 
additional 15%), but it wasn't completely saturating the machine.

Glad the patch fixed it!

Chris

On 2011/08/09 07:13, Jonathan Reams wrote:
> Chris,
>
> We didn't actually see any problems either until the KDC was under heavy load. The unpatched version of 1.9.1 was and still is running on our secondary KDC without issue, and we had been using 1.9.1 in testing and development for months without issue as well. During the period where we saw the performance degradation, the primary KDC handled 467000 distinct AS/TGS requests. Which means the KDC was handling roughly 43 requests per second (not counting lots of retransmits). That is typical of our primary production KDC's workload throughout the day, but we don't have any other KDC that gets that amount of traffic; by contrast, our secondary KDC gets a request once or twice a minute. So it would seem the performance problem only really comes into play when the KDC is under heavy load.
>
> Jonathan
>
> On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote:
>
>>
>> Just another data point:  I'm not seeing this on my locally built (but
>> not with the attached patch) 1.9.1:
>>
>> real    0m41.409s
>> user    0m3.358s
>> sys     0m3.683s
>> finished round 1
>>
>> real    0m35.036s
>> user    0m3.441s
>> sys     0m3.658s
>> finished round 2
>>
>> real    0m44.344s
>> user    0m3.363s
>> sys     0m3.728s
>> finished round 3
>>
>> real    0m40.930s
>> user    0m3.465s
>> sys     0m3.973s
>> finished round 4
>>
>> I had to reduce the number of inner iterations to 300 because my machine
>> is slow.  The variance in the above numbers is because there's a bunch
>> of stuff running on this machine.
>>
>> Chris
>>
>> On 2011/08/08 11:21, Greg Hudson wrote:
>>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote:
>>>> I did some performance testing on our test KDC and was able to
>>>> reproduce the performance issue with 1.9.1.
>>>
>>> I found a regression which would affect these tests, but I'm not sure it
>>> accounts for your global performance issues.
>>>
>>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
>>> but due to a bug, it is actually opening and reading a replay cache for
>>> every TGS request, which is significantly less efficient than the 1.8
>>> behavior (using a replay cache which stays open for the lifetime of the
>>> KDC).
>>>
>>> In a test which runs in under five minutes, this regression produces
>>> visible O(n^2) performance characteristics.  This would not necessarily
>>> account for performance degradation over hours, as the performance drag
>>> of the replay cache should become stable after five minutes.  It's
>>> possible that the constant drag was enough to cause the KDC to fall
>>> behind on the request load, but it's also possible that there's a second
>>> problem which isn't so easily reproduced.
>>>
>>> I've attached a patch.  Note that there is a second, in-memory
>>> "lookaside" cache with O(n^2) performance characteristics in the short
>>> term, which holds queries for up to two minutes.  You may see a slight
>>> degradation in performance in test cases due to this.  You can
>>> temporarily rebuild the kdc directory with "make clean;
>>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
>>> performance tests.
>>>
>>>
>>>
>>>
>>> ________________________________________________
>>> Kerberos mailing list           Kerberos at mit.edu
>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>> ________________________________________________
>> Kerberos mailing list           Kerberos at mit.edu
>> https://mailman.mit.edu/mailman/listinfo/kerberos
>>
>
>
> ________________________________________________
> Kerberos mailing list           Kerberos at mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>


------------------------------

_______________________________________________
Kerberos mailing list
Kerberos at mit.edu
https://mailman.mit.edu/mailman/listinfo/kerberos


End of Kerberos Digest, Vol 104, Issue 11
*****************************************




More information about the Kerberos mailing list