special ccache performance issue

[Charles Hedrick]

We have a workaround, although it wasn’t intended for this purpose.

In https://github.com/clhedrick/kerberos, look at krenew-wrapper. It builds a sharable library intended to be loaded with LD_PRELOAD. It wraps krb5_init_context with code that renews and copies the TGT into a memory cache, and sets KRB5CCNAME to point to that cache. For your purposes I wouldn’t do the renew, as that’s going to thrash the KDC. I’d just copy the TGT credentials.

The actual purpose is to solve a problem that may actually not be significant. The problem is that when you do ssh, it forwards the ticket to the new host with the current time as start, and the original expiration time. Suppose you have a TGT that you got at 9am, Jan 1. It will (at most sites) expire at 9am, Jan 2. Now you come in on Jan 2 at 8:45. Your ticket is still valid, but only has 15 minutes of life left. You ssh to another site. You now end up with a ticket starting at 8:45 and ending at 9am. If you renew it, you get another ticket with 15 min life.

This is significant for us because we automatically renew tickets as long as a user is logged in. But we only check every hour. We don’t want to have any tickets with stupidly short lifetimes. I don’t think with our current parameters that this issue is likely to occur in practice, but we used to use different parameters where the issue was a real one. But the code could still be used to solve your problem, though I’d recommend modifying it so you don't renew the credentials each time.

I think you want remove krb5_get_renewed_creds and creds_valid = 1; and replace krb5_cc_store_cred with

krb5_cc_copy_creds(ctx, ccache, newcache)

The main code then looks like this:

  cctype = krb5_cc_get_type(ctx, ccache);
  code = krb5_cc_get_principal(ctx, ccache, &user);   if (code) goto done;
  code = krb5_cc_resolve(ctx, "MEMORY:renewtmp", &newcache); if (code) goto done;
  code = krb5_cc_initialize(ctx, newcache, user); if (code) goto done;
  code = krb5_cc_copy_creds(ctx, ccache, newcache); if (code) goto done;

Then you want to configure ansible so instead of calling ssh directly it calls a script that does

export LD_PRELOAD=…./krenew-wrap.so
exec /bin/sh “$@"

I did actually test the modified code...


On May 13, 2019, at 11:07 AM, Greg Hudson <ghudson at mit.edu<mailto:ghudson at mit.edu>> wrote:

On 5/13/19 3:22 AM, Wang Jian wrote:
When using ansible with kerberos for thousands of targets, there is a
serious ccache performance issue.

Agreed.

Using file ccache (DIR:)
- from a cold ccache, running simple script on servers is fast, at 500-700
hosts/min with 2 or 4 concurrent ansible instance. But things change when
ccache has over 5000 host tickets. The speed drops to 10-30/min and sys CPU
keeps very high.
- High file lock intesion which consumes nearly all CPU

One small improvement we know we could make is to stop locking file
ccaches for reads, since it's an append-only format.  (We would have to
ignore truncated records at the end when reading, instead of erroring
out.)  This would only help a little bit, since the real problem is
O(n^2) performance.

A more ambitious possibility is to write a config entry into the cache
which acts as a hash table for service tickets, while old
implementations will read past it.  (On hash collision, simply overwrite
the old ticket.)  For resource and complexity reasons I'm not sure that
could be implemented any time soon.

Using kernel keyring ccahe
- fast from start, but eventually, continuous failure, and high sys CPU
- from klist -a, the output is empty now and then, which indicates that
keyring has kneed down under pressure

Users are only allowed to use a certain amount of keyring space, so
maybe you're running into this.  (It has been argued that the keyring
ccache type should simply store an encryption key which is then used for
a file-based ccache, and that could be done with a new name.  But then
you're back to needing a high-performance file-based ccache type.)

Using Heimdal KCM
- didn't try. Heimdal KCM uses sequential algorithm and single lock

SSSD also has a KCM server implementation, though I don't know much
about its performance characteristics.  Regardless, the KCM client
(Heimdal and MIT krb5) iterates over the ccache for get_principal,
making O(n) behavior impossible.  Heimdal's KCM server has a
currently-unused get_principal operation which makes its own TGS request
when a credential isn't found (similar to the Microsoft LSA ccache on
Windows); I am undecided on whether that's desirable behavior.

I know this is a special case, but perhaps it should be addressed.

It seems to be a rare case, but it has come up before.

On the user end, you can work around this by manually swapping out the
ccache for each request, either to a copy of the initial ccache
(sacrificing caching) or to a per-target-host ccache.  That's obviously
a lot of work you shouldn't have to do at a minimum, and could be
impractical in some cases.

We've talked about an environment variable which suppresses caching, but
it hasn't been implemented in either MIT krb5 or Heimdal.
________________________________________________
Kerberos mailing list           Kerberos at mit.edu<mailto:Kerberos at mit.edu>
https://mailman.mit.edu/mailman/listinfo/kerberos