[RFC] Improvements to KCM prococol and notification mechanism

[Pavel Březina]

Hi krbdev,
I would like to propose two (backward compatible) changes to the KCM 
protocol to improve performance of sssd-kcm. Please comment on the 
following design and let me know if this is something MIT Kerberos would 
accept. Also let me know if there is other channel where changes to KCM 
protocol should be discussed.

Thank you,
Pavel.

Improve KCM performance when listing credentials
################################################

Problem Statement
*****************

When Kerberos needs to list user’s credentials that are contained within
a specific ccache, it first obtains a list of ccache’s credentials UUIDs
and then iterates over each UUID to resolve it to a Kerberos credential.
This means that Kerberos sends a KCM request to the KCM daemon for each
credential within the ccache. These IO operations create a significant
bottleneck for large ccaches that contain hundreds of credentials.

The performance degradation can be seen in comparison with other ccache
types. For example, a gssapi-authenticated SSH connection to hundreds of
Ansible hosts can take several minutes with SSSD KCM, but takes only a
few seconds with the KEYRING ccache.

Note that bottlenecks in SSSD KCM itself were already addressed [1] and
the IO operations can be clearly identified as a bottleneck with
profiling tools like perf.

[1] https://github.com/SSSD/sssd/pull/5375

Solution overview
*****************

A new operation GET_CRED_LIST will be added. This operation will return
a snapshot of the current credentials in a given ccache, instead of
returning a list of UUIDs.

Implementation details
**********************

SSSD KCM will implement a new KCM operation GET_CRED_LIST that will
return all credentials available in the ccache.
* Input will be the same as for GET_CRED_UUID_LIST
* Reply will be: KCM_HEADER + NUM_CREDS + (LEN + CRED) …
   * KCM_HEADER is the standard reply header (len + return code)
   * NUM_CREDS is uint32, a total number of credentials in reply,
     big-endian
   * LEN is uint32, a credential length in bytes, big-endian
   * CRED  is binary blob representing the credential, same format as in 

     GET_CRED_BY_UUID

MIT Kerberos will use this operation when available and fallback to 
current behavior if not.
* kcm_start_seq_get will call GET_CRED_LIST and store obtained
   credentials. If the operation fails, it will fallback to current
   behavior.
* kcm_next_cred will iterate over cached credentials instead of UUID if
   available.

KCM/KRB5 Changes Notification Mechanism
#######################################

Problem Statement
*****************

Applications exist that need to act upon changes in the Kerberos ccache.
One such application is Gnome Online Accounts (GOA) which connects
multiple accounts and integrates them into various system services (such
as calendar). It supports connection to typical social media accounts
but also integrates Kerberos accounts. For this, GOA needs to be
notified about changes in user ccache in order to stay in sync with
available credentials. It uses inotify for FILE and DIR ccaches, but the
KEYRING and KCM are being polled every 5 seconds. The constant polling
is not desirable for the KCM daemon.

The goal is to provide a notification interface (preferably implemented
through Kerberos API) in order to avoid constant polling the KCM daemon
but keep fast synchronization and good user experience for GOA users.

Note: The main reason is to avoid performance issues and increased CPU
load of SSSD KCM. These, however, will be fixed as part of “Improve KCM
performance when listing credentials”. Nevertheless, KCM is a daemon
that should sleep most of its runtime and it is undesirable for it to be
polled so often.

Solution overview
*****************

Kerberos development library libkrb5 will gain a new call
krb5_request_notifications (or similar) that can be used by applications
to request notifications for changes in user ccache. The function will
be called on a ccache for which we want to receive notifications. The
ccache’s backend will setup a file based notification mechanism if the
caller has access to the ccache and fails otherwise.

This function will return a path to a file that can be watched for
changes by inotify. The notification support will be specific to each
backend and each backend is responsible for its own implementation.
The file will be created by the Kerberos ccache backend (if the backend
supports it) and it will be touched by the backend each time a ccache is
created, destroyed or changed to fire an inotify event.

KCM Implementation details
**************************

KCM will gain new operation REQUEST_NOTIFICATIONS that will be called by
krb5_request_notifications from libkrb5. The input will be UID for the
user that requested the notification. If the provided UID does not match
UID of the caller, it will fail.

* The operation will create XDG_RUNTIME_DIR like structure
   /run/sss/user/$UID for the user, if not already created and creates a
   file /run/sss/user/$UID/kcm-notify.
* The reply will be KCM_HEADER + strlen(path) + path

KCM will touch the file each time a ccache is created, destroyed or changed.

The SSSD runtime directory will be managed by systemd unit file that 
will be responsible for deleting /run/sss/users/$UID when the user’s 
session terminates. Its creation should be left on KCM daemon to avoid 
creating this directory for every user even if there is no interest in 
the notifications.