unexpected failure for GSS Pg server

[Dameon Wagner]

On Tue, Feb 08 2022 at 12:28:21 -0600, Matt Zagrabelny via Kerberos scribbled
 in "Re: unexpected failure for GSS Pg server":
> On Tue, Feb 8, 2022 at 11:54 AM Matt Zagrabelny <mzagrabe at d.umn.edu> wrote:
> 
> > Greetings,
> >
> > I'm experiencing a failure between a GSS enabled Postgresql server and my
> > CLI client.
> >
> > To my knowledge nothing has changed on the system to create this
> > failure.  I did modify some puppet configs, but according to the
> > puppet log output (and stat'ing
> > /etc/postgresql-common/krb5.keytab) file contents did not change.
> >
> > Here are the errors on on the Pg side:
> >
> > 2022-02-08 11:29:59.304 CST [19401] [unknown]@[unknown] FATAL:
> >  unsupported frontend protocol 1234.5680: server supports 2.0 to 3.0
> > 2022-02-08 11:29:59.315 CST [19402] mzagrabe at somedb FATAL:  accepting GSS
> > security context failed
> > 2022-02-08 11:29:59.315 CST [19402] mzagrabe at somedb DETAIL:  Unspecified
> > GSS failure.  Minor code may provide more information: Request ticket
> > server postgres/db.example.com at EXAMPLE.COM kvno 3 not found in keytab;
> > keytab is likely out of date
> >
> > NOTE: I still have some Pg servers where my GSS authentication is
> > not failing. On those systems, Pg still logs the first line above:
> > "unsupported frontend protocol 1234.5680: server supports 2.0 to
> > 3.0". I only included it above for full disclosure.
> >
> > Here is the error on the client side:
> >
> > GSSAPI continuation error: Unspecified GSS failure.  Minor code may
> > provide more information: Key version is not available
> >
> > Here is what klist has to say...
> >
> > Valid starting       Expires              Service principal
> > 02/08/2022 11:10:57  02/08/2022 21:10:57  krbtgt/EXAMPLE.COM at EXAMPLE.COM
> >         renew until 02/09/2022 11:10:46
> > 02/08/2022 11:11:05  02/08/2022 21:10:57  postgres/
> > db.example.com at EXAMPLE.COM
> >         renew until 02/09/2022 11:10:46
> >
> > Looking at the krb5kdc logs, I see no differences between successful GSS
> > Pg auths and the failure mentioned above:
> > Feb  8 11:36:42 auth krb5kdc[434]: TGS_REQ (8 etypes {18 17 20 19 16 23 25
> > 26}) [IPV6 redacted]: ISSUE: authtime 1644340257, etypes {rep=18 tkt=18
> > ses=18}, mzagrabe at EXAMPLE.COM for postgres/db.example.com at EXAMPLE.COM
> > Feb  8 11:36:42 auth krb5kdc[434]: closing down fd 14
> > Feb  8 11:42:45 auth krb5kdc[434]: TGS_REQ (8 etypes {18 17 20 19 16 23 25
> > 26}) [IPV6 redacted]: ISSUE: authtime 1644340257, etypes {rep=18 tkt=18
> > ses=18}, mzagrabe at EXAMPLE.COM for postgres/
> > successful.example.com at EXAMPLE.COM
> > Feb  8 11:42:45 auth krb5kdc[434]: closing down fd 14
> >
> > Does anyone have any ideas of where to look to start to debug this issue?
> >
> 
> After thinking a bit more on the problem. I believe the issue is that I
> have different kvno (key version numbers).

Based on the server-side logs from your initial post I would
completely agree with your recent thoughts.

To confirm the state of things there are two (or three) commands that
you will need to run, one on a client system, and one on the
postgresql server (and possibly, just for good measure, a third you
your master KDC):

*   On a client run `kvno postgres/db.example.com at EXAMPLE.COM`. This
    should return the key version number of the
    "postgres/db.example/com" principal as fetched from the KDC.
    Going by the your quoted logs, I would expect this to be "3", at
    least at the time you captured those logs.
*   On the postgres server run
    `klist -ek /etc/postgresql-common/krb5.keytab`.  This should
    provide output that looks something like the following:

        Keytab name: FILE:/etc/postgresql-common/krb5.keytab
        KVNO Principal
        ---- -------------------------------------------------------------
           2 postgres/db.example.com at EXAMPLE.COM (aes256-cts-hmac-sha1-96) 
           2 postgres/db.example.com at EXAMPLE.COM (aes128-cts-hmac-sha1-96) 
           2 postgres/db.example.com at EXAMPLE.COM (arcfour-hmac)

    which may have a different selection of keytypes for your keytab,
    and possibly a different value in the KVNO column.  I would
    guess that at the time you captured the logs, the KVNO listed
    would have been less than 3 (certainly should not be more, unless
    something rather weird has happened).

*   Optionally, you could also use `kadmin` to connect to your KDC,
    and run the subcommand: `get_principal postgres/db.example.com`.
    This will also give you a list of the current kvno (for each
    keytype), which should match the value from the first command you
    ran on the client.  More importantly or usefully though, it will
    also list when it was "Last modified", and by whom, which may shed
    light on why it has changed.

Armed with that information, the most likely solution would be to
extract a fresh keytab (using either the kadmin "ktadd" subcommand, or
the handy `k5srvutil` command).

One caveat however (and something that has caught many sysadmins out
at one time or another): If you're running a clustered system which is
attempting to use the same principal on more than one server, creating
a fresh keytab will increment the kvno in the KDC again, meaning it
will be out of date on any other system that may be using the
principal.  In this case you'd have to copy the keytab to those
affected systems by hand.

> For now (before anyone spends any time on my initial request for
> help) I would say that I need to do more research and confirm that
> my process is working as expected.
> 
> One request, where is the documentation that describes how/where the kvno
> is used between the kdc and principals?

It may appear a bit old, but the O'Reilly book is still a classic
resource for becoming familiar with Kerberos and how it functions.

Cheers.

Dameon.

-- 
><> ><> ><> ><> ><> ><> ooOoo <>< <>< <>< <>< <>< <><
Dr. Dameon Wagner, Unix Platform Services
IT Services, University of Oxford
><> ><> ><> ><> ><> ><> ooOoo <>< <>< <>< <>< <>< <><