unexpected failure for GSS Pg server

[Matt Zagrabelny]

Greetings,

I'm experiencing a failure between a GSS enabled Postgresql server and my
CLI client.

To my knowledge nothing has changed on the system to create this failure. I
did modify some puppet configs, but according to the puppet log output (and
stat'ing /etc/postgresql-common/krb5.keytab) file contents did not change.

Here are the errors on on the Pg side:

2022-02-08 11:29:59.304 CST [19401] [unknown]@[unknown] FATAL:  unsupported
frontend protocol 1234.5680: server supports 2.0 to 3.0
2022-02-08 11:29:59.315 CST [19402] mzagrabe at somedb FATAL:  accepting GSS
security context failed
2022-02-08 11:29:59.315 CST [19402] mzagrabe at somedb DETAIL:  Unspecified
GSS failure.  Minor code may provide more information: Request ticket
server postgres/db.example.com at EXAMPLE.COM kvno 3 not found in keytab;
keytab is likely out of date

NOTE: I still have some Pg servers where my GSS authentication is not
failing. On those systems, Pg still logs the first line above: "unsupported
frontend protocol 1234.5680: server supports 2.0 to 3.0". I only included
it above for full disclosure.

Here is the error on the client side:

GSSAPI continuation error: Unspecified GSS failure.  Minor code may provide
more information: Key version is not available

Here is what klist has to say...

Valid starting       Expires              Service principal
02/08/2022 11:10:57  02/08/2022 21:10:57  krbtgt/EXAMPLE.COM at EXAMPLE.COM
        renew until 02/09/2022 11:10:46
02/08/2022 11:11:05  02/08/2022 21:10:57  postgres/
db.example.com at EXAMPLE.COM
        renew until 02/09/2022 11:10:46

Looking at the krb5kdc logs, I see no differences between successful GSS Pg
auths and the failure mentioned above:
Feb  8 11:36:42 auth krb5kdc[434]: TGS_REQ (8 etypes {18 17 20 19 16 23 25
26}) [IPV6 redacted]: ISSUE: authtime 1644340257, etypes {rep=18 tkt=18
ses=18}, mzagrabe at EXAMPLE.COM for postgres/db.example.com at EXAMPLE.COM
Feb  8 11:36:42 auth krb5kdc[434]: closing down fd 14
Feb  8 11:42:45 auth krb5kdc[434]: TGS_REQ (8 etypes {18 17 20 19 16 23 25
26}) [IPV6 redacted]: ISSUE: authtime 1644340257, etypes {rep=18 tkt=18
ses=18}, mzagrabe at EXAMPLE.COM for postgres/
successful.example.com at EXAMPLE.COM
Feb  8 11:42:45 auth krb5kdc[434]: closing down fd 14

Does anyone have any ideas of where to look to start to debug this issue?

Thanks for any help!

-m