GSSAPI security context integrity check
Alexandr Nedvedicky
alexandr.nedvedicky at oracle.com
Wed May 6 13:18:28 EDT 2020
Hello,
not sure if it is the right place to ask questions related to GSSAPI, will be
glad for any useful pointers.
The issue described here is about libgss, which comes with MIT kerberos 1.16,
and SAP NetWeaver Application Server release 749. Customer runs SAP with
privacy and integrity protection. Everything used to work fine with old
kerberos 1.8.4. Customer switched to Solaris 11.4, which comes with kerberos
1.16. The clients are no longer able to connect to SAP server on Solaris
as long as integrity protection is enabled. The log on server shows
error below:
N Mon Mar 16 15:58:10:381 2020
N *** ERROR => SncPFrameIn()==SNCERR_GSSAPI [/bas/749_REL/sr 4454]
N GSS-API(maj): An expected per-message token was not received
N GSS-API(min): Unknown code 0
The error message matches GSS_S_GAP_TOKEN error. Grepping through source code
the error can come from g_seqstate_check() found in
lib/gssapi/generic/util_seqstate.c. This is the only function, which returns
GSS_S_GAP_TOKEN error, I could find. This was promising as kerberos 1.8.4
comes with different algorithm for sequence number checking.
To further analyze the issue I've sent customer a library with patch (see
attachment). The patched binary allowed me to gather detailed tracing
from g_seqstate_check(). The results I got from customer are kind of
surprising, because the results just confirm both algorithms (current and
1.8.4) are equal. However the data collected by customer shows some
odd behavior, which rather points to interoperability issue between
MIT GSSAPI implementation and GSSAPI on clients (?Windows 10?).
The attached patch tracks all sequence number since security context gets
created until it gets released. Once patched kerberos got installed
customer retry the login. There was a failure and the directory
with data captured two files:
lumpy$ ls -l |grep -v '0 Apr'
total 8
-rw-r----- 1 sashan wheel 131 Apr 8 01:33 ctx-10.c523660
-rw-r----- 1 sashan wheel 134 Apr 8 01:33 ctx-11.c523660
two security contexts attempted to use integrity protection. What's
worth to note are the file names. File names are constructed
in simple/naive fashion:
snprintf(path, sizeof (path), "%s/ctx-%u.%p", dbg_dir,
counter++, ctx);
fd = open(path, O_RDWR|O_CREAT|O_APPEND,
S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH);
The counter is static variable, ctx is pointer to g_seqnum_state
instance allocated in g_seqstate_init(). Let's take a look at
the content of files. The first file (ctx-10.c523660) looks OK,
there was a single line:
[ 04-07 16:33:31 ] in: 1509845287 base: 1509845287 \
mask: ffffffffffffffff relative: 0 \
map: 0 offset: 0 next: 1 OK (GSS_S_COMPLETE)
The second file ctx-11.c523660 matches the error we've seen in log from SAP:
[ 04-07 16:33:32 ] in: 1509845288 base: 1509845287 \
mask: ffffffffffffffff relative: 1 \
map: 0 offset: 1 next: 2 FAIL (GSS_S_GAP_TOKEN)
What's worth to note is that the context 11 and 10 do share same base/initial
sequence number. If I understand GSSAPI right the base sequence number
comes from peer, acceptor (kg_accept_krb5()) retrieves that here:
608 /* decode the message */
609
610 if ((code = krb5_auth_con_init(context, &auth_context))) {
611 major_status = GSS_S_FAILURE;
612 save_error_info((OM_uint32)code, context);
613 goto fail;
614 }
615 if (cred->rcache) {
616 cred_rcache = 1;
...
983 {
984 krb5_int32 seq_temp;
985 krb5_auth_con_getremoteseqnumber(context, auth_context, &seq_temp);
986 ctx->seq_recv = seq_temp;
987 }
988
989 if ((code = krb5_timeofday(context, &now))) {
990 major_status = GSS_S_FAILURE;
991 goto fail;
992 }
993
994 code = g_seqstate_init(&ctx->seqstate, ctx->seq_recv,
995 (ctx->gss_flags & GSS_C_REPLAY_FLAG) != 0,
996 (ctx->gss_flags & GSS_C_SEQUENCE_FLAG) != 0,
997 ctx->proto);
All it seems to me there is a some disagreement between acceptor (SAP server)
and client, whether there is one security context (they way window initiator
sees the session) or there is a new second security context, which is being
accepted as seen by acceptor.
I'll be glad for any further debugging tips like how to further trace GSSAPI to
hunt down the bug root cause the problem. I'll be also happy to discuss the
matter off-list and share the results.
thanks for any hints or pointers to get this moving forward.
regards
sasha
-------------- next part --------------
diff --git a/src/lib/gssapi/generic/util_seqdebug.c b/src/lib/gssapi/generic/util_seqdebug.c
new file mode 100644
index 000000000..4bd499033
--- /dev/null
+++ b/src/lib/gssapi/generic/util_seqdebug.c
@@ -0,0 +1,87 @@
+/* -*- mode: c; c-basic-offset: 4; indent-tabs-mode: nil -*- */
+/* lib/gssapi/generic/util_seqstate.c - sequence number checking */
+/*
+ * Copyright (C) 2020 by the Massachusetts Institute of Technology.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+#include <stdlib.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <stdarg.h>
+
+static unsigned int counter = 0;
+
+int
+g_seqstate_opendbg(void *ctx)
+{
+ const char *dbg_dir = getenv("GSSAPI_DBGDIR");
+ char path[1024];
+ int fd;
+
+ if (dbg_dir != NULL) {
+ snprintf(path, sizeof (path), "%s/ctx-%u.%p", dbg_dir,
+ counter++, ctx);
+ fd = open(path, O_RDWR|O_CREAT|O_APPEND,
+ S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH);
+ }
+
+ return (fd);
+}
+
+void
+g_seqstate_closedbg(int fd)
+{
+ if (fd == -1)
+ return;
+
+ close(fd);
+}
+
+void
+g_seqstate_writedbg(int fd, const char *fmt, ...)
+{
+ char buf[1024];
+ int len;
+ va_list args;
+
+ va_start(args, fmt);
+ if (fd == -1) {
+ va_end(args);
+ return;
+ }
+
+ len = vsnprintf(buf, sizeof (buf), fmt, args);
+ va_end(args);
+ if ((len > 0) && (len <= sizeof (buf)))
+ write(fd, buf, (size_t)len);
+}
diff --git a/src/lib/gssapi/generic/util_seqstate.c b/src/lib/gssapi/generic/util_seqstate.c
index a0bc2cc1c..bd7476c86 100644
--- a/src/lib/gssapi/generic/util_seqstate.c
+++ b/src/lib/gssapi/generic/util_seqstate.c
@@ -32,6 +32,8 @@
#include "gssapiP_generic.h"
#include <string.h>
+#include <sys/types.h>
+#include <time.h>
struct g_seqnum_state_st {
/* Flags to indicate whether we are supposed to check for replays or
@@ -59,6 +61,7 @@ struct g_seqnum_state_st {
* we advance next, we shift recvmap to the left.
*/
uint64_t recvmap;
+ int fd;
};
long
@@ -76,49 +79,94 @@ g_seqstate_init(g_seqnum_state *state_out, uint64_t seqnum, int do_replay,
state->seqmask = wide ? UINT64_MAX : UINT32_MAX;
state->base = seqnum;
state->next = state->recvmap = 0;
+ state->fd = g_seqstate_opendbg(state);;
*state_out = state;
return 0;
}
+/*
+ the ctx-11.c523660 file contains what I was looking for:
+
+ [ 04-07 16:33:32 ] in: 1509845288 base: 1509845287 mask: ffffffffffffffff
+ relative: 1 map: 0 offset: 1 next: 2 FAIL (GSS_S_GAP_TOKEN)
+*/
+
OM_uint32
g_seqstate_check(g_seqnum_state state, uint64_t seqnum)
{
uint64_t rel_seqnum, offset, bit;
+ char str[80];
+ struct tm *tm;
+ time_t tval;
if (!state->do_replay && !state->do_sequence)
return GSS_S_COMPLETE;
+ time(&tval);
+ tm = localtime(&tval);
+ strftime(str, sizeof (str), "%m-%d %H:%M:%S", tm);
+ g_seqstate_writedbg(state->fd, "[ %s ] ", str);
+
/* Use the difference from the base seqnum, to simplify wraparound. */
rel_seqnum = (seqnum - state->base) & state->seqmask;
+ g_seqstate_writedbg(state->fd, "in:\t%lu base:\t%lu mask:\t%lx relative:\t%lu map:\t%lx ",
+ seqnum, state->base, state->seqmask, rel_seqnum, state->recvmap);
if (rel_seqnum >= state->next) {
+ uint64_t next;
/* seqnum is the expected sequence number or in the future. Update the
* received bitmap and expected next sequence number. */
offset = rel_seqnum - state->next;
state->recvmap = (state->recvmap << (offset + 1)) | 1;
+ next = state->next;
state->next = (rel_seqnum + 1) & state->seqmask;
-
- return (offset > 0 && state->do_sequence) ? GSS_S_GAP_TOKEN :
- GSS_S_COMPLETE;
+ g_seqstate_writedbg(state->fd, "offset:\t%lu next:\t%lu ",
+ offset, state->next);
+
+ if (offset > 0 && state->do_sequence) {
+ g_seqstate_writedbg(state->fd, "FAIL (GSS_S_GAP_TOKEN)\n");
+ return (GSS_S_GAP_TOKEN);
+ } else {
+ g_seqstate_writedbg(state->fd, "OK (GSS_S_COMPLETE)\n");
+ return (GSS_S_COMPLETE);
+ }
}
/* seqnum is in the past. Check if it's too old for replay detection. */
offset = state->next - rel_seqnum;
- if (offset > 64)
- return state->do_sequence ? GSS_S_UNSEQ_TOKEN : GSS_S_OLD_TOKEN;
+ if (offset > 64) {
+ OM_uint32 rv;
+
+ g_seqstate_writedbg(state->fd, "offset:\t%lu ", offset);
+ rv = state->do_sequence ? GSS_S_UNSEQ_TOKEN : GSS_S_OLD_TOKEN;
+ if (rv == GSS_S_UNSEQ_TOKEN)
+ g_seqstate_writedbg(state->fd, "FAIL (GSS_S_UNSEQ_TOKEN)\n");
+ else
+ g_seqstate_writedbg(state->fd, "FAIL (GSS_S_OLD_TOKEN)\n");
+
+ return (rv);
+ }
/* Check for replay and mark as received. */
bit = (uint64_t)1 << (offset - 1);
- if (state->do_replay && (state->recvmap & bit))
+ g_seqstate_writedbg(state->fd, "bit:\t%lx ");
+ if (state->do_replay && (state->recvmap & bit)) {
+ g_seqstate_writedbg(state->fd, "FAIL (GSS_S_DUPLICATE_TOKEN)\n");
return GSS_S_DUPLICATE_TOKEN;
+ }
state->recvmap |= bit;
+ if (state->do_sequence)
+ g_seqstate_writedbg(state->fd, "OK (GSS_S_COMPLETE)\n");
+ else
+ g_seqstate_writedbg(state->fd, "FAIL (GSS_S_UNSEQ_TOKEN)\n");
return state->do_sequence ? GSS_S_UNSEQ_TOKEN : GSS_S_COMPLETE;
}
void
g_seqstate_free(g_seqnum_state state)
{
+ g_seqstate_closedbg(state->fd);
free(state);
}
More information about the krbdev
mailing list