mount error(126): Required key not available (using sec=krb5)

[aye coder]

My application needs to securely mount an Isilon share using CIFS and
Kerberos. My mount attempt returns: Required key not available:

--- mount -t cifs //fileserver.example.com/client123/files
/mnt/client123/files -o username=acoder,password=XXXXXX,sec=krb5

Response:

--- mount error(126): Required key not available
--- Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Here are corresponding entries from /var/log/messages

--- Sep 16 16:33:49 clientbox kernel: CIFS VFS: Send error in SessSetup = -126
--- Sep 16 16:33:49 clientbox kernel: CIFS VFS: cifs_mount failed
w/return code = -126

I enabled debugging in CIFS, and attempted to mount the share again.
Here's that dmesg output:

--- fs/cifs/cifsfs.c: Devname: //fileserver.example.com/client123/files flags: 0
--- fs/cifs/connect.c: prefix path /files
--- fs/cifs/connect.c: Username: acoder
--- fs/cifs/connect.c: file mode: 0x1ed  dir mode: 0x1ed
--- fs/cifs/connect.c: CIFS VFS: in cifs_mount as Xid: 8 with uid: 0
--- fs/cifs/connect.c: UNC: \\fileserver.example.com/client123/files ip: 1.2.3.4
--- fs/cifs/connect.c: Socket created
--- fs/cifs/connect.c: sndbuf 19800 rcvbuf 87380 rcvtimeo 0x1b58
--- fs/cifs/connect.c: CIFS VFS: in cifs_get_smb_ses as Xid: 9 with uid: 0
--- fs/cifs/connect.c: Demultiplex PID: 22937
--- fs/cifs/connect.c: Existing smb sess not found
--- fs/cifs/cifssmb.c: secFlags 0x9
--- fs/cifs/cifssmb.c: Kerberos only mechanism, enable extended security
--- fs/cifs/transport.c: For smb_command 114
--- fs/cifs/transport.c: Sending smb: smb_len=78
--- fs/cifs/connect.c: RFC1002 header 0xbc
--- fs/cifs/transport.c: cifs_sync_mid_result: cmd=114 mid=1 state=4
--- fs/cifs/cifssmb.c: Dialect: 2
--- fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 0x348 0x1bb92
--- fs/cifs/asn1.c: OID len = 6 oid = 0x1 0x3 0x5 0x1
--- fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 0x348 0xbb92
--- fs/cifs/asn1.c: OID len = 10 oid = 0x1 0x3 0x6 0x1
--- fs/cifs/asn1.c: Need to call asn1_octets_decode() function for
not_defined_in_RFC4178 at please_ignore
--- fs/cifs/cifssmb.c: negprot rc 0
--- fs/cifs/connect.c: Security Mode: 0x3 Capabilities: 0x8000e2fc TimeAdjust: 0
--- fs/cifs/sess.c: sess setup type 4
--- fs/cifs/cifs_spnego.c: key description =
ver=0x2;host=fileserver.example.com;ip4=1.2.3.4;sec=krb5;uid=0x0;creduid=0x0;user=acoder;pid=0xXXXXX
--- fs/cifs/sess.c: ssetup freeing small buf ffff8804359b02701
--- CIFS VFS: Send error in SessSetup = -126
--- fs/cifs/connect.c: CIFS VFS: leaving cifs_get_smb_ses (xid = 9) rc = -126
--- fs/cifs/connect.c: CIFS VFS: leaving cifs_mount (xid = 8) rc = -126
--- CIFS VFS: cifs_mount failed w/return code = -126




*** Background & Config ***

I added a keytab using:

--- /usr/bin/ktutil
--- addent -password -p acoder at EXAMPLE.COM -k 1 -e rc4-hmac
--- addent -password -p acoder at EXAMPLE.COM -k 1 -e aes256-cts
--- wkt /etc/krb5.keytab

Checked with klist -kte:

--- [acoder at clientbox]# klist -kte
--- Keytab name: FILE:/etc/krb5.keytab
--- KVNO Timestamp         Principal
--- ---- -----------------
--------------------------------------------------------
---    1 09/16/15 16:24:32 acoder at EXAMPLE.COM (arcfour-hmac)
---    1 09/16/15 16:25:46 acoder at EXAMPLE.COM (aes256-cts-hmac-sha1-96)

Here's request-key.conf:

--- #OP TYPE    DESCRIPTION CALLOUT INFO    PROGRAM ARG1 ARG2 ARG3 ...
--- #====== ======= =============== ===============
===============================
--- create  user        debug:*     negate      /bin/keyctl negate %k 30 %S
--- create  user        debug:loop:*    *       |/bin/cat
--- create  user        debug:*     *
/usr/share/keyutils/request-key-debug.sh %k %d %c %S
--- negate  *       *       *       /bin/keyctl negate %k 30 %S
--- create  cifs.spnego     *       *       /usr/sbin/cifs.upcall %k
--- create  dns_resolver    *       *       /usr/sbin/cifs.upcall %k

Ticket cache:

--- # klist | grep "Ticket cache:"
--- Ticket cache: FILE:/tmp/krb5cc_0


What could be causing the "Required key not available" error?