kerberized NFS mount fails if NFS server's DNS domain differs from clients' DNS domain

[Sascha Frey]

I need some help with Kerberos and NFS.
I have to extend an existing installation with one KDC, two NFS
servers and a couple of clients.

The kerberos realm is: FIRST-DOMAIN.COM

DNS (forward&reverse) of the first two NFS servers:
nfs-server1.First-Domain.COM
nfs-server2.First-Domain.COM

DNS of some NFS clients:
one.First-Domain.COM
four.First-Domain.COM
pc1.SUB.Other-Domain.NET

These principals do exist:
krb# kadmin.local -q "listprincs"
krbtgt/FIRST-DOMAIN.COM at FIRST-DOMAIN.COM
host/nfs-server1.first-domain.com at FIRST-DOMAIN.COM
nfs/nfs-server1.first-domain.com at FIRST-DOMAIN.COM
host/nfs-server2.first-domain.com at FIRST-DOMAIN.COM
nfs/nfs-server2.first-domain.com at FIRST-DOMAIN.COM
host/one.first-domain.com at FIRST-DOMAIN.COM
nfs/one.first-domain.com at FIRST-DOMAIN.COM
host/four.first-domain.com at FIRST-DOMAIN.COM
nfs/four.first-domain.com at FIRST-DOMAIN.COM
host/pc1.sub.other-domain.net at FIRST-DOMAIN.COM
nfs/pc1.sub.other-domain.net at FIRST-DOMAIN.COM

This setup works well.

Now I had to add a third NFS server. This ones hostname is:
nfsd.SUB.Other-Domain.NET


I created the principals the same way:
krb# kadmin.local -q 'addprinc -randkey
host/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM'
krb# kadmin.local -q 'addprinc -randkey
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM'
krb# kadmin.local -q 'ktadd -k /tmp/krb5.keytab
host/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM'
krb# kadmin.local -q 'ktadd -k /tmp/krb5.keytab
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM'
(/tmp/krb5.keytab was copied to /etc/krb5.keytab on the new server)


Mounting a NFS share from nfs-server1 or nfs-server2 does
work on all clients.

Mounting a NFS share from the new server 'nfsd' does only work
from clients with the same DNS domain. On the clients where the DNS
domain is the same as the realm, I get an error when trying to mount,


Summary:

Server: nfs-server1.First-Domain.COM, Client: one.First-Domain.COM -> OK
Server: nfs-server1.First-Domain.COM, Client: pc1.SUB.Other-Domain.NET -> OK
Server: nfsd.SUB.Other-Domain.NET, Client: pc1.SUB.Other-Domain.NET -> OK
Server: nfsd.SUB.Other-Domain.NET, Client: one.First-Domain.COM -> FAIL!


I started the rpc.gssd on the client (Debian Jessie) with debug
output:

Full hostname for 'nfsd.SUB.Other-Domain.NET' is 'nfsd.sub.other-domain.net'
Full hostname for 'one.First-Domain.COM' is 'one.first-domain.com'
No key table entry found for ONE$@FIRST-DOMAIN.COM while getting
keytab entry for 'ONE$@FIRST-DOMAIN.COM'
No key table entry found for
root/one.first-domain.com at FIRST-DOMAIN.COM while getting keytab entry
for 'root/one.first-domain.com at FIRST-DOMAIN.COM'
Success getting keytab entry for 'nfs/one.first-domain.com at FIRST-DOMAIN.COM'
INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM' are
good until 1423999935
INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM' are
good until 1423999935
using FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM as credentials cache
for machine creds
using environment variable to select krb5 ccache
FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM
creating context using fsuid 0 (save_uid 0)
creating tcp client for server nfsd.SUB.Other-Domain.NET
DEBUG: port already set to 2049
creating context with server nfs at nfsd.SUB.Other-Domain.NET
WARNING: Failed to create krb5 context for user with uid 0 for server
nfsd.SUB.Other-Domain.NET
WARNING: Failed to create machine krb5 context with credentials cache
FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM for server
nfsd.SUB.Other-Domain.NET
WARNING: Failed to create machine krb5 context with any credentials
cache for server nfsd.SUB.Other-Domain.NET
doing error downcall

one# klist -c /tmp/krb5ccmachine_FIRST-DOMAIN.COM
Ticket cache: FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM
Default principal: nfs/one.first-domain.com at FIRST-DOMAIN.COM

Valid starting       Expires              Service principal
02/14/2015 12:32:15  02/15/2015 12:32:15
krbtgt/FIRST-DOMAIN.COM at FIRST-DOMAIN.COM
    renew until 02/21/2015 12:32:15


I also see errors from the KDC:
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (1 etypes {1})
129.70.137.45: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (7 etypes {18 17 16 23 3 1
2}) 192.168.112.22: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (1 etypes {1})
129.70.137.45: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (7 etypes {18 17 16 23 3 1
2}) 192.168.112.22: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (1 etypes {1})
129.70.137.45: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (7 etypes {18 17 16 23 3 1
2}) 192.168.112.22: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (1 etypes {1})
129.70.137.45: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (7 etypes {18 17 16 23 3 1
2}) 192.168.112.22: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (1 etypes {1})
129.70.137.45: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (7 etypes {18 17 16 23 3 1
2}) 192.168.112.22: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (1 etypes {1})
129.70.137.45: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)
Feb 14 12:41:12 krb krb5kdc[31565]: TGS_REQ (7 etypes {18 17 16 23 3 1
2}) 192.168.112.22: NO PREAUTH: authtime 0,
nfs/one.first-domain.com at FIRST-DOMAIN.COM for
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM, Generic error (see
e-text)


Mouting on the clients with the same DNS domain does work:
creating tcp client for server nfsd.SUB.Other-Domain.NET
DEBUG: port already set to 2049
creating context with server nfs at nfsd.SUB.Other-Domain.NET
DEBUG: serialize_krb5_ctx: lucid version!
prepare_krb5_rfc4121_buffer: protocol 1
prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
doing downcall lifetime_rec 86400
destroying client /run/rpc_pipefs/nfs/clnt139
destroying client /run/rpc_pipefs/nfs/clnt138

pc1# klist -c /tmp/krb5ccmachine_FIRST-DOMAIN.COM
Ticket cache: FILE:/tmp/krb5ccmachine_FIRST-DOMAIN.COM
Default principal: nfs/pc1.sub.other-domain.net at FIRST-DOMAIN.COM
Valid starting       Expires              Service principal
02/14/2015 12:47:19  02/15/2015 12:47:19
krbtgt/FIRST-DOMAIN.COM at FIRST-DOMAIN.COM
02/14/2015 12:47:19  02/15/2015 12:47:19  nfs/nfsd.sub.other-domain.net@
02/14/2015 12:47:19  02/15/2015 12:47:19
nfs/nfsd.sub.other-domain.net at FIRST-DOMAIN.COM


The /etc/krb5.conf is the same on all clients and servers:
[libdefaults]
 default_realm = FIRST-DOMAIN.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 allow_weak_crypto = true
 default_tgs_enctypes = des-cbc-crc
 default_tkt_enctypes = des-cbc-crc
 permitted_enctypes = des-cbc-crc


[realms]
 FIRST-DOMAIN.COM = {
  kdc = krb.first-domain.com
  admin_server = krb.first-domain.com
 }

[domain_realm]
 first-domain.com = FIRST-DOMAIN.COM
 .first-domain.com = FIRST-DOMAIN.COM

-----------

I tried to add
sub.other-domain.net = FIRST-DOMAIN.COM
.sub.other-domain.net = FIRST-DOMAIN.COM
to [domain_realm] of all krb5.conf files, but that didn't help.


Where's my fault?


Thanks for your help!