From jshivers at redhat.com Thu Aug 4 09:54:03 2022 From: jshivers at redhat.com (Jacob Shivers) Date: Thu, 4 Aug 2022 09:54:03 -0400 Subject: cross-realm delegation via attempted RBCD fails with KRB5KRB_AP_ERR_ILL_CR_TKT In-Reply-To: References:

Message-ID: Hello, Reaching out again. Requesting any further input. As I have said, if something is poorly worded or requires further clarification I will be happy to elaborate and reword as necessary. Regards, On Wed, Apr 27, 2022 at 4:19 PM Jacob Shivers wrote: > > Sending this to the dev list to hope to get some traction there. > > Any input on this will be greatly appreciated. > > ---------- Forwarded message --------- > From: Jacob Shivers > Date: Fri, Apr 8, 2022 at 11:49 AM > Subject: Re: cross-realm delegation via attempted RBCD fails with > KRB5KRB_AP_ERR_ILL_CR_TKT > To: > > > Hello, > > Reaching out again. > > If something is poorly worded or requires further > clarification/explanation I am more than willing to try to elaborate. > I am a bit stuck on this issue and would greatly appreciate any > feedback of things to test or to look at further. > > Thank you _very_ much. > > On Mon, Mar 28, 2022 at 11:08 AM Jacob Shivers wrote: > > > > Hello All, > > > > My setup: > > > > * Parent realm (AD.TOB.COM) and child realm (TEST.AD.TOB.COM) with a two-way > > transitive trust in Active Directory. > > * NFS client (f35.ad.tob.com) in AD.TOB.COM > > * NFS server (8x1-nfs.ad.tob.com) in AD.TOB.COM exporting a Kerberized NFS > > share > > * User (data) in AD.TOB.COM > > * User (lore) in TEST.AD.TOB.COM > > > > I am trying to setup cross-realm Kerberos delegation via Resource Based > > Constrained Delegation (RBCD) within Active Directory 2K16. In this test, there > > are two domains that have a parent/child relationship. User in both the parent > > and the child domain are logging into a NFS client within the parent realm that > > has mounted a Kerberized NFS share from a NFS server also within the parent > > realm. No user logging in has a Kerberos ticket and there are no stored keytabs > > for users on the NFS client. > > > > Configuring gssproxy with 'impersonate = yes', users within the parent realm > > are able to access the Kerberized NFS share with no issue. However, users in > > the child realm are unable to access the share and gssproxy logs 'Illegal > > cross-realm ticket' as returned by krb5 libraries. I observe this behavior in > > RHEL 8.5 as well as Fedora 35 with Alexander Bokovoy's upstream copr build for > > krb5-libs that includes RBCD patches not yet in Fedora proper. > > > > I have found some sample packet captures from wireshark.org for RBCD, but even > > after viewing the captures, I still am not sure what the exact behavior should > > be for cross-realm delegation. That being said, the NFS client logs > > KRB5KRB_AP_ERR_ILL_CR_TKT before the point of delegation for the user in the > > child domain to the local NFS server. > > > > > > My limited understanding, and please excuse any misnaming, is that when the > > user in the child domain on the NFS client attempts to access the Kerberized > > NFS share with impersonation active the NFS client should: > > > > * Authenticate and receive a ticket granting service principal for its local > > realm which is the parent realm (krbtgt/AD.TOB.COM at AD.TOB.COM). > > > > * Obtain the remote ticket granting server principal pointing towards the > > child domain (krbtgt/TEST.AD.TOB.COM at AD.TOB.COM). > > > > * Obtain the remote ticket granting server principal pointing back towards the > > parent domain (krbtgt/AD.TOB.COM at TEST.AD.TOB.COM). > > > > * Authenticate on behalf of the user in the child domain to the parent domain > > using the cross realm TGT ticket (krbtgt/AD.TOB.COM at TEST.AD.TOB.COM) for the > > proxy_impersonator (F35$@AD.TOB.COM). > > > > * Use the proxy_impersonator key to obtain the endpoint credentials for the > > NFS server's nfs service (nfs/8x1-nfs.ad.tob.com at AD.TOB.COM) for the user in > > the child domain > > > > The client does _not_ reach the point of the actual RBCD bits of requesting the > > NFS ticket granting service ticket for the user based on comparing this failing > > traffic to that of a user in the same realm. `$ tshark` flags > > kerberos.KDCOptions.constrained.delegation and > > kerberos.PAC.OPTIONS.FLAGS.resource.based.constrained.delegation are set once > > this occurs. > > > > > > The below is present in /etc/krb5.conf by way of > > /var/lib/sss/pubconf/krb5.include.d/domain_realm_ad_tob_com: > > > > [capaths] > > TEST.AD.TOB.COM = { > > AD.TOB.COM = AD.TOB.COM > > } > > AD.TOB.COM = { > > TEST.AD.TOB.COM = AD.TOB.COM > > } > > > > > > I have collected a network trace, a `# strace` of gssproxy, journalctl output, > > as well as a KRB5_TRACE of gssproxy with debug_level set to 3. This lab > > contains no confidential data so I can capture and share any tracing. > > > > I can also perform any additional tests should it be requested. > > > > > > Thank you very much for any guidance that can be offered. > > > > > > > > -- > > > > Jacob Shivers > > > > -- > > Jacob Shivers -- Jacob Shivers From jes59 at cornell.edu Thu Aug 4 13:18:56 2022 From: jes59 at cornell.edu (Jerry Shipman) Date: Thu, 4 Aug 2022 13:18:56 -0400 Subject: can realms get "aliased" when there is a one-way trust? or, what is going on here? Message-ID: Hello, This might just be a microsoft implementation thing -- sorry. But I am scratching my head and wonder if somebody can help me understand what is going on. We have several different realms (both MIT KDCs and AD DCs) run by various departments. There are sometimes cross-realm trusts in one or both directions. There is an MIT realm (let's say MIT.FOO.CORNELL.EDU) and an AD realm (let's say FOO.CORNELL.EDU). MIT.FOO.CORNELL.EDU trusts FOO.CORNELL.EDU, but not vice-versa. The users are mostly in FOO.CORNELL.EDU and the service in question has a principal in MIT.FOO.CORNELL.EDU but not in FOO.CORNELL.EDU. It seems that when a user tries to get a service ticket for the afs/mit.foo.cornell.edu at FOO.CORNELL.EDU (which doesn't exist), he will wind up with two tickets, one for afs/mit.foo.cornell.edu at FOO.CORNELL.EDU and one for afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU. But this is odd, because afs/mit.foo.cornell.edu at FOO.CORNELL.EDU doesn't exist. I would think that the AD KDC there would just tell the client that the principal doesn't exist? (It seems like it is aliasing it somehow maybe? But that seems dangerous because e.g. jes59 at MIT.FOO.CORNELL.EDU and jes59 at FOO.CORNELL.EDU are probably different people.) More succinctly: $ kinit user at FOO.CORNELL.EDU $ kvno afs/mit.foo.cornell.edu at FOO.CORNELL.EDU $ klist [...] Valid starting Expires Service principal 08/03/2022 15:46:18 08/03/2022 22:26:13 krbtgt/FOO.CORNELL.EDU at FOO.CORNELL.EDU 08/03/2022 15:46:28 08/03/2022 22:26:13 krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU 08/03/2022 15:46:28 08/03/2022 22:26:13 afs/mit.foo.cornell.edu at FOO.CORNELL.EDU # <-- this doesn't exist! why is it here? 08/03/2022 15:46:28 08/03/2022 22:26:13 afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU A priori I would expect that my $ kvno afs/mit.foo.cornell.edu at FOO.CORNELL.EDU would just get a "not found in kerberos database" kind of error, since that principal doesn't exist in that realm (only afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU exist). If I get the trace like: $ KRB5_TRACE=/users/jes59/trace.txt kvno afs/mit.foo.cornell.edu at FOO.CORNELL.EDU It says: [13699] 1659556399.495222: Getting credentials user at FOO.CORNELL.EDU -> afs/mit.foo.cornell.edu at FOO.CORNELL.EDU using ccache FILE:/tmp/krb5cc_10811 [13699] 1659556399.495223: Retrieving user at FOO.CORNELL.EDU -> afs/mit.foo.cornell.edu at FOO.CORNELL.EDU from FILE:/tmp/krb5cc_10811 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_10811) [13699] 1659556399.495224: Retrieving user at FOO.CORNELL.EDU -> krbtgt/FOO.CORNELL.EDU at FOO.CORNELL.EDU from FILE:/tmp/krb5cc_10811 with result: 0/Success [13699] 1659556399.495225: Starting with TGT for client realm: user at FOO.CORNELL.EDU -> krbtgt/FOO.CORNELL.EDU at FOO.CORNELL.EDU [13699] 1659556399.495226: Requesting tickets for afs/mit.foo.cornell.edu at FOO.CORNELL.EDU, referrals on [13699] 1659556399.495227: Generated subkey for TGS request: aes256-cts/2FD6 [13699] 1659556399.495230: Encoding request body and padata into FAST request [13699] 1659556399.495231: Sending request (4086 bytes) to FOO.CORNELL.EDU [13699] 1659556399.495232: Resolving hostname [...] [13699] 1659556399.495253: Response was not from master KDC [13699] 1659556399.495254: Decoding FAST response [13699] 1659556399.495255: FAST reply key: aes256-cts/13D4 [13699] 1659556399.495256: Reply server krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU differs from requested afs/mit.foo.cornell.edu at FOO.CORNELL.EDU [13699] 1659556399.495257: TGS reply is for user at FOO.CORNELL.EDU -> krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU with session key aes256-cts/3CE3 [13699] 1659556399.495258: TGS request result: 0/Success [13699] 1659556399.495259: Storing user at FOO.CORNELL.EDU -> krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU in FILE:/tmp/krb5cc_10811 [13699] 1659556399.495257: TGS reply is for user at FOO.CORNELL.EDU -> krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU with session key aes256-cts/3CE3 [13699] 1659556399.495258: TGS request result: 0/Success [13699] 1659556399.495259: Storing user at FOO.CORNELL.EDU -> krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU in FILE:/tmp/krb5cc_10811 [13699] 1659556399.495260: Following referral TGT krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU [13699] 1659556399.495261: Requesting tickets for afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU, referrals on [13699] 1659556399.495265: Encoding request body and padata into FAST request [13699] 1659556399.495266: Sending request (4085 bytes) to SERVICE [13699] 1659556399.495267: Resolving hostname [...] [13699] 1659556399.495270: Sending initial UDP request to dgram [...] [13699] 1659556399.495271: Received answer (879 bytes) from dgram [...] [13699] 1659556399.495272: Response was not from master KDC [13699] 1659556399.495273: Decoding FAST response [13699] 1659556399.495274: FAST reply key: aes256-cts/66E3 [13699] 1659556399.495275: TGS reply is for user at FOO.CORNELL.EDU -> afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU with session key aes256-cts/72F0 [13699] 1659556399.495276: TGS request result: 0/Success [13699] 1659556399.495277: Received creds for desired service afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU [13699] 1659556399.495278: Storing user at FOO.CORNELL.EDU -> afs/mit.foo.cornell.edu at FOO.CORNELL.EDU in FILE:/tmp/krb5cc_10811 [13699] 1659556399.495279: Also storing user at FOOCORNELL.EDU -> afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU based on ticket [13699] 1659556399.495280: Removing user at FOO.CORNELL.EDU -> afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU from FILE:/tmp/krb5cc_10811 [...] I'm not sure how to read it. It got a referral, followed that. Got a krbtgt/ to do the cross-realm trust stuff. That makes sense. These lines seem important: [13699] 1659556399.495256: Reply server krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU differs from requested afs/mit.foo.cornell.edu at FOO.CORNELL.EDU [...] [13699] 1659556399.495277: Received creds for desired service afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU [13699] 1659556399.495278: Storing user at FOO.CORNELL.EDU -> afs/mit.foo.cornell.edu at FOO.CORNELL.EDU in FILE:/tmp/krb5cc_10811 [13699] 1659556399.495279: Also storing user at FOOCORNELL.EDU -> afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU based on ticket But I don't know why it would think it's OK to do that? A priori I would expect that my $ kvno afs/mit.foo.cornell.edu at FOO.CORNELL.EDU would just get a "not found in kerberos database" kind of error, since that principal doesn't exist in that realm (only afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU exist). If I instead do $ kvno afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU then I wind up with just: $ klist Valid starting Expires Service principal 08/03/2022 16:10:38 08/03/2022 22:50:33 krbtgt/FOO.CORNELL.EDU at FOO.CORNELL.EDU 08/03/2022 16:10:45 08/03/2022 22:50:33 krbtgt/MIT.FOO.CORNELL.EDU at FOO.CORNELL.EDU 08/03/2022 16:10:45 08/03/2022 22:50:33 afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU That makes sense to me. It seems like maybe AD is aliasing afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU to afs/mit.foo.cornell.edu at FOO.CORNELL.EDU because I asked for it and it didn't find it locally, and then giving me both tickets? Maybe as some kind of misguided compatibility thing? But isn't that dangerous, because bar at MIT.FOO.CORNELL.EDU and bar at FOO.CORNELL.EDU are totally different entities! Why would it do that? Is there a way to turn that off? Or, more generally... can you help me understand what is going on there? Thank you! Jerry From ghudson at mit.edu Thu Aug 4 15:27:10 2022 From: ghudson at mit.edu (Greg Hudson) Date: Thu, 4 Aug 2022 15:27:10 -0400 Subject: can realms get "aliased" when there is a one-way trust? or, what is going on here? In-Reply-To: References: Message-ID: <3c91f83c-c25a-156e-5c2a-ef6b66f3d9d4@mit.edu> On 8/4/22 13:18, Jerry Shipman wrote: > It seems that when a user tries to get a service ticket for the > afs/mit.foo.cornell.edu at FOO.CORNELL.EDU (which doesn't exist), he will > wind up with two tickets, one for > afs/mit.foo.cornell.edu at FOO.CORNELL.EDU and one for > afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU. But this is odd, because > afs/mit.foo.cornell.edu at FOO.CORNELL.EDU doesn't exist. Kerberos has the concept of "referrals" for TGS requests, which are defined in RFC 6806. If the FOO.CORNELL.EDU KDC finds that afs/mit.foo.cornell.edu doesn't exist in its database, but can determine that it should be in MIT.FOO.CORNELL.EDU, then instead of issuing a service ticket, it may issue a cross-realm TGT for MIT.FOO.CORNELL.EDU. The client will then follow the referral to MIT.FOO.CORNELL.EDU and get a afs/mit.foo.cornell.edu at MIT.FOO.CORNELL.EDU ticket. How the result is cached depends on the client software. For MIT krb5, prior to 1.18 the resulting ticket is cached under both the requested name and the final ticket service name. In 1.18 and later the ticket is only cached under the requested name, but klist shows the ticket service name if it differs, like this: 08/04/22 14:52:52 08/05/22 14:50:28 a/x.d at KRBTEST1.COM Ticket server: a/x.d at REFREALM > But isn't that dangerous, because bar at MIT.FOO.CORNELL.EDU and > bar at FOO.CORNELL.EDU are totally different entities! Why would it do > that? Is there a way to turn that off? The theory behind referrals is that service principal names are generally unique across realms, because they incorporate FQDNs. So a client looking for a ticket for afs/mit.foo.cornell.edu generally doesn't need to know which realm takes responsibility for authenticating that name; its KDC can make that decision. Client names are of course much less likely to be unique across realms; "fred" may refer to a completely different user in different realms even if they have cross-realm relationships.