From jake at poptart.org  Wed Jan 29 12:00:29 2025
From: jake at poptart.org (Jake Scott)
Date: Wed, 29 Jan 2025 12:00:29 -0500
Subject: Restoring DB to alternate LDAP suffix
Message-ID: <CAExmWcgam_XtE+32_weR_hrO1ZDb-irNpWCaP0QxA3_c9zfcyw@mail.gmail.com>

Hi there..

We are currently migrating data from an LDAP backend (MIT v1.18) to a new
suffix. We've dumped the data using kdb5_util and are attempting to restore
it using a new configuration with the updated suffix.

During the restore process, it appears that the principals are being added
back using their original DNs instead of under the new suffix. Is this
expected behavior? We were surprised to find the principal DNs included in
the dump file.

Any insight or advice would be much appreciated!


Thanks..

Jake

From ghudson at mit.edu  Wed Jan 29 15:25:04 2025
From: ghudson at mit.edu (Greg Hudson)
Date: Wed, 29 Jan 2025 15:25:04 -0500
Subject: Restoring DB to alternate LDAP suffix
In-Reply-To: <CAExmWcgam_XtE+32_weR_hrO1ZDb-irNpWCaP0QxA3_c9zfcyw@mail.gmail.com>
References: <CAExmWcgam_XtE+32_weR_hrO1ZDb-irNpWCaP0QxA3_c9zfcyw@mail.gmail.com>
Message-ID: <1b7e7376-528a-4ba7-a942-e43016ada39d@mit.edu>

On 1/29/25 12:00, Jake Scott wrote:
> We are currently migrating data from an LDAP backend (MIT v1.18) to a new
> suffix. We've dumped the data using kdb5_util and are attempting to restore
> it using a new configuration with the updated suffix.
> 
> During the restore process, it appears that the principals are being added
> back using their original DNs instead of under the new suffix. Is this
> expected behavior? We were surprised to find the principal DNs included in
> the dump file.

That is expected behavior.  I would speculate that the design intent was 
that administrators can set explicit DNs when creating principals, and 
that dumping and loading should preserve that DN structure rather than 
creating a new one based on the container DN and principal names.

I don't see an easier workaround for your use case than modifying the 
dump file.  Unfortunately, the principal DNs are hidden two layers deep:

* Within each principal line are fields representing zero or more 
type-length-data records.  The placement and structure of these records 
is documented in https://github.com/krb5/krb5/pull/1408 .

* Within the tl-data record of type 255 is an encoded series of LDAP 
type-length-data subrecords.  This encoding isn't documented anywhere as 
far as I know (I will hopefully add it to the PR), but it's one or more 
repetitions of a one-byte tag, a two-byte big-endian length, and then 
<length> bytes of data.  Tag 3 indicates a user DN.  You could change 
the tag of the subrecord to some nonsense value (like 255) to cause it 
to be ignored, or remove it and fix up the length of the containing field.

From ghudson at mit.edu  Thu Jan 30 00:33:29 2025
From: ghudson at mit.edu (Greg Hudson)
Date: Thu, 30 Jan 2025 00:33:29 -0500
Subject: Restoring DB to alternate LDAP suffix
In-Reply-To: <CAExmWciERhM+GREyppiJyvCo6_9Lf7hTvKKxR28Z2z6pYEmrVw@mail.gmail.com>
References: <CAExmWcgam_XtE+32_weR_hrO1ZDb-irNpWCaP0QxA3_c9zfcyw@mail.gmail.com>
 <1b7e7376-528a-4ba7-a942-e43016ada39d@mit.edu>
 <CAExmWciERhM+GREyppiJyvCo6_9Lf7hTvKKxR28Z2z6pYEmrVw@mail.gmail.com>
Message-ID: <33440bd5-c7d5-4868-a616-297ec20c116e@mit.edu>

On 1/29/25 21:57, Jake Scott wrote:
> One thing ? I did try restoring the dump to a file based database and 
> then dump/restoring again to LDAP and the same issue happened so I 
> assume that the LDAP data ends up in the file DB as well - is that also 
> what you expect?

That's also expected.  The LDAP KDB module synthesizes and interprets 
the type-255 tl-data; other modules don't know anything about it, so 
they just store it and replay it.

From jake at poptart.org  Wed Jan 29 21:57:40 2025
From: jake at poptart.org (Jake Scott)
Date: Wed, 29 Jan 2025 21:57:40 -0500
Subject: Restoring DB to alternate LDAP suffix
In-Reply-To: <1b7e7376-528a-4ba7-a942-e43016ada39d@mit.edu>
References: <CAExmWcgam_XtE+32_weR_hrO1ZDb-irNpWCaP0QxA3_c9zfcyw@mail.gmail.com>
 <1b7e7376-528a-4ba7-a942-e43016ada39d@mit.edu>
Message-ID: <CAExmWciERhM+GREyppiJyvCo6_9Lf7hTvKKxR28Z2z6pYEmrVw@mail.gmail.com>

Thank you for the detailed response, it is much appreciated.

We worked around the issue by using an LDAP level dump/restore with some
search/replace in between and that seemed to have done the job.  But it
would be nicer to use the kdb5_util interface for sure.  Now that I?m aware
that this data is in the dump and how it is encoded, thanks to you - I will
try to find time to create some dump manipulation tooling.

One thing ? I did try restoring the dump to a file based database and then
dump/restoring again to LDAP and the same issue happened so I assume that
the LDAP data ends up in the file DB as well - is that also what you expect?


Many thanks ? Jake



On Wed, Jan 29, 2025 at 15:25 Greg Hudson <ghudson at mit.edu> wrote:

> On 1/29/25 12:00, Jake Scott wrote:
> > We are currently migrating data from an LDAP backend (MIT v1.18) to a new
> > suffix. We've dumped the data using kdb5_util and are attempting to
> restore
> > it using a new configuration with the updated suffix.
> >
> > During the restore process, it appears that the principals are being
> added
> > back using their original DNs instead of under the new suffix. Is this
> > expected behavior? We were surprised to find the principal DNs included
> in
> > the dump file.
>
> That is expected behavior.  I would speculate that the design intent was
> that administrators can set explicit DNs when creating principals, and
> that dumping and loading should preserve that DN structure rather than
> creating a new one based on the container DN and principal names.
>
> I don't see an easier workaround for your use case than modifying the
> dump file.  Unfortunately, the principal DNs are hidden two layers deep:
>
> * Within each principal line are fields representing zero or more
> type-length-data records.  The placement and structure of these records
> is documented in https://github.com/krb5/krb5/pull/1408 .
>
> * Within the tl-data record of type 255 is an encoded series of LDAP
> type-length-data subrecords.  This encoding isn't documented anywhere as
> far as I know (I will hopefully add it to the PR), but it's one or more
> repetitions of a one-byte tag, a two-byte big-endian length, and then
> <length> bytes of data.  Tag 3 indicates a user DN.  You could change
> the tag of the subrecord to some nonsense value (like 255) to cause it
> to be ignored, or remove it and fix up the length of the containing field.
>