MIT Kerberos Master principal deletion

[D'Angelo, Jeff C]

Sorry, I must have misunderstood you.  When you said:
While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M at REALM principal
details in this file.
I thought you said you found another old copy of the principal database file in /var/log/kerberos/krb5kdc (or was that a typo for where the prod db lives in /var/lib/kerberos/krb5kdc?) from before the deletion of K/M occurred.  If you did, that would make this easier as it seems existing tools could remedy the problem (dump the K/M entry from that other DB and then load -update into your prod one).

The trick is to make sure it's the correct K/M with the original key you had, but I have some suggestions on how to test that before you affect prod.  If you happened to find the original password you used to create the database, that would also work.  Heck, even if you only have some guesses as to what the original password was, that could also help.

But still, finding the stash file (which is presumably the correct one) is a big help and avoids needing to hunt it down out of a core file.


--

Jeff

________________________________
From: Harshawardhan Kulkarni <harshawardhan.rk at gmail.com>
Sent: Friday, June 19, 2020 1:16 PM
To: D'Angelo, Jeff C <jcd at psu.edu>
Cc: kerberos at mit.edu <kerberos at mit.edu>
Subject: Re: MIT Kerberos Master principal deletion

Hi Jeff,

Many thanks for giving this a try. Really appreciated :)

Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.

My Kerberos version is (Kerberos 5 version 1.12.5),
Below is the error I get,

kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump
kdb5_util: No such entry in the database while retrieving master entry

Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file?
You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file.

Thanks again,
Harsh

On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C <jcd at psu.edu<mailto:jcd at psu.edu>> wrote:
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this.  I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try.

I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state.  This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump".  For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump).

Then I created a shorter dump file of just the header and K/M entry using grep [1]:
sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'

  [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files.

Make sure it's just those two lines:
sudo cat kdb5.dump.km_restore


Then do a kdb5_util incremental (-update) load with that file:

sudo kdb5_util load -update kdb5.dump.km_restore


Surprisingly, it worked.  I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one.


Harsh, what version kerb are you running?


Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db.  I'm not sure yet what loading a different K/M would do.


--

Jeff

________________________________
From: kerberos-bounces at mit.edu<mailto:kerberos-bounces at mit.edu> <kerberos-bounces at mit.edu<mailto:kerberos-bounces at mit.edu>> on behalf of Harshawardhan Kulkarni <harshawardhan.rk at gmail.com<mailto:harshawardhan.rk at gmail.com>>
Sent: Thursday, June 18, 2020 6:27 PM
To: kerberos at mit.edu<mailto:kerberos at mit.edu> <kerberos at mit.edu<mailto:kerberos at mit.edu>>
Subject: Re: MIT Kerberos Master principal deletion

Hi Team,

I am reaching out back again with my existing issue regarding master key
deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of
people are using it.

While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M at REALM principal
details in this file.

So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?

Thanks,
Harsh


On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
harshawardhan.rk at gmail.com<mailto:harshawardhan.rk at gmail.com>> wrote:

> Hi Team,
>
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/M at REALM.COM<mailto:M at REALM.COM>) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?
>
> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.
>
> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.
>
> Can anyone help me here? Looking forward for your reply.
>
> Thanks,
> Harsh Kulkarni
>
________________________________________________
Kerberos mailing list           Kerberos at mit.edu<mailto:Kerberos at mit.edu>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&amp;sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C24638044c9484278c86008d81474972a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281838273451942&sdata=nq3dpBcY%2BsC2gMW9bWw%2FnPMQiVkmz7RlYQfzQz48N3s%3D&reserved=0>