MIT Kerberos Master principal deletion

[D'Angelo, Jeff C]

A couple points:
Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.
Same for me in 1.17.  It seems that the following actions fail when K/M is missing from the db:
(1) A running kadmind cannot ank (create prinicipals) nor cpw (change passwords [1]), but at least other read and metadata modification functions still work (the old mkey is still in memory)
(2) A running krb5kdc will continue to issue tickets (the old mkey is still in memory)
(3) kadmind cannot be [re]started
(4) krb5kdc cannot be [re]started
(5) kadmin.local cannot open the database
(6) kdb5_util dump will fail the same way you saw

  [1] I got a misleading error (yes, I had permission in kadm5.acl): change_password: Operation requires ``change-password'' privilege while changing password for "princ at realm.name".

As I mentioned earlier, I was able to re-add K/M from a previous dump with [2]:
kdb5_util load -update file_with_just_km.dump

 [2] And this step doesn't seem to require you to have the correct stash file in place yet, but good to correct as an immediately next or previous step.

If it was the correct K/M (and I presume you still have the correct stash file), then life was seemingly back to normal.

I also found that when you use an "imposter" K/M (created from a different password/key) but with matching principal name, realm, enctypes and kvno (not sure those all need to match, necessarily, to achieve these results), and replace the stash file to match, I was able to do the following:
(1) kdb5_util dump - notably, only the K/M entry was different.  The other entries matched what I had before in a previous dump, so presumably still use the same encryption they had before with the old key; in other words, the current K/M does not appear to be used to decrypt or encrypt passwords as this point [3], but the content of the db file are simply dumped in a mostly readable text format. [4]
(2) No effect I saw on running krb5kdc [5]
(3) kadmind would again permit kadmin clients to ank and cpw again, however, they use the new master key to encrypt not the old, so restoring a database with mixed master keys at the same kvno is likely a recipe for further pain; I might next test using a different kvno (simulating a normal master key rotation) to see how that fares
(4) krb5kdc will [re]start again, but fail to issue tickets; kinit attempts are met with an immediate and slightly misleading but arguably technically correct error message "kinit: Password incorrect while getting initial credentials" (before I even get a chance to enter the password)
(5) kadmind will start again; I presume some functions may fail at this point such as `kadmin: ktadd -norandkey` due to the master key mismatch

  [3] Unless you used an option like -mkey_convert

  [4] If I left the stash file mismatched, it would still dump, but give the warning:
kdb5_util: Unable to decrypt latest master key with the provided master key
 while getting master key list
kdb5_util: Warning: proceeding without master key list

  [5] Not that I tested more than kinits.  I'm sure it might be worth testing the effect on updates to "Last successful authentication" and "Last failed authentication", but I presume that will succeed like a `kadmin: modprinc -expire now` did for me.

So it would seem a recovery procedure for you would best:
(1) first reclaim the correct K/M entry (or your guess at one) into a dump file
(2) verify it is the correct entry with a test database, probably best on another system so you don't accidentally break your operational server
(3) perhaps [temporarily?] load the correct (or an imposter) K/M onto your operation DB so you can make a dump for emergency backup purpose as well as to use to test on your test db; maybe delete it again so to avoid ank or cpw calls if you can't stop them otherwise
(4) See if you can fire up krb5kdc on your test system with a database loaded from prod and kinit against it with a known principal

Or something like it.  Again, it may be best to test the full scenario on your version of software before assuming it will work for you.

As for (1) reclaim the correct K/M entry, I wonder how hard it would be to extend `krb5_util create` to permit using an existing key in a keytab (or stash) file as alternative to the usual password prompt.  Otherwise, we may be getting into other esoterica like how to convert a keytab (stash) file to a kdb5_util load_dump format.

But if you can find an old copy of the database file, or guesses at the original password, that would also help.


--

Jeff

________________________________
From: Harshawardhan Kulkarni <harshawardhan.rk at gmail.com>
Sent: Friday, June 19, 2020 1:16 PM
To: D'Angelo, Jeff C <jcd at psu.edu>
Cc: kerberos at mit.edu <kerberos at mit.edu>
Subject: Re: MIT Kerberos Master principal deletion

Hi Jeff,

Many thanks for giving this a try. Really appreciated :)

Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.

My Kerberos version is (Kerberos 5 version 1.12.5),
Below is the error I get,

kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump
kdb5_util: No such entry in the database while retrieving master entry

Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file?
You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file.

Thanks again,
Harsh

On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C <jcd at psu.edu<mailto:jcd at psu.edu>> wrote:
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this.  I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try.

I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state.  This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump".  For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump).

Then I created a shorter dump file of just the header and K/M entry using grep [1]:
sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'

  [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files.

Make sure it's just those two lines:
sudo cat kdb5.dump.km_restore


Then do a kdb5_util incremental (-update) load with that file:

sudo kdb5_util load -update kdb5.dump.km_restore


Surprisingly, it worked.  I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one.


Harsh, what version kerb are you running?


Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db.  I'm not sure yet what loading a different K/M would do.


--

Jeff

________________________________
From: kerberos-bounces at mit.edu<mailto:kerberos-bounces at mit.edu> <kerberos-bounces at mit.edu<mailto:kerberos-bounces at mit.edu>> on behalf of Harshawardhan Kulkarni <harshawardhan.rk at gmail.com<mailto:harshawardhan.rk at gmail.com>>
Sent: Thursday, June 18, 2020 6:27 PM
To: kerberos at mit.edu<mailto:kerberos at mit.edu> <kerberos at mit.edu<mailto:kerberos at mit.edu>>
Subject: Re: MIT Kerberos Master principal deletion

Hi Team,

I am reaching out back again with my existing issue regarding master key
deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of
people are using it.

While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M at REALM principal
details in this file.

So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?

Thanks,
Harsh


On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
harshawardhan.rk at gmail.com<mailto:harshawardhan.rk at gmail.com>> wrote:

> Hi Team,
>
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/M at REALM.COM<mailto:M at REALM.COM>) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?
>
> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.
>
> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.
>
> Can anyone help me here? Looking forward for your reply.
>
> Thanks,
> Harsh Kulkarni
>
________________________________________________
Kerberos mailing list           Kerberos at mit.edu<mailto:Kerberos at mit.edu>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&amp;sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C24638044c9484278c86008d81474972a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281838273451942&sdata=nq3dpBcY%2BsC2gMW9bWw%2FnPMQiVkmz7RlYQfzQz48N3s%3D&reserved=0>