[krbdev.mit.edu #2873] test suite delays too much

[Ken Raeburn via RT]

Some thoughts on the test suite's performance:

We spend a bunch of time -- typically 1-2s -- in most .exp files
delaying to give whatever daemon time to start up.  We should consider
tweaking the daemon code to print out something once it starts
listening for connections, so we can start the client code running
immediately.  Starting the application daemons earlier in the process
(say, before getting user credentials, and maybe even before setting
up the Kerberos database, since the daemons won't need to look at the
keytab until a connection comes in) would also give more time for the
daemon to start up without unduly delaying the execution of the tests;
of course, this complicates the cleanup a bit if the database can't
then be set up properly.  See if we can make do without any "sleep"
calls, just timeouts on expect{} calls.

We're also delaying to let the "tail" on the log files start up.  I
vaguely recall it's because of a race condition in GNU tail where some
contents would be printed, then we'd dump the marker line into the
file, then tail would record the size/timestamp/whatever, and not
notice we added more stuff that it hadn't printed.  It might work to:
 - write a (blank?) line to the file to guarantee it's not empty
 - start tail
 - wait for output from tail to start
 - write the marker line
 - wait for the marker to be output by tail
 - if it's not after 1 second, prod with another blank line
 - repeat that last step a few times
Or something like that.  Experiment and see.  (Remember to flush
output.)

I'd guess that's 4-5s in each of 9 of the .exp files, in each of 13
passes, spent sleeping; totalling probably over 8 minutes if all
passes are run, with most of it probably unnecessary.

(Actually, I was assuming "sleep 2" pauses for 1.5s on average, which
I think is probably okay for the UNIX sleep command, but I think the
expect "sleep" built-in command may actually be more precise, so my
checkin earlier may have raised the sleep time by 1/3, bringing the
total up over 11 minutes.  Oops.)

We might also try harder to use one invocation of the KDC and kadmind
in each pass, at least for the bits where we're testing application
code and not aiming specifically to exercise the KDC startup code.
Just reset some passwords and re-extract some keys, if necessary.
Similarly for running kinit to get initial credentials; just assume
that one pass will run in less than N hours, or check the clock if we
want to be paranoid.  These should all be restarted on each pass
because of the changes in enctypes.

If the test suite fails to be able to get a root shell once, it
probably shouldn't bother trying again in the same run.  Also, if it's
not too painful or fragile a change, we should just use the one root
shell session for the whole run once we acquire it.  (The whole run,
not just one pass; that's 2 * 13 = 26 root login sessions or
attempts.)

Ken