[krbdev.mit.edu #3060] large default FD_SETSIZE and stack consumption in 1.3.5 krb5 library

[Public Submitter via RT]

Hi -

Under AIX 5.2, the default size of fd_set is 8KB or about 64K
descriptors.  This recently caused a thread stack overflow problem for
us when using the krb5 client library.

For the problematic threads, it turned out that the biggest contributer
to stack use was a couple of  select_state structures pushed on the
stack in the kerberos client library - used for example when contacting
the kdc in the krb5_gss_init_sec_context() call chain.  select_state
looks like this:

struct select_state {
    int max;
    int nfds;
    fd_set rfds;
    fd_set wfds;
    fd_set xfds;
    struct timeval end_time;
};

We were using 64KB thread stacks for credential management.  But the
call path from krb5_gss_init_sec_context() to krb5int_cm_call_select()
consumes about 73KB of stack on AIX 5.2. 

I'm not sure if there are other unix variants with a huge default
FD_SETSIZE.  But, it seems like addressing this by replacing select()
with poll(), putting fd_set structures on the heap instead of the stack
or even defining -DFD_SETSIZE=<something_reasonable> for systems with an
extremely large default FD_SETSIZE (is there anywhere in the kerberos
code that needs 64K open file descriptors?) would be worth doing.

Thanks,
-Damian