RPCSEC_GSS and namespaces

[Tom Yu]

This is another note concerning the RPCSEC_GSS merge, specifically
regarding namespace conflicts and OS integration issues.  Some issues
include conflicts with OS-provided RPC implementations and conflicts
with reserved namespace.  Additional issues include details of how OS
vendors should integrate RPCSEC_GSS functionality into their systems.

In the past, we've had trouble with namespace conflict between the OS
vendor's RPC library and the AUTH_GSSAPI-enabled RPC library in the
krb5 distribution.  These troubles included program crashes when
NIS-based lookups are enabled on a system attempting to run kadmin
clients, for example.  The current sources implement a partial
solution by renaming a number of external names (global object file
symbols) to have the "gssrpc_" prefix.  The header files also are
installed in $includedir/gssrpc rather than in $includedir/rpc.
Unfortunately, the renaming of external names is scattered somewhat,
and is difficult for OS integrators to undo if necessary.

The C standard effectively reserves a leading underscore for all names
(the rules are more complex than that, but that's the net effect).
RPSRC-4.0 uses a number of reserved names, and since it's meant to go
into an OS vendor's C library, it can do that.  As a user-level
library, the MIT krb5 GSSRPC library should not use the reserved
namespace, though OS vendors integrating GSSRPC into their OS should
probably perform some renaming of symbols into the reserved namespace
as appropriate.

My current proposal is to rename all external names using a "rename.h"
header, which defines macros to rename all external names to begin
with a prefix of "gssrpc_"; semi-private external names of the form
"_foo" get explicitly rewritten as "gssrpc__foo", to avoid conflicting
with reserved namespace.  (Declaring macro names in the reserved
namespace invokes undefined behavior, according to the C standard.)
Macros under "#if 0" map "gssrpc__foo" to "_foo", as a reminder to OS
integrators which names need to be explicitly rewritten in the .c
files.

The eventual goal is to have a GSSRPC library whose external names do
not conflict with OS-provided RPC facilities and do not conflict with
reserved names.  The external names should also not conflict with user
namespace except by using a prefix.  It is somewhat more difficult to
avoid conflict in the other namespaces; including a header associated
with GSSRPC may conflict with the OS-provided RPC.

One example of header conflict is the <netdb.h> problem.  Some systems
have a <netdb.h> which includes <rpc/netdb.h>; this caused problems
when MIT krb5 provided a <rpc/netdb.h>.  The renaming of the header
files to gssrpc/*.h helped this somewhat, but the potential for such
conflict remains, especially when both a system header and a gssrpc
header define identically-named typedef names.

OS vendors who integrate GSSRPC into their OS libraries should undo
the renaming, in order to avoid conflict with future changes to the
GSSRPC library.  Undoing the renaming mostly involves ceasing
inclusion of "rename.h" and explicitly rewriting the names of the
semi-private symbols which were rewritten by us to have the prefix
"gssrpc__".  Also, OS integrators should change the header
installation location to <rpc/*.h> rather than <gssrpc/*.h>.

---Tom