fopen(3) returning NULL (with workaround.)

[Douglas E. Engert]

I have seen a similiar problem with open failing in the profile
code but te profile code tried to use the handle. It has to do
with shared libs and the definition of errno as a function or
an extern.

With Solaris I build Kerberos with  -D_TS_ERRNO

See the /usr/include/errno.h


Roland C. Dowdeswell wrote:
> So, I've experienced a few core dumps with the kerberos libraries
> on Solaris 7.  After a bit of examination, I determined that on
> Solaris 7 [and I presume earlier versions as well], it is possible
> for fopen(3) to return NULL yet set errno to zero[1].  In
> krb5_ktfileint_open(), if fopen(3) returns NULL, the function will
> return errno.  This has the unfortunate effect of seeming like
> success to callers who later try to use the NULL FILE *.
> 
> Anyway, here's a small patch which just returns ENFILE if errno == 0.
> 
> [1]	Solaris 7's fopen(3) returns with errno == 0, if it can't
> 	get a fd < 256.  In the same situation Solaris 8 sets errno
> 	to ENFILE.
> 
> --
>     Roland Dowdeswell                      http://www.Imrryr.ORG/~elric/
> 
> ===================================================================
> RCS file: /ms/dev/sec/mitkrb5/cvs-dirs/mitkrb5-1.3.1/mitkrb5/src/lib/krb5/keytab/kt_file.c,v
> retrieving revision 1.1
> retrieving revision 1.2
> diff -u -r1.1 -r1.2
> --- src/lib/krb5/keytab/kt_file.c	29 Mar 2004 19:57:38 -0000	1.1
> +++ src/lib/krb5/keytab/kt_file.c	23 Sep 2004 04:24:22 -0000	1.2
> @@ -1008,10 +1008,10 @@
>              krb5_create_secure_file(context, KTFILENAME(id));
>  	    KTFILEP(id) = fopen(KTFILENAME(id), fopen_mode_rbplus);
>  	    if (!KTFILEP(id))
> -		return errno;
> +		return errno?errno:ENFILE;
>  	    writevno = 1;
>  	} else				/* some other error */
> -	    return errno;
> +	    return errno?errno:ENFILE;
>      }
>      if ((kerror = krb5_lock_file(context, fileno(KTFILEP(id)), mode))) {
>  	(void) fclose(KTFILEP(id));
> _______________________________________________
> krbdev mailing list             krbdev at mit.edu
> https://mailman.mit.edu/mailman/listinfo/krbdev
> 
> 
> 

-- 

  Douglas E. Engert  <DEEngert at anl.gov>
  Argonne National Laboratory
  9700 South Cass Avenue
  Argonne, Illinois  60439
  (630) 252-5444