[krbdev.mit.edu #2760] [Roland C. Dowdeswell] fopen(3) returning NULL and errno==0

[Tom Yu via RT]

-------------------- Start of forwarded message --------------------
To: krbdev at mit.edu
Date: Tue, 26 Oct 2004 18:54:07 -0400
From: "Roland C. Dowdeswell" <elric at imrryr.org>
Message-Id: <20041026225407.6098E174F8 at arioch.imrryr.org>
Subject: fopen(3) returning NULL (with workaround.)

So, I've experienced a few core dumps with the kerberos libraries
on Solaris 7.  After a bit of examination, I determined that on
Solaris 7 [and I presume earlier versions as well], it is possible
for fopen(3) to return NULL yet set errno to zero[1].  In
krb5_ktfileint_open(), if fopen(3) returns NULL, the function will
return errno.  This has the unfortunate effect of seeming like
success to callers who later try to use the NULL FILE *.

Anyway, here's a small patch which just returns ENFILE if errno == 0.

[1]	Solaris 7's fopen(3) returns with errno == 0, if it can't
	get a fd < 256.  In the same situation Solaris 8 sets errno
	to ENFILE.

--
    Roland Dowdeswell                      http://www.Imrryr.ORG/~elric/

===================================================================
RCS file: /ms/dev/sec/mitkrb5/cvs-dirs/mitkrb5-1.3.1/mitkrb5/src/lib/krb5/keytab/kt_file.c,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -r1.1 -r1.2
--- src/lib/krb5/keytab/kt_file.c	29 Mar 2004 19:57:38 -0000	1.1
+++ src/lib/krb5/keytab/kt_file.c	23 Sep 2004 04:24:22 -0000	1.2
@@ -1008,10 +1008,10 @@
             krb5_create_secure_file(context, KTFILENAME(id));
 	    KTFILEP(id) = fopen(KTFILENAME(id), fopen_mode_rbplus);
 	    if (!KTFILEP(id))
-		return errno;
+		return errno?errno:ENFILE;
 	    writevno = 1;
 	} else				/* some other error */
-	    return errno;
+	    return errno?errno:ENFILE;
     }
     if ((kerror = krb5_lock_file(context, fileno(KTFILEP(id)), mode))) {
 	(void) fclose(KTFILEP(id));
_______________________________________________
krbdev mailing list             krbdev at mit.edu
https://mailman.mit.edu/mailman/listinfo/krbdev

-------------------- End of forwarded message --------------------