How to prevent very very large ccaches?

[Ken Raeburn]

"Nicolas Williams" <Nicolas.Williams at ubsw.com> writes:

>> If the ccache is passed in, I think the flag state should be restored
>> at the end to the way the caller might have set it.  But I think
>> cc_retr.c is the right place for temporarily overriding the one flag.
>
> Note that there is no get_flags() function/macro for ccaches, so how can
> restore the flag to the previous setting if I cannot find out what that
> is? The comment in the patch states this too. The assumption seems to be
> made everywhere that OPENCLOSE is normally set all times, with special
> exceptions in cccopy.c and, now, I hope, cc_retr.c :)

So add it. :-)

Whether we want to add it to the advertised API (such as it is) may be
a different question, but certainly having the functionality available
within the library is fine.

>> So is performance acceptable at 1000 entries now, or should we look at
>> a db2 ccache format? :-)
>
> I'll actually do a full test tomorrow. For now I see that truss/strace
> confirms that the clients no longer open/fcntl lock/close for every
> damned ccache entry, so I'm reasonably confident that the problem is
> solved well enough for our needs.
>
> Would an indexed ccache layout be better? Maybe, but I think an
> IPC/daemon ccache where the daemon uses MEMORY ccaches would be even
> better (and it would avoid the need to do any work with db2 or any other
> db).

Yes, I think in the long run this is probably the better choice, and
probably by way of the CCAPI interface.  I wasn't that serious about
db, it's a lot to add in just for a little gain in ccache performance.

Another thing to look at in the short term would be the stdio version
of the ccache code.  Since it uses buffering, it wouldn't make nearly
as many i/o calls.  Currently it's turned on by using "STDIO:" instead
of "FILE:", but really I think it should be an implementation detail
which one of the two is used for "FILE:" since AFAIK the contents
should be the same regardless of the means used to access the disk
file.

Ken