[Dspace-general] [Dspace-tech] too many open files

[Jose Blanco]

So why do you think we are getting "too many open files" error?  It seems to
be happening when google is crawling our site.  It also seems like this
error message has to do with the kernel limits on the number of open files,
which by default is 1024 - which should be enough, no?  And we do just run
./filter-media nightly.

 

Thanks for you thoughts on this.

 

 

  _____  

From: dspace-tech-bounces at lists.sourceforge.net
[mailto:dspace-tech-bounces at lists.sourceforge.net] On Behalf Of Mark Diggory
Sent: Tuesday, December 05, 2006 3:44 PM
To: Jose Blanco
Cc: dspace-tech at lists.sourceforge.net; dspace-general at MIT.EDU
Subject: Re: [Dspace-tech] [Dspace-general] too many open files

 

Yes, that looks like an optimized search index. An unoptimized index would
many more files in it.

 

-Mark

 

On Dec 5, 2006, at 3:36 PM, Jose Blanco wrote:





Mark:

 

Thanks for answering this question.

 

We run index-all nightly, and when I go to the in <dspace>/search dir this
is what I see:

 

-bash-3.00$ pwd

/l1/dspace/repository/prod/search

-bash-3.00$ ls -la

total 2102880

drwxr-xr-x   2 dspace dspace       4096 Dec  5 06:07 .

drwxr-xr-x  13 dspace dspace       4096 Dec  1 10:52 ..

-rw-r--r--   1 dspace dspace          4 Dec  5 06:07 deletable

-rw-r--r--   1 dspace dspace 2151226568 Dec  5 06:07 _s12.cfs

-rw-r--r--   1 dspace dspace         29 Dec  5 06:07 segments

 

Does this look OK to you?

 

Thanks!!

 

  _____  

From: Mark Diggory [mailto:mdiggory at MIT.EDU] 
Sent: Tuesday, December 05, 2006 3:33 PM
To: Jose Blanco
Cc: dspace-tech at lists.sourceforge.net; dspace-general at MIT.EDU
Subject: Re: [Dspace-general] too many open files

 

FilterMedia doesn't actually interact with Lucene directly, only indirectly
in that any generated text bitstreams will get picked up later when
"index-all" is called. So, no, running filtermedia will not solve your too
many files open issue.

 

The current version of <dspace>/bin/index-all will rebuild your entire
lucene search index (and this will be completely optimized as welli). The
usual suggestion is to run it nightly in a cron job on your dspace server.
if you look in <dspace>/search and see many many "segment" files there, this
may suggest that your index is not optimized.

 

Cheers,

Mark

 

 

On Dec 5, 2006, at 2:43 PM, Jose Blanco wrote:






A day ago I posted that we were getting "too many files open" error and I
found this thread today discussing it:

 

http://sourceforge.net/mailarchive/forum.php?forum_id=39921
<http://sourceforge.net/mailarchive/forum.php?forum_id=39921&max_rows=25&sty
le=flat&viewmonth=200408> &max_rows=25&style=flat&viewmonth=200408

 

I'm a bit confused as to what I need to do.  I have version 1.4 of DSpace,
I'm not sure what version of Lucene I have.  Can some one tell me how I can
find that out?  Do I need to get the latest version of Lucene and run
./filter-media with a -f switch to force all items to be re-indexed to
create compound files and get rid of this error?

 

Thanks!

 

Jose

_______________________________________________

Dspace-general mailing list

Dspace-general at mit.edu

http://mailman.mit.edu/mailman/listinfo/dspace-general

 

Mark R. Diggory

~~~~~~~~~~~~~

DSpace Systems Manager

MIT Libraries, Systems and Technology Services

Massachusetts Institute of Technology






 

-------------------------------------------------------------------------

Take Surveys. Earn Cash. Influence the Future of IT

Join SourceForge.net's Techsay panel and you'll get the chance to share your

opinions on IT & business topics through brief surveys - and earn cash

http://www.techsay.com/default.php?page=join.php
<http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV__
_____________________________________________>
&p=sourceforge&CID=DEVDEV_______________________________________________

DSpace-tech mailing list

DSpace-tech at lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/dspace-tech

 

Mark R. Diggory

~~~~~~~~~~~~~

DSpace Systems Manager

MIT Libraries, Systems and Technology Services

Massachusetts Institute of Technology





 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/dspace-general/attachments/20061205/cb49bd63/attachment.htm