[Dspace-general] [Dspace-tech] too many open files

[Mark Diggory]

I'm going to stop duel posting to both lists in my next email and  
just post to dspace-tech for this issue.

So you have lots of inodes for your filesystems, the next big  
question is. How many are you allocating for open files? Which has  
allot to do with which kernel/O.S. your running. Can you post more  
detail on your Operating System?

-Mark

On Dec 5, 2006, at 3:47 PM, Jose Blanco wrote:

> Here are the results:
>
>
>
> -bash-3.00$ df -i
>
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
>
> /dev/sda7             128520   27978  100542   22% /
>
> /dev/sda1              64256      47   64209    1% /boot
>
> /dev/sda8            5013504   10060 5003444    1% /l
>
> /dev/sdb1            60325888 1881916 58443972    4% /l1
>
> /dev/sdc1            58621952  119448 58502504    1% /l2
>
> none                  223864       1  223863    1% /dev/shm
>
> /dev/sda5             262144     176  261968    1% /tmp
>
> /dev/sda2            3074176  107051 2967125    4% /usr
>
> /dev/sda3             262144    1278  260866    1% /var
>
> AFS                  9000000       0 9000000    0% /afs
>
>
>
> From: Mark Diggory [mailto:mdiggory at MIT.EDU]
> Sent: Tuesday, December 05, 2006 3:47 PM
> To: Jose Blanco
> Cc: dspace-tech at lists.sourceforge.net; dspace-general at MIT.EDU
> Subject: Re: [Dspace-tech] [Dspace-general] too many open files
>
>
>
> Jose,
>
>
>
> This may have much more to do with the number of available inodes  
> on that disk than on the search or filter media, you might do  
> something like "df -i" and post its result to this thread.
>
>
>
> -Mark
>
>
>
> On Dec 5, 2006, at 3:36 PM, Jose Blanco wrote:
>
>
>
>
> Mark:
>
>
>
> Thanks for answering this question.
>
>
>
> We run index-all nightly, and when I go to the in <dspace>/search  
> dir this is what I see:
>
>
>
> -bash-3.00$ pwd
>
> /l1/dspace/repository/prod/search
>
> -bash-3.00$ ls -la
>
> total 2102880
>
> drwxr-xr-x   2 dspace dspace       4096 Dec  5 06:07 .
>
> drwxr-xr-x  13 dspace dspace       4096 Dec  1 10:52 ..
>
> -rw-r--r--   1 dspace dspace          4 Dec  5 06:07 deletable
>
> -rw-r--r--   1 dspace dspace 2151226568 Dec  5 06:07 _s12.cfs
>
> -rw-r--r--   1 dspace dspace         29 Dec  5 06:07 segments
>
>
>
> Does this look OK to you?
>
>
>
> Thanks!!
>
>
>
> From: Mark Diggory [mailto:mdiggory at MIT.EDU]
> Sent: Tuesday, December 05, 2006 3:33 PM
> To: Jose Blanco
> Cc: dspace-tech at lists.sourceforge.net; dspace-general at MIT.EDU
> Subject: Re: [Dspace-general] too many open files
>
>
>
> FilterMedia doesn't actually interact with Lucene directly, only  
> indirectly in that any generated text bitstreams will get picked up  
> later when "index-all" is called. So, no, running filtermedia will  
> not solve your too many files open issue.
>
>
>
> The current version of <dspace>/bin/index-all will rebuild your  
> entire lucene search index (and this will be completely optimized  
> as welli). The usual suggestion is to run it nightly in a cron job  
> on your dspace server. if you look in <dspace>/search and see many  
> many "segment" files there, this may suggest that your index is not  
> optimized.
>
>
>
> Cheers,
>
> Mark
>
>
>
>
>
> On Dec 5, 2006, at 2:43 PM, Jose Blanco wrote:
>
>
>
>
>
> A day ago I posted that we were getting “too many files open” error  
> and I found this thread today discussing it:
>
>
>
> http://sourceforge.net/mailarchive/forum.php? 
> forum_id=39921&max_rows=25&style=flat&viewmonth=200408
>
>
>
> I’m a bit confused as to what I need to do.  I have version 1.4 of  
> DSpace, I’m not sure what version of Lucene I have.  Can some one  
> tell me how I can find that out?  Do I need to get the latest  
> version of Lucene and run ./filter-media with a –f switch to force  
> all items to be re-indexed to create compound files and get rid of  
> this error?
>
>
>
> Thanks!
>
>
>
> Jose
>
> _______________________________________________
>
> Dspace-general mailing list
>
> Dspace-general at mit.edu
>
> http://mailman.mit.edu/mailman/listinfo/dspace-general
>
>
>
> Mark R. Diggory
>
> ~~~~~~~~~~~~~
>
> DSpace Systems Manager
>
> MIT Libraries, Systems and Technology Services
>
> Massachusetts Institute of Technology
>
>
>
>
>
>
>
> ---------------------------------------------------------------------- 
> ---
>
> Take Surveys. Earn Cash. Influence the Future of IT
>
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
>
> opinions on IT & business topics through brief surveys - and earn cash
>
> http://www.techsay.com/default.php? 
> page=join.php&p=sourceforge&CID=DEVDEV________________________________ 
> _______________
>
> DSpace-tech mailing list
>
> DSpace-tech at lists.sourceforge.net
>
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
>
> Mark R. Diggory
>
> ~~~~~~~~~~~~~
>
> DSpace Systems Manager
>
> MIT Libraries, Systems and Technology Services
>
> Massachusetts Institute of Technology
>
>
>
>
>
>
> ---------------------------------------------------------------------- 
> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php? 
> page=join.php&p=sourceforge&CID=DEVDEV________________________________ 
> _______________
> DSpace-tech mailing list
> DSpace-tech at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech

Mark R. Diggory
~~~~~~~~~~~~~
DSpace Systems Manager
MIT Libraries, Systems and Technology Services
Massachusetts Institute of Technology


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/dspace-general/attachments/20061205/a2186c69/attachment.htm