<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div style="-webkit-text-size-adjust: auto; ">Thanks Chris,</div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">This is helpful. </div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">Your suggestion to change to in demand scheduling makes sense. If I set <b style="font-style: italic; background-color: rgba(255, 255, 255, 0); "><font size="3">flush_finish_sec </font></b>to 1 sec I will get faster submission of my faster jobs that take less than a second. Currently I have 20 slots available so this should allow processing about 1200 of these in a minute. What I am afraid of is that the scheduler will get called 20 times - once for each such job. Since grid engine seg faulted in the last day under pressure I am afraid of stressing it to this level - so I am inquiring about behavior before I make this change on a running simulation. Does each event trigger another instance of grid engine to launch? Or are all 20 events handled at once a second after the first event about completion is received?</div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">Also, your suggestions do not address the issue that I find disturbing. Deleting jobs just takes forever when the machine is loaded with jobs. So if there is trouble it takes me about an hour just to clean the queue and restart everything from scratch. If there is an easier way to clear a queue of jobs it would help. I have a single queue, will deleting it and creating a new one we faster?</div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">By the way, my queue type according to qmon is BIP. I am running 20 slots with 8 CPUs and arch is lx26amd64. </div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">Can this queue type handle 50k+ jobs efficiently for submission/deletion? I think the information you provided gets launching queued jobs covered, the real issue is submission/deletion overhead. And I would like to know from others who launch that many dependent jobs about their experiences. </div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">Also, is there a way to tell grid engine to resubmit interrupted jobs under the same name and job number in case of failure. I have many dependencies and if a machine breaks during a long run I want grid engine to relaunch the jobs it was working on in a way that will maintain dependencies defined at submission time. Currently I had to write a script to recover from interrupted runs - if grid engine has this capability it could make things easier. And by the way, I need to submit many little jobs rather than job arrays partially because of these dependencies. </div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">I would appreciate to learn about other people experiences with 50k+ jobs. </div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; "> Jacob</div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">Sent from my iPhone</div><div style="-webkit-text-size-adjust: auto; "><br>On Mar 30, 2014, at 6:35 AM, Chris Dagdigian <<a href="mailto:dag@bioteam.net">dag@bioteam.net</a>> wrote:<br><br></div><blockquote type="cite" style="-webkit-text-size-adjust: auto; "><div><span></span><br><span>The #1 suggestion is to change the default scheduling setup over to the</span><br><span>settings for "on demand scheduling" - you can do that live on a running</span><br><span>cluster I believe. Google can help you with the parameters but it</span><br><span>basically involves switching from the looping/cyclical scheduler</span><br><span>behavior to an event based method where SGE runs a scheduling/dispatch</span><br><span>cycle each time a job enters or leaves the system. This is purpose built</span><br><span>for the "many short jobs" use case.</span><br><span></span><br><span>The #2 suggestion is to make sure you use berkeleyDB based spooling</span><br><span>(this is one scenario where I'll switch from my preference for classic</span><br><span>spooling) AND make sure that you are spooling to local disk on each node</span><br><span>instead of spooling back to a shared filesystem. These settings cannot</span><br><span>easily be changed on a live system so in starcluster land you might have</span><br><span>to tweak the source code to come up with a different installation</span><br><span>template that would put these settings in. (Note that I'm not sure what</span><br><span>SC puts in by default so there is a chance that this is already set up</span><br><span>the way you'd want ...)</span><br><span></span><br><span>Other stuff:</span><br><span></span><br><span>- SGE Array Jobs are way more efficient than single jobs. Instead of</span><br><span>70,000 jobs can you submit a single array job that has *70,000 tasks*</span><br><span>instead? Or 7 jobs containing 10,000 tasks etc. ?</span><br><span></span><br><span>- Is there any way to batch up or collect your short jobs into bigger</span><br><span>collections so that the average job run time is longer? SGE is not the</span><br><span>best at jobs that run for mere seconds or less - most people would batch</span><br><span>those together to at least get a 60-90 second runtime ...</span><br><span></span><br><span></span><br><span></span><br><span>-Chris</span><br><span></span><br><span></span><br><span></span><br><span>Jacob Barhak wrote:</span><br><blockquote type="cite"><span>Hi to SGE experts,</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>This is less of a StarCluster specific issue. It is more of an SGE issue I encountered and was hoping someone here can help with. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>My system runs many many smaller jobs - tens of thousands. When I need a rushed solution to reach a deadline I use StarCluster. However, if I have time, I run the simulations on a single 8 core machine that has SGE installed over Ubuntu 12.04. This machine is new and fast with SSD drive and freshly installed yet I am encountering issues. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>1. When I launch about 70k jobs submitting a single new job to the queue takes a while - about a second or so, compared to fractions of a second when the queue is empty. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>2. Deleting all the jobs from the queue using qdel -u username takes a long time. It reports about 24 deletes to the screen every few seconds - at this rate it will take hours to delete the entire queue. It is still deleting while I am writing these words. Way too much time. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>3. The system was working ok for a few days yet now has trouble with qmaster. It report the following:</span><br></blockquote><blockquote type="cite"><span>error: commlib error: got select error (connection refused) </span><br></blockquote><blockquote type="cite"><span>unable to send message to qmaster using port 6444 on host 'localhost': got send error. </span><br></blockquote><blockquote type="cite"><span>Also qmon reported cannot reach qmaster. I had to restart and suspend and disable the queue. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Note that qstat -j currently reports:</span><br></blockquote><blockquote type="cite"><span>All queues dropped because of overload or full</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Note that I configured the schedule interval to 2 seconds since many of my jobs are so fast that even 2 seconds is very inefficient for them yet some are longer and memory consuming so I cannot allow more slots to launch too many jobs. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Am I overloading the system with too many jobs? What is the limit on a single strong machine? How will this scale when I run this on StarCluster?</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Any advice on how to efficiently handle many jobs, some of which are very short, will be appreciated. And I hope this interests the audience. </span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span> Jacob</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Sent from my iPhone</span><br></blockquote><blockquote type="cite"><span>_______________________________________________</span><br></blockquote><blockquote type="cite"><span>StarCluster mailing list</span><br></blockquote><blockquote type="cite"><span><a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a></span><br></blockquote><blockquote type="cite"><span><a href="http://mailman.mit.edu/mailman/listinfo/starcluster">http://mailman.mit.edu/mailman/listinfo/starcluster</a></span><br></blockquote></div></blockquote></body></html>