<div dir="ltr"><div>Hi Rayson,</div><div> </div><div>First thank you for replying quickly.</div><div> </div><div>The first issue I mentioned is intermittent - it is not reproducible - I was able to easily start a 5 node cluster and 20 node cluster just now. I assume this was due to communication problems and will forget about it. Never the less, it would help if there would be a tool like a log that helps find out what went wrong and diagnose if this is a network problem or something else.</div>
<div> </div><div>The second issue, however, is reproducible. I just tried again a 20 node cluster. </div><div><br>This time I posted 2497 jobs to the queue - each about 1 minute long. The system stopped sending jobs the queue about half point. There were 1310 jobs in the queue when the system stopped sending more jobs. </div>
<div>When running "qstat -j" the system provided the following answer:</div><div> </div><div>scheduling info: (Collecting of scheduler job information is turned off)</div><div><br>I am not familiar with the error messages, yet it seems I need to enable something that is turned off. If there is quick obvious solution for this please let me know what to do, otherwise are there any other diagnostics tools I can use? </div>
<div> </div><div>Again, thanks for the quick reply and I hope this is an easy fix.</div><div> </div><div> Jacob<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 22, 2013 at 2:51 PM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Sun, Jul 21, 2013 at 1:40 AM, Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com">jacob.barhak@gmail.com</a>> wrote:<br>
> 1. Sometime starcluster is unable to properly connect the instances on the<br>
> start command and cannot mount /home. It happened once when I asked for 5<br>
> m1.small machines and when I terminated this cluster and started again<br>
> things went fine. Is this intermittent due to cloud traffic or is this a<br>
> bug? Is there a way for me to check why?<br>
<br>
</div>Can be a problem with the actual hardware - can you ssh into the node<br>
and manually mount /home by hand next time you encounter this issue<br>
and see if you can reproduce it when run interactively?<br>
<div class="im"><br>
<br>
> 2. After launching 20 c1.xlarge machines and running about 2500 jobs, each<br>
> about 5 minutes long, I encountered a problem after and hour or so. It seems<br>
> that SGE stopped sending jobs from to the queue to the instances. No error<br>
> was found and the queue showed about 850 pending jobs. This did not change<br>
> for a while and I could not find any failure with qstat or qhost. No jobs<br>
> were running on any nodes and I waited a while for these to start without<br>
> success. I tried the same thing again after a few hours and it seems that<br>
> the cluster stops sending jobs from the queue after about 1600 jobs have<br>
> been submitted. This does not happen when SGE is installed on a single<br>
> Ubuntu machine I have at home. I am trying to figure out what is wrong. Did<br>
> you impose some limit on the number of jobs? Can this be fixed? I really<br>
> need to submit many jobs - tens of thousands jobs in my full runs. This one<br>
> was relatively small and still did not pass.<br>
<br>
</div>Looks like SGE thinks that the nodes are down or in alarm or error<br>
state? To find out why SGE thinks there are no nodes available, run:<br>
qstat -j<br>
<div class="im"><br>
<br>
<br>
> 3. I tried to start Star Cluster with 50 nodes and got an error about<br>
> exceeding a quota of 20. Is it your quota or Amazon quota? Are there any<br>
> other restrictions I should be aware of at the beginning? Also after the<br>
> system is unable start the cluster it thinks it is still running and a<br>
> terminate command is needed before another start can be issued - even though<br>
> nothing got started.<br>
<br>
</div>It's Amazon's quota. 50 is considered small by AWS standard, and they<br>
can give it to you almost right away... You need to request AWS to<br>
give you a higher limit:<br>
<a href="https://aws.amazon.com/contact-us/ec2-request/" target="_blank">https://aws.amazon.com/contact-us/ec2-request/</a><br>
<br>
Note that last year we requested for 10,000 nodes and the whole<br>
process took less than 1 day:<br>
<br>
<a href="http://blogs.scalablelogic.com/2012/11/running-10000-node-grid-engine-cluster.html" target="_blank">http://blogs.scalablelogic.com/2012/11/running-10000-node-grid-engine-cluster.html</a><br>
<br>
Rayson<br>
<br>
==================================================<br>
Open Grid Scheduler - The Official Open Source Grid Engine<br>
<a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br>
<div class="im"><br>
<br>
><br>
> This all happened on us-west-2 with the help of star cluster 0.93.3 and the<br>
> Anaconda AMI - ami-a4d64194<br>
><br>
> Here is some more information on what I am doing to help you answer the<br>
> above.<br>
><br>
> I am running Monte Carlo simulations to simulate chronic disease<br>
> progression. I am using MIST to run over the cloud:<br>
><br>
> <a href="https://github.com/scipy/scipy2013_talks/blob/master/talks/jacob_barhak/readme.md" target="_blank">https://github.com/scipy/scipy2013_talks/blob/master/talks/jacob_barhak/readme.md</a><br>
><br>
> The Reference Model is what I am running using MIST:<br>
><br>
> <a href="http://youtu.be/7qxPSgINaD8" target="_blank">http://youtu.be/7qxPSgINaD8</a><br>
><br>
> I am launching many simulations in parallel and it takes me days on a single<br>
> 8 core machine. The cloud allows me to cut down this time to hours. This is<br>
> why star cluster is so useful. In the past I did this over other clusters<br>
> yet the cloud is still new to me.<br>
><br>
> I will appreciate any recommendations I can get from you to improve the<br>
> behaviors I am experiencing.<br>
><br>
> --<br>
> Jacob Barhak Ph.D.<br>
> <a href="http://sites.google.com/site/jacobbarhak/" target="_blank">http://sites.google.com/site/jacobbarhak/</a><br>
><br>
><br>
> Sent from my iPhone<br>
><br>
</div>> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
><br>
</blockquote></div><br></div>