<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear all, </div><div><br></div><div>Thank you for your feedback, it has been very useful. The new StarCluster release 0.92 solves most of the problems. <br><br>It is much more stable, and node I don't get any error launching large clusters (with 100 or more instances). <br><br>Anyway the overall process is still very slow and, above all, the time required seems to be linear with the number of the instances used. <br><br>For examples: <br><br>- Launching 100 nodes, the configuration requires ~ 30 minutes to complete;<br>- Launching 200 nodes, it requires ~ 1 hour; <br><br>Since our target is launching such as number of nodes to run jobs that may require around 1 hour to be completed, it would be meaningless to spend 50% or more of the time only to configure the system. The addnode command does not help because this process is even longer, since for each added node StarCluster need to update the /etc/hosts for each node.</div><div><br></div><div><br></div><div>So the question is: would not be possible to use pre-configured node images, to shorten as much as possible to configuration steps (ideally only to the "/etc/hosts" files and the SGE updating) ?</div><div><br></div><div><br></div><div>I'm thinking something similar to: </div><div><br></div><div>1) Launch a 2-node configuration. </div><div>2) Save the master and the node instances as two new separate AMI images. </div><div>3) Use these images as pre-configured machines to deploy a large cluster, updating the "hosts" files (and whatever else is needed). </div><div><br></div><div>This would avoid to configure all the nodes from scratch and reduce the overall star-up time. </div><div><br></div><div><br></div><div>Does it make sense? Is it possible in some way? Maybe using a custom plugin ? </div><div><br></div><div><br></div><div>Cheers, </div><div><br></div><div>Paolo Di Tommaso</div><div><div><div>Software Engineer </div><div>Comparative Bioinformatics Group</div><div>Centre de Regulacio Genomica (CRG)</div><div>Dr. Aiguader, 88</div><div>08003 Barcelona, Spain</div></div>
</div>
<div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><div><div>On Oct 17, 2011, at 5:59 PM, Rayson Ho wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div>1) I agree with Matt, also a 20-node cluster should be relatively error free to bootstrap.<br></div><div><br></div><div><br></div><div>2) EC2 occasionally fails to start a node or 2 when requested to start a large number of nodes (instances), and I believe it has to do with how busy it is handling other requests as well. The best way to not overload EC2 is to start a few nodes at a time rather than the whole cluster all at once.</div><div><br></div><div>In 0.92rc2, there is the addnode command:</div><div><br></div><div>$ starcluster addnode mynewcluster</div><div><br></div><div>The latest trunk introduces the ability to add multiple nodes, e.g. 3 nodes:</div><div><br></div><div>$ starcluster addnode -n 3 mycluster</div><div><br></div><div>So instead of starting a 100-node cluster during start-up, try starting a 20 or 30-node
one first, and then grow the cluster. For 0.92rc2, you may want to script the addnode command unless you enjoy typing :-D<br></div><div><br></div><div><br></div><div>3) I will do more scalability testing and hope to contribute scalability related improvements to StarCluster in the near future. I am waiting for the EBS based AMI so that I can start a large number of instances without breaking the bank - I am going to use my own AWS account, so I am interested in minimizing cost by using t1.micro (which is slower when running real work, but I am interesting in the launch speed of EC2 itself, so t1.micro seems to be perfect for my need!).<br></div><div><br></div><div><a href="https://github.com/jtriley/StarCluster/issues/52">https://github.com/jtriley/StarCluster/issues/52</a><br></div><div><a href="http://mailman.mit.edu/pipermail/starcluster/2011-October/000818.html">http://mailman.mit.edu/pipermail/starcluster/2011-October/000818.html</a></div><div><br></div><div>(To Justin: no pressure in getting the EBS AMI, I will be busy till mid
Nov).</div><div><br></div><div><span>Rayson</span></div><div><br><span></span></div><div><span>=================================<br>Grid Engine / Open Grid Scheduler<br><a href="http://gridscheduler.sourceforge.net">http://gridscheduler.sourceforge.net</a><br></span></div><div><br><span></span></div><div><span></span></div><div><br></div><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;"><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;"><font size="2" face="Arial"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Matthew Summers <<a href="mailto:quantumsummers@gentoo.org">quantumsummers@gentoo.org</a>><br><b><span style="font-weight: bold;">To:</span></b> "<a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a>" <<a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a>><br><b><span style="font-weight: bold;">Sent:</span></b> Monday, October 17, 2011 10:58 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [StarCluster] 100 nodes cluster</font><br><br>Are you guys running a versioned release or the HEAD on git. I am
more<br>than fairly certain this has been optimized in the repo, iirc a few<br>months ago.<br><br>-- <br>Matthew W. Summers<br>Gentoo Foundation Inc.<br><br>_______________________________________________<br>StarCluster mailing list<br><a ymailto="mailto:StarCluster@mit.edu" href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br><a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>Barcelona, Spain<br>>><br>>><br>>> _______________________________________________<br>>> StarCluster mailing list<br>>> <a ymailto="mailto:StarCluster@mit.edu" href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>>> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>><br>> --<br>> Luis M. Carril<br>> Project Technician<br>> Galicia Supercomputing Center (CESGA)<br>> Avda. de Vigo s/n<br>> 15706 Santiago de
Compostela<br>> SPAIN<br>><br>> Tel: 34-981569810 ext 249<br>> <a ymailto="mailto:lmcarril@cesga.es" href="mailto:lmcarril@cesga.es">lmcarril@cesga.es</a><br>> <a target="_blank" href="http://www.cesga.es/">www.cesga.es</a><br>><br>><br>> ==================================================================<br>><br>> _______________________________________________<br>> StarCluster mailing list<br>> <a ymailto="mailto:StarCluster@mit.edu" href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>><br><br>Are you guys running a versioned release or the HEAD on git. I am more<br>than fairly certain this has been optimized in the repo, iirc a few<br>months ago.<br><br>-- <br>Matthew W. Summers<br>Gentoo Foundation
Inc.<br><br>_______________________________________________<br>StarCluster mailing list<br><a ymailto="mailto:StarCluster@mit.edu" href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br><a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br><br><br></div></div></div></div>_______________________________________________<br>StarCluster mailing list<br><a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>http://mailman.mit.edu/mailman/listinfo/starcluster<br></blockquote></div><br></body></html>