Hi Justin,<div><br></div><div>Thanks for your response.<br><div><br></div><div>1. Currently my account has an instance limit of 100. </div><div><br></div><div>2. I will with spot instance to check with the speed.</div><div>
<br></div><div>3. As per one of my query.</div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><span class="Apple-style-span" style="border-collapse: collapse; color: rgb(80, 0, 80); font-family: arial, sans-serif; font-size: 13px; "><div>
<br></div><div><span style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; "><div>Does starcluster wait for all the nodes to be up and then it starts configuring them all at one time.</div><div>
Is there any parameter in the config file or any options in the starcluster start command that says "<font color="#000099">configuration of the cluster and installing SGE/Configuring NFS to be a parallel operation. any node should not wait for the other nodes to be up for getiing configured that's if we post a job on that ready node it should start executing the job with the available no of nodes that are running and configured</font>."</div>
<div><br></div><div>If the above is not possible , is there any specific reason while starting a cluster, starcluster does the configuration of nodes only when all are running.</div><div><br></div><div><br></div><div>Regards</div>
<div>Sumita</div></span></div></span><br><div class="gmail_quote">On Wed, Nov 9, 2011 at 12:15 PM, Justin Riley <span dir="ltr"><<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi Sumita,<br>
<br>
</div>Unless you've specifically submitted a request to Amazon to increase<br>
your instance limit[1] I can't help but suspect that you're likely<br>
running into issues because of the default 20 instance limit for<br>
flat-rate instances I mentioned earlier.<br>
<br>
I would recommend trying with spot instances[2]; they're usually<br>
cheaper than the flat-rate(s) AND you can launch up to 100 of them. To<br>
request a spot cluster just pass the --bid option to the start command:<br>
<br>
$ starcluster start --bid 0.50 mycluster<br>
<br>
This will place a $0.50 spot bid on each node in the cluster except<br>
for the master. The master node is always launched as a flat-rate<br>
instance for stability.<br>
<br>
To help you decide a decent spot bid use the spot history command:<br>
<br>
$ starcluster spothistory m1.large<br>
<br>
With that said you can check which nodes have SSH up using:<br>
<br>
$ starcluster listclusters --show-ssh-status<br>
<br>
Also, you can *always* restart and reboot all nodes in the cluster and<br>
completely reconfigure the cluster using the "restart" commmand:<br>
<br>
$ starcluster restart mycluster<br>
<br>
HTH,<br>
<br>
~Justin<br>
<br>
[1] <a href="http://aws.amazon.com/contact-us/ec2-request/" target="_blank">http://aws.amazon.com/contact-us/ec2-request/</a><br>
[2] <a href="http://aws.amazon.com/ec2/spot-instances/" target="_blank">http://aws.amazon.com/ec2/spot-instances/</a><br>
<div class="im"><br>
On 11/08/2011 07:20 PM, Sumita Sinha wrote:<br>
> Hi Justin,<br>
><br>
> I again tried creating 30 nodes cluster and figured out something<br>
> new. I am waiting for last 20 min for the cluster to be up. I get<br>
> the below message. Currently in EC2 all the nodes are up and<br>
> running ,i don't know which node is taking time for SSH<br>
> configuration. so i am not able to restart or terminate a node.<br>
><br>
>>>> Using default cluster template: smallcluster Validating<br>
>>>> cluster template settings... Cluster template settings are<br>
>>>> valid Starting cluster... Launching a 30-node cluster...<br>
>>>> Creating security group @sc-smallcluster...<br>
> Reservation:r-0e2d7060<br>
>>>> Waiting for cluster to come up... (updating every 30s)<br>
>>>> Waiting for all nodes to be in a 'running' state...<br>
> 29/29<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> 100%<br>
>>>> Waiting for SSH to come up on all nodes...<br>
> 28/29<br>
> |------------------------------------------------------------- |<br>
> 96%<br>
><br>
><br>
> Regards Sumita<br>
><br>
> On Tue, Nov 8, 2011 at 7:42 PM, Justin Riley <<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a><br>
</div><div><div></div><div class="h5">> <mailto:<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>>> wrote:<br>
><br>
><br>
> Hi Sumita,<br>
><br>
> Were you using spot instances? If not I believe there's a default<br>
> limit of 20 instances by default for flat-rate instances which<br>
> *could* be related to your issue. With spot instances you can<br>
> create up to 100 instances by default. So, if you need more than<br>
> 20 nodes and do not wish to submit a request to Amazon to increase<br>
> your flat-rate instance limit, you should be using spot instances:<br>
><br>
> $ starcluster start -s 30 -b 0.50 mycluster<br>
><br>
> With that said, StarCluster has no limit to the number of nodes you<br>
> can create, however, as you've seen, sometimes EC2 instances can<br>
> take longer to become 'running' than usual. Unfortunately this is<br>
> purely an EC2 back-end issue that cannot be resolved directly by<br>
> StarCluster. In my experience 22 minutes *is* quite a while to wait<br>
> for any instance to come up, however, I have had instances take up<br>
> to 15 min before in the past so this is not a total surprise to<br>
> me.<br>
><br>
> In the future if you run into this problem of waiting for an<br>
> instance to change from 'pending' to 'running' for too long (e.g.<br>
> 15min+) I would recommend simply terminating the faulty instance<br>
> from the AWS console and then restart the cluster using:<br>
><br>
> $ starcluster restart mycluster<br>
><br>
> This should reboot all the currently running instances and begin<br>
> configuring the cluster and avoid having to terminate the entire<br>
> cluster and lose instance hours.<br>
><br>
> HTH,<br>
><br>
> ~Justin<br>
><br>
><br>
> On 11/8/11 6:39 AM, Sumita Sinha wrote:<br>
>> Hello ,<br>
><br>
>> Currently working with starcluster on EC2.<br>
><br>
>> Tried creating a cluster with 30 nodes of type m1.small using<br>
>> AMI -<br>
> ami-8cf913e5.<br>
>> Cluster creation was never completed as i found out that one<br>
>> node<br>
> node025 was showing pending status.<br>
>> I waited for almost 22 minutes then terminated the cluster.<br>
>> Cluster was terminated properly. Is there any limit to the<br>
>> creation<br>
> of nodes .<br>
><br>
><br>
><br>
><br>
>> -- Regards Sumita Sinha<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> -- Regards Sumita Sinha<br>
><br>
><br>
<br>
-----BEGIN PGP SIGNATURE-----<br>
</div></div>Version: GnuPG v2.0.17 (GNU/Linux)<br>
<div class="im">Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
<br>
</div>iEYEARECAAYFAk66IZYACgkQ4llAkMfDcrmZ5ACeIPTP8ZiFKTlTNxif6SgIKsWm<br>
SmoAnA08GWFcOcmpCF+MMHwLzhqzD0Va<br>
=KFye<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Regards<br>Sumita Sinha<br><br><br>
</div></div>