[StarCluster] 100 nodes cluster

Thu Oct 20 14:53:49 EDT 2011

It's interesting Mark -- I didn't know about the "Global Interpreter Lock" (GIL), and it can be a possible issue. But before modifying the StarCluster code to use xarg right away, may be we should find out if the bottleneck is caused by the GIL.

StarCluster 0.92 uses Workerpool (threadpool), and is the GIL really in the way when StarCluster executes ssh to do remote setup? And if it is, can StarCluster use the same tricks as zlib or hashlib to release the GIL when it is doing the ssh setup??

I really wanted to do the testing myself, but I am busy till Nov... :-(  However, if one can wrap around the real ssh with a fake ssh script that sleeps 30 seconds and then runs the real ssh, then we can see how good (or bad) the Workerpool handles long latency commands - and we will start from there to optimize the launch performance.

Also, congratulations to Justin & others who contribute to StarCluster - a few months ago, people were having problems with clusters smaller than 100 nodes, and now the issue is "just" performance when launching a 200-node cluster.

Rayson

=================================
Grid Engine / Open Grid Scheduler
http://gridscheduler.sourceforge.net

________________________________
From: Mark Gordon <mark.gordon at ualberta.ca>
To: "starcluster at mit.edu" <starcluster at mit.edu>
Sent: Thursday, October 20, 2011 10:31 AM
Subject: Re: [StarCluster] 100 nodes cluster

Hey Paolo:

Just throwing this out there, as a possibility: the xargs command when used with the P option could be used to launch all the worker nodes at once (I am sure there is some limit in each case as to the maximum number of concurrent conversations a single machine could drive). See this blog post. Python's GIL prevents true multi-threading in the language, so something like xarg is needed.

Using xargs would require a change to StarCluster. The source is available at GitHub..

Thanks to Luke Tymowski at Cybera for the tip re xargs.

cheers,
Mark

Systems Analyst
Department of Physics
University of Alberta

On Thu, Oct 20, 2011 at 5:32 AM, Paolo Di Tommaso <Paolo.DiTommaso at crg.eu> wrote:

Dear all, 
>
>
>Thank you for your feedback, it has been very useful. The new StarCluster release 0.92 solves most of the problems. 
>
>It is much more stable, and node I don't get any error launching large clusters (with 100 or more instances). 
>
>Anyway the overall process is still very slow and, above all, the time required seems to be linear with the number of the instances used. 
>
>For examples: 
>
>- Launching 100 nodes, the configuration requires ~ 30 minutes to complete;
>- Launching 200 nodes, it requires ~ 1 hour; 
>
>Since our target is launching such as number of nodes to run jobs that may require around 1 hour to be completed, it would be meaningless to spend 50% or more of the time only to configure the system. The addnode command does not help because this process is even longer, since for each added node StarCluster need to update the /etc/hosts for each node.
>
>
>
>
>So the question is: would not be possible to use pre-configured node images, to shorten as much as possible to configuration steps (ideally only to the "/etc/hosts" files and the SGE updating) ?
>
>
>
>
>I'm thinking something similar to: 
>
>
>1) Launch a 2-node configuration. 
>2) Save the master and the node instances as two new separate AMI images. 
>3) Use these images as pre-configured machines to deploy a large cluster, updating the "hosts" files (and whatever else is needed). 
>
>
>This would avoid to configure all the nodes from scratch and reduce the overall star-up time. 
>
>
>
>
>Does it make sense? Is it possible in some way? Maybe using a custom plugin ? 
>
>
>
>
>Cheers, 
>
>
>Paolo Di Tommaso
>Software Engineer 
>Comparative Bioinformatics Group
>Centre de Regulacio Genomica (CRG)
>Dr. Aiguader, 88
>08003 Barcelona, Spain
>
>
>
>
>
>
>
>
>
>
>
>On Oct 17, 2011, at 5:59 PM, Rayson Ho wrote:
>
>1) I agree with Matt, also a 20-node cluster should be relatively error free to bootstrap.
>>
>>
>>
>>
>>
>>2) EC2 occasionally fails to start a node or 2 when requested to start a large number of nodes (instances), and I believe it has to do with how busy it is handling other requests as well. The best way to not overload EC2 is to start a few nodes at a time rather than the whole cluster all at once.
>>
>>
>>In 0.92rc2, there is the addnode command:
>>
>>
>>$ starcluster addnode mynewcluster
>>
>>
>>The latest trunk introduces the ability to add multiple nodes, e.g. 3 nodes:
>>
>>
>>$ starcluster addnode -n 3 mycluster
>>
>>
>>So instead of starting a 100-node cluster during start-up, try starting a 20 or 30-node one first, and then grow the cluster. For 0.92rc2, you may want to script the addnode command unless you enjoy typing :-D
>>
>>
>>
>>
>>
>>3) I will do more scalability testing and hope to contribute scalability related improvements to StarCluster in the near future. I am waiting for the EBS based AMI so that I can start a large number of instances without breaking the bank - I am going to use my own AWS account, so I am interested in minimizing cost by using t1.micro (which is slower when running real work, but I am interesting in the launch speed of EC2 itself, so t1.micro seems to be perfect for my need!).
>>
>>
>>
>>https://github.com/jtriley/StarCluster/issues/52
>>
>>http://mailman.mit.edu/pipermail/starcluster/2011-October/000818.html
>>
>>
>>(To Justin: no pressure in getting the EBS AMI, I will be busy till mid Nov).
>>
>>
>>Rayson
>>
>>
>>=================================
>>Grid Engine / Open Grid Scheduler
>>http://gridscheduler.sourceforge.net
>>
>>
>>
>>
>>
>>
>>________________________________
>>From: Matthew Summers <quantumsummers at gentoo.org>
>>To: "starcluster at mit.edu" <starcluster at mit.edu>
>>Sent: Monday, October 17, 2011 10:58 AM
>>Subject: Re: [StarCluster] 100 nodes cluster
>>
>>Are you guys running a versioned release or the HEAD on git. I am
more
>>than fairly certain this has been optimized in the repo, iirc a few
>>months ago.
>>
>>-- 
>>Matthew W. Summers
>>Gentoo Foundation Inc.
>>
>>_______________________________________________
>>StarCluster mailing list
>>StarCluster at mit.edu
>>http://mailman.mit.edu/mailman/listinfo/starcluster
>>Barcelona, Spain
>>>>
>>>>
>>>> _______________________________________________
>>>> StarCluster mailing list
>>>> StarCluster at mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>> --
>>> Luis M. Carril
>>> Project Technician
>>> Galicia Supercomputing Center (CESGA)
>>> Avda. de Vigo s/n
>>> 15706 Santiago de
Compostela
>>> SPAIN
>>>
>>> Tel: 34-981569810 ext 249
>>> lmcarril at cesga.es
>>> www.cesga.es
>>>
>>>
>>> ==================================================================
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>
>>Are you guys running a versioned release or the HEAD on git. I am more
>>than fairly certain this has been optimized in the repo, iirc a few
>>months ago.
>>
>>-- 
>>Matthew W. Summers
>>Gentoo Foundation
Inc.
>>
>>_______________________________________________
>>StarCluster mailing list
>>StarCluster at mit.edu
>>http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>>_______________________________________________
>>StarCluster mailing list
>>StarCluster at mit.edu
>>http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>
>_______________________________________________
>StarCluster mailing list
>StarCluster at mit.edu
>http://mailman.mit.edu/mailman/listinfo/starcluster
>
>

Barcelona, Spain

On Oct 17, 2011, at 5:59 PM, Rayson Ho wrote:

1) I agree with Matt, also a 20-node cluster should be relatively error free to bootstrap.
>
>
>
>
>
>2) EC2 occasionally fails to start a node or 2 when requested to start a large number of nodes (instances), and I believe it has to do with how busy it is handling other requests as well. The best way to not overload EC2 is to start a few nodes at a time rather than the whole cluster all at once.
>
>
>In 0.92rc2, there is the addnode command:
>
>
>$ starcluster addnode mynewcluster
>
>
>The latest trunk introduces the ability to add multiple nodes, e.g. 3 nodes:
>
>
>$ starcluster addnode -n 3 mycluster
>
>
>So instead of starting a 100-node cluster during start-up, try starting a 20 or 30-node one first, and then grow the cluster. For 0.92rc2, you may want to script the addnode command unless you enjoy typing :-D
>
>
>
>
>
>3) I will do more scalability testing and hope to contribute scalability related improvements to StarCluster in the near future. I am waiting for the EBS based AMI so that I can start a large number of instances without breaking the bank - I am going to use my own AWS account, so I am interested in minimizing cost by using t1.micro (which is slower when running real work, but I am interesting in the launch speed of EC2 itself, so t1.micro seems to be perfect for my need!).
>
>
>
>https://github.com/jtriley/StarCluster/issues/52
>
>http://mailman.mit.edu/pipermail/starcluster/2011-October/000818.html
>
>
>(To Justin: no pressure in getting the EBS AMI, I will be busy till mid Nov).
>
>
>Rayson
>
>
>=================================
>Grid Engine / Open Grid Scheduler
>http://gridscheduler.sourceforge.net
>
>
>
>
>
>
>________________________________
>From: Matthew Summers <quantumsummers at gentoo.org>
>To: "starcluster at mit.edu" <starcluster at mit.edu>
>Sent: Monday, October 17, 2011 10:58 AM
>Subject: Re: [StarCluster] 100 nodes cluster
>
>Are you guys running a versioned release or the HEAD on git. I am
more
>than fairly certain this has been optimized in the repo, iirc a few
>months ago.
>
>-- 
>Matthew W. Summers
>Gentoo Foundation Inc.
>
>_______________________________________________
>StarCluster mailing list
>StarCluster at mit.edu
>http://mailman.mit.edu/mailman/listinfo/starcluster
>Barcelona, Spain
>>>
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>> --
>> Luis M. Carril
>> Project Technician
>> Galicia Supercomputing Center (CESGA)
>> Avda. de Vigo s/n
>> 15706 Santiago de
Compostela
>> SPAIN
>>
>> Tel: 34-981569810 ext 249
>> lmcarril at cesga.es
>> www.cesga.es
>>
>>
>> ==================================================================
>>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>
>Are you guys running a versioned release or the HEAD on git. I am more
>than fairly certain this has been optimized in the repo, iirc a few
>months ago.
>
>-- 
>Matthew W. Summers
>Gentoo Foundation
Inc.
>
>_______________________________________________
>StarCluster mailing list
>StarCluster at mit.edu
>http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
>_______________________________________________
>StarCluster mailing list
>StarCluster at mit.edu
>http://mailman.mit.edu/mailman/listinfo/starcluster
>

_______________________________________________
StarCluster mailing list
StarCluster at mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster

_______________________________________________
StarCluster mailing list
StarCluster at mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster