[StarCluster] Are heterogeneous clusters possible in StarCluster?

[Jennifer Staab]

I am not sure exactly what you are trying to do regarding a single 
cluster.  But in my experience the setup of the "master" node is 
propagated to the worker nodes.  I would set the master as I wanted and 
then add "workers" that had different attributes ( different AMI's and 
EC2 types, could be spots and/or on-demand).  The workers always 
inherited attributes from the master node, specifically NFS volumes from 
the master were always shared amongst the workers ( this included other 
EBS volumes I had mounted using Starcluster config file). When you 
submit the "starcluster addnode" command you just have to make sure you 
specify the attributes(AMI, instance type, spot/on-demand, etc.) you 
want for the added nodes.

In my experience, the added worker nodes have all their resources 
dedicated to the SGE queue(s). You would use "qconf" command to adjust 
how you want things set up regarding each added node and how you want to 
set up your queues. One good thing about this setup is you can have your 
master as an on-demand or reserved instance and your workers as spot 
instances (bid with cheaper hourly rates). This way jobs running on 
spots that are terminated (due to bid pricing) are just resubmitted back 
to the queue as long as you issue the resubmit option in your qsub calls.

One word of caution is that I didn't mixing and match on OS type or 
virtualization type (PV, HVM) within a single cluster. My thought was 
that there might be underlying incompatibilities in the systems such 
that the propagation of attributes from master to worker nodes might not 
work seamlessly.

Good Luck.

-Jennifer


On 12/9/14 11:30 AM, Fran Campillo wrote:
>
>     Hi Jin!
>
>     Thank you so much for your answer :) . Yes, I can try to do that, 
> but then I would need to do manually some stuff in the new nodes that 
> StarCluster usually takes care of (like the password-less ssh and 
> sharing /home and potentially other EBS volumes). Is my assumption right?
>
>     Thanks again!
>
>     Fran.
>
> On 09.12.2014 16:52, Jin Yu wrote:
>> Hi Fran,
>>
>> You can start a master node first and then add different types of 
>> nodes later. You may setup the SGE to define the job allocation 
>> behavior among these nodes.
>>
>> -Jin
>>
>> On Mon, Dec 8, 2014 at 6:19 AM, Fran Campillo <fran at spitch.ch 
>> <mailto:fran at spitch.ch>> wrote:
>>
>>
>>          Hi,
>>
>>          I began using StarCluster a couple of weeks ago for my research,
>>     and I find it really useful framework. I had to setup SGE myself
>>     several
>>     times in the past, and StarCluster makes our life way easier.
>>
>>          I still don't know many of the features of StarCluster, and
>>     I would
>>     like to ask the community whether certain things I want to do are
>>     actually possible with the current version from StarCluster. In
>>     particular, I would like to create a heterogeneous cluster on
>>     StarCluster (this is, with different kinds of instances, that I could
>>     have in different SGE queues. In the problem I have to solve
>>     there are
>>     some stages that need GPU and others that do not, and I would
>>     like to be
>>     able to setup the complete cluster at the beginning of the
>>     process and
>>     work like this:
>>
>>     ------------
>>
>>          1.- Init: setup of the heterogeneous cluster {gpu_clust,
>>     no_gpu_clust).
>>
>>          2.- No cuda tasks:
>>          2.1.- stop gpu_clust instances.
>>          2.2.- run tasks in no_gpu_clust.
>>
>>          3.- Cuda tasks:
>>          3.1.- stop no_gpu_clust.
>>          3.2.- start gpu_clust.
>>          3.3.- run tasks in gpu_clust.
>>
>>          4.- No cuda tasks:
>>          ...
>>
>>     ------------
>>
>>          Is this currently possible with StarCluster? I guess that I can
>>     already fake this behavior creating both clusters from another Amazon
>>     instance and run the tasks with one or another via ssh, but I
>>     find it a
>>     worse solution.
>>
>>          Thank you very much in advance!
>>
>>          Cheers!
>>
>>          Fran.
>>
>>     _______________________________________________
>>     StarCluster mailing list
>>     StarCluster at mit.edu <mailto:StarCluster at mit.edu>
>>     http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>
>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20141209/7e3ca568/attachment.htm