[Starcluster] Cluster with mixed NODE_INSTANCE_TYPE's

[Dan Yamins]

>
> We do have support for even more fine grained control of node instance
> types (thanks to Dan Yamins) if it's needed,


The syntax is pretty simple (correct me where I go wrong, Justin).  It is
basically governed y the following rules:

1)  To specify N instance-types "type1', ... "typeN",  with specific AMIs
and number of nodes for each type, you use a comma-separated list of
elements, with ":"-separation between declarations for instance type, AMI
ID, and number of nodes, e.g.:

    NODE_INSTANCE_TYPE = type1 : ami1 : num1 , type2 : ami2 : num 2, ... ,
typeN : amiN : numN


2) The AMI specification is optional, e.g.

    NODE_INSTANCE_TYPE = type1 : num1 , type2 :  num 2, ... , typeN :  numN

is a valid specification, with the effect that all instances will use the
ami specified by NODE_IMAGE_ID.


3)  The num specification  is optional as well, e.g.

    NODE_INSTANCE_TYPE = type1, type2, ... , typeN

is a valid specification, with the effect of bringing up 1 node each for
type1, type2, ...typeN-1.  If numN is unspecified, it is automatically set
to

     numN = CLUSTER_SIZE - num1 - num2 ... - numN-1


4) The numbers num1, num2, ... numN, must satisfy;

     CLUSTER_SIZE = num1 + num2 + ... + numN

If you specify too many or too few nodes, an error will be thrown.


Examples:

#########   this sets up a 4-node cluster with 1 m1.large, 1 c1.xlarge, and
3 m1.xlarge
CLUSTER_SIZE = 5
NODE_INSTANCE_TYPE = m1.large,c1.xlarge,m1.xlarge
#########

#########   this sets up a 10-node cluster with 4 m1.large, 4 c1.xlarge, and
2 m1.xlarge
CLUSTER_SIZE = 10
NODE_INSTANCE_TYPE = m1.large : 4 , c1.xlarge : 4 ,m1.xlarge
#########


#########   this sets up a 10-node cluster with 4 m1.large using ami
ami-7aca2213,
#########   and 6 c1.xlarge using the AMI specified by NODE_IMAGE ID
CLUSTER_SIZE = 10
NODE_INSTANCE_TYPE = m1.large : ami-7aca2213: 4 , c1.xlarge
#########


#########   this generates an error
CLUSTER_SIZE = 5
NODE_INSTANCE_TYPE = m1.large : 4 , c1.xlarge : 4
#########

If yo have any questions, feel free to ask

Dan






> however, I haven't yet documented this functionality. Let me know if the
> above doesn't meet your needs and I'll fill you in.
>
> ~Justin
> ________________________________________
> From: starcluster-bounces at MIT.EDU [starcluster-bounces at MIT.EDU] On Behalf
> Of Hesselberth, Jay [JAY.HESSELBERTH at ucdenver.edu]
> Sent: Monday, August 09, 2010 3:30 PM
> To: starcluster at mit.edu
> Subject: [Starcluster] Cluster with mixed NODE_INSTANCE_TYPE's
>
> I have an application where I'm running 1000's of jobs with small memory
> requirements (e.g. ~500 Mb / job), but there is a periodic job that needs
> substantially more RAM (~8 Gb).  I've been running clusters with c1.xlarge
> instances to maximize throughput of the small jobs, but these don't have the
> necessary RAM.
>
> I was thinking it would be nice to specify clusters with mixed
> NODE_INSTANCE_TYPES; e.g. configuring a cluster with 4 c1.xlarge instances
> for throughput and 1 m2.xlarge instance to handle large RAM jobs.  I don't
> see how this is currently possible (I'm using the experimental starcluster
> branch).   Is a feature that is planned for the future?
>
> Thanks,
> Jay Hesselberth
> _______________________________________________
> Starcluster mailing list
> Starcluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
> _______________________________________________
> Starcluster mailing list
> Starcluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20100809/732a1562/attachment.htm