[StarCluster] status of HPC instance type

[Fred Rotbart]

Hi Brian,

Since the storage performance of the virtual disk in either an S3 or EBS 
backed AMI is much worst than that of an attached EBS volume, and since 
this ami-30b84d59 was built with HPC in mind, the  image is configured 
with a minimum of storage. The assumption is that working storage will 
be attached.

The recommended way of getting the storage you need is to create an 
additional EBS volume and attach it, for example, as /home. This way you 
can define as much storage as you need and get a performance boost as 
well. The procedure to do so is documented in the StarCluster config file.

The EBS storage volume has the additional benefit, that if you do not 
explicitly configure it to terminate when your AMI does, your data 
storage is preserved and can be used in multiple runs.

- Fred


On 20/10/10 16:29 , Brian O'Connor wrote:
> Hi Justin,
>
> Thanks for the quick reply.  I did a checkout of StarCluster from git
> this evening, installed it following the directions on your site and
> ended up getting the following error:
>
> [boconnor at oslo StarCluster]$ starcluster listclusters
> StarCluster - (http://web.mit.edu/starcluster) (v. 0.9999)
> Software Tools for Academics and Researchers (STAR)
> Please submit bug reports to starcluster at mit.edu
>
> -------------------------------------
> masters (security group: @sc-masters)
> -------------------------------------
> cli.py:152 - ERROR - instance  has no alias
>
> I installed using python2.6 and this setup pulled in boto-2.0b3.
>
> I installed using (python 2.6):
>
> sudo python ez_setup.py
> sudo python setup.py install
>
> Do you know what could be wrong?
>
> I also tried "starcluster start -c dev-node dev-node-brian" and was
> able to launch a node without issues so this must be something in
> particular with listclusters.
>
> Finally, I tired launching an EBS backed AMI on an HPC node.  This
> worked but there are two issues.  First, some error messages:
>
>>>> Validating cluster template settings...
>>>> Cluster template settings are valid
>>>> Starting cluster...
>>>> Launching a 1-node cluster...
>>>> Launching master (ami: ami-30b84d59, type: cc1.4xlarge)
>>>> Creating security group @sc-dev-node-brian...
> EC2ResponseError: 400 Bad Request
> <?xml version="1.0" encoding="UTF-8"?>
> <Response><Errors><Error><Code>InvalidPlacementGroup.Unknown</Code><Message>The
> placement group '@sc-dev-node-brian' is
> unknown.</Message></Error></Errors><RequestID>00869090-d93a-4013-867f-6d5acc592420</RequestID></Response>
>>>> Creating placement group @sc-dev-node-brian...
> Reservation:r-5d31f237
>>>> Waiting for cluster to start...
>
> This seemed to be harmless since it eventually started.
>
> Second, and this is less a bug than a request, when I launch the
> ami-30b84d59 image I don't get the 1.7T of instance storage
> pre-configured as I do with the non-EBS starcluster images.  Here's
> the output from fdisk:
>
> Disk /dev/xvdb: 901.8 GB, 901874450432 bytes
> 255 heads, 63 sectors/track, 109646 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
> Disk /dev/xvdb doesn't contain a valid partition table
>
> Disk /dev/xvdc: 901.8 GB, 901874450432 bytes
> 255 heads, 63 sectors/track, 109646 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
> Disk /dev/xvdc doesn't contain a valid partition table
>
> I can write a plugin to partition and format this device but do you
> know if there's any way to get this to be either setup by Starcluster
> or by some other means?  Do you know if there's a way to avoid having
> to reformat the device every time I launch the nodes?  I found this
> posting helpful:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=186874
>
> Thanks for your help
>
> --Brian O'Connor
>
> On Mon, Oct 18, 2010 at 3:04 PM, Justin Riley<justin.t.riley at gmail.com>  wrote:
>> On 10/18/2010 03:02 PM, Justin Riley wrote:
>>> The github code now supports starting/stopping EBS clusters. The stop
>>> command will stop any EBS-backed instances rather than terminating them.
>>
>> This of course means that you can resume the EBS cluster later on using
>> the 'start' command without losing any data on the disks.
>>
>> ~Justin
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>