[Starcluster] Bug VolumeCreator _request_instance

Thu Jul 8 18:32:36 EDT 2010

Hi Justin,
     Yeah, my apologies, I meant createvolume.  I don't know that you 
could make it more clear.  I had just assumed that I had to do it 
myself.  Its documented pretty well.  Though, perhaps in the commands 
"Available Actions" output, if you said
"Create, partition, and format a new EBS volume for use with StarCluster 
(this launches an AMI to do the work)"
or something along those lines it would be more explicit.

I had really just assumed that they had to be made yourself.  I bet most 
users take the meaning correctly.

Austin

On 07/08/2010 06:25 PM, Justin Riley wrote:
> Hi Austin,
>
> Did you mean createvolume? If that's the case, yes it does and sorry 
> this wasn't
> immediately obvious to you. There are some docs here:
>
> http://web.mit.edu/stardev/cluster/docs/volumes.html#using-starcluster-to-create-partition-format-a-new-ebs-volume-recommended 
>
>
> Perhaps I should mention these things on the homepage (ie createvolume,
> createimage, etc) and possibly also the PyPI page. Any other 
> suggestions for
> making users aware of this functionality?
>
> Thanks,
>
> ~Justin
>
> Quoting Austin Godber <godber at uberhip.com>:
>
>> Oh man, I just implemented my own createcluster, I didn't realize
>> starcluster did it already.
>>
>> Austin
>>
>> On 07/08/2010 06:16 PM, Justin Riley wrote:
>>> Hi Joachim,
>>>
>>> Thanks for reporting this. Adding an --instance-type option to the
>>> createvolume
>>> command is on my todo list now and this should fix your issue.
>>>
>>> For the record though, there's no need to use a 64bit image to 
>>> create a new
>>> volume. The reason I chose m1.small as a default is because this 
>>> instance is
>>> *only* used to mount the volume and automate the 
>>> partitioning/formatting. For
>>> this there really isn't any advantage to using a 64bit image and using
>>> m1.small
>>> is your cheapest option.
>>>
>>> I would recommend using a 32bit alestic image (alestic.com), or 
>>> *any* 32bit
>>> image, in the eu-west-1 zone with the createvolume command. This avoids
>>> needing
>>> a different instance type. Regardless, I'll be adding checks for 
>>> invalid
>>> instance_type/image_id combos though in the VolumeCreator class in 
>>> addition to
>>> allowing instance_type to be passed to the createvolume action at 
>>> the command
>>> line.
>>>
>>> Also, I've decided to replicate the StarCluster AMIs in all 
>>> availability zones
>>> so that folks outside the us-east-1 zone do not have to migrate the 
>>> AMIs
>>> themselves. The whole point of StarCluster is to abstract most of 
>>> the details
>>> of using EC2 to create HPC clusters in the cloud. Having users 
>>> migrate the AMI
>>> themselves to other zones is suboptimal. I'll send a notice to the 
>>> list in the
>>> next week or so when the us-west/eu-west/ap-souteast AMIs are available
>>>
>>> Hope that helps and thanks again for reporting.
>>>
>>> ~Justin
>>>
>>> Quoting Joachim Van den Bogaert<joachim at ccl.kuleuven.be>:
>>>
>>>
>>>> Hi Justin,
>>>>
>>>>
>>>>
>>>> I tried to create a volume with the following command:
>>>>
>>>>
>>>>
>>>> joachim at tm1:~$ starcluster createvolume 100 eu-west-1b --image-id
>>>> ami-6be5cf1f
>>>>
>>>>
>>>>
>>>>
>>>>>>> Using keypair sge
>>>>>>>
>>>>
>>>>>>> Requesting host instance in zone eu-west-1b to attach volume to...
>>>>>>>
>>>>
>>>>>>> Creating security group @sc-volumecreator...
>>>>>>>
>>>>
>>>>>>> No instance in group @sc-volumecreator for zone eu-west-1b, 
>>>>>>> launching
>>>>>>>
>>>> one now.
>>>>
>>>>
>>>>
>>>> volume.py:186 - ERROR - exception thrown: EC2ResponseError: 400 Bad 
>>>> Request
>>>>
>>>> <?xml version="1.0"?>
>>>>
>>>> <Response>
>>>>
>>>> <Errors>
>>>>
>>>> <Error>
>>>>
>>>> <Code>InvalidParameterValue</Code>
>>>>
>>>> <Message>The requested instance type's architecture (i386) does not
>>>> match the architecture in the manifest for ami-6be5cf1f 
>>>> (x86_64)</Message>
>>>>
>>>> </Error>
>>>>
>>>> </Errors>
>>>>
>>>> <RequestID>4ff61a79-6a82-480e-b1dd-e26d8965eb70</RequestID>
>>>>
>>>> </Response>
>>>>
>>>>
>>>>
>>>> I was looking for an architecture parameter, but I couldn't find one
>>>>
>>>>
>>>>
>>>> (volume.py line 32 - 56)
>>>>
>>>>
>>>>
>>>> def _request_instance(self, zone):
>>>>
>>>>         for i in self.security_group.instances():
>>>>
>>>>             if i.state in ['pending','running'] and i.placement == 
>>>> zone:
>>>>
>>>>                 log.info("Using existing instance %s in group %s" % \
>>>>
>>>>                          (i.id,self.security_group.name))
>>>>
>>>>                 self._instance = Node(i, self._key_location, 
>>>> 'vol_host')
>>>>
>>>>                 break
>>>>
>>>>         if not self._instance:
>>>>
>>>>             log.info("No instance in group %s for zone %s, 
>>>> launching one
>>>> now." % \
>>>>
>>>>                      (self.security_group.name, zone))
>>>>
>>>>             self._resv = 
>>>> self._ec2.run_instances(image_id=self._image_id,
>>>>
>>>>                 instance_type='m1.small',
>>>>
>>>>                 min_count=1, max_count=1,
>>>>
>>>>                 security_groups=[self.security_group.name],
>>>>
>>>>                 key_name=self._keypair,
>>>>
>>>>                 placement=zone)
>>>>
>>>>             instance = self._resv.instances[0]
>>>>
>>>>             self._instance = Node(instance, self._key_location, 
>>>> 'vol_host')
>>>>
>>>>         s = Spinner()
>>>>
>>>>         log.info("Waiting for instance %s to come up..." %
>>>> self._instance.id)
>>>>
>>>>         s.start()
>>>>
>>>>         while not self._instance.is_up():
>>>>
>>>>             time.sleep(15)
>>>>
>>>>         s.stop()
>>>>
>>>>         return self._instance
>>>>
>>>>
>>>>
>>>> Apparently the code in line 43
>>>>
>>>>
>>>>
>>>> instance_type='m1.small'
>>>>
>>>>
>>>>
>>>> throws an error if you try to create a volume with a 64-bit 
>>>> StarCluster AMI,
>>>> since you can't launch an m1.small with 64 bit images.
>>>>
>>>>
>>>>
>>>> I am in the eu zone, so I can't use the AMI defined in
>>>> image_id=static.BASE_AMI_32.
>>>>
>>>> With the 64-bit image I use instead, an extra parameter could 
>>>> prevent this
>>>> error
>>>>
>>>>
>>>>
>>>> After I changed the line into
>>>>
>>>>
>>>>
>>>> instance_type='m1.large'
>>>>
>>>>
>>>>
>>>> the volume was created successfully.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Joachim
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Starcluster mailing list
>>> Starcluster at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>> _______________________________________________
>> Starcluster mailing list
>> Starcluster at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>
>