[Starcluster] Eucalyptus support

Wed Mar 24 17:46:10 EDT 2010

Hi Nicholas,

Excellent, these commands work for me as well with Eucalyptus. I know what's 
causing the AWS credentials issue and I'm working on it. I'll post again when 
it's ready for testing in github. 

I'm also going to try and bundle the starcluster iso into EMI on Eucalyptus to 
see if I can reproduce your ssh issues.

~Justin

On Wednesday 24 March 2010 2:49:43 pm Nicholas Ampazis wrote:
> Dear Justin,
> 
> We seem to be getting there. The following 4 starcluster commands worked
>  for me:
> 
> 1) starcluster listimages
> 
> 
> 2) starcluster listinstances
> 
> 
> 3) starcluster listvolumes
> 
> 
> 4) starcluster showconsole XXXXXX
> 
> WORKS!
> 
> 
> Unfortunately the "start" command failed with a "AWS credentials"
> error,  and I'm also still facing the problem of not being able to ssh
> in starcluster emi instances that I start manually.
> 
> 
> Best regards,
> 
> 
> Nicholas
> 
> On Wed, Mar 24, 2010 at 10:18 AM, Justin Riley <jtriley at mit.edu> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hi Nicholas,
> >
> > I've added support for eucalyptus to starcluster. Could you pull the
> > latest development code and give it a shot? In the [aws info] section
> > you will need to add the following settings:
> >
> > AWS_PORT=8773
> > AWS_IS_SECURE=False
> > AWS_REGION_NAME=eucalyptus
> > AWS_REGION_HOST=#eucalyptus host ip here
> > AWS_EC2_PATH=/services/Eucalyptus
> > AWS_S3_PATH=/services/Walrus
> >
> > I was able to use listimages, listvolumes, showimage, etc actions from
> > starcluster successfully. I have not tried to launch a cluster or mount
> > EBS volumes.
> >
> > If I were you I'd leave out the EBS stuff until a bare-bones cluster is
> > able to start without issues on your Eucalyptus cluster.
> >
> > BTW, did you have any luck ssh'ing into your StarCluster EMI on your
> > local eucalyptus cluster?
> >
> > ~Justin
> >
> > On 03/24/2010 04:43 AM, Nicholas Ampazis wrote:
> >> Dear Justin,
> >>
> >> Excellent news! I'm not sure though why you could not connect to our
> >> server...
> >>
> >> Yesterday I also downloaded the source code of euca2ools from here:
> >>
> >> http://open.eucalyptus.com/downloads
> >>
> >> and I compiled it within my own working environment where I've also
> >> compiled starcluster. All euca-* commands seem to be working allright
> >> and I can connect to my server. Unfortunately however I still have
> >> trouble connecting using the starcluster code, even though I've
> >> included all the necessary modifications which I've copied from the
> >> "__init__.py" file of the euca2ools source code (attached).
> >>
> >> In any case you seem to be making good progress with this and I really
> >> can't wait to hear for the github announcement.
> >>
> >>
> >> Many thanks for the support,
> >>
> >>
> >> Nicholas
> >>
> >> On Wed, Mar 24, 2010 at 6:45 AM, Justin Riley <justin.t.riley at gmail.com> 
wrote:
> >>> Hi Nicolas,
> >>>
> >>> I've cc'd the list on this one, hope you don't mind.
> >>>
> >>> Before modifying starcluster to connect to Eucalyptus I decided to try
> >>> it out manually using ipython shell and boto. I wasn't able to get it
> >>> to work using your config settings for some reason, however, I set up
> >>> my own single-node eucalyptus cluster today locally and I'm happy to
> >>> report that boto seems to work well with my local eucalyptus cluster.
> >>>
> >>> Keep in mind that I haven't done any rigorous testing outside of just
> >>> playing around with a few of the boto API calls and there very well may
> >>> be other issues lurking. Initial tests seem to indicate that it's
> >>> possible....we'll see. I'm adding in configuration support now and will
> >>> make the necessary modifications to awsutils.py in the github code so
> >>> that we can test this out further.
> >>>
> >>> Stay tuned,
> >>>
> >>> ~Justin
> >>>
> >>> On Sunday 21 March 2010 6:01:22 pm you wrote:
> >>>> Dear Justin,
> >>>>
> >>>> Many thanks for the Starcluster ISOs that you provided recently.
> >>>>
> >>>> I've successfully managed to bundle and upload the x86_64 iso to a UEC
> >>>> (eucalyptus) ami. I've also been successful in getting the instance
> >>>> running but I'm facing two problems:
> >>>>
> >>>> 1) I cannot ssh into the instance using e.g.
> >>>>
> >>>>  ssh -i ~/.ssh/blah.priv sgeadmin at XXX.XXX.XXX.XXX
> >>>>
> >>>> I've also tried "ubuntu" and "root" as usernames but still no luck.
> >>>> Please note that I can login to other ubuntu-based amis started with
> >>>> this keypair with no problem, so there's nothing wrong with my
> >>>> keypairs or with group permissions and such (e.g. ec2-authorize
> >>>> default -P tcp -p 22 -s 0.0.0.0/0, etc)
> >>>>
> >>>>
> >>>> 2) I installed the latest starcluster code using git and then I tried
> >>>> to modify the python code in order to make it work with Eucalyptus.
> >>>> I've discovered some extra parameters that I can pass to
> >>>> boto.connect_ec2 and  boto.connect_s3 in order to make them connect to
> >>>> my UEC front-end server rather than Amazon, and the code compiles
> >>>> allright. I've also set up a config file with my UEC credentials.
> >>>>
> >>>> However, when I run e.g. "starcluster start smallcluster test" I'm
> >>>> getting the following error:
> >>>>
> >>>> Traceback (most recent call last):
> >>>>   File "/usr/local/bin/starcluster", line 5, in <module>
> >>>>     pkg_resources.run_script('StarCluster==0.9999', 'starcluster')
> >>>>   File
> >>>>  "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/
> >>>>pytho n/pkg_resources.py", line 442, in run_script
> >>>>     self.require(requires)[0].run_script(script_name, ns)
> >>>>   File
> >>>>  "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/
> >>>>pytho n/pkg_resources.py", line 1167, in run_script
> >>>>     exec script_code in namespace, namespace
> >>>>   File
> >>>>  "/Library/Python/2.6/site-packages/StarCluster-0.9999-py2.6.egg/EGG-I
> >>>>NFO/s cripts/starcluster", line 6, in <module>
> >>>>
> >>>>   File "build/bdist.macosx-10.6-universal/egg/starcluster/cli.py",
> >>>> line 546, in main
> >>>>   File "build/bdist.macosx-10.6-universal/egg/starcluster/cli.py",
> >>>> line 157, in execute
> >>>>   File "build/bdist.macosx-10.6-universal/egg/starcluster/config.py",
> >>>> line 291, in get_cluster
> >>>>   File "build/bdist.macosx-10.6-universal/egg/starcluster/cluster.py",
> >>>> line 148, in __init__
> >>>>   File
> >>>> "build/bdist.macosx-10.6-universal/egg/starcluster/awsutils.py", line
> >>>> 51, in __init__
> >>>> TypeError: __init__() takes exactly 4 arguments (8 given)
> >>>>
> >>>> I understand that this error that I'm getting is due to the extra
> >>>> arguments that I've added but, unfortunately, my python skills are not
> >>>> that good and the _init__() is defined in multiple places. So, for
> >>>> example, I'm not sure whether I have to make a global change in "class
> >>>> EasyAWS(object)" or to each individual class (e.g. class
> >>>> EasyEC2(EasyAWS)). In addition, I'm not certain that more UEC related
> >>>> parameters aren't needed in other parts of the code.
> >>>>
> >>>> To this end, I've decided to ask for your help in case that you can
> >>>> enlighten me further. I'm attaching the file awsutils.py where I've
> >>>> made the changes and the config file that I'm using, so perhaps you
> >>>> can try it out for yourelf in our UEC server.
> >>>>
> >>>>
> >>>> Many thanks,
> >>>>
> >>>>
> >>>> Nicholas
> >>>
> >>> _______________________________________________
> >>> Starcluster mailing list
> >>> Starcluster at mit.edu
> >>> http://mailman.mit.edu/mailman/listinfo/starcluster
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.14 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >
> > iEYEARECAAYFAkuqV4wACgkQ4llAkMfDcrljrwCfUHJcW2PHcBdeKTxcB20eiOxx
> > hwMAn2NY0lW6F9IeUpWYzvWAAVrIESho
> > =GmyF
> > -----END PGP SIGNATURE-----
> 