[Starcluster] EC2 AMI creation

[Mark J. Pearrow]

Hi Justin and everyone,

Just tried running create_image.py, and created a .starclustercfg that I think should be ok, but I get an error message:

mjp at mcp:~$ ./create_image.py 
/usr/local/lib/python2.6/dist-packages/pycrypto-2.0.1-py2.6-linux-x86_64.egg/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
/usr/local/lib/python2.6/dist-packages/pycrypto-2.0.1-py2.6-linux-x86_64.egg/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
/home/mjp/mh-aws.pem
Traceback (most recent call last):
  File "./create_image.py", line 162, in <module>
    create_image.main()
  File "./create_image.py", line 55, in main
    EC2_PRIVATE_KEY = self.env_variables['EC2_PRIVATE_KEY']
KeyError: 'EC2_PRIVATE_KEY'

In my .starclustercfg file, I've put the following line about my keyfile:

KEYNAME = mh-aws
KEY_LOCATION = /home/mjp/mh-aws.pem

Where mh-aws is the name of the keypair as it appears in my AWS console and /home/mjp/mh-aws.pem is the corresponding RSA private key. I thought the problem might have been the control-M characters in the PEM file, but I nuked those and still get the error. Any pointers? 

Thanks,

mjp




On Dec 8, 2009, at 2:19 PM, Justin Riley wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Mark,
> 
> I forgot to package this with the first release. For now, you can get
> this script here:
> 
> http://github.com/jtriley/StarCluster/tree/master/utils/
> 
> My apologies for the inconvenience.
> 
> As I mentioned in the previous email, this stuff will be built into the
> next version of StarCluster which should be out sometime later this
> month/early next year.
> 
> Concerning ElasticFox, you can run it anywhere, you just need the right
> AWS credentials. I was merely suggesting using it to confirm that your
> new image was created because it is in general a very powerful AWS
> console for Firefox. You could as well run, with the current version of
> StarCluster,
> 
> $ starcluster -i
> 
> Hope that helps!
> 
> ~Justin
> 
> Mark J. Pearrow wrote:
>> Hi Justin,
>> 
>> Thanks for the reply. I had installed StarCluster using easy_install, but didn't find anything called create_image.py. So I downloaded the latest StarCluster from this link:
>> 
>> http://web.mit.edu/stardev/cluster/starcluster-0.90.tar.gz
>> 
>> I then unpacked it - but there is no utils directory. Is there some other package I should download? Also, do I need to run ElasticFox on the same computer where I am running create_image.py? 
>> 
>> mjp
>> 
>> 
>> On Dec 8, 2009, at 11:55 AM, Justin Riley wrote:
>> 
>> Hi Mark,
>> 
>> I've cc'd the StarCluster mailing list on this email, hope you don't mind.
>> 
>> First off, congrats on receiving AWS funding for your research and I'm
>> delighted you're choosing to use StarCluster to jump start your work on
>> EC2 :D
>> 
>> To answer your question concerning how to rebundle, if you download and
>> unpack StarCluster you will see a folder called utils. Inside of that
>> folder is a script called create_image.py.  Running:
>> 
>> python create_image.py --help
>> 
>> Should give you enough options to be self-explanatory. After it
>> completes you will have a new AMI available in ElasticFox (you ARE using
>> ElasticFox right? ;). To get StarCluster to use your new AMI, simply put
>> this new AMI id in your ~/.starclustercfg config file for
>> MASTER_IMAGE_ID/NODE_IMAGE_ID. Let me know if you need any help with
>> this command, I'd be glad to help.
>> 
>> Concerning creating persistent EBS, it's very easy to have it mounted on
>> /home on the entire cluster (via nfs) after the EBS volume has been
>> created. Right now the process for creating a new EBS for StarCluster is
>> a bit cumbersome but well-documented here:
>> 
>> http://web.mit.edu/stardev/cluster/documentation/ebsinit.html
>> 
>> I'm currently working on a new version of StarCluster that will make it
>> considerably easier to create both new AMIs as well as new EBS volumes
>> (which concerns both of your questions above). Stay Tuned!
>> 
>> Concering external LDAP authentication support on an AMI, I've not
>> looked into this but it certainly could be possible. If you find
>> something let me know and I'll look into it and consider adding it in as
>> optional functionality in StarCluster.
>> 
>> BTW, I'm doing a session over MIT's IAP period on using StarCluster if
>> you or anyone you know would be interested in attending. Details are
>> below. We should be in the IAP calendar within a few days or so.
>> 
>> Using StarCluster:  A Toolbox for Managing Amazon Elastic Compute Clouds
>> 
>> DATES:  Tuesday, January 19, 2010
>>       Tuesday, January 26, 2010
>> 
>> TIME:   11:00 a.m.  4:00 p.m.
>> 
>> ROOM:   14-0637 (17 ATHENA Workstations)
>> 
>> Thanks for your interest in StarCluster!
>> 
>> ~Justin
>> 
>> Mark J. Pearrow wrote:
>>>>> Hi Justin,
>>>>> 
>>>>> I was at the StarCluster presentation at the Stata Center back in September, and back then started experimenting with starcluster. I also applied for a research grant, which I just received, so I'm planning to start moving ahead with use. I have a question that is perhaps less starcluster-related: how do I modify the existing AMI and save it? And then how would I make star cluster launch that instance going forward?
>>>>> 
>>>>> I'm planning to create a persistent elastic block store and keep a lot of the neuroimaging software we use there, and hopefully I can mount that volume on all my instances.
>>>>> 
>>>>> Do you happen to know if it's possible to use external authentication methods on an AMI? i.e. can I use my LDAP server that authenticates my local cluster here?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> mjp
>>>>> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iEYEARECAAYFAkseptAACgkQ4llAkMfDcrkm7gCfXdTlEeZ9XlXUrrbb6oxxTiWc
> xyAAoIn03H0PLUSxSVN65C43vXdS89cA
> =8dE/
> -----END PGP SIGNATURE-----