[Starcluster] How to update numpy / rebundle Starcluster

[Justin Riley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Dan,

Let me start by sincerely apologizing for dropping the ball on your last
3 emails. To be honest, our new spam filter has been trashing all of my
list messages and I didn't notice until I just happened to check the
list archives out of curiousity. I've made sure this won't happen again
in the future.

Regarding your emails:

1. I'm glad to hear you figured out how to recompile numpy to build
against lapack/atlas on StarCluster.

I'm currently working on the next beta release of StarCluster that
should include multiple cluster support. With that in mind, I'll be
updating the StarCluster base AMIs before then with the latest versions
of ubuntu/sge/mpi/numpy/scipy/ipython etc and will likely ping you for
help in testing things like atlas/lapack support in numpy/scipy, etc.

I hope you don't mind :D

Also, I plan to add couchdb (with python bindings) and hadoop to the
list of base software.

2. Regarding rebundling the StarCluster AMI with your own software,
there is a script, although not documented...I need to do this, in the
source package under the 'utils' folder called create_image.py that will
take care of the whole rebundling process. Running:

python create_image.py --help

Should give you enough info about the options to get going with it.

I will likely build this in as an action in the command line interface
in the next version

3. Multiple cluster support is my first order of business at the moment.
 I had to switch focus to another project for the last month or two and
am now ready to come back to StarCluster and make it even better. With
that being said, the next version will also have some major differences
 briefly outlined below (potentially more to come):

StarCluster will switch to an action-based command line interface
(similar to manage.py if you're familiar with django) rather than
option-based:

For example:

starcluster listnodes

instead of:

starcluster -l

Each action will support it's own set of options that should be viewable
via:

starcluster someaction --help

This certainly reads better and is easier to remember for the user and
also allows me to expand my options namespace. (the way i was going, i
would have run out of options very quickly!)

There will also be changes to the configuration layout as well for
specifying multiple cluster configs.

There should also appear an action for initializing a new EBS volume
which will lessen the EBS volume docs on the web site.

If you'd be interested in testing these (and others) new features,
please let me know!

Thanks very much for your interest and enthusiasm (I saw your post:
http://groups.google.com/group/codenode-devel/browse_thread/thread/620f9a40e5e9e488)
and my apologies once again,

~Justin

>
>
>On Fri, Oct 30, 2009 at 7:28:09 PM, Dan Yamins <dyamins at gmail.com>
wrote:
>
>OK, let me add to that question by saying, I just read on your website
>that
>"Currently, StarCluster only supports launching a single cluster on >ec2.
>...  Support for the correct account of multiple clusters should come >in
>future versions."   Is that likely to arise any time soon?   (Also, >would
>mounting multiple clusters from the same EBS storage device be a >problem?
>Is there a write-lock? )  Sorry for all these questions ... I know I
>should
>have read your site more carefully before posting my previous question.
>
>best,
>Dan



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksVgzwACgkQ4llAkMfDcrnXiQCfbBKgfhU4q+UbhX/nFCpIwqiT
nlkAoIVqgvmq/wI0Tc4sfnpyAdOQE/gt
=mRD0
-----END PGP SIGNATURE-----