[StarCluster] Adding Python Modules

Mon Mar 14 23:55:09 EDT 2011

Hi Justin,

This definitely helped!

Your email reminded me I was experimenting with getting the X remote display
working around the time I tried to install a Python module.
So I had ssh'd in as opposed to using the starcluster command.

I found the point of confusion that led me down the wrong track using ssh
directly. In my config file, I defined a cluster called 'singleinstance'.
So I execute 'starcluster start singleinstance' to fire it up. At the end of
the startup process, I see the following output.

"The cluster has been started and configured.

Login to the master node as root by running:

    $ starcluster sshmaster singleinstance

or manually as sgeadmin:

    $ ssh -i <rsa file> sgeadmin@<machine name>

When you are finished using the cluster, run:

    $ starcluster stop singleinstance

to shutdown the cluster and stop paying for service"

Notice the instructions for making a manual connection using ssh. It says to
connect as sgeadmin, not as root.
When you connect using 'starcluster sshmaster singleinstance', you are
connected as root.

Once I substitute root for sgeadmin, all is well.

Should the starcluster commands get remote X display working properly? Even
with ssh -X from my Mac, I had no luck.
It took ssh -X -C -c blowfish -i <rsa file> root@<machine name> for me to
display a matplotlib figure remotely.
I was following the suggestion in this post.
http://blog.grayproductions.net/articles/load_an_ec2_gui_on_your_mac_os_x_box

Thanks for your help Justin.

Chris

On Mon, Mar 14, 2011 at 8:09 PM, Justin Riley <jtriley at mit.edu> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Chris,
>
> > I should have mentioned that I tried 'sudo easy_install ...' and it
> > prompted me for a password. Hitting return didn't do the trick...
> >
>
> How are you logging into the instance? Have you tried the
> sshmaster/sshnode/sshinstance commands? All of these commands should log
> you in as root by default:
>
> # login as root on the master node of 'mycluster'
> $ starcluster sshmaster mycluster
>
> # login as root to node001 of 'mycluster'
> $ starcluster sshnode mycluster node001
>
> # login as root to any instance by instance id (keypair must be defined
> in config)
> $ starcluster sshinstance i-14e9157c
>
> # login as root to any instance by public dns name (keypair must be
> defined in config)
> $ starcluster sshinstance ec2-<http://ec2-123-123-123-12.compute-1.amazonaws.com>
> 123-123-123-12.compute-1.amazonaws.com
>
> If these commands work you should be logged in as root and easy_install
> should work without any 'permission denied' errors. Pass --help to any
> of these commands for more details and a list of available options.
>
>
> > Once I am able to create a new custom AMI, shouldn't I be able to spin
> > up a cluster of VMs all using the same image?
>
> Absolutely. You can do this by putting your new AMI id as the
> NODE_IMAGE_ID setting in a cluster template defined in the config. Have
> a look at the comments for the NODE_IMAGE_ID setting in the example
> 'smallcluster' config here:
>
> http://web.mit.edu/stardev/cluster/docs/configuration.html#defining-cluster-templates
>
> However, please note that currently the best way to create a new AMI
> based on the StarCluster AMI is to launch a single instance of the AMI
> using the AWS Console instead of StarCluster. Then login to the instance
> as root, modify the install, and create a new AMI from the instance. If
> you want to create an S3-backed AMI you can use StarCluster's
> 'createimage' command to completely automate the process:
>
> $ starcluster createimage i-999999 my-new-image my-s3-bucket
>
> Please see http://web.mit.edu/stardev/cluster/docs/create_new_ami.html
> for more details.
>
> Hope that helps,
>
> ~Justin
>
>
>
> >
> > Chris
> >
> > On Mon, Mar 14, 2011 at 6:57 PM, Austin Godber <godber at uberhip.com
> > <mailto:godber at uberhip.com>> wrote:
> >
> >     You should be able to use sudo to do things as the root user:
> >
> >     sudo easy_install boto
> >
> >     I am pretty sure that works.
> >
> >     Though you should keep in mind that will only run it on the current
> >     node and /usr/local isn't share between nodes ... at least as far as
> >     I know of.
> >
> >     Austin
> >
> >
> >     On 03/14/2011 08:55 PM, Chris Diehl wrote:
> >>     Hello,
> >>
> >>     I just successfully fired up my first cluster in short order which
> >>     is brilliant!
> >>
> >>     My first step was to try and install additional modules so that I
> >>     could save a custom AMI for future use.
> >>
> >>     Doing an 'easy_install <packagename>' yielded:
> >>
> >>     -------------------------
> >>
> >>     error: can't create or remove files in install directory
> >>
> >>     The following error occurred while trying to add or remove files
> >>     in the
> >>     installation directory:
> >>
> >>         [Errno 13] Permission denied:
> >>
> '/usr/local/lib/python2.6/dist-packages/test-easy-install-5265.write-test'
> >>
> >>     The installation directory you specified (via --install-dir,
> >>     --prefix, or
> >>     the distutils default setting) was:
> >>
> >>         /usr/local/lib/python2.6/dist-packages/
> >>
> >>     Perhaps your account does not have write access to this directory?
> >>      If the
> >>     installation directory is a system-owned directory, you may need
> >>     to sign in
> >>     as the administrator or "root" account.  If you do not have
> >>     administrative
> >>     access to this machine, you may wish to choose a different
> >>     installation
> >>     directory, preferably one that is listed in your PYTHONPATH
> >>     environment
> >>     variable.
> >>
> >>     -------------------------
> >>
> >>     Does sgeadmin not have admin rights? Should I be able to install
> >>     software at will without permissions issues?
> >>
> >>     Chris
> >>
> >>
> >>
> >>     _______________________________________________
> >>     StarCluster mailing list
> >>     StarCluster at mit.edu <mailto:StarCluster at mit.edu>
> >>     http://mailman.mit.edu/mailman/listinfo/starcluster
> >
> >
> >     _______________________________________________
> >     StarCluster mailing list
> >     StarCluster at mit.edu <mailto:StarCluster at mit.edu>
> >     http://mailman.mit.edu/mailman/listinfo/starcluster
> >
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk1+2F0ACgkQ4llAkMfDcrmIUQCgjwAYW/u11uFbR3oBSHiAHxLx
> 8/8AoIjNpa27nPra3RvI7YWsxTQhVXwA
> =jbKi
> -----END PGP SIGNATURE-----
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20110314/386f2f31/attachment.htm