[StarCluster] using ipython as a cluster

[Justin Riley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Jonathan,

The plugin still needs documenting but here's the basic flow:

Put the plugin in the config:

[plugin ipcluster]
setup_class = starcluster.plugins.ipcluster.IPCluster

Then put the plugin in a cluster template:

[cluster smallcluster]
...
plugins = ipcluster

Then start a cluster:

$ starcluster start myipcluster

If you didn't put the plugin in the default cluster template (specified
by default_template in the [global] section) you can specify the
template like so:

$ starcluster start -c smallcluster myipcluster

This will start a cluster and run the ipcluster plugin which will setup
an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then
to use ipcluster:

$ starcluster sshmaster -u sgeadmin myipcluster
sgeadmin at master~ ipython
[~]> from IPython.kernel.client import MultiEngineClient
[~]> mec = MultiEngineClient()
[~]> mec.get_ids()
[0, 1]

In the above IPython session we see that we have two nodes in our
ipcluster. In general, you will have as many clients as you have nodes
in your StarCluster.

After you have a MultiEngineClient instance you can then do things like:

[~]> mec.map(lambda x: x**10, range(32)) # happens in parallel
[~]> mec.execute('d=10; print d') # executes on all nodes in parallel

For more details see:

http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism

HTH,

~Justin

On 07/26/2011 11:56 PM, Jonathan Taylor wrote:
> Hi,
> 
> I was wondering if someone had any examples / documentations on how
> the IPCluster plugin works in StarCluster.
> 
> Thanks,
> 
> Jonathan
> 
> -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137
>  390 Serra Mall Stanford, CA 94305 Tel:   650.723.9230 Fax:
> 650.725.8977 Web: http://www-stat.stanford.edu/~jtaylo

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ
yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy
=jnrj
-----END PGP SIGNATURE-----