I figured this is how it would work and tried it but I have no machines in my ipcluster so I was wondering if I had to invoke another command.<br><br>I did see starcluster say it started ipcluster:<br><br>>>> Running plugin ipcluster<br>
>>> Starting ipcluster...<br><br>But, when I ssh to master, I don't seem to have any machines in the MultiEngineClient:<br><br>jtaylo@gkf-ubuntu:~/work/regreg/code/regreg$ starcluster sshmaster --user sgeadmin myipcluster<br>
<br>sgeadmin@master:~$ ipython<br><br>In [1]: from IPython.kernel.client import MultiEngineClient<br>/usr/lib/pymodules/python2.6/IPython/kernel/clientconnector.py:43: DeprecationWarning: Importing class Tub directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Tub instead<br>
self.tub = Tub()<br><br>In [2]: mec = MultiEngineClient()<br>/usr/lib/pymodules/python2.6/IPython/kernel/multienginefc.py:86: DeprecationWarning: Importing class Referenceable1 directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Referenceable instead<br>
class FCSynchronousMultiEngineFromMultiEngine(Referenceable):<br><br>In [3]: mec.get_ids()<br>Out[3]: []<br><br>-- Jonathan<br><br><div class="gmail_quote">On Wed, Jul 27, 2011 at 12:14 PM, Justin Riley <span dir="ltr"><<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi Jonathan,<br>
<br>
The plugin still needs documenting but here's the basic flow:<br>
<br>
Put the plugin in the config:<br>
<br>
[plugin ipcluster]<br>
setup_class = starcluster.plugins.ipcluster.IPCluster<br>
<br>
Then put the plugin in a cluster template:<br>
<br>
[cluster smallcluster]<br>
...<br>
plugins = ipcluster<br>
<br>
Then start a cluster:<br>
<br>
$ starcluster start myipcluster<br>
<br>
If you didn't put the plugin in the default cluster template (specified<br>
by default_template in the [global] section) you can specify the<br>
template like so:<br>
<br>
$ starcluster start -c smallcluster myipcluster<br>
<br>
This will start a cluster and run the ipcluster plugin which will setup<br>
an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then<br>
to use ipcluster:<br>
<br>
$ starcluster sshmaster -u sgeadmin myipcluster<br>
sgeadmin@master~ ipython<br>
[~]> from IPython.kernel.client import MultiEngineClient<br>
[~]> mec = MultiEngineClient()<br>
[~]> mec.get_ids()<br>
[0, 1]<br>
<br>
In the above IPython session we see that we have two nodes in our<br>
ipcluster. In general, you will have as many clients as you have nodes<br>
in your StarCluster.<br>
<br>
After you have a MultiEngineClient instance you can then do things like:<br>
<br>
[~]> mec.map(lambda x: x**10, range(32)) # happens in parallel<br>
[~]> mec.execute('d=10; print d') # executes on all nodes in parallel<br>
<br>
For more details see:<br>
<br>
<a href="http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism" target="_blank">http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism</a><br>
<br>
HTH,<br>
<br>
~Justin<br>
<div><div></div><div class="h5"><br>
On 07/26/2011 11:56 PM, Jonathan Taylor wrote:<br>
> Hi,<br>
><br>
> I was wondering if someone had any examples / documentations on how<br>
> the IPCluster plugin works in StarCluster.<br>
><br>
> Thanks,<br>
><br>
> Jonathan<br>
><br>
> -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137<br>
> 390 Serra Mall Stanford, CA 94305 Tel: <a href="tel:650.723.9230" value="+16507239230">650.723.9230</a> Fax:<br>
> <a href="tel:650.725.8977" value="+16507258977">650.725.8977</a> Web: <a href="http://www-stat.stanford.edu/%7Ejtaylo" target="_blank">http://www-stat.stanford.edu/~jtaylo</a><br>
<br>
</div></div>-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2.0.17 (GNU/Linux)<br>
Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
<br>
iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ<br>
yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy<br>
=jnrj<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br><br clear="all"><br>-- <br>Jonathan Taylor <br>Dept. of Statistics <br>Sequoia Hall, 137 <br>390 Serra Mall<br>Stanford, CA 94305<br>
Tel: 650.723.9230<br>Fax: 650.725.8977<br>Web: <a href="http://www-stat.stanford.edu/~jtaylo">http://www-stat.stanford.edu/~jtaylo</a><br>