I tried these new plugins which seemed to install fine but something's still not quite right -- trying to get a Client hangs:<br><br>sgeadmin@master:~$ ipython<br>Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) <br>Type "copyright", "credits" or "license" for more information.<br>
<br>IPython 0.11.dev -- An enhanced Interactive Python.<br>? -> Introduction and overview of IPython's features.<br>%quickref -> Quick reference.<br>help -> Python's own help system.<br>object? -> Details about 'object', use 'object??' for extra details.<br>
<br>In [1]: from IPython.parallel import Client<br><br>In [2]: rc = Client('starcluster.json')<br>---------------------------------------------------------------------------<br>AssertionError Traceback (most recent call last)<br>
/home/sgeadmin/<ipython-input-2-c67e0a34263a> in <module>()<br>----> 1 rc = Client('starcluster.json')<br><br>/usr/local/lib/python2.6/dist-packages/ipython-0.11.dev-py2.6.egg/IPython/parallel/client/client.pyc in __init__(self, url_or_file, profile, profile_dir, ipython_dir, context, debug, exec_key, sshserver, sshkey, password, paramiko, timeout, **extra_args)<br>
313 if self._cd:<br> 314 url_or_file = os.path.join(self._cd.security_dir, url_or_file)<br>--> 315 assert os.path.exists(url_or_file), "Not a valid connection file or url: %r"%url_or_file<br>
316 with open(url_or_file) as f:<br> 317 cfg = json.loads(f.read())<br><br>AssertionError: Not a valid connection file or url: u'/home/sgeadmin/.ipython/profile_default/security/starcluster.json'<br>
<br>---------<br><br>There are only two JSON files on the master node:<br><br>sgeadmin@master:~$ !fi<br>find . -name \*json<br>./.ipython/profile_default/security/ipcontroller-client.json<br>./.ipython/profile_default/security/ipcontroller-engine.json<br>
<br><br><br>-- Jonathan<br><br><div class="gmail_quote">On Wed, Jul 27, 2011 at 12:31 PM, MinRK <span dir="ltr"><<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
And I've been using an updated version of the plugin to work with<br>
IPython 0.11 (parallel code has been completely rewritten, and is much<br>
improved for certain activities). My plugins:<br>
<br>
<a href="https://github.com/minrk/starclusterplugins" target="_blank">https://github.com/minrk/starclusterplugins</a><br>
<br>
-MinRK<br>
<div><div></div><div class="h5"><br>
On Wed, Jul 27, 2011 at 12:14, Justin Riley <<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>> wrote:<br>
> -----BEGIN PGP SIGNED MESSAGE-----<br>
> Hash: SHA1<br>
><br>
> Hi Jonathan,<br>
><br>
> The plugin still needs documenting but here's the basic flow:<br>
><br>
> Put the plugin in the config:<br>
><br>
> [plugin ipcluster]<br>
> setup_class = starcluster.plugins.ipcluster.IPCluster<br>
><br>
> Then put the plugin in a cluster template:<br>
><br>
> [cluster smallcluster]<br>
> ...<br>
> plugins = ipcluster<br>
><br>
> Then start a cluster:<br>
><br>
> $ starcluster start myipcluster<br>
><br>
> If you didn't put the plugin in the default cluster template (specified<br>
> by default_template in the [global] section) you can specify the<br>
> template like so:<br>
><br>
> $ starcluster start -c smallcluster myipcluster<br>
><br>
> This will start a cluster and run the ipcluster plugin which will setup<br>
> an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then<br>
> to use ipcluster:<br>
><br>
> $ starcluster sshmaster -u sgeadmin myipcluster<br>
> sgeadmin@master~ ipython<br>
> [~]> from IPython.kernel.client import MultiEngineClient<br>
> [~]> mec = MultiEngineClient()<br>
> [~]> mec.get_ids()<br>
> [0, 1]<br>
><br>
> In the above IPython session we see that we have two nodes in our<br>
> ipcluster. In general, you will have as many clients as you have nodes<br>
> in your StarCluster.<br>
><br>
> After you have a MultiEngineClient instance you can then do things like:<br>
><br>
> [~]> mec.map(lambda x: x**10, range(32)) # happens in parallel<br>
> [~]> mec.execute('d=10; print d') # executes on all nodes in parallel<br>
><br>
> For more details see:<br>
><br>
> <a href="http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism" target="_blank">http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism</a><br>
><br>
> HTH,<br>
><br>
> ~Justin<br>
><br>
> On 07/26/2011 11:56 PM, Jonathan Taylor wrote:<br>
>> Hi,<br>
>><br>
>> I was wondering if someone had any examples / documentations on how<br>
>> the IPCluster plugin works in StarCluster.<br>
>><br>
>> Thanks,<br>
>><br>
>> Jonathan<br>
>><br>
>> -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137<br>
>> 390 Serra Mall Stanford, CA 94305 Tel: <a href="tel:650.723.9230" value="+16507239230">650.723.9230</a> Fax:<br>
>> <a href="tel:650.725.8977" value="+16507258977">650.725.8977</a> Web: <a href="http://www-stat.stanford.edu/%7Ejtaylo" target="_blank">http://www-stat.stanford.edu/~jtaylo</a><br>
><br>
> -----BEGIN PGP SIGNATURE-----<br>
> Version: GnuPG v2.0.17 (GNU/Linux)<br>
> Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
><br>
> iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ<br>
> yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy<br>
> =jnrj<br>
> -----END PGP SIGNATURE-----<br>
</div></div>> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
><br>
</blockquote></div><br><br clear="all"><br>-- <br>Jonathan Taylor <br>Dept. of Statistics <br>Sequoia Hall, 137 <br>390 Serra Mall<br>Stanford, CA 94305<br>
Tel: 650.723.9230<br>Fax: 650.725.8977<br>Web: <a href="http://www-stat.stanford.edu/~jtaylo">http://www-stat.stanford.edu/~jtaylo</a><br>