<div dir="ltr">Hi all,<div><br></div><div>I just wanted to ping on this same question, in case one of the experts is around...</div><div><br></div><div>Can this be configured at the starcluster level?</div><div><br></div><div>
Thanks!</div><div><br></div><div>f</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <span dir="ltr"><<a href="mailto:jascha@stanford.edu" target="_blank">jascha@stanford.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
Due to limitations on the number of simultaneous active network connections, IPython.parallel is unable to handle more than about 250 engines. (see the discussion at <a href="http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html" target="_blank">http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html</a> )<br>
<br>
On an mx.large instance, with 8 cores, and with an engine started for each core, this means that an ipcluster is limited to around 30 nodes. However, a single ipengine is capable of using all the cores on a node (for instance, if it's doing matrix algebra). I would like to be able to start a cluster with fewer ipengines than cores. That is, I would like the ipcontroller call on each node to have a "-n [constant number of engines]" rather than "-n [number of cores]".<br>
<br>
This is probably easy to change in a config file somewhere, so apologies if this should be obvious, but I don't (yet) know where to look.<br>
<br>
Thank you very much for any help!<br>
<br>
-Jascha<br>
<br>
<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Fernando Perez (@fperez_org; <a href="http://fperez.org" target="_blank">http://fperez.org</a>)<br>fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)<br>
fernando.perez-at-berkeley: contact me here for any direct mail<br>
</div>