<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Jascha and Fernando,<div><br></div><div>I am not an expert. However I did take a look... and while there doesn't seem to be an option built in for this, thanks to your detailed instructions it was easy to parameterize this as an option for the plugin (I named it MAX_ENGINES_PER_NODE). You can see the changes needed here:</div><div><br></div><div><a href="https://github.com/nkrumm/StarCluster/commit/9670dde20b0ed8fbd8a62ed97650892faea01703">https://github.com/nkrumm/StarCluster/commit/9670dde20b0ed8fbd8a62ed97650892faea01703</a></div><div><br></div><div>and then just add the variable to the [plugin ipcluster] section of your config file, eg:</div><div>MAX_ENGINES_PER_NODE=1</div><div><br></div><div>Full disclosure...I have not yet tested these changes! good luck :)</div><div>~Nik</div><div><br></div><div><div><div>On Oct 31, 2013, at 6:03 PM, Fernando Perez wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hi all,<div><br></div><div>I just wanted to ping on this same question, in case one of the experts is around...</div><div><br></div><div>Can this be configured at the starcluster level?</div><div><br></div><div>
Thanks!</div><div><br></div><div>f</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <span dir="ltr"><<a href="mailto:jascha@stanford.edu" target="_blank">jascha@stanford.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
Due to limitations on the number of simultaneous active network connections, IPython.parallel is unable to handle more than about 250 engines. (see the discussion at <a href="http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html" target="_blank">http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html</a> )<br>
<br>
On an mx.large instance, with 8 cores, and with an engine started for each core, this means that an ipcluster is limited to around 30 nodes. However, a single ipengine is capable of using all the cores on a node (for instance, if it's doing matrix algebra). I would like to be able to start a cluster with fewer ipengines than cores. That is, I would like the ipcontroller call on each node to have a "-n [constant number of engines]" rather than "-n [number of cores]".<br>
<br>
This is probably easy to change in a config file somewhere, so apologies if this should be obvious, but I don't (yet) know where to look.<br>
<br>
Thank you very much for any help!<br>
<br>
-Jascha<br>
<br>
<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Fernando Perez (@fperez_org; <a href="http://fperez.org/" target="_blank">http://fperez.org</a>)<br>fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)<br>
fernando.perez-at-berkeley: contact me here for any direct mail<br>
</div>
_______________________________________________<br>StarCluster mailing list<br><a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>http://mailman.mit.edu/mailman/listinfo/starcluster<br></blockquote></div><br></div></body></html>