[StarCluster] limiting the number of ipengines started on each node?

[Niklas Krumm]

Hi Jascha and Fernando,

I am not an expert. However I did take a look... and while there doesn't seem to be an option built in for this,  thanks to your detailed instructions it was easy to parameterize this as an option for the plugin (I named it MAX_ENGINES_PER_NODE). You can see the changes needed here:

https://github.com/nkrumm/StarCluster/commit/9670dde20b0ed8fbd8a62ed97650892faea01703

and then just add the variable to the [plugin ipcluster] section of your config file, eg:
MAX_ENGINES_PER_NODE=1

Full disclosure...I have not yet tested these changes! good luck :)
~Nik

On Oct 31, 2013, at 6:03 PM, Fernando Perez wrote:

> Hi all,
> 
> I just wanted to ping on this same question, in case one of the experts is around...
> 
> Can this be configured at the starcluster level?
> 
> Thanks!
> 
> f
> 
> 
> On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <jascha at stanford.edu> wrote:
> Hello,
> 
> Due to limitations on the number of simultaneous active network connections, IPython.parallel is unable to handle more than about 250 engines.  (see the discussion at http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html )
> 
> On an mx.large instance, with 8 cores, and with an engine started for each core, this means that an ipcluster is limited to around 30 nodes.  However, a single ipengine is capable of using all the cores on a node (for instance, if it's doing matrix algebra).  I would like to be able to start a cluster with fewer ipengines than cores.  That is, I would like the ipcontroller call on each node to have a "-n [constant number of engines]" rather than "-n [number of cores]".
> 
> This is probably easy to change in a config file somewhere, so apologies if this should be obvious, but I don't (yet) know where to look.
> 
> Thank you very much for any help!
> 
> -Jascha
> 
> 
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
> 
> 
> 
> -- 
> Fernando Perez (@fperez_org; http://fperez.org)
> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> fernando.perez-at-berkeley: contact me here for any direct mail
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20131101/a5c822be/attachment.htm