[StarCluster] limiting the number of ipengines started on each node?

[Fernando Perez]

Hi all,

I just wanted to ping on this same question, in case one of the experts is
around...

Can this be configured at the starcluster level?

Thanks!

f


On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein
<jascha at stanford.edu>wrote:

> Hello,
>
> Due to limitations on the number of simultaneous active network
> connections, IPython.parallel is unable to handle more than about 250
> engines.  (see the discussion at
> http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html)
>
> On an mx.large instance, with 8 cores, and with an engine started for each
> core, this means that an ipcluster is limited to around 30 nodes.  However,
> a single ipengine is capable of using all the cores on a node (for
> instance, if it's doing matrix algebra).  I would like to be able to start
> a cluster with fewer ipengines than cores.  That is, I would like the
> ipcontroller call on each node to have a "-n [constant number of engines]"
> rather than "-n [number of cores]".
>
> This is probably easy to change in a config file somewhere, so apologies
> if this should be obvious, but I don't (yet) know where to look.
>
> Thank you very much for any help!
>
> -Jascha
>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>



-- 
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20131031/b4c765be/attachment.htm