[StarCluster] limiting the number of ipengines started on each node?

[Fernando Perez]

Hi Niklas,

thanks so much! I'll test it and will report back if there are any issues.

Best,

f


On Fri, Nov 1, 2013 at 12:14 AM, Niklas Krumm <nkrumm at gmail.com> wrote:

> Hi Jascha and Fernando,
>
> I am not an expert. However I did take a look... and while there doesn't
> seem to be an option built in for this,  thanks to your detailed
> instructions it was easy to parameterize this as an option for the plugin
> (I named it MAX_ENGINES_PER_NODE). You can see the changes needed here:
>
>
> https://github.com/nkrumm/StarCluster/commit/9670dde20b0ed8fbd8a62ed97650892faea01703
>
> and then just add the variable to the [plugin ipcluster] section of your
> config file, eg:
> MAX_ENGINES_PER_NODE=1
>
> Full disclosure...I have not yet tested these changes! good luck :)
> ~Nik
>
> On Oct 31, 2013, at 6:03 PM, Fernando Perez wrote:
>
> Hi all,
>
> I just wanted to ping on this same question, in case one of the experts is
> around...
>
> Can this be configured at the starcluster level?
>
> Thanks!
>
> f
>
>
> On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <
> jascha at stanford.edu> wrote:
>
>> Hello,
>>
>> Due to limitations on the number of simultaneous active network
>> connections, IPython.parallel is unable to handle more than about 250
>> engines.  (see the discussion at
>> http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html)
>>
>> On an mx.large instance, with 8 cores, and with an engine started for
>> each core, this means that an ipcluster is limited to around 30 nodes.
>>  However, a single ipengine is capable of using all the cores on a node
>> (for instance, if it's doing matrix algebra).  I would like to be able to
>> start a cluster with fewer ipengines than cores.  That is, I would like the
>> ipcontroller call on each node to have a "-n [constant number of engines]"
>> rather than "-n [number of cores]".
>>
>> This is probably easy to change in a config file somewhere, so apologies
>> if this should be obvious, but I don't (yet) know where to look.
>>
>> Thank you very much for any help!
>>
>> -Jascha
>>
>>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>
>
>
> --
> Fernando Perez (@fperez_org; http://fperez.org)
> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> fernando.perez-at-berkeley: contact me here for any direct mail
>  _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
>


-- 
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20131101/614cc540/attachment-0001.htm