[StarCluster] limiting the number of ipengines started on each node?

[Fernando Perez]

ps - did you submit this for upstreaming into the official repo?


On Fri, Nov 1, 2013 at 9:30 AM, Fernando Perez <fperez.net at gmail.com> wrote:

> Hi Niklas,
>
> thanks so much! I'll test it and will report back if there are any issues.
>
> Best,
>
> f
>
>
> On Fri, Nov 1, 2013 at 12:14 AM, Niklas Krumm <nkrumm at gmail.com> wrote:
>
>> Hi Jascha and Fernando,
>>
>> I am not an expert. However I did take a look... and while there doesn't
>> seem to be an option built in for this,  thanks to your detailed
>> instructions it was easy to parameterize this as an option for the plugin
>> (I named it MAX_ENGINES_PER_NODE). You can see the changes needed here:
>>
>>
>> https://github.com/nkrumm/StarCluster/commit/9670dde20b0ed8fbd8a62ed97650892faea01703
>>
>> and then just add the variable to the [plugin ipcluster] section of your
>> config file, eg:
>> MAX_ENGINES_PER_NODE=1
>>
>> Full disclosure...I have not yet tested these changes! good luck :)
>> ~Nik
>>
>> On Oct 31, 2013, at 6:03 PM, Fernando Perez wrote:
>>
>> Hi all,
>>
>> I just wanted to ping on this same question, in case one of the experts
>> is around...
>>
>> Can this be configured at the starcluster level?
>>
>> Thanks!
>>
>> f
>>
>>
>> On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <
>> jascha at stanford.edu> wrote:
>>
>>> Hello,
>>>
>>> Due to limitations on the number of simultaneous active network
>>> connections, IPython.parallel is unable to handle more than about 250
>>> engines.  (see the discussion at
>>> http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html)
>>>
>>> On an mx.large instance, with 8 cores, and with an engine started for
>>> each core, this means that an ipcluster is limited to around 30 nodes.
>>>  However, a single ipengine is capable of using all the cores on a node
>>> (for instance, if it's doing matrix algebra).  I would like to be able to
>>> start a cluster with fewer ipengines than cores.  That is, I would like the
>>> ipcontroller call on each node to have a "-n [constant number of engines]"
>>> rather than "-n [number of cores]".
>>>
>>> This is probably easy to change in a config file somewhere, so apologies
>>> if this should be obvious, but I don't (yet) know where to look.
>>>
>>> Thank you very much for any help!
>>>
>>> -Jascha
>>>
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>
>>
>>
>> --
>> Fernando Perez (@fperez_org; http://fperez.org)
>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
>> fernando.perez-at-berkeley: contact me here for any direct mail
>>  _______________________________________________
>> StarCluster mailing list
>> StarCluster at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>>
>
>
> --
> Fernando Perez (@fperez_org; http://fperez.org)
> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> fernando.perez-at-berkeley: contact me here for any direct mail
>



-- 
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20131101/72e2f461/attachment.htm