[StarCluster] loadbalancer race condition with plugin.on_add_node()

[Justin Riley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Jeff,

My apologies for the delay. I agree there should be a way to execute a
plugin before a node is added to SGE, however, I think there are also
valid use cases for wanting to be able to run after SGE has been
configured. For instance a plugin may need to submit new jobs/cleanup
old jobs or inspect the current state of the queue. Fortunately there's
a solution that meets both needs and does not require adding more plugin
'hooks'.

I've had requests in the past for an option to disable SGE altogether
which I've implemented in the latest github code (see
http://mailman.mit.edu/pipermail/starcluster/2010-December/000578.html).
I'm working now to extract the SGE-related code into a completely
separate plugin that, for now, gets executed by StarCluster by default
unless disable_queue=True in the config. This helps to gradually migrate
SGE to a plugin given that some folks still expect it to be there
out-of-the-box.

With SGE as a plugin you could then completely control the order like so:

[plugin SGE]
setup_class = starcluster.plugins.sge.SGEPlugin

[cluster smallcluster]
...
# this line tells StarCluster not to configure SGE by default
disable_queue = True
# now add your plugins, including SGE, in the order you want executed
plugins = myplugfirst, SGE, myotherplugin

Does this make sense? I'll send an update to the list once the SGE code
has been converted to a plugin on github...

~Justin

On 02/21/2011 02:50 PM, Jeff White wrote:
> I believe there is a race condition when nodes are added via the
> loadbalancer. The configured plugins are run after the new node is added
> to SGE. The issue is our plugin needs to do some setup on the new node
> before jobs get submitted to the new node. It would be better if plugins
> were run before the node is available to run jobs. If there is a use
> case for running a plugin after the node is added to SGE, then more
> hooks should be added, e.g. on_add_node_before_sge(),
> on_add_node_after_sge().
> 
> jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1/4PYACgkQ4llAkMfDcrmzKQCgkuB/0a9//pQ77iKZW25jcQHO
8ogAn2GZc8JGG6ZGeZuPMu4jZd97YOi1
=o3GS
-----END PGP SIGNATURE-----