[StarCluster] StarCluster Plugins

Fri Mar 28 14:37:02 EDT 2014

This brings up something I've been thinking about. How hard would it be to rewrite the loadbalancer to work in the context of Amazon's AWS Autoscaling instead of as its own standalone piece? I am not familiar with autoscaling yet, but seems like there is a bit of an overlap in functionality. Does that sound like a good direction?

Also, it would be great to abstract the loadbalancer's queue sensor and make SGE a plugin. Other implementations can include Celery or perhaps ipython. Just brainstorming for now.

________________________________
From: Cory Dolphin <wcdolphin at gmail.com>
Sent: Friday, March 28, 2014 10:56 AM
To: Sergio Mafra
Cc: Dmitry Serenbrennikov; starcluster at mit.edu
Subject: Re: [StarCluster] StarCluster Plugins

Is there a central repository of user-contributed plugins anywhere? Regardless, I have both a contribution and a question:

1. Has anyone experimented with using SGE profile in an IPython cluster? I wish to enable starcluster's load balancing, but since it depends on SGE's queue length, I do not believe it will behave as expected.
2. I had some issues with remote engines dying during a particular job, so I edited the ipcluster plugin to use supervisord to watch the processes, if anyone else finds it useful, it is on Github with the creative name   StarClusterReliableIPCluster<https://github.com/wcdolphin/StarClusterReliableIPCluster>

Cheers,
Cory

On Mon, Mar 17, 2014 at 3:29 PM, Sergio Mafra <sergiohmafra at gmail.com<mailto:sergiohmafra at gmail.com>> wrote:
Good move Dmitry..

If someone has more plugins who wants to share.. be welcomed here.

All best,

Sergio

On Mon, Mar 17, 2014 at 3:45 PM, Dmitry Serenbrennikov <dmitry at adchemy.com<mailto:dmitry at adchemy.com>> wrote:

Same here.

Here's another plugin that I have written. It integrates Chef such that master nodes are configured as chef clients/nodes.

https://gist.github.com/nitecoder/9605618

[https://1.gravatar.com/avatar/4456a0f0eda53e81fcd6ede8f2bf6dec?d=https%3A%2F%2Fidenticons.github.com%2Fdf64630c486a992b347c268ad6a445b5.png&s=140]<https://gist.github.com/nitecoder/9605618>

This is a Chef plugin for StarCluster. It's really my first ever StarCluster plugin, plus I'm very new to Chef as well. So please take with a grain of salt. Any feedback is welcome! Especially, a few things are not quite to my liking: chef's authentication system seems to presume that validation.pem file is widely distributed. I'm attempting to limit the need for this file being everywhere (by u
This is a Chef plugin for StarCluster. It's really my first ever StarCluster plugin, plus I'm very new to Chef as well. So please take with a grain of salt. Any feedback is welcome! Especially, a few things are not quite to my liking: chef's authentication system seems to presume that validation.pem file is widely distributed. I'm attempting to limit the need for this file being everywhere (by using chef bootstrap). It works, but doesn't seem like the right approach. Am I just misunderstanding Chef somehow? currently it's not possible to tell if starcluster node is terminated or just stopped. If one deletes chef client on stop, then start doesn't work. I chose to delete the node but leave the client. But this also doesn't seem like the right approach. - Gist is a simple way to share snippets of text and code with others.
Read more...<https://gist.github.com/nitecoder/9605618>

It's kind of clunky but seems to work. In particular, I've attempted to de-provision the node (if not the client) when cluster terminates. But I'm new to Chef and StarCluster both, so this could be totally missing the way Chef is supposed to operate.

Any feedback is welcome!

Thanks!

-Dmitry

________________________________
From: starcluster-bounces at mit.edu<mailto:starcluster-bounces at mit.edu> <starcluster-bounces at mit.edu<mailto:starcluster-bounces at mit.edu>> on behalf of Jacob Barhak <jacob.barhak at gmail.com<mailto:jacob.barhak at gmail.com>>
Sent: Sunday, March 16, 2014 9:21 PM
To: Sergio Mafra
Cc: starcluster at mit.edu<mailto:starcluster at mit.edu>
Subject: Re: [StarCluster] StarCluster Plugins

Thanks Sergio,

You have a second there.

People were very helpful to me in the past when I asked questions and they suggested plugins that solved my problems. However, I still do not understand too much about them to the point I can write one on my own, and I saw others in the list that ask similar questions about plugins.

The real question is if anyone who understands enough is willing to take this effort of a centralized repository?

If someone does choose to invest in this it will help for sure and may grow to be a useful tool set.

           Jacob

On Sat, Mar 15, 2014 at 3:08 PM, Sergio Mafra <sergiohmafra at gmail.com<mailto:sergiohmafra at gmail.com>> wrote:
Hi fellows,

I?ve got no doubt that StarCluster is a fantastic tool. This helps lots of people on creating and managing an array of clusters.
But there is more.. the plugins.. They can help us improve what is really great.
The problem is knowing how to really develop and find them. They are scattered on internet.
I think that if someone (loving soul) creates a catalog of plugins, starting with "How to develop a StarCluster plugin for dummies"... this should be something. Better if Justin and Rayson cloud enhance the StarCluster?s site (http://star.mit.edu/cluster/docs/latest/manual/plugins.html#plugin-system) to be like a micro git.
Tell me what do you think.

Sergio

_______________________________________________
StarCluster mailing list
StarCluster at mit.edu<mailto:StarCluster at mit.edu>
http://mailman.mit.edu/mailman/listinfo/starcluster

_______________________________________________
StarCluster mailing list
StarCluster at mit.edu<mailto:StarCluster at mit.edu>
http://mailman.mit.edu/mailman/listinfo/starcluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20140328/eb960f7f/attachment-0001.htm