<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} .ms-cui-menu {background-color:#ffffff;border:1px rgb(171, 171, 171) solid;font-family:'Segoe UI WPC', 'Segoe UI', Tahoma, 'Microsoft Sans Serif', Verdana, sans-serif;font-size:11pt;color:rgb(51, 51, 51);} .ms-cui-menusection-title {display:none;} .ms-cui-ctl {vertical-align:text-top;text-decoration:none;color:rgb(51, 51, 51);} .ms-cui-ctl-on {background-color:rgb(223, 237, 250);opacity: 0.8;} .ms-cui-img-cont-float {display:inline-block;margin-top:2px} .ms-cui-smenu-inner {padding-top:0px;} .ms-owa-paste-option-icon {margin: 2px 4px 0px 4px;vertical-align:sub;padding-bottom: 2px;display:inline-block;} .ms-rtePasteFlyout-option:hover {background-color:rgb(223, 237, 250) !important;opacity:1 !important;} .ms-rtePasteFlyout-option {padding:8px 4px 8px 4px;outline:none;} .ms-cui-menusection {float:left; width:85px;height:24px;overflow:hidden}.wf {speak:none; font-weight:normal; font-variant:normal; text-transform:none; -webkit-font-smoothing:antialiased; vertical-align:middle; display:inline-block;}.wf-family-owa {font-family:'o365Icons'}@font-face { font-family:'o365IconsIE8'; src:url('prem/15.0.898.11/resources/styles/office365icons.ie8.eot?#iefix') format('embedded-opentype'), url('prem/15.0.898.11/resources/styles/office365icons.ie8.woff') format('woff'), url('prem/15.0.898.11/resources/styles/office365icons.ie8.ttf') format('truetype'); font-weight:normal; font-style:normal;}@font-face { font-family:'o365IconsMouse'; src:url('prem/15.0.898.11/resources/styles/office365icons.mouse.eot?#iefix') format('embedded-opentype'), url('prem/15.0.898.11/resources/styles/office365icons.mouse.woff') format('woff'), url('prem/15.0.898.11/resources/styles/office365icons.mouse.ttf') format('truetype'); font-weight:normal; font-style:normal;}.wf-family-owa {font-family:'o365IconsMouse'}.ie8 .wf-family-owa {font-family:'o365IconsIE8'}.ie8 .wf-owa-play-large:before {content:'\e254';}.notIE8 .wf-owa-play-large:before {content:'\e054';}.ie8 .wf-owa-play-large {color:#FFFFFF/*$WFWhiteColor*/;}.notIE8 .wf-owa-play-large {border-color:#FFFFFF/*$WFWhiteColor*/; width:1.4em; height:1.4em; border-width:.1em; border-style:solid; border-radius:.8em; text-align:center; box-sizing:border-box; -moz-box-sizing:border-box; padding:0.1em; color:#FFFFFF/*$WFWhiteColor*/;}.ie8 .wf-size-play-large {width:40px; height:40px; font-size:30px}.notIE8 .wf-size-play-large {width:40px; height:40px; font-size:30px}--></style>
</head>
<body>
<div style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>This brings up something I've been thinking about. How hard would it be to rewrite the loadbalancer to work in the context of Amazon's AWS Autoscaling instead of as its own standalone piece? I am not familiar with autoscaling yet, but seems like there is
a bit of an overlap in functionality. Does that sound like a good direction?<br>
</p>
<p><br>
</p>
<p>Also, it would be great to abstract the loadbalancer's queue sensor and make SGE a plugin. Other implementations can include Celery or perhaps ipython. Just brainstorming for now.<br>
</p>
<p><br>
</p>
<div style="color: #282828;">
<hr tabindex="-1" style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> Cory Dolphin <wcdolphin@gmail.com><br>
<b>Sent:</b> Friday, March 28, 2014 10:56 AM<br>
<b>To:</b> Sergio Mafra<br>
<b>Cc:</b> Dmitry Serenbrennikov; starcluster@mit.edu<br>
<b>Subject:</b> Re: [StarCluster] StarCluster Plugins</font>
<div> </div>
</div>
<div>
<div dir="ltr">Is there a central repository of user-contributed plugins anywhere? Regardless, I have both a contribution and a question:
<div><br>
</div>
<div>1. Has anyone experimented with using SGE profile in an IPython cluster? I wish to enable starcluster's load balancing, but since it depends on SGE's queue length, I do not believe it will behave as expected.</div>
<div>2. I had some issues with remote engines dying during a particular job, so I edited the ipcluster plugin to use supervisord to watch the processes, if anyone else finds it useful, it is on Github with the creative name <a href="https://github.com/wcdolphin/StarClusterReliableIPCluster">StarClusterReliableIPCluster</a></div>
<div><br>
</div>
<div>Cheers,</div>
<div>Cory</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Mar 17, 2014 at 3:29 PM, Sergio Mafra <span dir="ltr">
<<a href="mailto:sergiohmafra@gmail.com" target="_blank">sergiohmafra@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">
<div dir="ltr">Good move Dmitry..
<div><br>
</div>
<div>If someone has more plugins who wants to share.. be welcomed here.</div>
<div><br>
</div>
<div>All best,</div>
<div><br>
</div>
<div>Sergio</div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Mar 17, 2014 at 3:45 PM, Dmitry Serenbrennikov <span dir="ltr">
<<a href="mailto:dmitry@adchemy.com" target="_blank">dmitry@adchemy.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">
<div>
<div style="font-size: 12pt; font-family: calibri, arial, helvetica, sans-serif;">
<p>Same here. <br>
</p>
<p><br>
</p>
<p>Here's another plugin that I have written. It integrates Chef such that master nodes are configured as chef clients/nodes.<br>
</p>
<p><a href="https://gist.github.com/nitecoder/9605618" target="_blank">https://gist.github.com/nitecoder/9605618</a></p>
<div style="margin-top: 20px; margin-bottom: 20px;">
<table style="border-bottom-style: solid; border-bottom-width: 1px; border-top-style: solid; border-top-color: #cccccc; width: 80%; border-bottom-color: #cccccc; border-top-width: 1px;">
<tbody>
<tr valign="top">
<td style="width: 140px; display: inline;">
<div style="width: 140px; display: table; margin-top: 12px; min-height: 140px;"><a href="https://gist.github.com/nitecoder/9605618" target="_blank" style="display: table-cell;"><img width="140" height="140" alt="" style="display: block; margin-left: auto; margin-right: auto; max-width: 140px; max-height: 140px; min-height: 140px; width: 140px; border-width: 0px;" src="https://1.gravatar.com/avatar/4456a0f0eda53e81fcd6ede8f2bf6dec?d=https%3A%2F%2Fidenticons.github.com%2Fdf64630c486a992b347c268ad6a445b5.png&s=140"></a></div>
</td>
<td>
<div style="margin-top: 8px; font-size: 21px; font-family: 'segoe ui wpc semilight', 'segoe ui semilight', tahoma, 'microsoft sans serif', verdana, sans-serif; color: #333333; margin-left: 14px; margin-right: 14px;">
This is a Chef plugin for StarCluster. It's really my first ever StarCluster plugin, plus I'm very new to Chef as well. So please take with a grain of salt. Any feedback is welcome! Especially, a few things are not quite to my liking: chef's authentication
system seems to presume that validation.pem file is widely distributed. I'm attempting to limit the need for this file being everywhere (by u</div>
<div style="margin-top: 8px; font-size: 13px; font-family: 'segoe ui wpc', 'segoe ui', tahoma, 'microsoft sans serif', verdana, sans-serif; color: #666666; margin-left: 14px; margin-right: 14px;">
This is a Chef plugin for StarCluster. It's really my first ever StarCluster plugin, plus I'm very new to Chef as well. So please take with a grain of salt. Any feedback is welcome! Especially, a few things are not quite to my liking: chef's authentication
system seems to presume that validation.pem file is widely distributed. I'm attempting to limit the need for this file being everywhere (by using chef bootstrap). It works, but doesn't seem like the right approach. Am I just misunderstanding Chef somehow?
currently it's not possible to tell if starcluster node is terminated or just stopped. If one deletes chef client on stop, then start doesn't work. I chose to delete the node but leave the client. But this also doesn't seem like the right approach. - Gist
is a simple way to share snippets of text and code with others.</div>
<div style="margin: 8px 14px 10px; min-height: 18px; text-overflow: ellipsis; overflow: hidden; white-space: nowrap;">
<a href="https://gist.github.com/nitecoder/9605618" target="_blank" style="font-size: 11px; font-family: 'segoe ui wpc', 'segoe ui', tahoma, 'microsoft sans serif', verdana, sans-serif; text-decoration: none;">Read more...</a></div>
</td>
</tr>
</tbody>
</table>
</div>
It's kind of clunky but seems to work. In particular, I've attempted to de-provision the node (if not the client) when cluster terminates. But I'm new to Chef and StarCluster both, so this could be totally missing the way Chef is supposed to operate. <br>
<p></p>
<p><br>
</p>
<p>Any feedback is welcome!<br>
</p>
<p><br>
</p>
<p>Thanks!<br>
</p>
<p>-Dmitry<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div style="color: #282828;">
<hr style="display: inline-block; width: 98%;">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b>
<a href="mailto:starcluster-bounces@mit.edu" target="_blank">starcluster-bounces@mit.edu</a> <<a href="mailto:starcluster-bounces@mit.edu" target="_blank">starcluster-bounces@mit.edu</a>> on behalf of Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" target="_blank">jacob.barhak@gmail.com</a>><br>
<b>Sent:</b> Sunday, March 16, 2014 9:21 PM<br>
<b>To:</b> Sergio Mafra<br>
<b>Cc:</b> <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a><br>
<b>Subject:</b> Re: [StarCluster] StarCluster Plugins</font>
<div> </div>
</div>
<div>
<div>
<div>
<div dir="ltr">
<div>Thanks Sergio,</div>
<div><br>
</div>
<div>You have a second there. </div>
<div><br>
</div>
<div>People were very helpful to me in the past when I asked questions and they suggested plugins that solved my problems. However, I still do not understand too much about them to the point I can write one on my own, and I saw others in the list that ask similar
questions about plugins.</div>
<div><br>
</div>
<div>The real question is if anyone who understands enough is willing to take this effort of a centralized repository?
</div>
<div><br>
</div>
<div>If someone does choose to invest in this it will help for sure and may grow to be a useful tool set.</div>
<div><br>
</div>
<div> Jacob</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Sat, Mar 15, 2014 at 3:08 PM, Sergio Mafra <span dir="ltr">
<<a href="mailto:sergiohmafra@gmail.com" target="_blank">sergiohmafra@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">
<div dir="ltr">Hi fellows,
<div><br>
</div>
<div>I´ve got no doubt that StarCluster is a fantastic tool. This helps lots of people on creating and managing an array of clusters.</div>
<div>But there is more.. the plugins.. They can help us improve what is really great.</div>
<div>The problem is knowing how to really develop and find them. They are scattered on internet.</div>
<div>I think that if someone (loving soul) creates a catalog of plugins, starting with "How to develop a StarCluster plugin for dummies"... this should be something. Better if Justin and Rayson cloud enhance the StarCluster´s site (<a href="http://star.mit.edu/cluster/docs/latest/manual/plugins.html#plugin-system" target="_blank">http://star.mit.edu/cluster/docs/latest/manual/plugins.html#plugin-system</a>)
to be like a micro git.</div>
<div>Tell me what do you think.</div>
<span><font color="#888888">
<div><br>
Sergio</div>
</font></span></div>
<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>