<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Amanda:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I agree with Rajat's t2 suggestion … even just a t2.micro will help over a t1.micro … and it's cheaper!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">And are you running jobs on the master as well as the nodes? If so, you could disable that:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[cluster mycluster]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">DISABLE_QUEUE=True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">PLUGINS = sge<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[plugin sge]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">setup_class = starcluster.plugins.sge.SGEPlugin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">master_is_exec_host = False<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">That might help with stability a good bit.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">You can also use spot pricing for the master to get a beefier master for a much lower price … but of course with the risk of losing the whole cluster if you
are outbid.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Good luck with the project!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Hugh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> starcluster-bounces@mit.edu [mailto:starcluster-bounces@mit.edu]
<b>On Behalf Of </b>Rajat Banerjee<br>
<b>Sent:</b> Tuesday, September 23, 2014 1:11 PM<br>
<b>To:</b> Amanda Joy Kedaigle<br>
<b>Cc:</b> starcluster@mit.edu<br>
<b>Subject:</b> Re: [StarCluster] FW: commlib error<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">HI Amanda,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I googled your error and found a few pages that suggest that sge service on the master node went down:<br>
<br>
<a href="http://verahill.blogspot.com/2012/08/sun-gridengine-commlib-error-got-select.html">http://verahill.blogspot.com/2012/08/sun-gridengine-commlib-error-got-select.html</a><br>
<br>
<a href="https://supcom.hgc.jp/english/utili_info/manual/faq.html">https://supcom.hgc.jp/english/utili_info/manual/faq.html</a><br>
<br>
<a href="http://comments.gmane.org/gmane.comp.clustering.gridengine.users/17283">http://comments.gmane.org/gmane.comp.clustering.gridengine.users/17283</a><o:p></o:p></p>
</div>
<p class="MsoNormal">If your OpenBLAS command is killing the process on master that could cause your issues according to those authors. Sorry I don't have anything more helpful, but the t2.small is still less than $.03 per hour now. That may not increase your
costs too much.<br>
<br>
Raj<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Sep 23, 2014 at 12:55 PM, Amanda Joy Kedaigle <<a href="mailto:mandyjoy@mit.edu" target="_blank">mandyjoy@mit.edu</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma","sans-serif";color:black">Thanks, Raj. I can communicate with the master node, it just looks like SGE is failing. I restarted the cluster and everything seemed to be working, but then it just failed in
the same way again.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">> starcluster listclusters (should list status of all your active clusters and running nodes)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p><span style="color:black">-----------------------------------------------------<o:p></o:p></span></p>
<p><span style="color:black">fraenkelcluster (security group: @sc-fraenkelcluster)<o:p></o:p></span></p>
<p><span style="color:black">-----------------------------------------------------<o:p></o:p></span></p>
<p><span style="color:black">Launch time: 2014-09-23 11:59:43<o:p></o:p></span></p>
<p><span style="color:black">Uptime: 0 days, 00:45:58<o:p></o:p></span></p>
<p><span style="color:black">VPC: vpc-c71f0fa5<o:p></o:p></span></p>
<p><span style="color:black">Subnet: subnet-e6b8c8ce<o:p></o:p></span></p>
<p><span style="color:black">Zone: us-east-1c<o:p></o:p></span></p>
<p><span style="color:black">Keypair: fraenkel-keypair<o:p></o:p></span></p>
<p><span style="color:black">EBS volumes:<o:p></o:p></span></p>
<p><span style="color:black"> vol-5e75ba11 on master:/dev/sdz (status: attached)<o:p></o:p></span></p>
<p><span style="color:black">Cluster nodes:<o:p></o:p></span></p>
<p><span style="color:black"> master running i-acc76242 <a href="http://ec2-54-164-81-80.compute-1.amazonaws.com" target="_blank">
ec2-54-164-81-80.compute-1.amazonaws.com</a><o:p></o:p></span></p>
<p><span style="color:black"> node001 running i-5177ddbf <a href="http://ec2-54-164-98-38.compute-1.amazonaws.com" target="_blank">
ec2-54-164-98-38.compute-1.amazonaws.com</a><o:p></o:p></span></p>
<p><span style="color:black"> node002 running i-9976c077 <a href="http://ec2-54-164-88-184.compute-1.amazonaws.com" target="_blank">
ec2-54-164-88-184.compute-1.amazonaws.com</a><o:p></o:p></span></p>
<p><span style="color:black"> node003 running i-9e76c070 <a href="http://ec2-54-164-38-146.compute-1.amazonaws.com" target="_blank">
ec2-54-164-38-146.compute-1.amazonaws.com</a><o:p></o:p></span></p>
<p><span style="color:black"> node004 running i-1776c0f9 <a href="http://ec2-54-86-252-119.compute-1.amazonaws.com" target="_blank">
ec2-54-86-252-119.compute-1.amazonaws.com</a><o:p></o:p></span></p>
<p><span style="color:black"> node005 running i-1676c0f8 <a href="http://ec2-54-165-66-3.compute-1.amazonaws.com" target="_blank">
ec2-54-165-66-3.compute-1.amazonaws.com</a><o:p></o:p></span></p>
<p><span style="color:black">Total nodes: 6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">> starcluster sshmaster <your cluster name><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">works just fine, I am ssh'd into master under root user.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Some more details: I am wondering if this is because my master node is a t1.micro - either it is an older generation and not updated, or doesn't have enough memory
to run the queue? When doing my initial tests, running thousands of simple jobs, it worked fine, and the load balancer added and deleted nodes as expected. However, when running slightly more intensive jobs, including the python module networkx, the jobs give
this error and then SGE dies:<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Segoe UI","sans-serif";color:black">OpenBLAS : Your OS does not support AVX instructions. OpenBLAS is using Nehalem kernels as a fallback, which may give poorer performance.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Segoe UI","sans-serif";color:black">Killed<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Segoe UI","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">I would really like to have a very cheap master node since I expect to keep it running 24/7, but only use the cluster in bursts. <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">On Mon, Sep 22, 2014 at 5:13 PM, Amanda Joy Kedaigle <<a href="mailto:mandyjoy@mit.edu" target="_blank">mandyjoy@mit.edu</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Hi,
<br>
<br>
I am trying to run starcluster's loadbalancer to keep only one node running until jobs are submitted to the cluster. I know it's an experimental feature, but I'm wondering if anyone has run into this error before, or has any suggestions. The cluster has been
whittled down to 1 node after a weekend of inactivity, and now it seems that when jobs are submitted to the queue, instead of adding nodes, SGE fails.<br>
<br>
>>> Loading full job history<br>
*** WARNING - Failed to retrieve stats (1/5):<br>
Traceback (most recent call last):<br>
File "/net/dorsal/apps/python2.7/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/balancers/sge/__init__.py", line 552, in get_stats<br>
return self._get_stats()<br>
File "/net/dorsal/apps/python2.7/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/balancers/sge/__init__.py", line 522, in _get_stats<br>
qhostxml = '\n'.join(master.ssh.execute('qhost -xml'))<br>
File "/net/dorsal/apps/python2.7/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/sshutils.py", line 578, in execute<br>
msg, command, exit_status, out_str)<br>
RemoteCommandFailed: remote command 'source /etc/profile && qhost -xml' failed with status 1:<br>
error: commlib error: got select error (Connection refused)<br>
error: unable to send message to qmaster using port 63231 on host "master": got send error<br>
<br>
Thanks for any help!</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:#888888"><br>
Amanda</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black"><br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>