<div dir="ltr"><div><div>HI Amanda,<br></div>I googled your error and found a few pages that suggest that sge service on the master node went down:<br><br><a href="http://verahill.blogspot.com/2012/08/sun-gridengine-commlib-error-got-select.html">http://verahill.blogspot.com/2012/08/sun-gridengine-commlib-error-got-select.html</a><br><br><a href="https://supcom.hgc.jp/english/utili_info/manual/faq.html">https://supcom.hgc.jp/english/utili_info/manual/faq.html</a><br><br><a href="http://comments.gmane.org/gmane.comp.clustering.gridengine.users/17283">http://comments.gmane.org/gmane.comp.clustering.gridengine.users/17283</a><br><br></div>If your OpenBLAS command is killing the process on master that could cause your issues according to those authors. Sorry I don't have anything more helpful, but the t2.small is still less than $.03 per hour now. That may not increase your costs too much.<br><br>Raj<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 23, 2014 at 12:55 PM, Amanda Joy Kedaigle <span dir="ltr"><<a href="mailto:mandyjoy@mit.edu" target="_blank">mandyjoy@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<div style="direction:ltr"><span style="font-size:small;font-family:Tahoma">Thanks, Raj. I can communicate with the master node, it just looks like SGE is failing. I restarted the cluster and everything seemed to be working, but then
it just failed in the same way again.</span></div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div style="font-size:16px;font-family:'Times New Roman'">
<div>
<div><br>
</div>
> starcluster listclusters (should list status of all your active clusters and running nodes)</div>
<div><br>
</div>
<div>
<p>-----------------------------------------------------</p>
<p>fraenkelcluster (security group: @sc-fraenkelcluster)</p>
<p>-----------------------------------------------------</p>
<p>Launch time: 2014-09-23 11:59:43</p>
<p>Uptime: 0 days, 00:45:58</p>
<p>VPC: vpc-c71f0fa5</p>
<p>Subnet: subnet-e6b8c8ce</p>
<p>Zone: us-east-1c</p>
<p>Keypair: fraenkel-keypair</p>
<p>EBS volumes:</p>
<p> vol-5e75ba11 on master:/dev/sdz (status: attached)</p>
<p>Cluster nodes:</p>
<p> master running i-acc76242 <a href="http://ec2-54-164-81-80.compute-1.amazonaws.com" target="_blank">ec2-54-164-81-80.compute-1.amazonaws.com</a></p>
<p> node001 running i-5177ddbf <a href="http://ec2-54-164-98-38.compute-1.amazonaws.com" target="_blank">ec2-54-164-98-38.compute-1.amazonaws.com</a></p>
<p> node002 running i-9976c077 <a href="http://ec2-54-164-88-184.compute-1.amazonaws.com" target="_blank">ec2-54-164-88-184.compute-1.amazonaws.com</a></p>
<p> node003 running i-9e76c070 <a href="http://ec2-54-164-38-146.compute-1.amazonaws.com" target="_blank">ec2-54-164-38-146.compute-1.amazonaws.com</a></p>
<p> node004 running i-1776c0f9 <a href="http://ec2-54-86-252-119.compute-1.amazonaws.com" target="_blank">ec2-54-86-252-119.compute-1.amazonaws.com</a></p>
<p> node005 running i-1676c0f8 <a href="http://ec2-54-165-66-3.compute-1.amazonaws.com" target="_blank">ec2-54-165-66-3.compute-1.amazonaws.com</a></p>
<p>Total nodes: 6</p>
<br>
</div>
> starcluster sshmaster <your cluster name></div>
<div style="font-size:16px;font-family:'Times New Roman'"><br>
</div>
<div><font>works just fine, I am ssh'd into master under root user.</font></div>
<div><font><br>
</font></div>
<div><font>Some more details: I am wondering if this is because my master node is a t1.micro - either it is an older generation and not updated, or doesn't have enough memory to run the queue? When doing my initial tests, running thousands of simple
jobs, it worked fine, and the load balancer added and deleted nodes as expected. However, when running slightly more intensive jobs, including the python module networkx, the jobs give this error and then SGE dies:</font></div>
<div style="font-size:16px;font-family:'Times New Roman'">
<div style="font-family:'Segoe UI',Helvetica,Arial,sans-serif;font-size:medium">
OpenBLAS : Your OS does not support AVX instructions. OpenBLAS is using Nehalem kernels as a fallback, which may give poorer performance.</div>
<div style="font-family:'Segoe UI',Helvetica,Arial,sans-serif;font-size:medium">
Killed</div>
</div>
<div style="font-size:medium;font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<br>
</div>
<div><font>I would really like to have a very cheap master node since I expect to keep it running 24/7, but only use the cluster in bursts. </font></div>
</div>
<div class="gmail_extra" style="font-size:16px;font-family:'Times New Roman'"><br>
<div class="gmail_quote">On Mon, Sep 22, 2014 at 5:13 PM, Amanda Joy Kedaigle <span dir="ltr">
<<a href="mailto:mandyjoy@mit.edu" target="_blank">mandyjoy@mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Hi,
<br>
<br>
I am trying to run starcluster's loadbalancer to keep only one node running until jobs are submitted to the cluster. I know it's an experimental feature, but I'm wondering if anyone has run into this error before, or has any suggestions. The cluster has been
whittled down to 1 node after a weekend of inactivity, and now it seems that when jobs are submitted to the queue, instead of adding nodes, SGE fails.<br>
<br>
>>> Loading full job history<br>
*** WARNING - Failed to retrieve stats (1/5):<br>
Traceback (most recent call last):<br>
File "/net/dorsal/apps/python2.7/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/balancers/sge/__init__.py", line 552, in get_stats<br>
return self._get_stats()<br>
File "/net/dorsal/apps/python2.7/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/balancers/sge/__init__.py", line 522, in _get_stats<br>
qhostxml = '\n'.join(master.ssh.execute('qhost -xml'))<br>
File "/net/dorsal/apps/python2.7/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/sshutils.py", line 578, in execute<br>
msg, command, exit_status, out_str)<br>
RemoteCommandFailed: remote command 'source /etc/profile && qhost -xml' failed with status 1:<br>
error: commlib error: got select error (Connection refused)<br>
error: unable to send message to qmaster using port 63231 on host "master": got send error<br>
<br>
Thanks for any help!<span><font color="#888888"><br>
Amanda<br>
</font></span></div>
</div>
<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div><br>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div>