<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Update: It seems like it might start happening whenever the cluster gets up to maximum capacity, which is 16 nodes. Any ideas of what to look for would be appreciated, this is getting
expensive.
<div><br>
</div>
<div>Amanda</div>
<div><br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF627870" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Amanda Joy Kedaigle<br>
<b>Sent:</b> Thursday, October 02, 2014 12:36 PM<br>
<b>To:</b> starcluster@mit.edu<br>
<b>Subject:</b> Loadbalancer error - node does not exist<br>
</font><br>
</div>
<div></div>
<div>
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">Hi, I'm running the Elastic LoadBalancer to keep our cluster down to one node when we're not using it, and then ramp up as needed. Generally (i.e. when I run tests and watch it),
it works just fine. But twice now, we've had it fail to remove nodes overnight and give the following error, leaving the cluster at full blast with no jobs to run. It says the nodes don't exist, but they are there both on the AWS EC2 console and when I run
qhost on the cluster. Any ideas as to the cause? Thanks!
<div><br>
</div>
<div>
<p class="p1">>>> Removing node013 from SGE</p>
<p class="p1">!!! ERROR - Error occured while running plugin 'starcluster.plugins.sge.SGEPlugin':</p>
<p class="p1">!!! ERROR - Failed to remove node node013</p>
<p class="p1">Traceback (most recent call last):</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/balancers/sge/__init__.py", line 754, in _eval_remove_node</p>
<p class="p1"> self._cluster.remove_node(node)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1050, in remove_node</p>
<p class="p1"> force=force)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1076, in remove_nodes</p>
<p class="p1"> reverse=True)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1690, in run_plugins</p>
<p class="p1"> self.run_plugin(plug, method_name=method_name, node=node)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1715, in run_plugin</p>
<p class="p1"> func(*args)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/plugins/sge.py", line 204, in on_remove_node</p>
<p class="p1"> self._remove_from_sge(node)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/plugins/sge.py", line 166, in _remove_from_sge</p>
<p class="p1"> master.ssh.execute('qconf -de %s' % node.alias)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/sshutils.py", line 579, in execute</p>
<p class="p1"> msg, command, exit_status, out_str)</p>
<p class="p1">RemoteCommandFailed: remote command 'source /etc/profile && qconf -de node013' failed with status 1:</p>
<p class="p1">denied: execution host "node013" does not exist</p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>