<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hi Amanda,</div><div><br></div><div>Did you check your I/O when running your application?</div><div><br></div><div>In the past I had too much traffic to the NFS due to many files and it caused things to slow down considerably. </div><div><br></div><div>It may not be your issue, yet it is worth checking anyway. </div><div><br></div><div>I hope you resolve your issue regardless. </div><div><br></div><div> Jacob<br><br>Sent from my iPhone</div><div><br>On Oct 4, 2014, at 1:07 PM, Amanda Joy Kedaigle <<a href="mailto:mandyjoy@mit.edu">mandyjoy@mit.edu</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Update: It seems like it might start happening whenever the cluster gets up to maximum capacity, which is 16 nodes. Any ideas of what to look for would be appreciated, this is getting
expensive.
<div><br>
</div>
<div>Amanda</div>
<div><br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF627870" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Amanda Joy Kedaigle<br>
<b>Sent:</b> Thursday, October 02, 2014 12:36 PM<br>
<b>To:</b> <a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a><br>
<b>Subject:</b> Loadbalancer error - node does not exist<br>
</font><br>
</div>
<div></div>
<div>
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt">Hi, I'm running the Elastic LoadBalancer to keep our cluster down to one node when we're not using it, and then ramp up as needed. Generally (i.e. when I run tests and watch it),
it works just fine. But twice now, we've had it fail to remove nodes overnight and give the following error, leaving the cluster at full blast with no jobs to run. It says the nodes don't exist, but they are there both on the AWS EC2 console and when I run
qhost on the cluster. Any ideas as to the cause? Thanks!
<div><br>
</div>
<div>
<p class="p1">>>> Removing node013 from SGE</p>
<p class="p1">!!! ERROR - Error occured while running plugin 'starcluster.plugins.sge.SGEPlugin':</p>
<p class="p1">!!! ERROR - Failed to remove node node013</p>
<p class="p1">Traceback (most recent call last):</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/balancers/sge/__init__.py", line 754, in _eval_remove_node</p>
<p class="p1"> self._cluster.remove_node(node)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1050, in remove_node</p>
<p class="p1"> force=force)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1076, in remove_nodes</p>
<p class="p1"> reverse=True)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1690, in run_plugins</p>
<p class="p1"> self.run_plugin(plug, method_name=method_name, node=node)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/cluster.py", line 1715, in run_plugin</p>
<p class="p1"> func(*args)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/plugins/sge.py", line 204, in on_remove_node</p>
<p class="p1"> self._remove_from_sge(node)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/plugins/sge.py", line 166, in _remove_from_sge</p>
<p class="p1"> master.ssh.execute('qconf -de %s' % node.alias)</p>
<p class="p1"> File "/home/mandyjoy/ENV/lib/python2.7/site-packages/StarCluster-0.95.5-py2.7.egg/starcluster/sshutils.py", line 579, in execute</p>
<p class="p1"> msg, command, exit_status, out_str)</p>
<p class="p1">RemoteCommandFailed: remote command 'source /etc/profile && qconf -de node013' failed with status 1:</p>
<p class="p1">denied: execution host "node013" does not exist</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>StarCluster mailing list</span><br><span><a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a></span><br><span><a href="http://mailman.mit.edu/mailman/listinfo/starcluster">http://mailman.mit.edu/mailman/listinfo/starcluster</a></span><br></div></blockquote></body></html>