<div dir="ltr">Update/Close: Strangely, this particular issue was resolved by going to the master and zeroing the known_hosts file (as in "> known_hosts").</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 7, 2015 at 11:39 AM, Lyn Gerner <span dir="ltr"><<a href="mailto:schedulerqueen@gmail.com" target="_blank">schedulerqueen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Developers,<div><br></div><div>Sorry for the Fri afternoon query, but I'm getting an error never before seen on an addnode, and it recurs even on a -x retry. Appreciate any workaround/recovery suggestions for the following:</div><div><br></div><div><p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b># sc an -x -a node002 w2c</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>StarCluster - (<a href="http://star.mit.edu/cluster" target="_blank">http://star.mit.edu/cluster</a>) (v. 0.95.6)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>Software Tools for Academics and Researchers (STAR)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>Please submit bug reports to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208);min-height:16px"><b></b><br></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Waiting for node(s) to come up... (updating every 30s)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Waiting for all nodes to be in a 'running' state...</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>3/3 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Waiting for SSH to come up on all nodes...</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>3/3 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Waiting for cluster to come up took 0.206 mins</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Running plugin starcluster.clustersetup.DefaultClusterSetup</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring hostnames...</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring /etc/hosts on each node</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>3/3 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring NFS exports path(s):</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>/home /jobs/ /usr/share/jobs/ /pipe/</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Mounting all NFS export path(s) on 1 worker node(s)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Setting up NFS took 0.021 mins</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring scratch space for user(s): sgeadmin</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring passwordless ssh for root</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring passwordless ssh for sgeadmin</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Running plugin swap_addnode_w2c.VISwapConfigurator</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Configuring Swap on node002</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Running plugin starcluster.plugins.users.CreateUsers</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Creating 1 users on node002</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>>>> Adding node002 to known_hosts for 1 users</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>!!! ERROR - Error occured while running plugin 'starcluster.plugins.users.CreateUsers':</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>!!! ERROR - Unhandled exception occured</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>Traceback (most recent call last):</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/cli.py", line 274, in main</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> sc.execute(args)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/commands/addnode.py", line 128, in execute</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> no_create=self.opts.no_create)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/cluster.py", line 189, in add_nodes</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> no_create=no_create)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/cluster.py", line 1042, in add_nodes</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> self.run_plugins(method_name="on_add_node", node=node)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/cluster.py", line 1690, in run_plugins</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> self.run_plugin(plug, method_name=method_name, node=node)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/cluster.py", line 1715, in run_plugin</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> func(*args)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/plugins/users.py", line 164, in on_add_node</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> master.add_to_known_hosts(user, [node])</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/node.py", line 578, in add_to_known_hosts</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> khostsf = self.ssh.remote_file(known_hosts_file, 'a')</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/StarCluster-0.95.6-py2.6.egg/starcluster/sshutils.py", line 320, in remote_file</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> rfile = self.sftp.open(file, mode)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/paramiko-1.15.1-py2.6.egg/paramiko/sftp_client.py", line 327, in open</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> t, msg = self._request(CMD_OPEN, filename, imode, attrblock)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/paramiko-1.15.1-py2.6.egg/paramiko/sftp_client.py", line 729, in _request</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> return self._read_response(num)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/paramiko-1.15.1-py2.6.egg/paramiko/sftp_client.py", line 776, in _read_response</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> self._convert_status(msg)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> File "/usr/lib/python2.6/site-packages/paramiko-1.15.1-py2.6.egg/paramiko/sftp_client.py", line 802, in _convert_status</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b> raise IOError(errno.ENOENT, text)</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>IOError: [Errno 2] No such file</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208);min-height:16px"><b></b><br></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>!!! ERROR - Oops! Looks like you've found a bug in StarCluster</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>!!! ERROR - Crash report written to: /root/.starcluster/logs/crash-report-11317.txt</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>!!! ERROR - Please remove any sensitive data from the crash report</b></p>
<p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b>!!! ERROR - and submit it to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></b></p><p style="margin:0px;font-size:14px;font-family:Menlo;color:rgb(76,47,45);background-color:rgb(167,221,208)"><b><br></b></p><p style="margin:0px;font-size:14px;color:rgb(76,47,45);background-color:rgb(167,221,208)"><font face="arial, helvetica, sans-serif">There's not much more in the crash report, but I can send it, if it will help. Thanks in advance.</font></p><p style="margin:0px;font-size:14px;color:rgb(76,47,45);background-color:rgb(167,221,208)"><font face="arial, helvetica, sans-serif"><br></font></p><p style="margin:0px;font-size:14px;color:rgb(76,47,45);background-color:rgb(167,221,208)"><font face="arial, helvetica, sans-serif">Best,</font></p><p style="margin:0px;font-size:14px;color:rgb(76,47,45);background-color:rgb(167,221,208)"><font face="arial, helvetica, sans-serif">Lyn</font></p></div></div>
</blockquote></div><br></div>