<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>
<div><br>
</div>
<div><br>
</div>
<div>Hi Starcluster </div>
<div><br>
</div>
<div>I have been building some plugins that take a while to run because they are building and installing large libraries. As a result I have seen issues with ssh terminating my connection while the process is still running. Which seems to return exit code 1,
although the process continues on the cluster.</div>
<div><br>
</div>
<div>For my custom plugins I added the following line to apply a keep alive to the ssh transport. </div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre">def run(self, nodes, master, user, user_shell, volumes):</span></div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>…</div>
<div>
<div> <span class="Apple-tab-span" style="white-space:pre"></span>for node in nodes:</div>
<div> <span class="Apple-tab-span" style="white-space:pre"></span>node.ssh.transport.set_keepalive(30)</div>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>…</div>
<div><br>
</div>
<div>This can be done this way, but you might consider adding it somewhere in starcluster, probably in the connect method of SSHClient:</div>
<div><a href="https://github.com/jtriley/StarCluster/blob/develop/starcluster/sshutils.py#L100">https://github.com/jtriley/StarCluster/blob/develop/starcluster/sshutils.py#L100</a></div>
<div><br>
</div>
<div><br>
</div>
<div>Here are the methods form paramiko:</div>
<div><a href="https://github.com/paramiko/paramiko/blob/master/paramiko/packet.py#L175">https://github.com/paramiko/paramiko/blob/master/paramiko/packet.py#L175</a></div>
<div><a href="https://github.com/paramiko/paramiko/blob/master/paramiko/transport.py#L762">https://github.com/paramiko/paramiko/blob/master/paramiko/transport.py#L762</a></div>
<div><br>
</div>
<div><br>
</div>
<div>Another step that would help is to add a longer disconnect to the default /etc/ssh/sshd_config in the cluster ami.</div>
<div><br>
</div>
<div>For instance I have used one of my plugins to set:</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>ClientAliveInterval 600</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>ClientAliveCountMax 3</div>
<div><br>
</div>
<div>That should keep ssh connections open for half an hour. </div>
<div><br>
</div>
<div>
<div>
<p class="MsoNormal" style="color: rgb(0, 0, 0); margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; line-height: 17px; font-size: 11pt; font-family: Calibri, sans-serif; ">
<b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; ">David</span></b><b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; "> Stuebe</span></b></p>
<p class="MsoNormal" style="color: rgb(0, 0, 0); margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; line-height: 17px; font-size: 11pt; font-family: Calibri, sans-serif; ">
<b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; ">Scientist & Software Engineer </span></b><b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; "><br>
<br>
</span></b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: black; font-family: Arial, sans-serif; ">55 Village Square Drive <br>
South Kingstown, RI 02879-8248<br>
<br>
</span><b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; ">Tel: </span></b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: black; font-family: Arial, sans-serif; ">+1 (401) 789-6224</span></p>
<p class="MsoNormal" style="margin-top: 0in; margin-right: 0in; margin-bottom: 10pt; margin-left: 0in; ">
<span lang="EN-GB"><b style="color: rgb(0, 0, 0); font-family: Arial, sans-serif; font-size: 8pt; line-height: 12px; "><span style="color: navy; ">Email: </span></b><span style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; "><a href="mailto:David.Stuebe@rpsgroup.com" style="color: blue; text-decoration: underline; ">David.Stuebe@rpsgroup.com</a></span></span><span style="color: rgb(0, 0, 0); font-size: 12pt; line-height: 18px; font-family: 'Times New Roman', serif; "><br>
</span><b style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 11pt; line-height: 17px; "><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; ">www:</span></b><span lang="EN-GB" style="font-size: 8pt; line-height: 12px; color: navy; font-family: Arial, sans-serif; "> <a href="http://www.asascience.com/" style="color: blue; text-decoration: underline; ">asascience.com</a> | <a href="http://www.rpsgroup.com/" style="color: blue; text-decoration: underline; ">rpsgroup.com</a></span><span style="color: rgb(0, 0, 0); font-size: 12pt; line-height: 18px; font-family: 'Times New Roman', serif; "><br>
<br>
</span><i style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 11pt; line-height: 17px; "><span style="font-size: 8pt; line-height: 12px; color: black; font-family: Arial, sans-serif; "><o:p></o:p></span></i></p>
<p class="MsoNormal" style="color: rgb(0, 0, 0); margin-top: 0in; margin-right: 0in; margin-bottom: 10pt; margin-left: 0in; line-height: 17px; font-size: 11pt; font-family: Calibri, sans-serif; ">
<i><span style="font-size: 8pt; line-height: 12px; color: black; font-family: Arial, sans-serif; ">A member of the RPS Group plc</span></i></p>
</div>
</div>
</div>
</div>
</body>
</html>