<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} p
{margin-top:0;
margin-bottom:0}--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>since 'qstat -j 2' shows me<br>
</p>
<p><br>
</p>
<p>...</p>
<p>error reason 8: 08/22/2016 15:31:35 [1000:44925]: unable to find job file "/opt/sge6/default/spool/exec_spool_local/mynew1-node001/job_scripts/2"<br>
error reason 9: 08/22/2016 15:31:35 [1000:44926]: unable to find job file "/opt/sge6/default/spool/exec_spool_local/mynew1-node001/job_scripts/2"<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>this sounds a *lot* like the race condition described at</p>
<p><br>
</p>
<p><a href="https://confluence.si.edu/display/HPC/Job+Arrays#JobArrays-ParallelJobArrays">https://confluence.si.edu/display/HPC/Job+Arrays#JobArrays-ParallelJobArrays</a></p>
<p><br>
</p>
<p>and</p>
<p><br>
</p>
<p><a href="http://users.gridengine.sunsource.narkive.com/66KtbRva/sporadic-errors-in-array-tasks-with-a-pe">http://users.gridengine.sunsource.narkive.com/66KtbRva/sporadic-errors-in-array-tasks-with-a-pe</a></p>
<p><br>
</p>
<p>but adding '-b yes' doesn't seem to fix the problem. (there were no </p>
<p><span style="color: rgb(34, 34, 34); font-family: "Helvetica Neue", Arial, sans-serif; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">embedded
SGE options in my scriptfile)</span><br>
</p>
<p><br>
</p>
<p>Has anyone else encountered this? Found a work around?<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>also fwiw:<br>
</p>
<p><br>
</p>
<p>less /opt/sge6/default/spool/exec_spool_local/mynew1-node001/messages<br>
</p>
<p><br>
</p>
<p>08/22/2016 15:31:36| main|mynew1-node001|E|shepherd of job 2.8 exited with exit status = 11<br>
08/22/2016 15:31:36| main|mynew1-node001|C|exec of mailer "/bin/mail" failed: "No such file or directory"<br>
08/22/2016 15:31:36| main|mynew1-node001|E|shepherd of job 2.9 exited with exit status = 11<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><font color="000000"></font><br>
<font color="000000"></font></p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font color="000000"></font>
<div style="font-family:Tahoma; font-size:13px"><span lang="en-US"><font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" size="2"><span style="font-size:10pt"><b><font face="Arial,Helvetica,sans-serif">Michael Cariaso</font><br>
</b></span></font></span></font></div>
<font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"><span lang="en-US"><font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font color="000000"><a id="NoLP" href="mailto:michael.cariaso@keygene.com" tabindex="0"><font face="Consolas" size="2"><span style="font-size:10pt"><span lang="en-US"></span></span></font></a></font><font face="Consolas" size="2"><font color="000000"></font>
<div style="margin:0"><i><font face="Arial,Helvetica,sans-serif"><font color="000000" size="2"><span style="font-size:11pt"><font size="2"><span style="font-size:10pt">Bioinformatician</span></font></span></font><font size="2"><span style="font-size:11pt"></span></font><font size="2"><a id="NoLP" href="http://www.keygene.com" tabindex="0"><font size="2"><span style="font-size:10pt"></span></font></a><font color="#7F7F7F" size="2"><span style="font-size:10pt"></span></font></font></font></i>
</div>
</font></span><font face="Consolas" size="2"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"></span></font></span></font></font></font></div>
</span></span></font></span></font></div>
</span><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"><i></i></span></font></span></font><span lang="en-US"></span></div>
</div>
</div>
</div>
<div style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> starcluster-bounces@mit.edu <starcluster-bounces@mit.edu> on behalf of Mike Cariaso <mike.cariaso@keygene.com><br>
<b>Sent:</b> Tuesday, August 23, 2016 12:13 AM<br>
<b>To:</b> starcluster@mit.edu<br>
<b>Subject:</b> [StarCluster] workers go idle until a new worker is added ... ?</font>
<div> </div>
</div>
<div>
<p>using the latest version from</p>
<p><a href="https://github.com/datacratic/StarCluster/blob/vanilla_improvements/starcluster/plugins/sge.py">https://github.com/datacratic</a><br>
</p>
<p><br>
</p>
<p>I start a master node, and zero workers, and put an array job into the queue. I then then gradually add workers nodes. A new worker accepts as many tasks as the slots allow, but after they complete it never picks up additional work. When I add a new worker
machine, it accepts some tasks and runs them successfully, but never goes back for more. Usually during this time one of the idle previous machines will also pickup some more tasks, but once those are finished it again sits waiting.<br>
</p>
<p><br>
</p>
<p>qstat -j 1.19 shows me 'unable to find job file "/opt/sge6/default/spool/exec_spool_local/mynew1-node002/job_scripts/1"'</p>
<p><br>
</p>
<p>and it's true that no file is there. When I add a new machine, the job appears, suggesting this isn't a file permission issue.</p>
<p><br>
</p>
<p>some nodes remain out of action. </p>
<p>starcluster addnode -x -a nodename clustername</p>
<p>doesn't seem to help.</p>
<p><font color="000000"><br>
</font></p>
<p><font color="000000"></font><br>
<font color="000000"></font></p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font color="000000"></font>
<div style="font-family:Tahoma; font-size:13px"><span lang="en-US"><font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" size="2"><span style="font-size:10pt"><b><font face="Arial,Helvetica,sans-serif">Michael Cariaso</font><br>
</b></span></font></span></font></div>
<font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"><span lang="en-US"><font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font color="000000"><a id="NoLP" href="mailto:michael.cariaso@keygene.com" tabindex="0"><font face="Consolas" size="2"><span style="font-size:10pt"><span lang="en-US"></span></span></font></a></font><font face="Consolas" size="2"><font color="000000"></font>
<div style="margin:0"><i><font face="Arial,Helvetica,sans-serif"><font color="000000" size="2"><span style="font-size:11pt"><font size="2"><span style="font-size:10pt">Bioinformatician</span></font></span></font><font size="2"><span style="font-size:11pt"></span></font><font size="2"><a id="NoLP" href="http://www.keygene.com" tabindex="0"><font size="2"><span style="font-size:10pt"></span></font></a><font color="#7F7F7F" size="2"><span style="font-size:10pt"></span></font></font></font></i>
</div>
</font></span><font face="Consolas" size="2"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"></span></font></span></font></font></font></div>
</span></span></font></span></font></div>
</span><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"><i></i></span></font></span></font><span lang="en-US"></span></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>