<div class="gmail_quote">Hi all,<div><br></div><div>From time to time, when I tried to add nodes to a running starcluster using either the loadbalance or addnodes, starcluster would miss fire. For example, I set "-a 5" in loadbalance, </div>
<div><br></div><div>command:</div><div> starcluster loadbalance -m 20 -a 5 -n 1 <mycluster></div><div><br></div><div>here is what I got:</div><div><br></div><div><div>>>> Loading full job history</div><div>
Cluster size: 10</div><div>Queued jobs: 361</div><div>Oldest queued job: <a href="tel:2012-01-03%2020" value="+12012010320" target="_blank">2012-01-03 20</a>:13:56</div><div>Avg job duration: 256 secs</div><div>Avg job wait time: 167 secs</div>
<div>Last cluster modification time: <a href="tel:2012-01-03%2020" value="+12012010320" target="_blank">2012-01-03 20</a>:17:07</div>
<div>>>> A job has been waiting for 963 sec, longer than max 900</div><div>>>> *** ADDING 5 NODES at <a href="tel:2012-01-03%2020" value="+12012010320" target="_blank">2012-01-03 20</a>:29:59.623917</div>
<div>>>> Launching node(s): node010, node011, node012, node013, node014</div>
<div>SpotInstanceRequest:sir-29586e14</div><div>SpotInstanceRequest:sir-46e90414</div><div>SpotInstanceRequest:sir-314a9814</div><div>SpotInstanceRequest:sir-99387e14</div><div>SpotInstanceRequest:sir-9ad72a14</div><div>
SpotInstanceRequest:sir-089dcc11</div>
<div>SpotInstanceRequest:sir-09d28011</div><div>SpotInstanceRequest:sir-64d4dc11</div><div>SpotInstanceRequest:sir-45516411</div><div>SpotInstanceRequest:sir-f2b31a11</div><div>SpotInstanceRequest:sir-0198f214</div><div>
SpotInstanceRequest:sir-1db0a014</div>
<div>SpotInstanceRequest:sir-49c97814</div><div>SpotInstanceRequest:sir-94fdd414</div><div>SpotInstanceRequest:sir-69db0014</div><div>SpotInstanceRequest:sir-6f410612</div></div><div><div>SpotInstanceRequest:sir-93c1c012</div>
<div>SpotInstanceRequest:sir-e44c7c12</div><div>SpotInstanceRequest:sir-dbc51012</div><div>SpotInstanceRequest:sir-aa52dc12</div><div>SpotInstanceRequest:sir-9f9e6811</div><div>SpotInstanceRequest:sir-50053011</div><div>
SpotInstanceRequest:sir-33455211</div>
<div>SpotInstanceRequest:sir-ffcdd011</div><div>SpotInstanceRequest:sir-c1d7ee11</div><div>>>> Waiting for node(s) to come up... (updating every 30s)</div><div>>>> Waiting for open spot requests to become active...</div>
<div>34/34 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </div><div>>>> Waiting for all nodes to be in a 'running' state...</div><div>35/35 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </div>
<div>>>> Waiting for SSH to come up on all nodes...</div><div>^C/35 ||||||||||||||||||||||||||||||||||||||||||||||||||||||| | 85% </div></div><div><br></div><div>Instead of 5 nodes, 25 nodes were fired up. Did anyone experience similar issue? Is this a bug in the code or I miss something in my command?</div>
<div><br></div><div>Thanks!</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br clear="all"><div><br></div>-- <br>Wei Tao, Ph.D.<br>TSI Biocomputing LLC<br><a href="tel:617-564-0934" value="+16175640934" target="_blank">617-564-0934</a><br>
</div>
</font></span></div><br>