Hi All,<div><br></div><div>I am experimenting with StarCluster loadbalancing of SGE using spot-priced instances.</div><div><br></div><div>After the wait time threshold is exceeded, starcluster puts in a new spot instance request, and when it becomes available and SSHable, it then errors out as it tries to add the node to the cluster. Since the first spot-instance fails to join, after stabilization, it tries to add another node, with the same error result:</div>
<div><br></div><div><div>>>> Loading full job history</div><div>Execution hosts: 2</div><div>Queued jobs: 18</div><div>Oldest queued job: 2013-03-07 18:18:54</div><div>Avg job duration: 252 secs</div><div>Avg job wait time: 7 secs</div>
<div>Last cluster modification time: 2013-03-07 18:03:11</div><div>>>> A job has been waiting for 454 sec, longer than max 300</div><div>*** WARNING - Adding 1 nodes at 2013-03-07 18:26:32.471812</div><div>>>> Launching node(s): node002</div>
<div>SpotInstanceRequest:sir-4763a012</div><div>>>> Waiting for node(s) to come up... (updating every 30s)</div><div>>>> Waiting for all nodes to be in a 'running' state...</div><div>2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </div>
<div>>>> Waiting for SSH to come up on all nodes...</div><div>2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </div><div>>>> Waiting for cluster to come up took 0.017 mins</div>
<div>!!! ERROR - Failed to add new host</div><div>>>> Sleeping...(looping again in 60 secs)</div><div><br></div><div>>>> Waiting for all nodes to come up...</div><div>>>> Waiting for all nodes to come up...</div>
<div><snip>...</div><div><br></div><div>>>> Loading full job history</div><div>Execution hosts: 2</div><div>Queued jobs: 17</div><div>Oldest queued job: 2013-03-07 18:20:48</div><div>Avg job duration: 282 secs</div>
<div>Avg job wait time: 252 secs</div><div>Last cluster modification time: 2013-03-07 18:03:11</div><div>>>> A job has been waiting for 798 sec, longer than max 300</div><div>*** WARNING - Adding 1 nodes at 2013-03-07 18:34:10.519812</div>
<div>>>> Launching node(s): node003</div><div>SpotInstanceRequest:sir-02a87c11</div><div>>>> Waiting for node(s) to come up... (updating every 30s)</div><div>>>> Waiting for all nodes to be in a 'running' state...</div>
<div>3/3 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </div><div>>>> Waiting for SSH to come up on all nodes...</div><div>3/3 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100% </div>
<div>>>> Waiting for cluster to come up took 0.021 mins</div><div>!!! ERROR - Failed to add new host</div><div>>>> Sleeping...(looping again in 60 secs)</div><div><br></div><div>>>> Waiting for all nodes to come up...</div>
<div>...</div></div><div><br></div><div>/etc/host on the master confirms the nodes haven't joined:</div><div><br></div><div><div>[prod@master scripts]$ cat /etc/hosts</div><div>127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4</div>
<div>::1 localhost localhost.localdomain localhost6 localhost6.localdomain6</div><div>10.76.35.21 master</div><div>10.119.19.94 node001</div></div><div><br></div><div>Unfortunately, the extra nodes are up and being charged by AWS, but are not doing the cluster's work.</div>
<div><br></div><div>And starcluster listclusters shows the cluster w/all four nodes:</div><div><br></div><div><div>-----------------------------</div><div>e1d (security group: @sc-e1d)</div><div>-----------------------------</div>
<div>Launch time: 2013-03-07 09:50:57</div><div>Uptime: 0 days, 00:48:50</div><div>Zone: us-east-1d</div><div>Keypair: lapuserkey</div><div>EBS volumes:</div><div> <snip> ...</div><div>Spot requests: 3 active</div>
<div>Cluster nodes:</div><div> master running i-f8ad948b <a href="http://ec2-50-17-99-136.compute-1.amazonaws.com">ec2-50-17-99-136.compute-1.amazonaws.com</a></div><div> node001 running i-c4a79eb7 <a href="http://ec2-50-17-178-183.compute-1.amazonaws.com">ec2-50-17-178-183.compute-1.amazonaws.com</a> (spot sir-88e00c14)</div>
<div> node002 running i-7a437b09 <a href="http://ec2-54-234-178-59.compute-1.amazonaws.com">ec2-54-234-178-59.compute-1.amazonaws.com</a> (spot sir-4763a012)</div><div> node003 running i-f82a128b <a href="http://ec2-23-20-158-195.compute-1.amazonaws.com">ec2-23-20-158-195.compute-1.amazonaws.com</a> (spot sir-02a87c11)</div>
<div>Total nodes: 4</div></div><div><br></div><div>Here is the debug log's relevant info; same error mode for the next attempt to add node003:</div><div><br></div><div><div>2013-03-07 10:26:34,302 PID: 94806 __init__.py:680 - ERROR - Failed to add new host</div>
<div>2013-03-07 10:26:34,308 PID: 94806 __init__.py:681 - DEBUG - Traceback (most recent call last):</div><div> File "/Library/Python/2.7/site-packages/StarCluster-0.93.3-py2.7.egg/starcluster/balancers/sge/__init__.py", line 675, in _eval_add_node</div>
<div> self._cluster.add_nodes(need_to_add)</div><div> File "/Library/Python/2.7/site-packages/StarCluster-0.93.3-py2.7.egg/starcluster/cluster.py", line 838, in add_nodes</div><div> node = self.get_node_by_alias(alias)</div>
<div> File "/Library/Python/2.7/site-packages/StarCluster-0.93.3-py2.7.egg/starcluster/cluster.py", line 707, in get_node_by_alias</div><div> raise exception.InstanceDoesNotExist(alias, label='node')</div>
<div>InstanceDoesNotExist: node 'node002' does not exist</div></div><div><br></div><div>Is there a place in the node adding functionality where I could productively add a delay so there's no further action til after the hostname has been added to /etc/hosts, to try to overcome this failure? I'll appreciate any feedback.</div>
<div><br></div><div>Thanks,</div><div>Lyn</div>