<div dir="ltr">Thanks so much Antonio. It does seem like this is the issue (and I hope it is). I was very worried there was a larger issue, or that my AWS account got into some error state that would be difficult to fix.<div><br></div><div>Hopefully AWS will resolve these issues and everything will start working again.<br>Avner</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 24, 2015 at 5:18 PM, Antonio Osorio <span dir="ltr"><<a href="mailto:antonio.osorio@schrodinger.com" target="_blank">antonio.osorio@schrodinger.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><span class=""><div>Avner,</div><div><br></div>AWS is having some API problems<div><br></div><div><a href="http://status.aws.amazon.com/" target="_blank">http://status.aws.amazon.com/</a></div><div><br></div><div>Amazon Elastic Compute Cloud (N. Virginia) -> Increased API Error Rates:</div><div><br></div></span><div><div style="margin-top:8px"><span class=""><div><span>12:22 PM PST</span> We are investigating increased API error rates in the US-EAST-1 Region.</div><div><span>12:53 PM PST</span> We can confirm increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.</div><div><span>1:58 PM PST</span> We continue to work toward resolving increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.</div><div><br></div></span><span class="HOEnZb"><font color="#888888"><div>-Antonio</div></font></span></div></div><div><div class="h5"><div><br style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br>
</div>
<br><div><div>On Feb 24, 2015, at 5:04 PM, Avner May <<a href="mailto:avnermay@cs.columbia.edu" target="_blank">avnermay@cs.columbia.edu</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">And now this error message when calling "listclusters"...<div><br></div><div><div><i>C:\Windows\system32>starcluster listclusters</i></div><div><i>StarCluster - (<a href="http://star.mit.edu/cluster" target="_blank">http://star.mit.edu/cluster</a>) (v. 0.95.6)</i></div><div><i>Software Tools for Academics and Researchers (STAR)</i></div><div><i>Please submit bug reports to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></i></div><div><i><br></i></div><div><i>!!! ERROR - ************************************************************</i></div><div><i>!!! ERROR - INCOMPATIBLE CLUSTER: babel</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - The cluster 'babel' is not compatible with StarCluster</i></div><div><i>!!! ERROR - 0.95.6. Possible reasons are:</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 1. The '@sc-babel' group was created using an incompatible</i></div><div><i>!!! ERROR - version of StarCluster (stable or development).</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 2. The '@sc-babel' group was manually created outside of</i></div><div><i>!!! ERROR - StarCluster.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 3. One of the nodes belonging to '@sc-babel' was manually</i></div><div><i>!!! ERROR - created outside of StarCluster.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 4. StarCluster was interrupted very early on when first</i></div><div><i>!!! ERROR - creating the cluster's security group.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - In any case 'babel' and its nodes cannot be used with this</i></div><div><i>!!! ERROR - version of StarCluster (0.95.6).</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - The cluster 'babel' currently has 0 active nodes.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - Please terminate the cluster using:</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - $ starcluster terminate --force babel</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - ************************************************************</i></div><div><i><br></i></div><div><i>!!! ERROR - ************************************************************</i></div><div><i>!!! ERROR - INCOMPATIBLE CLUSTER: babel2</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - The cluster 'babel2' is not compatible with StarCluster</i></div><div><i>!!! ERROR - 0.95.6. Possible reasons are:</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 1. The '@sc-babel2' group was created using an incompatible</i></div><div><i>!!! ERROR - version of StarCluster (stable or development).</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 2. The '@sc-babel2' group was manually created outside of</i></div><div><i>!!! ERROR - StarCluster.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 3. One of the nodes belonging to '@sc-babel2' was manually</i></div><div><i>!!! ERROR - created outside of StarCluster.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - 4. StarCluster was interrupted very early on when first</i></div><div><i>!!! ERROR - creating the cluster's security group.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - In any case 'babel2' and its nodes cannot be used with this</i></div><div><i>!!! ERROR - version of StarCluster (0.95.6).</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - The cluster 'babel2' currently has 0 active nodes.</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - Please terminate the cluster using:</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - $ starcluster terminate --force babel2</i></div><div><i>!!! ERROR -</i></div><div><i>!!! ERROR - ************************************************************</i></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 24, 2015 at 4:58 PM, Avner May <span dir="ltr"><<a href="mailto:avnermay@cs.columbia.edu" target="_blank">avnermay@cs.columbia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Here's the latest error I received when trying to start a cluster. What does it mean that my default VPC is 'none'??<div><br></div><div><b>!!! ERROR - InvalidGroup.NotFound: The security group '@sc-babel2' does not exist in default VPC 'none'</b><br><div><br></div><div>======== FULL ERROR MESSAGE / CALL STACK ===========</div><div><span><div>C:\Windows\system32>starcluster start babel2</div><div>StarCluster - (<a href="http://star.mit.edu/cluster" target="_blank">http://star.mit.edu/cluster</a>) (v. 0.95.6)</div><div>Software Tools for Academics and Researchers (STAR)</div><div>Please submit bug reports to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></div><div><br></div><div>>>> Using default cluster template: main</div><div>>>> Validating cluster template settings...</div><div>>>> Cluster template settings are valid</div><div>>>> Starting cluster...</div><div>>>> Launching a 20-node cluster...</div><div>>>> Creating security group @sc-babel2...</div></span><div>>>> Waiting for security group @sc-babel2...</div><div>!!! ERROR - InvalidGroup.NotFound: The security group '@sc-babel2' does not exist in default VPC 'none'</div><span><div>Traceback (most recent call last):</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cli.py", line 274, in main</div><div> sc.execute(args)</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\commands\start.py", line 244, in execute</div><div> validate_running=validate_running)</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1628, in start</div><div> return self._start(create=create, create_only=create_only)</div><div> File "<string>", line 2, in _start</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\utils.py", line 112, in wrap_f</div><div> res = func(*arg, **kargs)</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1643, in _start</div><div> self.create_cluster()</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1163, in create_cluster</div><div> self._create_flat_rate_cluster()</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1185, in _create_flat_rate_cluster</div><div> force_flat=True)[0]</div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 926, in create_nodes</div><div> cluster_sg = <a href="http://self.cluster_group.name" target="_blank">self.cluster_group.name</a></div><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 655, in cluster_group</div><div> vpc_id=vpc_id)</div></span><div> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\awsutils.py", line 314, in create_group</div><div> to_port=65535)</div><div> File "C:\Python27\lib\site-packages\boto\ec2\securitygroup.py", line 203, in authorize</div><div> dry_run=dry_run)</div><div> File "C:\Python27\lib\site-packages\boto\ec2\connection.py", line 3191, in <b>authorize_security_group</b></div><div> params, verb='POST')</div><div> File "C:\Python27\lib\site-packages\boto\connection.py", line 1226, in get_status</div><span><div> raise self.ResponseError(response.status, response.reason, body)</div><div>EC2ResponseError: EC2ResponseError: <b>400 Bad Request</b></div><div><?xml version="1.0" encoding="UTF-8"?></div></span><div><Response><Errors><Error><Code>InvalidGroup.NotFound</Code><Message><b>The security group '@sc-babel2' does not exist in default VPC 'none'</b></Message></Er</div><div>ror></Errors><RequestID>94012bc0-dba2-4c0b-b9fe-00ac38a45ed0</RequestID></Response></div></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 24, 2015 at 4:55 PM, Avner May <span dir="ltr"><<a href="mailto:avnermay@cs.columbia.edu" target="_blank">avnermay@cs.columbia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I'm getting the errors below when I try to start a cluster, or listclusters. This all started after I terminated a cluster. I got an error during termination, so it told me to use the "-f" flag to force termination. I did that, but it was taking a very long time to erase the security group. So I interrupted the "terminate -f" command, and I've been having issues ever since. Basically, if I try to start a cluster, it is taking forever in the step where it says "waiting for a security group @sc-cluster-name" (I've been waiting like 20+ minutes for a cluster to start...). It then generally gives me some error like the one below. It also fails in the "listclusters" command. At the heart of this there seem to be issues in the "<b>get_all_security_groups</b>" and "<b>create_security_group</b>" functions in "C:\Python27\lib\site-packages\boto\ec2\connection.py". Any idea what might be going on? Help would be very appreciated, as this is totally blocking my progress on my work.</div><div><br>Thanks a lot,</div><div>Avner</div><div><br></div><div><div><i>C:\Windows\system32>starcluster start babel</i></div><div><i>StarCluster - (<a href="http://star.mit.edu/cluster" target="_blank">http://star.mit.edu/cluster</a>) (v. 0.95.6)</i></div><div><i>Software Tools for Academics and Researchers (STAR)</i></div><div><i>Please submit bug reports to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></i></div><div><i><br></i></div><div><i>>>> Using default cluster template: main</i></div><div><i>>>> Validating cluster template settings...</i></div><div><i>>>> Cluster template settings are valid</i></div><div><i>>>> Starting cluster...</i></div><div><i>>>> Launching a 20-node cluster...</i></div><div><i>>>> Creating security group @sc-babel...</i></div><div><i>>>> Waiting for security group @sc-babel...</i></div><div><i>!!! ERROR - InternalError: An internal error has occurred</i></div><div><i>Traceback (most recent call last):</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cli.py", line 274, in main</i></div><div><i> sc.execute(args)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\commands\start.py", line 244, in execute</i></div><div><i> validate_running=validate_running)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1628, in start</i></div><div><i> return self._start(create=create, create_only=create_only)</i></div><div><i> File "<string>", line 2, in _start</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\utils.py", line 112, in wrap_f</i></div><div><i> res = func(*arg, **kargs)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1643, in _start</i></div><div><i> self.create_cluster()</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1163, in create_cluster</i></div><div><i> self._create_flat_rate_cluster()</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 1185, in _create_flat_rate_cluster</i></div><div><i> force_flat=True)[0]</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 926, in create_nodes</i></div><div><i> cluster_sg = <a href="http://self.cluster_group.name" target="_blank">self.cluster_group.name</a></i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 655, in cluster_group</i></div><div><i> vpc_id=vpc_id)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\awsutils.py", line 300, in create_group</i></div><div><i> while not self.get_group_or_none(name):</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\awsutils.py", line 333, in get_group_or_none</i></div><div><i> return self.get_security_group(name)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\awsutils.py", line 357, in get_security_group</i></div><div><i> filters={'group-name': groupname})[0]</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\awsutils.py", line 369, in get_security_groups</i></div><div><i> return self.conn.get_all_security_groups(filters=filters)</i></div><div><i> File "C:\Python27\lib\site-packages\boto\ec2\connection.py", line 2968, in <b>get_all_security_groups</b></i></div><div><i> [('item', SecurityGroup)], verb='POST')</i></div><div><i> File "C:\Python27\lib\site-packages\boto\connection.py", line 1169, in get_list</i></div><div><i> response = self.make_request(action, params, path, verb)</i></div><div><i> File "C:\Python27\lib\site-packages\boto\connection.py", line 1115, in make_request</i></div><div><i> return self._mexe(http_request)</i></div><div><i> File "C:\Python27\lib\site-packages\boto\connection.py", line 1027, in _mexe</i></div><div><i> raise BotoServerError(response.status, response.reason, body)</i></div><div><i>BotoServerError: BotoServerError: 500 Internal Server Error</i></div><div><i><?xml version="1.0" encoding="UTF-8"?></i></div><div><i><Response><Errors><Error><Code>InternalError</Code><Message>An internal error has occurred</Message></Error></Errors><RequestID>808ce646-9203-412f-8fa</i></div><div><i>9-0d994e74e418</RequestID></Response></i></div></div><div><br></div><div>I am also seeing the following error</div><div><div><i>C:\Windows\system32>starcluster listclusters</i></div><div><i>StarCluster - (<a href="http://star.mit.edu/cluster" target="_blank">http://star.mit.edu/cluster</a>) (v. 0.95.6)</i></div><div><i>Software Tools for Academics and Researchers (STAR)</i></div><div><i>Please submit bug reports to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></i></div><div><i><br></i></div><div><i>!!! ERROR - InternalError: An internal error has occurred</i></div><div><i>Traceback (most recent call last):</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cli.py", line 274, in main</i></div><div><i> sc.execute(args)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\commands\listclusters.py", line 36, in execute</i></div><div><i> show_ssh_status=self.opts.show_ssh_status)</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 280, in list_clusters</i></div><div><i> cluster_groups = self.get_cluster_security_groups()</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\cluster.py", line 253, in get_cluster_security_groups</i></div><div><i> sgs = self.ec2.get_security_groups(filters={'group-name': glob})</i></div><div><i> File "C:\Python27\lib\site-packages\starcluster-0.95.6-py2.7.egg\starcluster\awsutils.py", line 369, in get_security_groups</i></div><div><i> return self.conn.get_all_security_groups(filters=filters)</i></div><div><i> File "C:\Python27\lib\site-packages\boto\ec2\connection.py", line 2968, in <b>get_all_security_groups</b></i></div><div><i> [('item', SecurityGroup)], verb='POST')</i></div><div><i> File "C:\Python27\lib\site-packages\boto\connection.py", line 1169, in get_list</i></div><div><i> response = self.make_request(action, params, path, verb)</i></div><div><i> File "C:\Python27\lib\site-packages\boto\connection.py", line 1115, in make_request</i></div><div><i> return self._mexe(http_request)</i></div><div><i> File "C:\Python27\lib\site-packages\boto\connection.py", line 1027, in _mexe</i></div><div><i> raise BotoServerError(response.status, response.reason, body)</i></div><div><i>BotoServerError: BotoServerError: 500 Internal Server Error</i></div><div><i><?xml version="1.0" encoding="UTF-8"?></i></div><div><i><Response><Errors><Error><Code>InternalError</Code><Message>An internal error has occurred</Message></Error></Errors><RequestID>c18d1a11-a6a6-4a8d-a74</i></div><div><i>2-f3f69b593189</RequestID></Response></i></div></div><div><br></div><div>I also got this error recently:</div><div><div><i>C:\Windows\system32>starcluster start babel2</i></div><div><i>StarCluster - (<a href="http://star.mit.edu/cluster" target="_blank">http://star.mit.edu/cluster</a>) (v. 0.95.6)</i></div><div><i>Software Tools for Academics and Researchers (STAR)</i></div><div><i>Please submit bug reports to <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a></i></div><div><br></div></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div><br></div></div></div><br>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div>