Justin,<br><br>I don't use the installer. I just untar the package and set my PATH and PYTHONPATH appropriately. I didn't notice the updated boto requirement (README.rst needs an update). <br>After updating boto, it works fantastically. I tested 25 and then 50 nodes. Only 9 minutes to create a 50-node cluster of small instances. And I verified they are all registered as exec nodes. <br>
So far so good. We'll be giving it more rigorous testing over the next few days.<br><br>Thanks again for your effort in fixing the issue and making it so much faster.<br><br>jeff<br><br><div class="gmail_quote">On Wed, Apr 6, 2011 at 10:49 AM, Justin Riley <span dir="ltr"><<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
</div>Hi Jeff,<br>
<br>
Did you reinstall after pulling the latest code using "python setup.py<br>
install"? If so, what version of boto do you have installed? You can<br>
check this with:<br>
<br>
% python -c 'import boto; print boto.Version'<br>
2.0b4<br>
<br>
The version should be 2.0b4 as above.<br>
<br>
~Justin<br>
<div><div></div><div class="h5"><br>
On 04/06/2011 01:04 PM, Jeff White wrote:<br>
> Hi Justin,<br>
><br>
> Thanks much for your effort on this. I got this error upon running<br>
> 'starcluster -s 25 start jswtest'. I have not altered my config file<br>
> from the one I sent you previously.<br>
><br>
> PID: 5530 config.py:515 - DEBUG - Loading config<br>
> PID: 5530 config.py:108 - DEBUG - Loading file:<br>
> /home/jsw/.starcluster/config<br>
> PID: 5530 config.py:515 - DEBUG - Loading config<br>
> PID: 5530 config.py:108 - DEBUG - Loading file:<br>
> /home/jsw/.starcluster/config<br>
> PID: 5530 awsutils.py:54 - DEBUG - creating self._conn w/<br>
> connection_authenticator kwargs = {'path': '/', 'region': None, 'port':<br>
> None, 'is_secure': True}<br>
> PID: 5530 start.py:167 - INFO - Using default cluster template: smallcluster<br>
> PID: 5530 cluster.py:1310 - INFO - Validating cluster template settings...<br>
> PID: 5530 cli.py:184 - DEBUG - Traceback (most recent call last):<br>
> File "/home/jsw/jtriley-StarCluster-dfba6ef/starcluster/cli.py", line<br>
> 160, in main<br>
> sc.execute(args)<br>
> File<br>
> "/home/jsw/jtriley-StarCluster-dfba6ef/starcluster/commands/start.py",<br>
> line 175, in execute<br>
> scluster._validate(validate_running=validate_running)<br>
> File "/home/jsw/jtriley-StarCluster-dfba6ef/starcluster/cluster.py",<br>
> line 1322, in _validate<br>
> self._validate_instance_types()<br>
> File "/home/jsw/jtriley-StarCluster-dfba6ef/starcluster/cluster.py",<br>
> line 1458, in _validate_instance_types<br>
> self.__check_platform(node_image_id, node_instance_type)<br>
> File "/home/jsw/jtriley-StarCluster-dfba6ef/starcluster/cluster.py",<br>
> line 1419, in __check_platform<br>
> image_is_hvm = (image.virtualization_type == "hvm")<br>
> AttributeError: 'Image' object has no attribute 'virtualization_type'<br>
><br>
> PID: 5530 cli.py:129 - ERROR - Oops! Looks like you've found a bug in<br>
> StarCluster<br>
> PID: 5530 cli.py:130 - ERROR - Debug file written to:<br>
> /tmp/starcluster-debug-jsw.log<br>
> PID: 5530 cli.py:131 - ERROR - Look for lines starting with PID: 5530<br>
> PID: 5530 cli.py:132 - ERROR - Please submit this file, minus any<br>
> private information,<br>
> PID: 5530 cli.py:133 - ERROR - to <a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a><br>
</div></div>> <mailto:<a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a>><br>
<div class="im">><br>
><br>
><br>
> On Wed, Apr 6, 2011 at 8:09 AM, Justin Riley <<a href="mailto:justin.t.riley@gmail.com">justin.t.riley@gmail.com</a><br>
</div><div class="im">> <mailto:<a href="mailto:justin.t.riley@gmail.com">justin.t.riley@gmail.com</a>>> wrote:<br>
><br>
> Jeff/Joseph,<br>
><br>
> Sorry for taking so long to follow up with this but I believe I've<br>
> fixed this issue for good and you should now be able to launch 50+<br>
> node clusters without issue. My original feeling was that the SGE<br>
> install script was at fault, however, after several hours of digging I<br>
> discovered that ssh-keyscan was failing when there were a large number<br>
> of nodes. Long story short this meant that passwordless-ssh wasn't<br>
> being setup fully for all nodes and so the SGE installer script could<br>
> not connect to those nodes to add them to the queue. I found a much<br>
> better way to populate the known_hosts file with all the nodes using<br>
> paramiko instead of ssh-keyscan which is much faster in this case.<br>
><br>
> If you haven't already please re-run 'python setup.py install' after<br>
> pulling the latest code to test out the latest changes. I've also<br>
> updated StarCluster perform the setup on all nodes concurrently using<br>
> a thread pool so you should notice it's much faster for larger<br>
> clusters. Please let me know if you have issues.<br>
><br>
> Thanks,<br>
><br>
> ~Justin<br>
><br>
> On Wed, Mar 16, 2011 at 1:37 PM, Kyeong Soo (Joseph) Kim<br>
</div><div class="im">> <<a href="mailto:kyeongsoo.kim@gmail.com">kyeongsoo.kim@gmail.com</a> <mailto:<a href="mailto:kyeongsoo.kim@gmail.com">kyeongsoo.kim@gmail.com</a>>> wrote:<br>
> > Justin,<br>
> > Please, find attached the said file.<br>
> ><br>
> > Regards,<br>
> > Joseph<br>
> ><br>
> ><br>
> > On Wed, Mar 16, 2011 at 4:38 PM, Justin Riley <<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a><br>
</div><div><div></div><div class="h5">> <mailto:<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>>> wrote:<br>
> Joseph,<br>
><br>
> Great thanks, can you also send me the /opt/sge6/ec2_sge.conf<br>
>> file please?<br>
><br>
> ~Justin<br>
><br>
> On 03/16/2011 12:29 PM, Kyeong Soo (Joseph) Kim wrote:<br>
>> >>> Hi Justin,<br>
>> >>><br>
>> >>> Please, find attached the gzipped tar file of the logfiles under<br>
>> >>> install_logs directory.<br>
>> >>><br>
>> >>> Note that the configuration is for 25-node (1 master and 24<br>
>> slaves) cluster.<br>
>> >>><br>
>> >>> Below is the time-sorted listing of log files under the same<br>
>> directory:<br>
>> >>><br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 13:23<br>
>> >>> execd_install_node024_2011-03-16_13:23:11.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node023_2011-03-16_11:13:37.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node022_2011-03-16_11:13:36.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node021_2011-03-16_11:13:36.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node020_2011-03-16_11:13:32.log<br>
>> >>> -rw-r--r-- 1 kks kks 18K 2011-03-16 11:13<br>
>> >>> execd_install_master_2011-03-16_11:13:10.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node017_2011-03-16_11:13:27.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node018_2011-03-16_11:13:27.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node019_2011-03-16_11:13:28.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node016_2011-03-16_11:13:26.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node014_2011-03-16_11:13:25.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node015_2011-03-16_11:13:26.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node012_2011-03-16_11:13:24.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node013_2011-03-16_11:13:25.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node010_2011-03-16_11:13:23.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node011_2011-03-16_11:13:24.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node008_2011-03-16_11:13:22.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node009_2011-03-16_11:13:22.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node006_2011-03-16_11:13:21.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node007_2011-03-16_11:13:21.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node004_2011-03-16_11:13:20.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node005_2011-03-16_11:13:20.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node003_2011-03-16_11:13:19.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node001_2011-03-16_11:13:18.log<br>
>> >>> -rw-r--r-- 1 kks kks 2.9K 2011-03-16 11:13<br>
>> >>> execd_install_node002_2011-03-16_11:13:19.log<br>
>> >>> -rw-r--r-- 1 kks kks 3.1K 2011-03-16 11:13<br>
>> >>> execd_install_master_2011-03-16_11:13:17.log<br>
>> >>> -rw-r--r-- 1 kks kks 8.4K 2011-03-16 11:13<br>
>> >>> qmaster_install_master_2011-03-16_11:13:05.log<br>
>> >>><br>
>> >>> As you can see, the installation of master has been duplicated<br>
>> and it<br>
>> >>> ended up with master, node001~node023; the top-most log for node024<br>
>> >>> was for the manual addition through "addnode" command later (i.e., 1<br>
>> >>> hour 10 mins after).<br>
>> >>><br>
>> >>> Even with this slimmed down version of configurations (compared<br>
>> to the<br>
>> >>> original 125-node one), the chances that all nodes are properly<br>
>> >>> installed (i.e., 25 out of 25) were about 50% (last night and this<br>
>> >>> morning, I tried it about 10 times to set total five of 25-node<br>
>> >>> clusters).<br>
>> >>><br>
>> >>> Regards,<br>
>> >>> Joseph<br>
>> >>><br>
>> >>><br>
>> >>> On Wed, Mar 16, 2011 at 3:57 PM, Justin Riley <<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a><br>
</div></div><div><div></div><div class="h5">>> <mailto:<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>>> wrote:<br>
>> >>> Hi Jeff/Joseph,<br>
>> >>><br>
>> >>> I just requested to up my EC2 instance limit so that I can test<br>
>> things<br>
>> >>> out at this scale and see what the issue is. In the mean time<br>
>> would you<br>
>> >>> mind sending me any logs found in<br>
>> /opt/sge6/default/common/install_logs<br>
>> >>> and also the /opt/sge6/ec2_sge.conf for a failed run?<br>
>> >>><br>
>> >>> Also if this happens again you could try reinstalling SGE manually<br>
>> >>> assuming all the nodes are up:<br>
>> >>><br>
>> >>> $ starcluster sshmaster mycluster<br>
>> >>> $ cd /opt/sge6<br>
>> >>> $ ./inst_sge -m -x -auto ./ec2_sge.conf<br>
>> >>><br>
>> >>> ~Justin<br>
>> >>><br>
>> >>> On 03/15/2011 06:30 PM, Kyeong Soo (Joseph) Kim wrote:<br>
>> >>>>>> Hi Jeff,<br>
>> >>>>>><br>
>> >>>>>> I experienced the same thing with my 50-node configuration<br>
>> (c1.xlarge).<br>
>> >>>>>> Out of 50 nodes, only 29 nodes are successfully identified by<br>
>> the SGE.<br>
>> >>>>>><br>
>> >>>>>> Regards,<br>
>> >>>>>> Joseph<br>
>> >>>>>><br>
>> >>>>>> On Sat, Mar 5, 2011 at 10:15 PM, Jeff White <<a href="mailto:jeff@decide.com">jeff@decide.com</a><br>
</div></div><div><div></div><div class="h5">>> <mailto:<a href="mailto:jeff@decide.com">jeff@decide.com</a>>> wrote:<br>
>> >>>>>>> I can frequently reproduce an issue where 'starcluster<br>
>> start' completes<br>
>> >>>>>>> without error, but not all nodes are added to the SGE pool,<br>
>> which I verify<br>
>> >>>>>>> by running 'qconf -sel' on the master. The latest example I<br>
>> have is creating<br>
>> >>>>>>> a 25-node cluster, where only the first 12 nodes are<br>
>> successfully installed.<br>
>> >>>>>>> The remaining instances are running and I can ssh to them<br>
>> but they aren't<br>
>> >>>>>>> running sge_execd. There are only install log files for the<br>
>> first 12 nodes<br>
>> >>>>>>> in /opt/sge6/default/common/install_logs. I have not found<br>
>> any clues in the<br>
>> >>>>>>> starcluster debug log or the logs inside master:/opt/sge6/.<br>
>> >>>>>>><br>
>> >>>>>>> I am running starcluster development snapshot 8ef48a3<br>
>> downloaded on<br>
>> >>>>>>> 2011-02-15, with the following relevant settings:<br>
>> >>>>>>><br>
>> >>>>>>> NODE_IMAGE_ID=ami-8cf913e5<br>
>> >>>>>>> NODE_INSTANCE_TYPE = m1.small<br>
>> >>>>>>><br>
>> >>>>>>> I have seen this behavior with the latest 32-bit and 64-bit<br>
>> starcluster<br>
>> >>>>>>> AMIs. Our workaround is to start a small cluster and<br>
>> progressively add nodes<br>
>> >>>>>>> one at a time, which is time-consuming.<br>
>> >>>>>>><br>
>> >>>>>>> Has anyone else noticed this and have a better workaround or<br>
>> an idea for a<br>
>> >>>>>>> fix?<br>
>> >>>>>>><br>
>> >>>>>>> jeff<br>
>> >>>>>>><br>
>> >>>>>>><br>
>> >>>>>>> _______________________________________________<br>
>> >>>>>>> StarCluster mailing list<br>
</div></div>>> >>>>>>> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a> <mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a>><br>
<div class="im">>> >>>>>>> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
>> >>>>>>><br>
>> >>>>>>><br>
>> >>>>>> _______________________________________________<br>
>> >>>>>> StarCluster mailing list<br>
</div>>> >>>>>> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a> <mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a>><br>
<div class="im">>> >>>>>> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
>> >>><br>
>> >>>><br>
><br>
> >><br>
> ><br>
> > _______________________________________________<br>
> > StarCluster mailing list<br>
</div>> > <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a> <mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a>><br>
<div class="im">> > <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
> ><br>
> ><br>
<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2.0.17 (GNU/Linux)<br>
Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
<br>
</div>iEYEARECAAYFAk2cp6sACgkQ4llAkMfDcrkEPQCeKo3XQ8ilWlv89E76NTReqBaz<br>
k68AoIur9985wTYnBKP4+cnKkKwMyL9i<br>
=QBz+<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br>