<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">Folks:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">First: any good way to search the archives? I tried various google strings to no good effect. I hate to duplicate effort/messages …<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">More importantly: A possible bug? Sometimes when starting SPOT_BID clusters (~30% of the time?) I'm seeing ‘start’ skip (apparently) “Waiting for open spot requests to become active…” and just process the master. When it works correctly,
I see:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching node001 (ami: ami-12b6477b, type: cc1.4xlarge)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SpotInstanceRequest:sir-9f38a214<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching node002 (ami: ami-12b6477b, type: cc1.4xlarge)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SpotInstanceRequest:sir-c4505a11<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching node003 (ami: ami-12b6477b, type: cc1.4xlarge)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SpotInstanceRequest:sir-cbb32414<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Waiting for cluster to come up... (updating every 20s)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0;background:yellow;mso-highlight:yellow">>>> Waiting for open spot requests to become active...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0;background:yellow;mso-highlight:yellow">0/3 | | 0%</span><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">
<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">When it doesn’t work correctly, I see the following, where it skips the highlighted section above and goes straight to ‘Waiting for all nodes’, and the count is /1 instead of /4 (or whatever the CLUSTER_SIZE is).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"># starcluster start -c spottest spottest<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">Software Tools for Academics and Researchers (STAR)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">Please submit bug reports to starcluster@mit.edu<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Validating cluster template settings...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Cluster template settings are valid<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Starting cluster...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching a 4-node cluster...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching master node (ami: ami-12b6477b, type: cc1.4xlarge)...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Creating security group @sc-spottest...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Opening tcp port range 22-22 for CIDR XXXXXXXXXX/22<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Creating placement group @sc-spottest...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">Reservation:r-02fbac61<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching node001 (ami: ami-12b6477b, type: cc1.4xlarge)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SpotInstanceRequest:sir-6cb0f014<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching node002 (ami: ami-12b6477b, type: cc1.4xlarge)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SpotInstanceRequest:sir-b0ff9e11<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Launching node003 (ami: ami-12b6477b, type: cc1.4xlarge)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SpotInstanceRequest:sir-2ef6f814<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Waiting for cluster to come up... (updating every 20s)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Waiting for all nodes to be in a 'running' state...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Waiting for SSH to come up on all nodes...<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> Waiting for cluster to come up took 3.547 mins<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">>>> The master node is ec2-184-72-156-11.compute-1.amazonaws.com<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I haven’t tried this with anything but ‘bigger’ stuff (cc1 & cc2), so don’t know if that has any bearing on the situation. My config:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[global]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">DEFAULT_TEMPLATE=Rcluster<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">ENABLE_EXPERIMENTAL=True<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">REFRESH_INTERVAL=20<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[aws info]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">AWS_ACCESS_KEY_ID = XXXXXXXXXXXX<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">AWS_SECRET_ACCESS_KEY = XXXXXXXXXXXXX<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">AWS_USER_ID = XXXXXXXXXXX<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">EC2_CERT = XXXXXXXXXXX.pem<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">EC2_PRIVATE_KEY = XXXXXXXXXXXXX.pem<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[key mykey]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">KEY_LOCATION=XXXXXXXXXXXXXXX.pem<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[cluster spottest]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">KEYNAME = mykey<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">CLUSTER_SIZE = 4<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">CLUSTER_USER = sgeadmin<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">CLUSTER_SHELL = bash<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">NODE_IMAGE_ID = ami-12b6477b<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">NODE_INSTANCE_TYPE = cc1.4xlarge<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">AVAILABILITY_ZONE = us-east-1c<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">VOLUMES = Rlocal-spottest<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">PLUGINS = setup-centos<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">PERMISSIONS = ssh-local<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">SPOT_BID = 1.50<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[volume Rlocal-spottest]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">VOLUME_ID = vol-XXXXXXXXXX<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">MOUNT_PATH = /usr/local<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[plugin setup-centos]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">setup_class = setup-centos.PackageInstaller<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">pkg_to_install = R<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">[permission ssh-local]<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">protocol = tcp<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">from_port = 22<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">to_port = 22<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:"Courier New";color:#7030A0">cidr_ip = XXXXXXXXXXX/22<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">This exact config works sometimes, other times not. Thanks for listening, or any advice you might have.<o:p></o:p></p>
<p class="MsoPlainText">-Hugh<o:p></o:p></p>
</div>
</body>
</html>