<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><br>
</p>
<p><a href="https://github.com/datacratic/StarCluster/issues/56">https://github.com/datacratic/StarCluster/issues/56</a></p>
<p><br>
</p>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
My immediate need is ubuntu14 or 16 amis in non-us-east-1 regions, and I'm sure I'm not the only one. The current situation with broken ubuntu13 amis still present in listpublic is definitely suboptimal.</p>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
So I'm thinking about introducing a new command into starcluster which will allow it to make it's own amis. This might also allow you to 'bake' an ami after all of your plugin installs have completed, to reduce setup time for new worker nodes.</p>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
it would largely follow the steps from<br style="box-sizing: border-box;">
<a href="http://star.mit.edu/cluster/mlarchives/2545.html" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">http://star.mit.edu/cluster/mlarchives/2545.html</a><br style="box-sizing: border-box;">
and would probably have a command line interface similar to</p>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
starcluster makeami --baseami ami-ed82e39e</p>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
obviously a new command is a more dramatic change than a new plugin. So before I jump in I thought I'd see if others would weigh in.</p>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 0px !important; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
Is it useful? or perhaps impractical for reasons I'm overlooking?</p>
<p><br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font color="000000"><br>
</font>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font color="000000"></font>
<div style="font-family:Tahoma; font-size:13px"><span lang="en-US"><font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" size="2"><span style="font-size:10pt"><b><font face="Arial,Helvetica,sans-serif">Michael Cariaso</font><br>
</b></span></font></span></font></div>
<font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"><span lang="en-US"><font color="000000"></font>
<div style="margin:0"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font color="000000"><a id="NoLP" href="mailto:michael.cariaso@keygene.com" tabindex="0"><font face="Consolas" size="2"><span style="font-size:10pt"><span lang="en-US"></span></span></font></a></font><font face="Consolas" size="2"><font color="000000"></font>
<div style="margin:0"><i><font face="Arial,Helvetica,sans-serif"><font color="000000" size="2"><span style="font-size:11pt"><font size="2"><span style="font-size:10pt">Bioinformatician</span></font></span></font><font size="2"><span style="font-size:11pt"></span></font><font size="2"><a id="NoLP" href="http://www.keygene.com" tabindex="0"><font size="2"><span style="font-size:10pt"></span></font></a><font color="#7F7F7F" size="2"><span style="font-size:10pt"></span></font></font></font></i>
</div>
</font></span><font face="Consolas" size="2"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"></span></font></span></font></font></font></div>
</span></span></font></span></font></div>
</span><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><font face="Consolas" color="#7F7F7F" size="2"><span style="font-size:10pt"><i></i></span></font></span></font><span lang="en-US"></span></div>
</div>
</div>
</div>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> starcluster-bounces@mit.edu <starcluster-bounces@mit.edu> on behalf of Avner May <avnermay@cs.columbia.edu><br>
<b>Sent:</b> Saturday, January 30, 2016 1:58 AM<br>
<b>To:</b> Ed Gray<br>
<b>Cc:</b> starcluster@mit.edu; Don Morton<br>
<b>Subject:</b> Re: [StarCluster] Updating public AMIs to have Ubuntu 14.04? Or how to install starcluster on AWS AMI?</font>
<div> </div>
</div>
<div>
<div dir="ltr">I just wanted to give one final request for there being public Ubuntu 14.04 StarCluster AMIs (EBS and HVM-EBS), which are listed in the output of the "starcluster listpublic", so that other people don't go through the painful experience I had
to go through.
<div><br>
</div>
<div>What is the process for adding a new public AMI to StarCluster?<br>
<div><br>
</div>
<div>Sergio Mafra has pointed out to me that ami-38b99850 is available and has StarCluster installed on Ubuntu 14.04.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jan 28, 2016 at 9:34 PM, Ed Gray <span dir="ltr">
<<a href="mailto:gray_ed@hotmail.com" target="_blank">gray_ed@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div>
<div>Having worked with Jennifer, I can tell you she's a star.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div style="font-size:10px">Sent from my Smartphone; please forgive any spelling and grammar mishaps.</div>
</div>
<div>
<div><br>
<br>
-------- Original message --------<br>
From: Avner May <<a href="mailto:avnermay@cs.columbia.edu" target="_blank">avnermay@cs.columbia.edu</a>>
<br>
Date: 01/28/2016 6:13 PM (GMT-05:00) <br>
To: Don Morton <<a href="mailto:don.morton@borealscicomp.com" target="_blank">don.morton@borealscicomp.com</a>>, Jennifer Staab <<a href="mailto:jstaab@cs.unc.edu" target="_blank">jstaab@cs.unc.edu</a>>
<br>
Cc: <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a> <br>
Subject: Re: [StarCluster] Updating public AMIs to have Ubuntu 14.04? Or how to install starcluster on AWS AMI?
<br>
<br>
<div>
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I have carefully followed Jennifer Staab's instructions <a href="http://star.mit.edu/cluster/mlarchives/2545.html" target="_blank">
here</a>, and things are working great (*** THANK YOU SO MUCH JENNIFER!! ***). One small hiccup I ran into is that at that link, one of the commands is long and is split into 2 lines ('apt-get install nfs-kernel-server nfs-common rpcbind libgssglue1 libnfsidmap2
libtirpc1 -y") -- if this is copy-pasted as 2 lines, it will fail quietly when instance is launched, and it eventually leads to an error starting NFS on the master (see error message below).</div>
<div><br>
</div>
<div>I also followed the instructions <a href="https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)" target="_blank">
here</a> in order to install NVIDIA drivers and CUDA (though I installed the latest versions, not the ones included in that post).</div>
<div><br>
</div>
<div>I would recommend including a modified version of Jennifer's instructions directly linked from the primary StarCluster website in order to explain how to set-up a non-StarCluster AMI to work with StarCluster. This is super valuable information.</div>
<div><br>
</div>
<div>Now everything is working!!!</div>
<div>Avner</div>
<div><br>
</div>
<div>========================</div>
<div>
<div>Starting NFS server on master</div>
<div>!!! ERROR - Error occured while running plugin 'starcluster.clustersetup.DefaultClusterSetup':</div>
<div>!!! ERROR - remote command 'source /etc/profile && /etc/init.d/nfs</div>
<div>!!! ERROR - start' failed with status 127:</div>
<div>!!! ERROR - bash: /etc/init.d/nfs: No such file or directory</div>
</div>
</div>
<div><br>
<div>On Thu, Jan 28, 2016 at 3:01 PM, Don Morton <span dir="ltr"><<a href="mailto:don.morton@borealscicomp.com" target="_blank">don.morton@borealscicomp.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">Hello,
<div><br>
</div>
<div>I faced this same issue a few months ago, and somehow succeeded after bumping into lots of walls (most of it based on my own ignorance). I haven't done anything with this since, and these notes are extremely rough (hopefully I didn't cuss too much!),
but you are welcome to use what you need. Maybe someone will have the time to make something more formal, but I'm afraid it won't be me for some time now.</div>
<div><br>
</div>
<div>All the best,</div>
<div><br>
</div>
<div>Don</div>
</div>
<div><br clear="all">
<div>
<div>
<div dir="ltr">---
<div>Don Morton, Owner/Manager</div>
<div>Boreal Scientific Computing LLC</div>
<div>Fairbanks, Alaska USA</div>
<div><a href="http://www.borealscicomp.com/" target="_blank">http://www.borealscicomp.com/</a></div>
<div><a href="http://www.borealscicomp.com/Miscellaneous/MortonBio/" target="_blank">http://www.borealscicomp.com/Miscellaneous/MortonBio/</a></div>
</div>
</div>
</div>
<br>
<div>
<div>
<div>On Thu, Jan 28, 2016 at 7:42 PM, Avner May <span dir="ltr"><<a href="mailto:avnermay@cs.columbia.edu" target="_blank">avnermay@cs.columbia.edu</a>></span> wrote:<br>
</div>
</div>
<blockquote style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div>
<div>
<div dir="ltr">I see that this has already been asked several times: <a href="http://star.mit.edu/cluster/mlarchives/2690.html" target="_blank">http://star.mit.edu/cluster/mlarchives/2690.html</a>, <a href="http://star.mit.edu/cluster/mlarchives/2545.html" target="_blank">http://star.mit.edu/cluster/mlarchives/2545.html</a>, <a href="https://www.youtube.com/watch?v=2RBupgpi_ec" target="_blank">https://www.youtube.com/watch?v=2RBupgpi_ec</a>,
etc.
<div><br>
</div>
<div>Which of these guides should I use? Why hasn't an Ubuntu 14.04 AMI been made public by the StarCluster development team yet? Are there plans to do this? Meanwhile, could someone make their AMI public?</div>
<div><br>
</div>
<div>It seems there is widespread interest in updated AMIs (EBS and HVM-EBS versions).</div>
<div><br>
</div>
<div>Thanks so much,</div>
<div>Avner</div>
<div>
<div>
<div><br>
<div>On Thu, Jan 28, 2016 at 2:30 PM, Avner May <span dir="ltr"><<a href="mailto:avnermay@cs.columbia.edu" target="_blank">avnermay@cs.columbia.edu</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>My goal is to have a StarCluster AMI with Ubuntu 14.04, CUDA 7.5, and the latest NVIDIA drivers (I want to use g2.2xlarge instance type).<br>
<div><br>
</div>
<div>I have noticed that the public StarCluster AMIs are pretty outdated, with the most up-to-date AMIs using Ubuntu 13.04, which has already reached its "<a href="http://fridge.ubuntu.com/2014/01/28/ubuntu-13-04-raring-ringtail-end-of-life-reached-on-january-27-2014/" target="_blank">end
of life</a>". Could an Ubuntu 14.04 + 64-bit + HVM-EBS public AMI be created, with CUDA 7.5 and the latest
<a href="http://www.nvidia.com/download/driverResults.aspx/97645/en-us" target="_blank">
NVIDIA driver</a> installed? This would be incredibly useful. I am having a very hard time upgrading ami-6b211202 for my needs (I have run into issues updating ubuntu, as well as installing NVIDIA drivers and CUDA).</div>
<div><br>
</div>
<div><a href="http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/" target="_blank">Here</a> are instructions on how to install NVIDIA drivers and CUDA on Ubuntu 14.04 AMI (ami-9eaa1cf6). So another option which
would satisfy my needs would be if you simply gave me instructions on how to install StarCluster on this AMI.</div>
<div><br>
</div>
<div>Any advice on the easiest way to get a working StarCluster AMI, with an up to date version of Ubuntu, would be great.<br>
</div>
<div><br>
</div>
<div>Thanks!<span><font color="#888888"><br>
Avner</font></span></div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" rel="noreferrer" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>