[StarCluster] Updating public AMIs to have Ubuntu 14.04? Or how to install starcluster on AWS AMI?

Thu Aug 25 09:24:44 EDT 2016

Hi Mike,

This is a very specific use case and I'm not sure if it will add additional
functionality beyond the existing ebsimage command. It would definitely be
great if we could prune the list of broken AMIs in listpublic and add 14.04
and 16.04. I have a working image in eu-central for 16.04  that I'm happy
to share if anyone's interested.

Cheers,
Vasisht

On Thu, Aug 25, 2016 at 12:14 AM, Mike Cariaso <mike.cariaso at keygene.com>
wrote:

>
> https://github.com/datacratic/StarCluster/issues/56
>
>
> My immediate need is ubuntu14 or 16 amis in non-us-east-1 regions, and I'm
> sure I'm not the only one. The current situation with broken ubuntu13 amis
> still present in listpublic is definitely suboptimal.
>
> So I'm thinking about introducing a new command into starcluster which
> will allow it to make it's own amis. This might also allow you to 'bake' an
> ami after all of your plugin installs have completed, to reduce setup time
> for new worker nodes.
>
> it would largely follow the steps from
> http://star.mit.edu/cluster/mlarchives/2545.html
> and would probably have a command line interface similar to
>
> starcluster makeami --baseami ami-ed82e39e
>
> obviously a new command is a more dramatic change than a new plugin. So
> before I jump in I thought I'd see if others would weigh in.
>
> Is it useful? or perhaps impractical for reasons I'm overlooking?
>
>
>
>
>
> *Michael Cariaso *
> <michael.cariaso at keygene.com>
> *Bioinformatician <http://www.keygene.com>*
> ------------------------------
> *From:* starcluster-bounces at mit.edu <starcluster-bounces at mit.edu> on
> behalf of Avner May <avnermay at cs.columbia.edu>
> *Sent:* Saturday, January 30, 2016 1:58 AM
> *To:* Ed Gray
> *Cc:* starcluster at mit.edu; Don Morton
> *Subject:* Re: [StarCluster] Updating public AMIs to have Ubuntu 14.04?
> Or how to install starcluster on AWS AMI?
>
> I just wanted to give one final request for there being public Ubuntu
> 14.04 StarCluster AMIs (EBS and HVM-EBS), which are listed in the output of
> the "starcluster listpublic", so that other people don't go through the
> painful experience I had to go through.
>
> What is the process for adding a new public AMI to StarCluster?
>
> Sergio Mafra has pointed out to me that ami-38b99850 is available and has
> StarCluster installed on Ubuntu 14.04.
>
> On Thu, Jan 28, 2016 at 9:34 PM, Ed Gray <gray_ed at hotmail.com> wrote:
>
>> Having worked with Jennifer, I can tell you she's a star.
>>
>>
>>
>> Sent from my Smartphone; please forgive any spelling and grammar mishaps.
>>
>>
>> -------- Original message --------
>> From: Avner May <avnermay at cs.columbia.edu>
>> Date: 01/28/2016 6:13 PM (GMT-05:00)
>> To: Don Morton <don.morton at borealscicomp.com>, Jennifer Staab <
>> jstaab at cs.unc.edu>
>> Cc: starcluster at mit.edu
>> Subject: Re: [StarCluster] Updating public AMIs to have Ubuntu 14.04? Or
>> how to install starcluster on AWS AMI?
>>
>> Hi all,
>>
>> I have carefully followed Jennifer Staab's instructions here
>> <http://star.mit.edu/cluster/mlarchives/2545.html>, and things are
>> working great (*** THANK YOU SO MUCH JENNIFER!! ***).  One small hiccup I
>> ran into is that at that link, one of the commands is long and is split
>> into 2 lines ('apt-get install nfs-kernel-server nfs-common rpcbind
>> libgssglue1 libnfsidmap2 libtirpc1 -y") -- if this is copy-pasted as 2
>> lines, it will fail quietly when instance is launched, and it eventually
>> leads to an error starting NFS on the master (see error message below).
>>
>> I also followed the instructions here
>> <https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)>
>> in order to install NVIDIA drivers and CUDA (though I installed the latest
>> versions, not the ones included in that post).
>>
>> I would recommend including a modified version of Jennifer's instructions
>> directly linked from the primary StarCluster website in order to explain
>> how to set-up a non-StarCluster AMI to work with StarCluster.  This is
>> super valuable information.
>>
>> Now everything is working!!!
>> Avner
>>
>> ========================
>> Starting NFS server on master
>> !!! ERROR - Error occured while running plugin 'starcluster.clustersetup.
>> DefaultClusterSetup':
>> !!! ERROR - remote command 'source /etc/profile && /etc/init.d/nfs
>> !!! ERROR - start' failed with status 127:
>> !!! ERROR - bash: /etc/init.d/nfs: No such file or directory
>>
>> On Thu, Jan 28, 2016 at 3:01 PM, Don Morton <don.morton at borealscicomp.com
>> > wrote:
>>
>> Hello,
>>
>> I faced this same issue a few months ago, and somehow succeeded after
>> bumping into lots of walls (most of it based on my own ignorance).  I
>> haven't done anything with this since, and these notes are extremely rough
>> (hopefully I didn't cuss too much!), but you are welcome to use what you
>> need.  Maybe someone will have the time to make something more formal, but
>> I'm afraid it won't be me for some time now.
>>
>> All the best,
>>
>> Don
>>
>> ---
>> Don Morton, Owner/Manager
>> Boreal Scientific Computing LLC
>> Fairbanks, Alaska USA
>> http://www.borealscicomp.com/
>> http://www.borealscicomp.com/Miscellaneous/MortonBio/
>>
>> On Thu, Jan 28, 2016 at 7:42 PM, Avner May <avnermay at cs.columbia.edu>
>> wrote:
>>
>> I see that this has already been asked several times:
>> http://star.mit.edu/cluster/mlarchives/2690.html,
>> http://star.mit.edu/cluster/mlarchives/2545.html, https://
>> www.youtube.com/watch?v=2RBupgpi_ec, etc.
>>
>> Which of these guides should I use?  Why hasn't an Ubuntu 14.04 AMI been
>> made public by the StarCluster development team yet?  Are there plans to do
>> this?  Meanwhile, could someone make their AMI public?
>>
>> It seems there is widespread interest in updated AMIs (EBS and HVM-EBS
>> versions).
>>
>> Thanks so much,
>> Avner
>>
>> On Thu, Jan 28, 2016 at 2:30 PM, Avner May <avnermay at cs.columbia.edu>
>> wrote:
>>
>> Hi,
>>
>> My goal is to have a StarCluster AMI with Ubuntu 14.04, CUDA 7.5, and the
>> latest NVIDIA drivers (I want to use g2.2xlarge instance type).
>>
>> I have noticed that the public StarCluster AMIs are pretty outdated, with
>> the most up-to-date AMIs using Ubuntu 13.04, which has already reached its "end
>> of life
>> <http://fridge.ubuntu.com/2014/01/28/ubuntu-13-04-raring-ringtail-end-of-life-reached-on-january-27-2014/>".
>> Could an Ubuntu 14.04 + 64-bit + HVM-EBS public AMI be created, with CUDA
>> 7.5 and the latest NVIDIA driver
>> <http://www.nvidia.com/download/driverResults.aspx/97645/en-us> installed?
>> This would be incredibly useful.  I am having a very hard time
>> upgrading ami-6b211202 for my needs (I have run into issues updating
>> ubuntu, as well as installing NVIDIA drivers and CUDA).
>>
>> Here
>> <http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/>
>> are instructions on how to install NVIDIA drivers and CUDA on Ubuntu 14.04
>> AMI (ami-9eaa1cf6).  So another option which would satisfy my needs would
>> be if you simply gave me instructions on how to install StarCluster on this
>> AMI.
>>
>> Any advice on the easiest way to get a working StarCluster AMI, with an
>> up to date version of Ubuntu, would be great.
>>
>> Thanks!
>> Avner
>>
>>
>>
>>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>>
>>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>

-- 
Pour trouver les limites du possible il faut tenter l'impossible.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20160825/944a7cbb/attachment-0001.html