[StarCluster] Little question

Tue Jun 26 13:16:35 EDT 2012

Hi Sergio,

Your welcome.

One of the things that you might want to think about is automating the
build of your starcluster node.  That way you have documentation on just
what is in it as well as an easy way to rebuild.  There is always a danger
in depending upon a hand crafted image that has become a 'work of art'.

There are several tools that can help you to do this.  The two prime examples
are puppet and chef.  However, we did not go that way.  Both tools look like
they take quite awhile to learn and they do more than what I was looking for.

So we wrote a small tool in Python using boto and fabric that reads a json
configuration script.  The challenge is not the functionality, but making it
robust with all the variable timings that arises from manipulating
images in EC2.

Finally, I'd highly recommend the book "Continuous Delivery"  by Jez Humble
and David Farley.  Bottom line message is "if it moves, automate it" (with the
corollary that everything is in revision control).

Anyway, this is just food for thought.  Hope you find it useful.

Regards,

Don

On Tue, Jun 26, 2012 at 7:43 AM, Sergio Mafra <sergiohmafra at gmail.com> wrote:
> Hi Don,
>
> Thanks for the info and what a relief! It´s been done a lot of job on the
> cluster itself and losing everything would be a waste of precious time. I
> was afraid of StarCluster restriction on that based on somebody´s blog that
> tells you should start fresh (not in my case).
>
> best regards,
>
> Sergio
>
> Extract from the blog...
>
> Start up an instance of the type you want to customize
>
> For large instances, I start with the x86_64 AMI on the StarCluster home
> page. That’s currently ami-999d49f0. I start it using the starcluster
> command line, from my local machine:
>
> %% starcluster start -o -s 1 -i m1.large -n ami-999d49f0 customize_me
>
> Then, once it’s started, I log in and install software, configure boot, etc.
> (See Reusing course materials for an example.) A few notes –
>
> don’t put anything in /root that’s not in a hidden (dot) directory; the
> script for building a new AMI will erase everything in that directory.
> don’t put anything under /mnt if you want it saved as part of the AMI, as
> this will not get baked into the new AMI.
> don’t mount any EBS volumes as they, too, will get baked into the new AMI.
>
> Generally I put things under /usr/local/src (if it’s a program) or elsewhere
> under /usr if not. These will get saved as part of the new AMI.
>
> Once you’ve customized things so that everything starts up on boot as you
> want – TEST THIS!! – tell starcluster to make a new EBS image:
>
> %% starcluster ebsimage i-9999999 my-new-image
>
> Here you should name it something other than ‘my-new-image’. You can figure
> out the correct instance ID by logging into the AWS console and looking, or
> by doing
>
> %% starcluster listclusters
>
> and voila, it will show up under your AMIs!
>
>
> On Mon, Jun 25, 2012 at 10:15 PM, Don MacMillen <macd at nimbic.com> wrote:
>>
>> HI Sergio,
>>
>> Yes, of course that should work just fine.  In fact you can if you wish,
>> build
>> from the bare Ubuntu server images based on the recipe posted on the
>> starcluster site along with your own software, but taking a snapshot is
>> much easier.  We build from the bare server images so I can look
>> customer's
>> in the eye and tell them we know _exactly_ what is on the image but we
>> started out taking snapshots of the images provided by Justin.
>>
>> Regards,
>>
>> Don
>>
>> On Mon, Jun 25, 2012 at 5:21 PM, Sergio Mafra <sergiohmafra at gmail.com>
>> wrote:
>> > C'mon friends.
>> >
>> > Need some help here.
>> > Simple question: Building an image from the current instance that I'm
>> > using
>> > in the StarCluster will work or not as a golden image for next clusters
>> > configuration?
>> >
>> > Best Regards,
>> >
>> > Sergio
>> >
>> >
>> > Em quarta-feira, 13 de junho de 2012, Sergio Mafra escreveu:
>> >>
>> >> Hi fellows,
>> >>
>> >> After lauching the cluster using StarCluster, we did some custom
>> >> installation on that. Now we want to make it our customized AMI since
>> >> we
>> >> will need it as base for adding and removing nodes as the load of
>> >> cluster
>> >> asks for.
>> >> If I build an image of this current AMI using AWS Console, terminate
>> >> the
>> >> current cluster and relaunch it using the new customized AMI... will it
>> >> work?
>> >>
>> >> All the best and sorry for this newbie questions..
>> >>
>> >> Sergio
>> >>
>> >>
>> >> On Tue, Jun 12, 2012 at 9:29 AM, Sergio Mafra <sergiohmafra at gmail.com>
>> >> wrote:
>> >>>
>> >>> Thanks.
>> >>>
>> >>> Em segunda-feira, 11 de junho de 2012, Ron Chen escreveu:
>> >>>
>> >>>> Basically, I think the idea is that StarCluster users can do
>> >>>> everything
>> >>>> with the StarCluster command line, with no dependencies on the AWS
>> >>>> Console.
>> >>>>
>> >>>>  -Ron
>> >>>>
>> >>>>
>> >>>>
>> >>>> ----- Original Message -----
>> >>>> From: Dustin Machi <dmachi at vbi.vt.edu>
>> >>>> To: Sergio Mafra <sergiohmafra at gmail.com>
>> >>>> Cc: starcluster at mit.edu
>> >>>> Sent: Monday, June 11, 2012 2:39 PM
>> >>>> Subject: Re: [StarCluster] Little question
>> >>>>
>> >>>> It is a multistep process, so the star cluster command just makes it
>> >>>> convenient.
>> >>>>
>> >>>> Dustin
>> >>>>
>> >>>> On Jun 11, 2012, at 1:12 PM, Sergio Mafra wrote:
>> >>>>
>> >>>> > Dear Friends,
>> >>>> >
>> >>>> > We will need to have a AMI customized for our particular needs.
>> >>>> > Reading the StarCluster docs a doubt arises...What is the
>> >>>> > difference of
>> >>>> > preparing a customized AMI using command "starcluster ebsimage
>> >>>> > i-999999
>> >>>> > mynewAMI" or create image from AWS Console?
>> >>>> >
>> >>>> > All the best,
>> >>>> > Sergio
>> >>>> > _______________________________________________
>> >>>> > StarCluster mailing list
>> >>>> > StarCluster at mit.edu
>> >>>> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> StarCluster mailing list
>> >>>> StarCluster at mit.edu
>> >>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>> >>>>
>> >>
>> >
>> > _______________________________________________
>> > StarCluster mailing list
>> > StarCluster at mit.edu
>> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >
>
>