[StarCluster] Creating a StarCluster AMI

[Pierre Mariani]

> Hi All,
>
> I need to generate an ami that comply with the company golden image and
> would like it to be used in a star cluster , SGE environment.
>
> Are there any instructions as for how to bake a starcluster ami rather than
> using those that are there ?
>
> What is the minimum packages and configuration that is required to be
> installed on an image so it can be used as a Starcluster node ?

Disclosure: I am a recent user of StarCluster and may be missing
something important.

I went through the same issue very recently, and this is what I found out:

The most relevant documentation I was able to find is
http://star.mit.edu/cluster/docs/0.95.5/manual/create_new_ami.html. It
is helpful for some aspects, but unfortunately it doesn't go into
details of what should be installed on the AMI.

So I ended up reading StarCluster code at
https://github.com/jtriley/StarCluster/blob/0.95.5/starcluster/. This
is helpful for getting an idea of which packages the framework is
expecting to find on the AMI, although an actual list would be nicer
to work with.

But, a more serious issue is that the code is making some low level
assumptions about these packages.

Let's take the example of NFS. The code is assuming that NFS is being
started with /etc/init.d/nfs start, and /etc/init.d/portmap start.
In the case of Ubuntu, it was correct up to 13.04. But in 14.04, it's
different. These files don't exist anymore. There are new files that
seem to replace them, but I don't know if there are more fundamental
changes beside just the file name.

I tried to work around this particular NFS issue by customizing the
AMI and symlinking the actual NFS files to what StarCluster was
expecting, but I bumped into another issue right after that.

At that point I felt I was on a wild goose chase and decided to fork,
update and maintain my own version of the Ubuntu 12.04 AMI and work
with that. (13.04 has reached EOL, 14.04 isn't officially supported
yet, so 12.04 is best choice available).

I don't know what StarCluster contributors' opinions are on this.
Personally, I think that it is unmaintainable to manually handle the
many different ways to interface with NFS, package managers and other
required packages across multiple distributions and multiple versions
of each distribution. Ideally StarCluster could leverage a 3rd party
library that did that, but I don't know if such a library exists.

I would be very interested in reading the experiences and opinions of
other members of the community on this.

Thank you.
Pierre