[StarCluster] Plugin for mounting disks properly and easily

[David Johnston]

Hi,
I just started using star cluster. It's fantastic.

One issue I am seeing though is that there is some difficulty getting disks
up and mounted in a reasonable place. I was wondering if anyone has written
a plugin for handling this. I have some work-arounds in terms of a quick
and dirty plugin but I don't know how general purpose it would be.

What I would like is the following:

I want to use EBS volumes for everything except for scratch space as I want
to be able to shutdown (to save money) and bring back up just as if it were
non-virtual hardware even with spot instances. I want it to survive (no
persisted data loss) having my spot instances being seized due to a spot
price surge. I believe this is possible. I am wrong?

I would like /home NFS cross-mounted EBS.

I'd like a "restart" command to find the right EBS volumes and mount them
in the usual places. In the case of a fresh "start", I'd like it to create
new EBS volumes. I believe the current EBS volume attachment functionality
(in config file), assumes that they already exists and that you know their
id.

I believe that when you change instance types from say r3.2xlarge to
m3.xlarge, the storage devices (/dev/xdvv etc) change names so this
requires some knowledge of what the devices will be names for each
instance.

Also, I want to avoid over-mounting a device on a directory with files and
subdirectories inside.

I can probably do all of this with a DIY solution but I figured everyone
must be dealing with similar issues. Is anyone working on a general plugin
for this?

I can imagine a simple plugin (similar to an stab file) that looks like
this in the config file
[volumesetup]
/home 100GB cross
/data 100GB cross
/hdfs1 200GB separate
/hdfs2 200GB separate

with the remaining local storage being automatically assigned to /scratch1
/scratch2 etc depending on the number of partitions.

Anyone have any ideas of suggestions?
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20140527/74050026/attachment.htm