[StarCluster] EBS storage does not mount <newbie>

Thu Aug 23 15:11:07 EDT 2012

I am a newbie and am just getting my feet wet on StarCluster and AWS

I have successfully installed 0.93.3 on Windows and associated it with my
AWS login. I can create clusters and I have used the documented way of
created an EBS store for use by my application.

My problem is that none of the instances in the cluster will mount the EBS
and the master complains about it having more than one partition. It was
created according to the method here:
http://web.mit.edu/star/cluster/docs/latest/manual/volumes.html#create-and-format-a-new-ebs-volume–
and to the best of my knowledge correctly so. I think I am doing all
the
things I am supposed to do, but obviously I am missing something. The
configuration file is attached but it does not seem to be causing
StarCluster any problems.

The runtime output is  however not so great:

C:\Windows\System32>starcluster start imagingcluster

StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)

Software Tools for Academics and Researchers (STAR)

Please submit bug reports to starcluster at mit.edu

>>> Using default cluster template: smallcluster

>>> Validating cluster template settings...

>>> Cluster template settings are valid

>>> Starting cluster...

>>> Launching a 3-node cluster...

>>> Creating security group @sc-imagingcluster...

Reservation:r-953beddd

>>> Waiting for cluster to come up... (updating every 30s)

>>> Waiting for all nodes to be in a 'running' state...

3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Waiting for SSH to come up on all nodes...

3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Waiting for cluster to come up took 1.383 mins

>>> The master node is ec2-46-137-154-117.eu-west-1.compute.amazonaws.com

>>> Setting up the cluster...

>>> Attaching volume vol-ae98c1c6 to master node on /dev/sdz ...

>>> Configuring hostnames...

3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

!!! ERROR - volume has more than one partition, please specify which
partition t

o use (e.g. partition=0, partition=1, etc.) in the volume's config

>>> Creating cluster user: None (uid: 1001, gid: 1001)

3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Configuring scratch space for user(s): qadmin

3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Configuring /etc/hosts on each node

3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Starting NFS server on master

>>> Configuring NFS exports path(s):

/home /mr_data

>>> Mounting all NFS export path(s) on 2 worker node(s)

!!! ERROR - command 'mount /mr_data' failed with status 32             |
0%

!!! ERROR - command 'mount /mr_data' failed with status 32

2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Setting up NFS took 0.145 mins

>>> Configuring passwordless ssh for root

>>> Configuring passwordless ssh for qadmin

>>> Shutting down threads...

20/20 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%

>>> Configuring cluster took 0.889 mins

>>> Starting cluster took 2.375 mins

A clue could be that when I sshmaster in, the /dev looks very odd as sda
etc. are missing:

root at master:~# ls -l /dev/s*

lrwxrwxrwx 1 root root       8 2012-08-23 18:44  /dev/shm->/run/shm

crw------- 1 root root 10, 231 2012-08-23 18:44  /dev/snapshot

lrwxrwxrwx 1 root root      15 2012-08-23 18:44 /dev/stderr->
/proc/self/fd/2

lrwxrwxrwx 1 root root      15 2012-08-23 18:44 /dev/stdin->/proc/self/fd/0

lrwxrwxrwx 1 root root      15 2012-08-23 18:44 /dev/stdout ->
proc/self/fd/1

However is seems that the disks are not sda1/sdz but xvd1 and xvdz:

root at master:~# ls -l /dev/x*

ls -l /dev/x*

brw-rw---- 1 root disk 202,   1 2012-08-23 18:44 /dev/xvda1

brw-rw---- 1 root disk 202, 400 2012-08-23 18:45 /dev/xvdz

Would that trip up StarCluster?

BTW, does anybody know why the output from the ssh session via Python SSH
are all filled by garbage control characters? The outputs above I had to
clean manually for it to be readable) – remember: total newbie.

All and any help is much appreciated.

Regards,

Anders Nielsen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20120823/4688b8c9/attachment-0001.htm
-------------- next part --------------
####################################
## StarCluster Configuration File ##
####################################
[global]
# Configure the default cluster template to use when starting a cluster
# defaults to 'smallcluster' defined below. This template should be usable
# out-of-the-box provided you've configured your keypair correctly
DEFAULT_TEMPLATE=smallcluster

# enable experimental features for this release
#ENABLE_EXPERIMENTAL=True

# number of seconds to wait when polling instances (default: 30s)
#REFRESH_INTERVAL=15

# specify a web browser to launch when viewing spot history plots
#WEB_BROWSER=chromium

# split the config into multiple files
#INCLUDE=~/.starcluster/aws, ~/.starcluster/keys, ~/.starcluster/vols

#############################################
## AWS Credentials and Connection Settings ##
#############################################
[aws info]
# This is the AWS credentials section (required).
# These settings apply to all clusters
# replace these with your AWS keys
AWS_ACCESS_KEY_ID = deleted
AWS_SECRET_ACCESS_KEY = deleted

# replace this with your account number
AWS_USER_ID= deleted

# Uncomment to specify a different Amazon AWS region  (OPTIONAL)
# (defaults to us-east-1 if not specified)
# NOTE: AMIs have to be migrated!
AWS_REGION_NAME = eu-west-1
AWS_REGION_HOST = ec2.eu-west-1.amazonaws.com

# Uncomment these settings when creating an instance-store (S3) AMI (OPTIONAL)
#EC2_CERT = /path/to/your/cert-asdf0as9df092039asdfi02089.pem
#EC2_PRIVATE_KEY = /path/to/your/pk-asdfasd890f200909.pem

# Uncomment these settings to use a proxy host when connecting to AWS
#AWS_PROXY = your.proxyhost.com
#AWS_PROXY_PORT = 8080
#AWS_PROXY_USER = yourproxyuser
#AWS_PROXY_PASS = yourproxypass

###########################
## Defining EC2 Keypairs ##
###########################
# Sections starting with "key" define your keypairs. See "starcluster createkey
# --help" for instructions on how to create a new keypair. Section name should
# match your key name e.g.:

# EU-west-1 keys
[key BIOMEDIQ]
KEY_LOCATION=~/.ssh/BIOMEDIQ_EC2.rsa

################################
## Defining Cluster Templates ##
################################
# Sections starting with "cluster" represent a cluster template. These
# "templates" are a collection of settings that define a single cluster
# configuration and are used when creating and configuring a cluster. You can
# change which template to use when creating your cluster using the -c option
# to the start command:
#
#     $ starcluster start -c mediumcluster mycluster
#
# If a template is not specified then the template defined by DEFAULT_TEMPLATE
# in the [global] section above is used. Below is the "default" template named
# "smallcluster". You can rename it but dont forget to update the
# DEFAULT_TEMPLATE setting in the [global] section above. See the next section
# on defining multiple templates.

[cluster smallcluster]
KEYNAME = BIOMEDIQ

# number of ec2 instances to launch
CLUSTER_SIZE = 3

# create the following user on the cluster
CLUSTER_USER = qadmin

# optionally specify shell (defaults to bash)
# (options: tcsh, zsh, csh, bash, ksh)
CLUSTER_SHELL = bash

# AMI to use for cluster nodes. These AMIs are for the us-east-1 region.
# Use the 'listpublic' command to list StarCluster AMIs in other regions
# The base i386 StarCluster AMI is ami-899d49e0
# The base x86_64 StarCluster AMI is ami-999d49f0
# The base HVM StarCluster AMI is ami-4583572c
# EU-west images:
# i386:
# ami-a5447bd1 eu-west-1 starcluster-base-ubuntu-11.10-x86 (EBS)
# ami-a4d7fdd0 eu-west-1 starcluster-base-ubuntu-9.10-x86-rc8
# x64:
# ami-ab447bdf eu-west-1 starcluster-base-ubuntu-11.10-x86_64 (EBS)
# ami-38d5ff4c eu-west-1 starcluster-base-ubuntu-9.10-x86_64-rc4

NODE_IMAGE_ID = ami-ab447bdf

# instance type for all cluster nodes
# (options: cg1.4xlarge, c1.xlarge, m1.small, c1.medium, m2.xlarge, t1.micro, cc1.4xlarge, m1.medium, cc2.8xlarge, m1.large, m1.xlarge, m2.4xlarge, m2.2xlarge)
NODE_INSTANCE_TYPE = t1.micro

# Uncomment to disable installing/configuring a queueing system on the
# cluster (SGE)
DISABLE_QUEUE=True

# Uncomment to specify a different instance type for the master node (OPTIONAL)
# (defaults to NODE_INSTANCE_TYPE if not specified)
#MASTER_INSTANCE_TYPE = t1.micro

# Uncomment to specify a separate AMI to use for the master node. (OPTIONAL)
# (defaults to NODE_IMAGE_ID if not specified)
#MASTER_IMAGE_ID = ami-ab447bdf (OPTIONAL)

# availability zone to launch the cluster in (OPTIONAL)
# (automatically determined based on volumes (if any) or
# selected by Amazon if not specified)
# AVAILABILITY_ZONE = eu-west-1c

# list of volumes to attach to the master node (OPTIONAL)
# these volumes, if any, will be NFS shared to the worker nodes
# see "Configuring EBS Volumes" below on how to define volume sections
VOLUMES = mrdata

# list of plugins to load after StarCluster's default setup routines (OPTIONAL)
# see "Configuring StarCluster Plugins" below on how to define plugin sections
#PLUGINS = myplugin, myplugin2

# list of permissions (or firewall rules) to apply to the cluster's security
# group (OPTIONAL).
#PERMISSIONS = ssh, http

# Uncomment to always create a spot cluster when creating a new cluster from
# this template. The following example will place a $0.050 bid for each spot
# request.
#SPOT_BID = 0.050

###########################################
## Defining Additional Cluster Templates ##
###########################################
# You can also define multiple cluster templates. You can either supply all
# configuration options as with smallcluster above, or create an
# EXTENDS=<cluster_name> variable in the new cluster section to use all
# settings from <cluster_name> as defaults. Below are example templates that
# use the EXTENDS feature:

# [cluster mediumcluster]
# Declares that this cluster uses smallcluster as defaults
# EXTENDS=smallcluster
# This section is the same as smallcluster except for the following settings:
# KEYNAME=myotherkey
# NODE_INSTANCE_TYPE = c1.xlarge
# CLUSTER_SIZE=8
# VOLUMES = biodata2

# [cluster largecluster]
# Declares that this cluster uses mediumcluster as defaults
# EXTENDS=mediumcluster
# This section is the same as mediumcluster except for the following variables:
# CLUSTER_SIZE=16

#############################
## Configuring EBS Volumes ##
#############################
# StarCluster can attach one or more EBS volumes to the master and then
# NFS_share these volumes to all of the worker nodes. A new [volume] section
# must be created for each EBS volume you wish to use with StarCluser. The
# section name is a tag for your volume. This tag is used in the VOLUMES
# setting of a cluster template to declare that an EBS volume is to be mounted
# and nfs shared on the cluster. (see the commented VOLUMES setting in the
# example 'smallcluster' template above) Below are some examples of defining
# and configuring EBS volumes to be used with StarCluster:

# Sections starting with "volume" define your EBS volumes
[volume mrdata]
# attach vol-c9999999 to /mr_data on master node and NFS-shre to worker nodes
VOLUME_ID = vol-ae98c1c6
MOUNT_PATH = /mr_data

# Same volume as above, but mounts to different location
# [volume biodata2]
# VOLUME_ID = vol-c999999
# MOUNT_PATH = /opt/

# Another volume example
# [volume oceandata]
# VOLUME_ID = vol-d7777777
# MOUNT_PATH = /mydata

# By default StarCluster will attempt first to mount the entire volume device,
# failing that it will try the first partition. If you have more than one
# partition you will need to set the PARTITION number, e.g.:
# [volume oceandata]
# VOLUME_ID = vol-d7777777
# MOUNT_PATH = /mydata
# PARTITION = 2

############################################
## Configuring Security Group Permissions ##
############################################
# Sections starting with "permission" define security group rules to
# automatically apply to newly created clusters. PROTOCOL in the following
# examples can be can be: tcp, udp, or icmp. CIDR_IP defaults to 0.0.0.0/0 or
# "open to the # world"

# open port 80 on the cluster to the world
# [permission http]
# PROTOCOL = tcp
# FROM_PORT = 80
# TO_PORT = 80

# open https on the cluster to the world
# [permission https]
# PROTOCOL = tcp
# FROM_PORT = 443
# TO_PORT = 443

# open port 80 on the cluster to an ip range using CIDR_IP
# [permission http]
# PROTOCOL = tcp
# FROM_PORT = 80
# TO_PORT = 80
# CIDR_IP = 18.0.0.0/8

# restrict ssh access to a single ip address (<your_ip>)
# [permission ssh]
# PROTOCOL = tcp
# FROM_PORT = 22
# TO_PORT = 22
# CIDR_IP = <your_ip>/32

#####################################
## Configuring StarCluster Plugins ##
#####################################
# Sections starting with "plugin" define a custom python class which perform
# additional configurations to StarCluster's default routines. These plugins
# can be assigned to a cluster template to customize the setup procedure when
# starting a cluster from this template (see the commented PLUGINS setting in
# the 'smallcluster' template above). Below is an example of defining a user
# plugin called 'myplugin':

# [plugin myplugin]
# NOTE: myplugin module must either live in ~/.starcluster/plugins or be
# on your PYTHONPATH
# SETUP_CLASS = myplugin.SetupClass
# extra settings are passed as __init__ arguments to your plugin:
# SOME_PARAM_FOR_MY_PLUGIN = 1
# SOME_OTHER_PARAM = 2

######################
## Built-in Plugins ##
######################
# The following plugins ship with StarCluster and should work out-of-the-box.
# Uncomment as needed. Don't forget to update your PLUGINS list!
# See http://web.mit.edu/star/cluster/docs/latest/plugins for plugin details.
#
# Use this plugin to install one or more packages on all nodes
# [plugin pkginstaller]
# SETUP_CLASS = starcluster.plugins.pkginstaller.PackageInstaller
# # list of apt-get installable packages
# PACKAGES = mongodb, python-pymongo
#
# Use this plugin to create one or more cluster users and download all user ssh
# keys to $HOME/.starcluster/user_keys/<cluster>-<region>.tar.gz
# [plugin createusers]
# SETUP_CLASS = starcluster.plugins.users.CreateUsers
# NUM_USERS = 30
# # you can also comment out NUM_USERS and specify exact usernames, e.g.
# # usernames = linus, tux, larry
# DOWNLOAD_KEYS = True
#
# Use this plugin to configure the Condor queueing system
# [plugin condor]
# SETUP_CLASS = starcluster.plugins.condor.CondorPlugin
#
# The SGE plugin is enabled by default and not strictly required. Only use this
# if you want to tweak advanced settings in which case you should also set
# DISABLE_QUEUE=TRUE in your cluster template. See the plugin doc for more
# details.
# [plugin sge]
# SETUP_CLASS = starcluster.plugins.sge.SGEPlugin
# MASTER_IS_EXEC_HOST = False
#
# The IPCluster plugin configures a parallel IPython cluster with optional
# web notebook support. This allows you to run Python code in parallel with low
# latency message passing via ZeroMQ.
# [plugin ipcluster]
# SETUP_CLASS = starcluster.plugins.ipcluster.IPCluster
# ENABLE_NOTEBOOK = True
# #set a password for the notebook for increased security
# NOTEBOOK_PASSWD = a-secret-password
#
# Use this plugin to create a cluster SSH "dashboard" using tmux. The plugin
# creates a tmux session on the master node that automatically connects to all
# the worker nodes over SSH. Attaching to the session shows a separate window
# for each node and each window is logged into the node via SSH.
# [plugin tmux]
# SETUP_CLASS = starcluster.plugins.tmux.TmuxControlCenter
#
# Use this plugin to change the default MPI implementation on the
# cluster from OpenMPI to MPICH2.
# [plugin mpich2]
# SETUP_CLASS = starcluster.plugins.mpich2.MPICH2Setup
#
# Configure a hadoop cluster. (includes dumbo setup)
# [plugin hadoop]
# SETUP_CLASS = starcluster.plugins.hadoop.Hadoop
#
# Configure a distributed MySQL Cluster
# [plugin mysqlcluster]
# SETUP_CLASS = starcluster.plugins.mysql.MysqlCluster
# NUM_REPLICAS = 2
# DATA_MEMORY = 80M
# INDEX_MEMORY = 18M
# DUMP_FILE = test.sql
# DUMP_INTERVAL = 60
# DEDICATED_QUERY = True
# NUM_DATA_NODES = 2
#
# Install and setup an Xvfb server on each cluster node
# [plugin xvfb]
# SETUP_CLASS = starcluster.plugins.xvfb.XvfbSetup