[StarCluster] EBS storage does not mount <newbie> - debug log
Anders Nielsen
an at biomediq.com
Mon Aug 27 08:28:58 EDT 2012
Just wanted to add a little information on my problem: the debug log show
definite problems with mounting the files system, I do hope someone can
tell me what I might be doing wrong here?
2012-08-02 15:06:33,573 PID: 10056 node.py:606 - INFO - Starting NFS server
on master
2012-08-02 15:06:33,704 PID: 10056 __init__.py:543 - DEBUG - Rather than
invoking init scripts through /etc/init.d, use the service(8)
2012-08-02 15:06:33,706 PID: 10056 __init__.py:543 - DEBUG - utility, e.g.
service portmap start
2012-08-02 15:06:33,711 PID: 10056 __init__.py:543 - DEBUG -
2012-08-02 15:06:33,717 PID: 10056 __init__.py:543 - DEBUG - Since the
script you are attempting to invoke has been converted to an
2012-08-02 15:06:33,719 PID: 10056 __init__.py:543 - DEBUG - Upstart job,
you may also use the start(8) utility, e.g. start portmap
2012-08-02 15:06:33,884 PID: 10056 __init__.py:540 - DEBUG - command 'mount
-t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs/' failed with status 32
2012-08-02 15:06:33,891 PID: 10056 __init__.py:543 - DEBUG - mount: sunrpc
already mounted or /var/lib/nfs/rpc_pipefs/ busy
2012-08-02 15:06:33,897 PID: 10056 __init__.py:543 - DEBUG - mount:
according to mtab, rpc_pipefs is already mounted on /var/lib/nfs/rpc_pipefs
2012-08-02 15:06:34,131 PID: 10056 __init__.py:543 - DEBUG - * Exporting
directories for NFS kernel daemon...
2012-08-02 15:06:34,137 PID: 10056 __init__.py:543 - DEBUG - ...done.
2012-08-02 15:06:34,142 PID: 10056 __init__.py:543 - DEBUG - * Starting NFS
kernel daemon
2012-08-02 15:06:34,142 PID: 10056 __init__.py:543 - DEBUG - ...done.
2012-08-02 15:06:34,144 PID: 10056 __init__.py:543 - DEBUG - exportfs:
scandir /etc/exports.d: No such file or directory
2012-08-02 15:06:34,144 PID: 10056 __init__.py:543 - DEBUG -
2012-08-02 15:06:34,308 PID: 10056 __init__.py:543 - DEBUG - exportfs:
scandir /etc/exports.d: No such file or directory
2012-08-02 15:06:34,313 PID: 10056 __init__.py:543 - DEBUG -
2012-08-02 15:06:34,315 PID: 10056 node.py:577 - INFO - Configuring NFS
exports path(s):
/home
2012-08-02 15:06:34,815 PID: 10056 __init__.py:543 - DEBUG - exportfs:
scandir /etc/exports.d: No such file or directory
2012-08-02 15:06:34,821 PID: 10056 __init__.py:543 - DEBUG -
2012-08-02 15:06:34,828 PID: 10056 clustersetup.py:313 - INFO - Mounting
all NFS export path(s) on 2 worker node(s)
2012-08-02 15:06:34,828 PID: 10056 threadpool.py:135 - DEBUG -
unfinished_tasks = 2
2012-08-02 15:06:34,967 PID: 10056 __init__.py:543 - DEBUG - Rather than
invoking init scripts through /etc/init.d, use the service(8)
2012-08-02 15:06:34,973 PID: 10056 __init__.py:543 - DEBUG - utility, e.g.
service portmap start
2012-08-02 15:06:34,980 PID: 10056 __init__.py:543 - DEBUG -
2012-08-02 15:06:34,983 PID: 10056 __init__.py:543 - DEBUG - Since the
script you are attempting to invoke has been converted to an
2012-08-02 15:06:34,983 PID: 10056 __init__.py:543 - DEBUG - Upstart job,
you may also use the start(8) utility, e.g. start portmap
2012-08-02 15:06:34,984 PID: 10056 __init__.py:543 - DEBUG - Rather than
invoking init scripts through /etc/init.d, use the service(8)
2012-08-02 15:06:34,990 PID: 10056 __init__.py:543 - DEBUG - utility, e.g.
service portmap start
2012-08-02 15:06:34,997 PID: 10056 __init__.py:543 - DEBUG -
2012-08-02 15:06:35,003 PID: 10056 __init__.py:543 - DEBUG - Since the
script you are attempting to invoke has been converted to an
2012-08-02 15:06:35,005 PID: 10056 __init__.py:543 - DEBUG - Upstart job,
you may also use the start(8) utility, e.g. start portmap
2012-08-02 15:06:35,154 PID: 10056 __init__.py:540 - DEBUG - command 'mount
-t devpts none /dev/pts' failed with status 32
2012-08-02 15:06:35,160 PID: 10056 __init__.py:543 - DEBUG - mount: none
already mounted or /dev/pts busy
2012-08-02 15:06:35,165 PID: 10056 __init__.py:543 - DEBUG - mount:
according to mtab, devpts is already mounted on /dev/pts
2012-08-02 15:06:35,176 PID: 10056 __init__.py:540 - DEBUG - command 'mount
-t devpts none /dev/pts' failed with status 32
2012-08-02 15:06:35,176 PID: 10056 __init__.py:543 - DEBUG - mount: none
already mounted or /dev/pts busy
2012-08-02 15:06:35,187 PID: 10056 __init__.py:543 - DEBUG - mount:
according to mtab, devpts is already mounted on /dev/pts
2012-08-02 15:06:35,328 PID: 10056 __init__.py:543 - DEBUG - /dev/xvda1 on
/ type ext4 (rw)
2012-08-02 15:06:35,328 PID: 10056 __init__.py:543 - DEBUG - proc on /proc
type proc (rw,noexec,nosuid,nodev)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - sysfs on /sys
type sysfs (rw,noexec,nosuid,nodev)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - fusectl on
/sys/fs/fuse/connections type fusectl (rw)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - none on
/sys/kernel/debug type debugfs (rw)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - none on
/sys/kernel/security type securityfs (rw)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - udev on /dev
type devtmpfs (rw,mode=0755)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - devpts on
/dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - tmpfs on /run
type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
2012-08-02 15:06:35,329 PID: 10056 __init__.py:543 - DEBUG - none on
/run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
2012-08-02 15:06:35,331 PID: 10056 __init__.py:543 - DEBUG - none on
/run/shm type tmpfs (rw,nosuid,nodev)
2012-08-02 15:06:35,349 PID: 10056 __init__.py:543 - DEBUG - /dev/xvda1 on
/ type ext4 (rw)
2012-08-02 15:06:35,349 PID: 10056 __init__.py:543 - DEBUG - proc on /proc
type proc (rw,noexec,nosuid,nodev)
2012-08-02 15:06:35,349 PID: 10056 __init__.py:543 - DEBUG - sysfs on /sys
type sysfs (rw,noexec,nosuid,nodev)
2012-08-02 15:06:35,349 PID: 10056 __init__.py:543 - DEBUG - fusectl on
/sys/fs/fuse/connections type fusectl (rw)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - none on
/sys/kernel/debug type debugfs (rw)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - none on
/sys/kernel/security type securityfs (rw)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - udev on /dev
type devtmpfs (rw,mode=0755)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - devpts on
/dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - tmpfs on /run
type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - none on
/run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
2012-08-02 15:06:35,351 PID: 10056 __init__.py:543 - DEBUG - none on
/run/shm type tmpfs (rw,nosuid,nodev)
2012-08-02 15:06:35,575 PID: 10056 __init__.py:284 - DEBUG - new /etc/fstab
after removing regex ( /home ) matches:
LABEL=cloudimg-rootfs /
ext4 defaults 0 0
2012-08-02 15:06:35,588 PID: 10056 __init__.py:284 - DEBUG - new /etc/fstab
after removing regex ( /home ) matches:
LABEL=cloudimg-rootfs /
ext4 defaults 0 0
2012-08-02 15:06:35,836 PID: 10056 threadpool.py:135 - DEBUG -
unfinished_tasks = 2
2012-08-02 15:06:36,838 PID: 10056 threadpool.py:135 - DEBUG -
unfinished_tasks = 2
2012-08-02 15:06:37,838 PID: 10056 utils.py:93 - INFO - Setting up NFS took
0.071 mins
Regards
Anders Nielsen
*From:* Anders Nielsen [mailto:an at biomediq.com]
*Sent:* 23. august 2012 21:11
*To:* starcluster at mit.edu
*Subject:* [StarCluster] EBS storage does not mount <newbie>
I am a newbie and am just getting my feet wet on StarCluster and AWS
I have successfully installed 0.93.3 on Windows and associated it with my
AWS login. I can create clusters and I have used the documented way of
created an EBS store for use by my application.
My problem is that none of the instances in the cluster will mount the EBS
and the master complains about it having more than one partition. It was
created according to the method here:
http://web.mit.edu/star/cluster/docs/latest/manual/volumes.html#create-and-format-a-new-ebs-volume–
and to the best of my knowledge correctly so. I think I am doing all
the
things I am supposed to do, but obviously I am missing something. The
configuration file is attached but it does not seem to be causing
StarCluster any problems.
The runtime output is however not so great:
C:\Windows\System32>starcluster start imagingcluster
StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster at mit.edu
>>> Using default cluster template: smallcluster
>>> Validating cluster template settings...
>>> Cluster template settings are valid
>>> Starting cluster...
>>> Launching a 3-node cluster...
>>> Creating security group @sc-imagingcluster...
Reservation:r-953beddd
>>> Waiting for cluster to come up... (updating every 30s)
>>> Waiting for all nodes to be in a 'running' state...
3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Waiting for SSH to come up on all nodes...
3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Waiting for cluster to come up took 1.383 mins
>>> The master node is ec2-46-137-154-117.eu-west-1.compute.amazonaws.com
>>> Setting up the cluster...
>>> Attaching volume vol-ae98c1c6 to master node on /dev/sdz ...
>>> Configuring hostnames...
3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
!!! ERROR - volume has more than one partition, please specify which
partition t
o use (e.g. partition=0, partition=1, etc.) in the volume's config
>>> Creating cluster user: None (uid: 1001, gid: 1001)
3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Configuring scratch space for user(s): qadmin
3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Configuring /etc/hosts on each node
3/3 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Starting NFS server on master
>>> Configuring NFS exports path(s):
/home /mr_data
>>> Mounting all NFS export path(s) on 2 worker node(s)
!!! ERROR - command 'mount /mr_data' failed with status 32 |
0%
!!! ERROR - command 'mount /mr_data' failed with status 32
2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Setting up NFS took 0.145 mins
>>> Configuring passwordless ssh for root
>>> Configuring passwordless ssh for qadmin
>>> Shutting down threads...
20/20 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
>>> Configuring cluster took 0.889 mins
>>> Starting cluster took 2.375 mins
A clue could be that when I sshmaster in, the /dev looks very odd as sda
etc. are missing:
root at master:~# ls -l /dev/s*
lrwxrwxrwx 1 root root 8 2012-08-23 18:44 /dev/shm->/run/shm
crw------- 1 root root 10, 231 2012-08-23 18:44 /dev/snapshot
lrwxrwxrwx 1 root root 15 2012-08-23 18:44 /dev/stderr->
/proc/self/fd/2
lrwxrwxrwx 1 root root 15 2012-08-23 18:44 /dev/stdin->/proc/self/fd/0
lrwxrwxrwx 1 root root 15 2012-08-23 18:44 /dev/stdout ->
proc/self/fd/1
However is seems that the disks are not sda1/sdz but xvd1 and xvdz:
root at master:~# ls -l /dev/x*
ls -l /dev/x*
brw-rw---- 1 root disk 202, 1 2012-08-23 18:44 /dev/xvda1
brw-rw---- 1 root disk 202, 400 2012-08-23 18:45 /dev/xvdz
Would that trip up StarCluster?
BTW, does anybody know why the output from the ssh session via Python SSH
are all filled by garbage control characters? The outputs above I had to
clean manually for it to be readable) – remember: total newbie.
All and any help is much appreciated.
Regards,
Anders Nielsen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20120827/054c4319/attachment-0001.htm
More information about the StarCluster
mailing list