<div dir="ltr">Something doesn't seem right with how you are mounting your volumes. If you are mounting a volume make sure that volume exists in your AWS system and is unattached to any EC2's. Next make sure the mount point let's say it's called /MyData exists as an empty directory on both Master and Worker AMIs. Make certain your OS is up-to-date on all AMIs (workers and master). Also make sure your volumes and EC2's are in the same availability zone. <div><br></div><div>Try this, create an empty volume, attach to the EC2 that's based upon your Master Node AMI. Then mount it to that Master Node EC2 (not using Starcluster) -- make sure it mounts successfully and you can write some temporary text files to it. Next unmount it and unattach it from the EC2. Once it is unmounted and unattached create a new cluster and see if you can get that cluster to successfully NFS that volume between Master & Workers. If you can get that volume to mount and NFS successfully between Master and Workers, then it is likely something about the volumes themselves and/or how you are mounting them that's causing the problem.</div><div><br></div><div>If you can't get that empty volume to NFS between Master & Workers -- then it is likely something regarding NFS or the OS you have running on the AMIs. Make sure you are using up-to-date OS and software and that your mount points are empty directories. Also don't mount under an existing mount point unless you are using crossmnt option in /etc/exports. Starcluster doesn't use the crossmnt option -- I altered my version of Starcluster as to allow crossmnt option - but it isn't a native feature.</div><div><br></div><div>Next I would try running NFS on an EC2 based upon the Master node AMI with that empty volume mounted and seeing if you can successfully NFS that volume with an EC2 based upon your Worker Node. My guess is if you can set NFS up successfully without running Starcluster -- in the process you will likely see where the NFS problem is.</div><div><br></div><div>Good Luck!</div><div><br></div><div>-Jennifer</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 5, 2015 at 9:28 AM, Andrey Evtikhov <span dir="ltr"><<a href="mailto:Andrey_Evtikhov@epam.com" target="_blank">Andrey_Evtikhov@epam.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div>
<p class="MsoNormal"><span style="color:#1f497d">Really need help with this issue – just created new cluster – still the same mess – everything nice on master and complete mess on NFS client: <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Except – /home ! Home mounted Ok !<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">root@node001:/geodata# ls -l<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">ls: cannot access biodata: Too many levels of symbolic links<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">ls: cannot access geodata: Too many levels of symbolic links<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">in config :<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUMES = geodata, biodata<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">[volume geodata]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUME_ID = vol-17feadfc<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">MOUNT_PATH = /geodata<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">[volume biodata]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUME_ID = vol-f3fdae18<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">MOUNT_PATH = /biodata<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">It is how it looks on client /biodata<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">total 8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">d????????? ? ? ? ? ? biodata<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">d????????? ? ? ? ? ? geodata<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">drwxr-xr-x 4 root root 4096 Jun 5 13:11 home<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">drwxr-xr-x 4 root root 4096 Jun 5 13:11 opt<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase">Andrey Evtikhov</span></b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#464547">Lead Software Maintenance Engineer</span></b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:5.0pt"><span style="font-size:5.0pt;font-family:"Arial",sans-serif;color:#1f497d"> <u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Email: </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><a href="mailto:andrey_evtikhov@epam.com" target="_blank"><span style="color:#666666;text-decoration:none">andrey_evtikhov@epam.com</span></a></span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Saint-Petersburg,</span></b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d"> </span><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Russia </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">(GMT+3)</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d">
</span><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><a href="http://www.epam.com" target="_blank"><span style="color:#666666;text-decoration:none">epam.com</span></a></span></b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:10.0pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#999999">CONFIDENTIALITY CAUTION AND DISCLAIMER<br>
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:6.0pt;font-family:"Arial",sans-serif;color:#1f497d"> </span><span style="color:#1f497d"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Andrey Evtikhov <br>
<b>Sent:</b> Tuesday, June 2, 2015 9:20 PM<br>
<b>To:</b> '<a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a>'<br>
<b>Subject:</b> Cannot mount properly /home over NFS<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Cannot mount properly /home <u></u>
<u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUMES = mycompany-data<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">[volume mycompany-data]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUME_ID = vol-xxxxxxx<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">MOUNT_PATH = /home<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">Master seems to be OK , but nodes has mounted home inside home !<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d">root@prod-sc-triad-node001:/home# cd /home<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">root@prod-sc-triad-node001:/home# ls -l<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">ls: cannot access home: Too many levels of symbolic links<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">total 4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">d????????? ? ? ? ? ? home<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">drwxr-xr-x 4 root root 4096 Apr 30 17:55 opt</span><span class="HOEnZb"><font color="#888888"><span style="color:#1f497d"><u></u><u></u></span></font></span></p><span class="HOEnZb"><font color="#888888">
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase">Andrey Evtikhov</span></b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:#1f497d">
<u></u><u></u></span></p>
</div>
</font></span></div>
</div>
<br>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div>