<div dir="ltr">Glad it worked with the new volume. Could be there is some issue with the data you have on the volume and not Starcluster itself. If you get the same error when you copy the data over to the new volume and try to mount the volume with Starcluster, I would guess this is the source of your issues. <div><br></div><div>If you google your error message (like <a href="https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=ls%3A+cannot+access+biodata%3A+Too+many+levels+of+symbolic+links">here</a>), you might be able to figure out what the issue is with the data you have on the volume. Seems like it might be as simple as having a circular symbolic link some how created/discovered when NFS attempts to mount the volumes on the client / worker nodes.<div><br></div><div>Good Luck!</div><div>-Jennifer<br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 5, 2015 at 2:33 PM, Andrey Evtikhov <span dir="ltr"><<a href="mailto:Andrey_Evtikhov@epam.com" target="_blank">Andrey_Evtikhov@epam.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thank you , Jennifer !<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Built from scratch Volume seems OK - with NO and WITH starclsuter .<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Just need to copy data over
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">J</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">What troubled me – on master everything looked normal…..
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">All good now when I made new volume ….<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">root@node001:/geodata# mkdir TESTFROMCLENT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">root@node001:/geodata# ls -l<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">total 32<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwx------ 2 root root 16384 Jun 5 17:47 lost+found<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:48 TEST4CHEVRON<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:48 TEST4CHEVRON2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:48 TEST4CHEVRON3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:51 TESTFROMCLENT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">root@node001:/geodata# exit<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">logout<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Connection to node001 closed.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">root@master:/geodata# ls -l<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">total 32<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwx------ 2 root root 16384 Jun 5 17:47 lost+found<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:48 TEST4CHEVRON<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:48 TEST4CHEVRON2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:48 TEST4CHEVRON3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">drwxr-xr-x 2 root root 4096 Jun 5 17:51 TESTFROMCLENT<u></u><u></u></span></p><span class="">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase">Andrey Evtikhov</span></b><span style="color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#464547">Lead Software Maintenance Engineer</span></b><span style="color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:5.0pt"><span style="font-size:5.0pt;font-family:"Arial",sans-serif;color:#1f497d"> <u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Email: </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><a href="mailto:andrey_evtikhov@epam.com" target="_blank"><span style="color:#666666;text-decoration:none">andrey_evtikhov@epam.com</span></a></span><span style="color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Saint-Petersburg,</span></b><span style="color:#1f497d"> </span><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Russia </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">(GMT+3)</span><span style="color:#1f497d">
</span><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><a href="http://www.epam.com" target="_blank"><span style="color:#666666;text-decoration:none">epam.com</span></a></span></b><span style="color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:10.0pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#999999">CONFIDENTIALITY CAUTION AND DISCLAIMER<br>
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
</span><span style="color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:6.0pt;font-family:"Arial",sans-serif;color:#1f497d"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u><u></u></span></p>
</span><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Jennifer Staab [mailto:<a href="mailto:jstaab@cs.unc.edu" target="_blank">jstaab@cs.unc.edu</a>]
<br>
<b>Sent:</b> Friday, June 5, 2015 6:48 PM<br>
<b>To:</b> Andrey Evtikhov<br>
<b>Cc:</b> <a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a><br>
<b>Subject:</b> Re: [StarCluster] NFS Help !!!!<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Something doesn't seem right with how you are mounting your volumes. If you are mounting a volume make sure that volume exists in your AWS system and is unattached to any EC2's. Next make sure the mount point let's say it's called /MyData
exists as an empty directory on both Master and Worker AMIs. Make certain your OS is up-to-date on all AMIs (workers and master). Also make sure your volumes and EC2's are in the same availability zone. <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Try this, create an empty volume, attach to the EC2 that's based upon your Master Node AMI. Then mount it to that Master Node EC2 (not using Starcluster) -- make sure it mounts successfully and you can write some temporary text files to
it. Next unmount it and unattach it from the EC2. Once it is unmounted and unattached create a new cluster and see if you can get that cluster to successfully NFS that volume between Master & Workers. If you can get that volume to mount and NFS successfully
between Master and Workers, then it is likely something about the volumes themselves and/or how you are mounting them that's causing the problem.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you can't get that empty volume to NFS between Master & Workers -- then it is likely something regarding NFS or the OS you have running on the AMIs. Make sure you are using up-to-date OS and software and that your mount points are empty
directories. Also don't mount under an existing mount point unless you are using crossmnt option in /etc/exports. Starcluster doesn't use the crossmnt option -- I altered my version of Starcluster as to allow crossmnt option - but it isn't a native feature.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Next I would try running NFS on an EC2 based upon the Master node AMI with that empty volume mounted and seeing if you can successfully NFS that volume with an EC2 based upon your Worker Node. My guess is if you can set NFS up successfully
without running Starcluster -- in the process you will likely see where the NFS problem is.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Good Luck!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">-Jennifer<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Jun 5, 2015 at 9:28 AM, Andrey Evtikhov <<a href="mailto:Andrey_Evtikhov@epam.com" target="_blank">Andrey_Evtikhov@epam.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="color:#1f497d">Really need help with this issue – just created new cluster – still the same mess – everything nice on master and complete mess on NFS client: </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Except – /home ! Home mounted Ok !</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">root@node001:/geodata# ls -l</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">ls: cannot access biodata: Too many levels of symbolic links</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">ls: cannot access geodata: Too many levels of symbolic links</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">in config :</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUMES = geodata, biodata</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">[volume geodata]</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUME_ID = vol-17feadfc</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">MOUNT_PATH = /geodata</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">[volume biodata]</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUME_ID = vol-f3fdae18</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">MOUNT_PATH = /biodata</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">It is how it looks on client /biodata</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">total 8</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">d????????? ? ? ? ? ? biodata</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">d????????? ? ? ? ? ? geodata</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">drwxr-xr-x 4 root root 4096 Jun 5 13:11 home</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">drwxr-xr-x 4 root root 4096 Jun 5 13:11 opt</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase">Andrey Evtikhov</span></b><span style="color:#1f497d">
</span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#464547">Lead Software Maintenance Engineer</span></b><span style="color:#1f497d">
</span><u></u><u></u></p>
<p class="MsoNormal" style="line-height:5.0pt">
<span style="font-size:5.0pt;font-family:"Arial",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Email: </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><a href="mailto:andrey_evtikhov@epam.com" target="_blank"><span style="color:#666666;text-decoration:none">andrey_evtikhov@epam.com</span></a></span><span style="color:#1f497d">
</span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Saint-Petersburg,</span></b><span style="color:#1f497d"> </span><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">Russia </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666">(GMT+3)</span><span style="color:#1f497d">
</span><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><a href="http://www.epam.com" target="_blank"><span style="color:#666666;text-decoration:none">epam.com</span></a></span></b><span style="color:#1f497d">
</span><u></u><u></u></p>
<p class="MsoNormal" style="line-height:10.0pt">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#999999">CONFIDENTIALITY CAUTION AND DISCLAIMER<br>
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:6.0pt;font-family:"Arial",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Andrey Evtikhov
<br>
<b>Sent:</b> Tuesday, June 2, 2015 9:20 PM<br>
<b>To:</b> '<a href="mailto:starcluster@mit.edu" target="_blank">starcluster@mit.edu</a>'<br>
<b>Subject:</b> Cannot mount properly /home over NFS<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Cannot mount properly /home
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUMES = mycompany-data</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">[volume mycompany-data]</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">VOLUME_ID = vol-xxxxxxx</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">MOUNT_PATH = /home</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Master seems to be OK , but nodes has mounted home inside home !</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">root@prod-sc-triad-node001:/home# cd /home</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">root@prod-sc-triad-node001:/home# ls -l</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">ls: cannot access home: Too many levels of symbolic links</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">total 4</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">d????????? ? ? ? ? ? home</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d;background:yellow">drwxr-xr-x 4 root root 4096 Apr 30 17:55 opt</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase"> </span></b><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase"> </span></b><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#39c2d7;text-transform:uppercase">Andrey Evtikhov</span></b><span style="color:#1f497d">
</span><span style="color:#888888"><u></u><u></u></span></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div></div></div></div>