<div>Hello Justin,</div><div><br></div><div>I checked out the develop branch from the link you send me and can confirm this is fixed. I started a cluster with one master node, then added one node and another node. /etc/exports is not clobbered anymore and everything NFS-wise seems to work.</div>
<div><br></div><div>cheers</div><div>Paul</div><div><br></div><div>Here is how the /etc/exports file looked like:</div><div><br></div><div>root@master:/data# cat /etc/exports</div><div># /etc/exports: the access control list for filesystems which may be exported</div>
<div>#<span class="Apple-tab-span" style="white-space:pre"> </span>to NFS clients. See exports(5).</div><div>#</div><div># Example for NFSv2 and NFSv3:</div><div># /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)</div>
<div>#</div><div># Example for NFSv4:</div><div># /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)</div><div># /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)</div><div>#</div><div><br></div><div>
root@master:/data# cat /etc/exports</div><div># /etc/exports: the access control list for filesystems which may be exported</div><div>#<span class="Apple-tab-span" style="white-space:pre"> </span>to NFS clients. See exports(5).</div>
<div>#</div><div># Example for NFSv2 and NFSv3:</div><div># /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)</div><div>#</div><div># Example for NFSv4:</div><div># /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)</div>
<div># /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)</div><div>#</div><div>/home node001(async,no_root_squash,no_subtree_check,rw)</div><div>/opt/sge6 node001(async,no_root_squash,no_subtree_check,rw)</div><div>/data node001(async,no_root_squash,no_subtree_check,rw)</div>
<div><br></div><div>root@master:/data# cat /etc/exports</div><div># /etc/exports: the access control list for filesystems which may be exported</div><div>#<span class="Apple-tab-span" style="white-space:pre"> </span>to NFS clients. See exports(5).</div>
<div>#</div><div># Example for NFSv2 and NFSv3:</div><div># /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)</div><div>#</div><div># Example for NFSv4:</div><div># /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)</div>
<div># /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)</div><div>#</div><div>/home node001(async,no_root_squash,no_subtree_check,rw)</div><div>/opt/sge6 node001(async,no_root_squash,no_subtree_check,rw)</div><div>/data node001(async,no_root_squash,no_subtree_check,rw)</div>
<div>/home node002(async,no_root_squash,no_subtree_check,rw)</div><div>/opt/sge6 node002(async,no_root_squash,no_subtree_check,rw)</div><div>/data node002(async,no_root_squash,no_subtree_check,rw)</div><div><br></div><div class="gmail_quote">
On Wed, Jan 18, 2012 at 23:45, Paul Koerbitz <span dir="ltr"><<a href="mailto:paul.koerbitz@gmail.com">paul.koerbitz@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Justin,<div><br></div><div>ok great. I have something running right now that I don't want to interrupt, but I might be able to take a stab at it tomorrow and will report back then.</div><div><br></div><div>cheers </div>
<span class="HOEnZb"><font color="#888888">
<div>Paul</div></font></span><div class="HOEnZb"><div class="h5"><div><br><div class="gmail_quote">On Wed, Jan 18, 2012 at 23:17, Justin Riley <span dir="ltr"><<a href="mailto:jtriley@mit.edu" target="_blank">jtriley@mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi Paul,<br>
<br>
</div>No problem at all and thanks for the kind words. From my limited<br>
testing I believe this is fixed in the latest github code which will<br>
be included in tomorrow's patch release:<br>
<br>
<a href="http://tinyurl.com/8axmckc" target="_blank">http://tinyurl.com/8axmckc</a><br>
<br>
If you could test the latest github code and report back whether it<br>
fixes the issue for you or not that'd be very helpful.<br>
<br>
~Justin<br>
<div><br>
On 01/18/2012 03:44 PM, Paul Koerbitz wrote:<br>
> Hi Justin,<br>
><br>
> thanks for the fast response and the great work. I thought about<br>
> taking a crack at a fix myself, but Im not familiar with the<br>
> codebase and don't have very little time right now.<br>
><br>
> thanks Paul<br>
><br>
> On Wed, Jan 18, 2012 at 21:33, Justin Riley <<a href="mailto:jtriley@mit.edu" target="_blank">jtriley@mit.edu</a><br>
</div><div><div>> <mailto:<a href="mailto:jtriley@mit.edu" target="_blank">jtriley@mit.edu</a>>> wrote:<br>
><br>
> Hi Paul,<br>
><br>
> I just tested for myself and I can confirm that /etc/exports is<br>
> indeed being clobbered when running the 'addnode' command. I'm<br>
> working on a patch release to fix this and other minor things.<br>
> Should be out tomorrow.<br>
><br>
> Thanks for reporting!<br>
><br>
> ~Justin<br>
><br>
> On 01/18/2012 02:08 PM, Paul Koerbitz wrote:<br>
>> Dear starcluster team,<br>
><br>
>> I tripped over what might be an error with the NFS setup when<br>
>> adding new nodes to a cluster.<br>
><br>
>> I set up my cluster with initially one root node only and then<br>
>> first added one node and subsequently 4 more nodes. I noticed<br>
>> that my ebsvolume wasn't getting mounted correctly on the nodes,<br>
>> calling 'df' reported 'stale filehandle' for /home /opt/sge6 and<br>
>> /data<br>
><br>
>> My impression is that as nodes get added, the /etc/exports file<br>
>> which is responsible for allowing NFS access gets overwritten.<br>
>> Therefore only the last added node can access the shared file<br>
>> systems.<br>
><br>
>> Here is how I resloved the issue. First I unmounted all the<br>
>> volumes:<br>
><br>
>> root@node001:~# umount -f /data<br>
><br>
>> At this point remounting doesn't work:<br>
><br>
>> root@node001:~# mount -t nfs master:/data /data<br>
><br>
>> mount.nfs: access denied by server while mounting master:/data<br>
><br>
><br>
>> I then edited /etc/exports on the master node. Here only the<br>
>> last node was listed:<br>
><br>
>> /home node005(async,no_root_squash,no_subtree_check,rw)<br>
>> /opt/sge6 node005(async,no_root_squash,no_subtree_check,rw)<br>
>> /data node005(async,no_root_squash,no_subtree_check,rw)<br>
><br>
>> I changed this to /home<br>
>> node001(async,no_root_squash,no_subtree_check,rw) /opt/sge6<br>
>> node001(async,no_root_squash,no_subtree_check,rw) /data<br>
>> node001(async,no_root_squash,no_subtree_check,rw) /home<br>
>> node002(async,no_root_squash,no_subtree_check,rw) /opt/sge6<br>
>> node002(async,no_root_squash,no_subtree_check,rw) /data<br>
>> node002(async,no_root_squash,no_subtree_check,rw) /home<br>
>> node003(async,no_root_squash,no_subtree_check,rw) /opt/sge6<br>
>> node003(async,no_root_squash,no_subtree_check,rw) /data<br>
>> node003(async,no_root_squash,no_subtree_check,rw) /home<br>
>> node004(async,no_root_squash,no_subtree_check,rw) /opt/sge6<br>
>> node004(async,no_root_squash,no_subtree_check,rw) /data<br>
>> node004(async,no_root_squash,no_subtree_check,rw) /home<br>
>> node005(async,no_root_squash,no_subtree_check,rw) /opt/sge6<br>
>> node005(async,no_root_squash,no_subtree_check,rw) /data<br>
>> node005(async,no_root_squash,no_subtree_check,rw)<br>
><br>
>> then restart the nfs-server:<br>
><br>
>> $ /etc/init.d/nfs-kernel-server restart<br>
><br>
>> After that running 'df' on each node showed the nfs now working<br>
>> correctly.<br>
><br>
>> kind regards Paul<br>
><br>
><br>
>> This body part will be downloaded on demand.<br>
><br>
><br>
><br>
<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2.0.17 (GNU/Linux)<br>
Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
<br>
</div></div>iEYEARECAAYFAk8XRPwACgkQ4llAkMfDcrlJWACgjNwy6KVMywbiP6aVggOgQVqm<br>
OD8AnA/1fwt04oGIhEtA7i3kq8KLMr0y<br>
=9mnL<br>
-----END PGP SIGNATURE-----<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br>