<div dir="ltr"><div><div><div><div><div>Dear Justin and Gonçalo, <br><br></div>thank you very much. It worked!<br><br></div>I first ran the two suggested commands:<br><br>$ update-alternatives --config mpi<br>
$ update-alternatives --config mpirun<br>
<br></div>and then I reconfigured and recompiled the application. <br><br></div>Regards,<br></div>Torstein<br><br><div><div><div><div><br></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Apr 28, 2014 at 4:43 PM, Justin Riley <span dir="ltr"><<a href="mailto:jtriley@mit.edu" target="_blank">jtriley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Gonçalo,<br>
<br>
Ah, I thought this sounded familiar:<br>
<br>
<a href="https://github.com/jtriley/StarCluster/issues/370" target="_blank">https://github.com/jtriley/StarCluster/issues/370</a><br>
<br>
Thanks for responding. This will be fixed in the upcoming 14.04 AMIs.<br>
<br>
Torstein, you can update the MPI links interactively by running the<br>
following commands as root:<br>
<br>
$ update-alternatives --config mpi<br>
$ update-alternatives --config mpirun<br>
<br>
Select either all openmpi or all mpich paths at the interactive prompts.<br>
<br>
~Justin<br>
<div><div class="h5"><br>
On Mon, Apr 28, 2014 at 02:06:50PM +0200, Gonçalo Albuquerque wrote:<br>
> Â Â Hi,<br>
>   When using AMI ami-6b211202 in us-east I stumbled across the same issue<br>
> Â Â you're experiencing.<br>
> Â Â The symbolic links in the alternatives system are mixing MPICH and<br>
> Â Â OpenMPI:<br>
> Â Â root@master:/etc/alternatives# update-alternatives --display mpi<br>
> Â Â mpi - auto mode<br>
> Â Â Â link currently points to /usr/include/mpich2<br>
> Â Â /usr/include/mpich2 - priority 40<br>
> Â Â Â slave libmpi++.so: /usr/lib/libmpichcxx.so<br>
> Â Â Â slave libmpi.so: /usr/lib/libmpich.so<br>
> Â Â Â slave libmpif77.so: /usr/lib/libfmpich.so<br>
> Â Â Â slave libmpif90.so: /usr/lib/libmpichf90.so<br>
> Â Â Â slave mpic++: /usr/bin/mpic++.mpich2<br>
> Â Â Â slave mpic++.1.gz: /usr/share/man/man1/mpic++.mpich2.1.gz<br>
> Â Â Â slave mpicc: /usr/bin/mpicc.mpich2<br>
> Â Â Â slave mpicc.1.gz: /usr/share/man/man1/mpicc.mpich2.1.gz<br>
> Â Â Â slave mpicxx: /usr/bin/mpicxx.mpich2<br>
> Â Â Â slave mpicxx.1.gz: /usr/share/man/man1/mpicxx.mpich2.1.gz<br>
> Â Â Â slave mpif77: /usr/bin/mpif77.mpich2<br>
> Â Â Â slave mpif77.1.gz: /usr/share/man/man1/mpif77.mpich2.1.gz<br>
> Â Â Â slave mpif90: /usr/bin/mpif90.mpich2<br>
> Â Â Â slave mpif90.1.gz: /usr/share/man/man1/mpif90.mpich2.1.gz<br>
> Â Â /usr/lib/openmpi/include - priority 40<br>
> Â Â Â slave libmpi++.so: /usr/lib/openmpi/lib/libmpi_cxx.so<br>
> Â Â Â slave libmpi.so: /usr/lib/openmpi/lib/libmpi.so<br>
> Â Â Â slave libmpif77.so: /usr/lib/openmpi/lib/libmpi_f77.so<br>
> Â Â Â slave libmpif90.so: /usr/lib/openmpi/lib/libmpi_f90.so<br>
> Â Â Â slave mpiCC: /usr/bin/mpic++.openmpi<br>
> Â Â Â slave mpiCC.1.gz: /usr/share/man/man1/mpiCC.openmpi.1.gz<br>
> Â Â Â slave mpic++: /usr/bin/mpic++.openmpi<br>
> Â Â Â slave mpic++.1.gz: /usr/share/man/man1/mpic++.openmpi.1.gz<br>
> Â Â Â slave mpicc: /usr/bin/mpicc.openmpi<br>
> Â Â Â slave mpicc.1.gz: /usr/share/man/man1/mpicc.openmpi.1.gz<br>
> Â Â Â slave mpicxx: /usr/bin/mpic++.openmpi<br>
> Â Â Â slave mpicxx.1.gz: /usr/share/man/man1/mpicxx.openmpi.1.gz<br>
> Â Â Â slave mpif77: /usr/bin/mpif77.openmpi<br>
> Â Â Â slave mpif77.1.gz: /usr/share/man/man1/mpif77.openmpi.1.gz<br>
> Â Â Â slave mpif90: /usr/bin/mpif90.openmpi<br>
> Â Â Â slave mpif90.1.gz: /usr/share/man/man1/mpif90.openmpi.1.gz<br>
> Â Â Current 'best' version is '/usr/include/mpich2'.<br>
> Â Â root@master:/etc/alternatives# update-alternatives --display mpirun<br>
> Â Â mpirun - auto mode<br>
> Â Â Â link currently points to /usr/bin/mpirun.openmpi<br>
> Â Â /usr/bin/mpirun.mpich2 - priority 40<br>
> Â Â Â slave mpiexec: /usr/bin/mpiexec.mpich2<br>
> Â Â Â slave mpiexec.1.gz: /usr/share/man/man1/mpiexec.mpich2.1.gz<br>
> Â Â Â slave mpirun.1.gz: /usr/share/man/man1/mpirun.mpich2.1.gz<br>
> Â Â /usr/bin/mpirun.openmpi - priority 50<br>
> Â Â Â slave mpiexec: /usr/bin/mpiexec.openmpi<br>
> Â Â Â slave mpiexec.1.gz: /usr/share/man/man1/mpiexec.openmpi.1.gz<br>
> Â Â Â slave mpirun.1.gz: /usr/share/man/man1/mpirun.openmpi.1.gz<br>
> Â Â Current 'best' version is '/usr/bin/mpirun.openmpi'.<br>
> Â Â You do compile it with MPICH and try to run with OpenMPI. The solution is<br>
> Â Â to change the symbolic links by using the update-alternatives command. For<br>
> Â Â the runtime link (mpirun), it must be done in all the nodes of the<br>
> Â Â cluster.<br>
> Â Â No doubt this will be corrected in upcoming versions of the AMIs.<br>
> Â Â Regards,<br>
>   Gonçalo<br>
><br>
> Â Â On Mon, Apr 28, 2014 at 1:09 PM, Torstein Fjermestad<br>
</div></div><div class="">> Â Â <[1]<a href="mailto:tfjermestad@gmail.com">tfjermestad@gmail.com</a>> wrote:<br>
><br>
> Â Â Â Dear Justin,<br>
> Â Â Â Â <br>
> Â Â Â during the compilation, the cluster only consisted of the master node<br>
> Â Â Â which is of instance type c3.large. In order to run a test parallel<br>
> Â Â Â calculation, I added a node of instance type c3.4xlarge (16 processors).<br>
><br>
> Â Â Â The cluster is created form the following AMI:Â <br>
> Â Â Â [0] ami-044abf73 eu-west-1 starcluster-base-ubuntu-13.04-x86_64 (EBS)<br>
><br>
> Â Â Â Executing the application outside the queuing system like<br>
><br>
> Â Â Â mpirun -np 2 -hostfile hosts ./pw.x -in inputfile.inp<br>
><br>
> Â Â Â did not change anything.<br>
><br>
> Â Â Â The output of the command "mpirun --version" is the following:<br>
><br>
> Â Â Â mpirun (Open MPI) 1.4.5<br>
><br>
</div>> Â Â Â Report bugs to [2]<a href="http://www.open-mpi.org/community/help/" target="_blank">http://www.open-mpi.org/community/help/</a><br>
<div><div class="h5">><br>
> Â Â Â After investigating the matter a little bit, I found that mpif90 is<br>
> Â Â Â likely compiled with an MPI version different from mpirun.<br>
> Â Â Â The first line of the output of the command "mpif90 -v" is the<br>
> Â Â Â following:<br>
> Â Â Â Â <br>
> Â Â Â mpif90 for MPICH2 version 1.4.1<br>
><br>
> Â Â Â Furthermore, the output of the command "ldd pw.x" indicates that pw.x is<br>
> Â Â Â compiled with mpich2 and not with Open MPI. The output is the following:<br>
> Â Â Â Â <br>
> Â Â Â linux-vdso.so.1 =>Â (0x00007fffd35fe000)<br>
> Â Â Â Â Â Â liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007ff38fb18000)<br>
> Â Â Â Â Â Â libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007ff38e2f5000)<br>
> Â Â Â Â Â Â libmpich.so.3 => /usr/lib/libmpich.so.3 (0x00007ff38df16000)<br>
> Â Â Â Â Â Â libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0<br>
> Â Â Â (0x00007ff38dcf9000)<br>
> Â Â Â Â Â Â libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3<br>
> Â Â Â (0x00007ff38d9e5000)<br>
> Â Â Â Â Â Â libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff38d6df000)<br>
> Â Â Â Â Â Â libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1<br>
> Â Â Â (0x00007ff38d4c9000)<br>
> Â Â Â Â Â Â libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff38d100000)<br>
> Â Â Â Â Â Â librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff38cef7000)<br>
> Â Â Â Â Â Â libcr.so.0 => /usr/lib/libcr.so.0 (0x00007ff38cced000)<br>
> Â Â Â Â Â Â libmpl.so.1 => /usr/lib/libmpl.so.1 (0x00007ff38cae8000)<br>
> Â Â Â Â Â Â /lib64/ld-linux-x86-64.so.2 (0x00007ff390820000)<br>
> Â Â Â Â Â Â libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0<br>
> Â Â Â (0x00007ff38c8b2000)<br>
> Â Â Â Â Â Â libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff38c6ae000)<br>
><br>
> Â Â Â The feedback I got from the Quantum Espresso mailing list suggested that<br>
> Â Â Â the cause of the error could be that pw.x (the executable) was not<br>
> Â Â Â compiled with the same version of mpi as mpirun.<br>
> Â Â Â The output of the commands "mpirun --version", "mpif90 -v" and "ldd<br>
> Â Â Â pw.x" above have lead me to suspect that this is indeed the case.<br>
> Â Â Â I therefore wonder whether it is possible to control which mpi version I<br>
> Â Â Â compile my applications with.<br>
><br>
> Â Â Â If, with the current mpi installation, the applications are compiled<br>
> Â Â Â with a different mpi version than mpirun, then I will likely have<br>
> Â Â Â similar problems when compiling other applications as well. I would<br>
> Â Â Â therefore very much appreciate if you could give me some hints on how I<br>
> Â Â Â can solve this problem.<br>
><br>
> Â Â Â Thanks in advance.<br>
><br>
> Â Â Â Regards,<br>
> Â Â Â Torstein<br>
><br>
</div></div>> Â Â Â On Thu, Apr 24, 2014 at 5:13 PM, Justin Riley <[3]<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>><br>
<div class="">> Â Â Â wrote:<br>
><br>
> Â Â Â Â Hi Torstein,<br>
><br>
> Â Â Â Â Can you please describe your cluster configuration (ie size, image<br>
> Â Â Â Â id(s),<br>
> Â Â Â Â instance type(s))? Also, you're currently using the SGE/OpenMPI<br>
> Â Â Â Â integration. Have you tried just using mpirun only as described in the<br>
> Â Â Â Â first part of:<br>
><br>
</div>> Â Â Â Â [4]<a href="http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment" target="_blank">http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment</a><br>
<div class="">><br>
> Â Â Â Â Also, what does 'mpirun --version' show?<br>
><br>
> Â Â Â Â ~Justin<br>
> Â Â Â Â On Thu, Apr 17, 2014 at 07:19:28PM +0200, Torstein Fjermestad wrote:<br>
> Â Â Â Â > Â Â Dear all,<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â I recently tried to compile an application (Quantum Espresso,<br>
</div>> Â Â Â Â > Â Â [1][5]<a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a>) to be used for parallel<br>
<div><div class="h5">> Â Â Â Â computations<br>
> Â Â Â Â > Â Â on StarCluster. The installation procedure of the application<br>
> Â Â Â Â consists of<br>
>     >   the standard "./configure + make" steps. At the end of the<br>
> Â Â Â Â output from<br>
> Â Â Â Â > Â Â ./configure, the statement "Parallel environment detected<br>
> Â Â Â Â successfully.\<br>
> Â Â Â Â > Â Â Configured for compilation of parallel executables." appears.<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â The compilation with "make" completes without errors. I then run<br>
> Â Â Â Â the<br>
> Â Â Â Â > Â Â application in the following way:<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â I first write a submit script (submit.sh) with the following<br>
> Â Â Â Â content:<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â cp /path/to/executable/pw.x .<br>
> Â Â Â Â > Â Â mpirun ./pw.x -in input.inp<br>
> Â Â Â Â > Â Â I then submit the job to the queueing system with the following<br>
> Â Â Â Â command<br>
> Â Â Â Â > Â Â Â <br>
> Â Â Â Â > Â Â qsub -cwd -pe orte 16 ./submit.sh<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â However, in the output of the calculation, the following line is<br>
> Â Â Â Â repeated<br>
> Â Â Â Â > Â Â 16 times:<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â Parallel version (MPI), running on 1 processors<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â It therefore seems like the program runs 16 1 processor<br>
> Â Â Â Â calculations that<br>
> Â Â Â Â > Â Â all write to the same output.<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â I wrote about this problem to the mailing list of Quantum<br>
> Â Â Â Â Espresso, and I<br>
> Â Â Â Â > Â Â got the suggestion that perhaps the mpirun belonged to a<br>
> Â Â Â Â different MPI<br>
> Â Â Â Â > Â Â library than pw.x (a particular package of Quantum Espresso) was<br>
> Â Â Â Â compiled<br>
> Â Â Â Â > Â Â with.<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â I compiled pw.x on the same cluster as I executed mpirun. Are<br>
> Â Â Â Â there<br>
> Â Â Â Â > Â Â several versions of openMPI on the AMIs provided by StarCluster?<br>
> Â Â Â Â In that<br>
> Â Â Â Â > Â Â case, how can I choose the correct one.<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â Perhaps the problem has a different cause. Does anyone have<br>
> Â Â Â Â suggestions on<br>
> Â Â Â Â > Â Â how to solve it?<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â Thanks in advance for your help.<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â Yours sincerely,<br>
> Â Â Â Â > Â Â Torstein Fjermestad<br>
> Â Â Â Â ><br>
> Â Â Â Â > References<br>
> Â Â Â Â ><br>
> Â Â Â Â > Â Â Visible links<br>
</div></div>> Â Â Â Â > Â Â 1. [6]<a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a><br>
><br>
> Â Â Â Â > _______________________________________________<br>
> Â Â Â Â > StarCluster mailing list<br>
> Â Â Â Â > [7]<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> Â Â Â Â > [8]<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
><br>
> Â Â Â _______________________________________________<br>
> Â Â Â StarCluster mailing list<br>
> Â Â Â [9]<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> Â Â Â [10]<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
><br>
> References<br>
><br>
> Â Â Visible links<br>
> Â Â 1. mailto:<a href="mailto:tfjermestad@gmail.com">tfjermestad@gmail.com</a><br>
> Â Â 2. <a href="http://www.open-mpi.org/community/help/" target="_blank">http://www.open-mpi.org/community/help/</a><br>
> Â Â 3. mailto:<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a><br>
> Â Â 4. <a href="http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment" target="_blank">http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment</a><br>
> Â Â 5. <a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a><br>
> Â Â 6. <a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a><br>
> Â Â 7. mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> Â Â 8. <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
> Â Â 9. mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> Â 10. <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<div class="HOEnZb"><div class="h5"><br>
> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br>
</div></div><br>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div>