<div dir="ltr"><div><div><div><div><div>Dear Justin and Gonçalo, <br><br></div>thank you very much. It worked!<br><br></div>I first ran the two suggested commands:<br><br>$ update-alternatives --config mpi<br>
$ update-alternatives --config mpirun<br>
<br></div>and then I reconfigured and recompiled the application. <br><br></div>Regards,<br></div>Torstein<br><br><div><div><div><div><br></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Apr 28, 2014 at 4:43 PM, Justin Riley <span dir="ltr"><<a href="mailto:jtriley@mit.edu" target="_blank">jtriley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Gonçalo,<br>
<br>
Ah, I thought this sounded familiar:<br>
<br>
<a href="https://github.com/jtriley/StarCluster/issues/370" target="_blank">https://github.com/jtriley/StarCluster/issues/370</a><br>
<br>
Thanks for responding. This will be fixed in the upcoming 14.04 AMIs.<br>
<br>
Torstein, you can update the MPI links interactively by running the<br>
following commands as root:<br>
<br>
$ update-alternatives --config mpi<br>
$ update-alternatives --config mpirun<br>
<br>
Select either all openmpi or all mpich paths at the interactive prompts.<br>
<br>
~Justin<br>
<div><div class="h5"><br>
On Mon, Apr 28, 2014 at 02:06:50PM +0200, Gonçalo Albuquerque wrote:<br>
> Hi,<br>
> When using AMI ami-6b211202 in us-east I stumbled across the same issue<br>
> you're experiencing.<br>
> The symbolic links in the alternatives system are mixing MPICH and<br>
> OpenMPI:<br>
> root@master:/etc/alternatives# update-alternatives --display mpi<br>
> mpi - auto mode<br>
> link currently points to /usr/include/mpich2<br>
> /usr/include/mpich2 - priority 40<br>
> slave libmpi++.so: /usr/lib/libmpichcxx.so<br>
> slave libmpi.so: /usr/lib/libmpich.so<br>
> slave libmpif77.so: /usr/lib/libfmpich.so<br>
> slave libmpif90.so: /usr/lib/libmpichf90.so<br>
> slave mpic++: /usr/bin/mpic++.mpich2<br>
> slave mpic++.1.gz: /usr/share/man/man1/mpic++.mpich2.1.gz<br>
> slave mpicc: /usr/bin/mpicc.mpich2<br>
> slave mpicc.1.gz: /usr/share/man/man1/mpicc.mpich2.1.gz<br>
> slave mpicxx: /usr/bin/mpicxx.mpich2<br>
> slave mpicxx.1.gz: /usr/share/man/man1/mpicxx.mpich2.1.gz<br>
> slave mpif77: /usr/bin/mpif77.mpich2<br>
> slave mpif77.1.gz: /usr/share/man/man1/mpif77.mpich2.1.gz<br>
> slave mpif90: /usr/bin/mpif90.mpich2<br>
> slave mpif90.1.gz: /usr/share/man/man1/mpif90.mpich2.1.gz<br>
> /usr/lib/openmpi/include - priority 40<br>
> slave libmpi++.so: /usr/lib/openmpi/lib/libmpi_cxx.so<br>
> slave libmpi.so: /usr/lib/openmpi/lib/libmpi.so<br>
> slave libmpif77.so: /usr/lib/openmpi/lib/libmpi_f77.so<br>
> slave libmpif90.so: /usr/lib/openmpi/lib/libmpi_f90.so<br>
> slave mpiCC: /usr/bin/mpic++.openmpi<br>
> slave mpiCC.1.gz: /usr/share/man/man1/mpiCC.openmpi.1.gz<br>
> slave mpic++: /usr/bin/mpic++.openmpi<br>
> slave mpic++.1.gz: /usr/share/man/man1/mpic++.openmpi.1.gz<br>
> slave mpicc: /usr/bin/mpicc.openmpi<br>
> slave mpicc.1.gz: /usr/share/man/man1/mpicc.openmpi.1.gz<br>
> slave mpicxx: /usr/bin/mpic++.openmpi<br>
> slave mpicxx.1.gz: /usr/share/man/man1/mpicxx.openmpi.1.gz<br>
> slave mpif77: /usr/bin/mpif77.openmpi<br>
> slave mpif77.1.gz: /usr/share/man/man1/mpif77.openmpi.1.gz<br>
> slave mpif90: /usr/bin/mpif90.openmpi<br>
> slave mpif90.1.gz: /usr/share/man/man1/mpif90.openmpi.1.gz<br>
> Current 'best' version is '/usr/include/mpich2'.<br>
> root@master:/etc/alternatives# update-alternatives --display mpirun<br>
> mpirun - auto mode<br>
> link currently points to /usr/bin/mpirun.openmpi<br>
> /usr/bin/mpirun.mpich2 - priority 40<br>
> slave mpiexec: /usr/bin/mpiexec.mpich2<br>
> slave mpiexec.1.gz: /usr/share/man/man1/mpiexec.mpich2.1.gz<br>
> slave mpirun.1.gz: /usr/share/man/man1/mpirun.mpich2.1.gz<br>
> /usr/bin/mpirun.openmpi - priority 50<br>
> slave mpiexec: /usr/bin/mpiexec.openmpi<br>
> slave mpiexec.1.gz: /usr/share/man/man1/mpiexec.openmpi.1.gz<br>
> slave mpirun.1.gz: /usr/share/man/man1/mpirun.openmpi.1.gz<br>
> Current 'best' version is '/usr/bin/mpirun.openmpi'.<br>
> You do compile it with MPICH and try to run with OpenMPI. The solution is<br>
> to change the symbolic links by using the update-alternatives command. For<br>
> the runtime link (mpirun), it must be done in all the nodes of the<br>
> cluster.<br>
> No doubt this will be corrected in upcoming versions of the AMIs.<br>
> Regards,<br>
> Gonçalo<br>
><br>
> On Mon, Apr 28, 2014 at 1:09 PM, Torstein Fjermestad<br>
</div></div><div class="">> <[1]<a href="mailto:tfjermestad@gmail.com">tfjermestad@gmail.com</a>> wrote:<br>
><br>
> Dear Justin,<br>
> <br>
> during the compilation, the cluster only consisted of the master node<br>
> which is of instance type c3.large. In order to run a test parallel<br>
> calculation, I added a node of instance type c3.4xlarge (16 processors).<br>
><br>
> The cluster is created form the following AMI: <br>
> [0] ami-044abf73 eu-west-1 starcluster-base-ubuntu-13.04-x86_64 (EBS)<br>
><br>
> Executing the application outside the queuing system like<br>
><br>
> mpirun -np 2 -hostfile hosts ./pw.x -in inputfile.inp<br>
><br>
> did not change anything.<br>
><br>
> The output of the command "mpirun --version" is the following:<br>
><br>
> mpirun (Open MPI) 1.4.5<br>
><br>
</div>> Report bugs to [2]<a href="http://www.open-mpi.org/community/help/" target="_blank">http://www.open-mpi.org/community/help/</a><br>
<div><div class="h5">><br>
> After investigating the matter a little bit, I found that mpif90 is<br>
> likely compiled with an MPI version different from mpirun.<br>
> The first line of the output of the command "mpif90 -v" is the<br>
> following:<br>
> <br>
> mpif90 for MPICH2 version 1.4.1<br>
><br>
> Furthermore, the output of the command "ldd pw.x" indicates that pw.x is<br>
> compiled with mpich2 and not with Open MPI. The output is the following:<br>
> <br>
> linux-vdso.so.1 => (0x00007fffd35fe000)<br>
> liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007ff38fb18000)<br>
> libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007ff38e2f5000)<br>
> libmpich.so.3 => /usr/lib/libmpich.so.3 (0x00007ff38df16000)<br>
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0<br>
> (0x00007ff38dcf9000)<br>
> libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3<br>
> (0x00007ff38d9e5000)<br>
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff38d6df000)<br>
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1<br>
> (0x00007ff38d4c9000)<br>
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff38d100000)<br>
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff38cef7000)<br>
> libcr.so.0 => /usr/lib/libcr.so.0 (0x00007ff38cced000)<br>
> libmpl.so.1 => /usr/lib/libmpl.so.1 (0x00007ff38cae8000)<br>
> /lib64/ld-linux-x86-64.so.2 (0x00007ff390820000)<br>
> libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0<br>
> (0x00007ff38c8b2000)<br>
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff38c6ae000)<br>
><br>
> The feedback I got from the Quantum Espresso mailing list suggested that<br>
> the cause of the error could be that pw.x (the executable) was not<br>
> compiled with the same version of mpi as mpirun.<br>
> The output of the commands "mpirun --version", "mpif90 -v" and "ldd<br>
> pw.x" above have lead me to suspect that this is indeed the case.<br>
> I therefore wonder whether it is possible to control which mpi version I<br>
> compile my applications with.<br>
><br>
> If, with the current mpi installation, the applications are compiled<br>
> with a different mpi version than mpirun, then I will likely have<br>
> similar problems when compiling other applications as well. I would<br>
> therefore very much appreciate if you could give me some hints on how I<br>
> can solve this problem.<br>
><br>
> Thanks in advance.<br>
><br>
> Regards,<br>
> Torstein<br>
><br>
</div></div>> On Thu, Apr 24, 2014 at 5:13 PM, Justin Riley <[3]<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a>><br>
<div class="">> wrote:<br>
><br>
> Hi Torstein,<br>
><br>
> Can you please describe your cluster configuration (ie size, image<br>
> id(s),<br>
> instance type(s))? Also, you're currently using the SGE/OpenMPI<br>
> integration. Have you tried just using mpirun only as described in the<br>
> first part of:<br>
><br>
</div>> [4]<a href="http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment" target="_blank">http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment</a><br>
<div class="">><br>
> Also, what does 'mpirun --version' show?<br>
><br>
> ~Justin<br>
> On Thu, Apr 17, 2014 at 07:19:28PM +0200, Torstein Fjermestad wrote:<br>
> > Dear all,<br>
> ><br>
> > I recently tried to compile an application (Quantum Espresso,<br>
</div>> > [1][5]<a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a>) to be used for parallel<br>
<div><div class="h5">> computations<br>
> > on StarCluster. The installation procedure of the application<br>
> consists of<br>
> > the standard "./configure + make" steps. At the end of the<br>
> output from<br>
> > ./configure, the statement "Parallel environment detected<br>
> successfully.\<br>
> > Configured for compilation of parallel executables." appears.<br>
> ><br>
> > The compilation with "make" completes without errors. I then run<br>
> the<br>
> > application in the following way:<br>
> ><br>
> > I first write a submit script (submit.sh) with the following<br>
> content:<br>
> ><br>
> > cp /path/to/executable/pw.x .<br>
> > mpirun ./pw.x -in input.inp<br>
> > I then submit the job to the queueing system with the following<br>
> command<br>
> > <br>
> > qsub -cwd -pe orte 16 ./submit.sh<br>
> ><br>
> > However, in the output of the calculation, the following line is<br>
> repeated<br>
> > 16 times:<br>
> ><br>
> > Parallel version (MPI), running on 1 processors<br>
> ><br>
> > It therefore seems like the program runs 16 1 processor<br>
> calculations that<br>
> > all write to the same output.<br>
> ><br>
> > I wrote about this problem to the mailing list of Quantum<br>
> Espresso, and I<br>
> > got the suggestion that perhaps the mpirun belonged to a<br>
> different MPI<br>
> > library than pw.x (a particular package of Quantum Espresso) was<br>
> compiled<br>
> > with.<br>
> ><br>
> > I compiled pw.x on the same cluster as I executed mpirun. Are<br>
> there<br>
> > several versions of openMPI on the AMIs provided by StarCluster?<br>
> In that<br>
> > case, how can I choose the correct one.<br>
> ><br>
> > Perhaps the problem has a different cause. Does anyone have<br>
> suggestions on<br>
> > how to solve it?<br>
> ><br>
> > Thanks in advance for your help.<br>
> ><br>
> > Yours sincerely,<br>
> > Torstein Fjermestad<br>
> ><br>
> > References<br>
> ><br>
> > Visible links<br>
</div></div>> > 1. [6]<a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a><br>
><br>
> > _______________________________________________<br>
> > StarCluster mailing list<br>
> > [7]<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> > [8]<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
><br>
> _______________________________________________<br>
> StarCluster mailing list<br>
> [9]<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> [10]<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
><br>
> References<br>
><br>
> Visible links<br>
> 1. mailto:<a href="mailto:tfjermestad@gmail.com">tfjermestad@gmail.com</a><br>
> 2. <a href="http://www.open-mpi.org/community/help/" target="_blank">http://www.open-mpi.org/community/help/</a><br>
> 3. mailto:<a href="mailto:jtriley@mit.edu">jtriley@mit.edu</a><br>
> 4. <a href="http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment" target="_blank">http://star.mit.edu/cluster/docs/latest/guides/sge.html#submitting-openmpi-jobs-using-a-parallel-environment</a><br>
> 5. <a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a><br>
> 6. <a href="http://www.quantum-espresso.org/" target="_blank">http://www.quantum-espresso.org/</a><br>
> 7. mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> 8. <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
> 9. mailto:<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> 10. <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<div class="HOEnZb"><div class="h5"><br>
> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br>
</div></div><br>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div>