[Starcluster] NumPy and SciPy tests fail on Ubuntu 9.10 Karmic

Sat Apr 17 18:16:55 EDT 2010

Hi Damian,

Good to hear you fixed the issue. I will certainly need to add a note about 
installing numpy/scipy/lapack/openmpi from ubuntu. These packages have been 
custom compiled and installed (see 
http://starcluster.scripts.mit.edu/~starcluster/wiki/index.php?title=StarCluster_AMI_Cookbook). 
Openmpi happens to be recompiled into a deb package so it's easy to reinstall, 
however, numpy/scipy/lapack require a bit more work if you clobber them with 
apt-get. 

Thanks for reporting this to the list,

~Justin

On Saturday 17 April 2010 6:07:14 pm Damian Eads wrote:
> I meant to include the list in my reply. I fixed my problem. The
> ptcblas error when running the NumPy and SciPy regression suites was a
> result of installing liblapack-dev, an optional dependency of one of
> my computer vision package. By not installing it this time, I no
> longer get the same error.
> 
> Cheers,
> 
> Damian
> 
> On Sat, Apr 17, 2010 at 4:24 PM, Damian Eads <eads at soe.ucsc.edu> wrote:
> > Hi Justin,
> >
> > As we discussed last night, I may have built my AMI off of yours
> > incorrectly by starting a cluster and building with
> > ec2-bundle-instance without excluding NFS-mounted directories
> > (whoops). I decided to rebuild my AMI off of yours again and the first
> > thing I did was run the NumPy and SciPy regression tests. This time,
> > I've launched your AMI using the EC2 web interface. For
> > reproducibility sake, the architecture I'm trying is c1.large.
> >
> > $ nosetests numpy
> > ...
> > ----------------------------------------------------------------------
> > Ran 2036 tests in 3.687s
> >
> > FAILED (SKIP=11, errors=7)
> >
> > $ nosetests scipy
> > ----------------------------------------------------------------------
> > Ran 4117 tests in 385.762s
> >
> > FAILED (SKIP=17, errors=10)
> >
> > Seven and ten failures respectively for NumPy and SciPy. Not bad. As
> > for the ptcblas error, it may have been a result of one of my computer
> > vision packages needing liblapack-dev. This may explain the complete
> > failure of the regressions as I explained in my earlier e-mail.
> >
> > Damian
> >
> > On Fri, Apr 16, 2010 at 11:11 PM, Justin Riley <jtriley at mit.edu> wrote:
> >> Hi Damian,
> >>
> >> I just launched the 32bit version on m1.small and ran the scipy/numpy
> >> tests in ipython:
> >>
> >> ~> numpy.test()
> >> ......
> >> .......
> >> OK (KNOWNFAIL=1, SKIP=11)
> >> Out[3]: <nose.result.TextTestResult run=2030 errors=0 failures=0>
> >>
> >>
> >> ~> scipy.test()
> >> ...
> >> ....
> >> OK (KNOWNFAIL=4, SKIP=17)
> >> Out[5]: <nose.result.TextTestResult run=3490 errors=0 failures=0>
> >>
> >> However when I run using "nosetests numpy scipy" like you I get a
> >> summary saying things failed. However, I don't see any issues with
> >> ptcblas and the likes so I'm not sure what's up.
> >>
> >> I've always used numpy/scipy.test() and numpy/scipy.show_config() to
> >> validate the installation (eg proper detection of atlas and company) and
> >> I get a test pass when using them as you see above. So now I'm confused
> >> at which is the preferred method and why they produce different results.
> >>
> >> Regardless, the numpy/scipy that comes with ubuntu did not pass the
> >> tests that pass above in ipython using our custom compiled
> >> atlas/blas/lapack/numpy/scipy on the AMI. I compiled atlas on the
> >> largest instance in order to optimize for the c1.xlarge given that the
> >> atlas support from ubuntu is not optimized. Also, I may be incorrect
> >> here, but I seem to recall that ubuntu's atlas was also not threaded (ie
> >> missing the libpt*.so's).
> >>
> >> BTW, here are the (pseudo) steps I used to create the AMI which includes
> >> the numpy/scipy/lapack/atlas setup:
> >>
> >> http://starcluster.scripts.mit.edu/~starcluster/wiki/index.php?title=Sta
> >>rCluster_AMI_Cookbook
> >>
> >> I'll try the numpy/scipy.test()'s with the 64bit ami and c1.xlarge
> >> instance.
> >>
> >> ~Justin
> >>
> >> On Friday 16 April 2010 7:13:08 pm Damian Eads wrote:
> >>> Hi Justin,
> >>>
> >>> I decided to run my usual computer vision regressions but before I got
> >>> there, I ran the NumPy and SciPy regression suite.
> >>>
> >>> root at domU-12-31-38-01-58-B1:/data# nosetests numpy scipy
> >>> EE
> >>> ======================================================================
> >>> ERROR: Failure: ImportError (/usr/lib/libptcblas.so: undefined symbol:
> >>> ATL_cpttrsm)
> >>> ----------------------------------------------------------------------
> >>> Traceback (most recent call last):
> >>>   File "/usr/local/lib/python2.6/dist-packages/nose/loader.py", line
> >>> 368, in loadTestsFromName
> >>>     module = resolve_name(addr.module)
> >>>   File "/usr/local/lib/python2.6/dist-packages/nose/util.py", line
> >>> 334, in resolve_name
> >>>     module = __import__('.'.join(parts_copy))
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py",
> >>> line 130, in <module>
> >>>     import add_newdocs
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py",
> >>> line 9, in <module>
> >>>     from lib import add_newdoc
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py",
> >>> line 13, in <module>
> >>>     from polynomial import *
> >>>   File
> >>> "/usr/local/lib/python2.6/dist-packages/numpy/lib/polynomial.py", line
> >>> 18, in <module>
> >>>     from numpy.linalg import eigvals, lstsq
> >>>   File
> >>> "/usr/local/lib/python2.6/dist-packages/numpy/linalg/__init__.py", line
> >>> 47, in <module>
> >>>     from linalg import *
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py",
> >>> line 22, in <module>
> >>>     from numpy.linalg import lapack_lite
> >>> ImportError: /usr/lib/libptcblas.so: undefined symbol: ATL_cpttrsm
> >>>
> >>> ======================================================================
> >>> ERROR: Failure: AttributeError ('module' object has no attribute
> >>> 'core')
> >>> ----------------------------------------------------------------------
> >>> Traceback (most recent call last):
> >>>   File "/usr/local/lib/python2.6/dist-packages/nose/loader.py", line
> >>> 368, in loadTestsFromName
> >>>     module = resolve_name(addr.module)
> >>>   File "/usr/local/lib/python2.6/dist-packages/nose/util.py", line
> >>> 334, in resolve_name
> >>>     module = __import__('.'.join(parts_copy))
> >>>   File "/usr/local/lib/python2.6/dist-packages/scipy/__init__.py",
> >>> line 11, in <module>
> >>>     from numpy import show_config as show_numpy_config
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py",
> >>> line 130, in <module>
> >>>     import add_newdocs
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py",
> >>> line 9, in <module>
> >>>     from lib import add_newdoc
> >>>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py",
> >>> line 13, in <module>
> >>>     from polynomial import *
> >>>   File
> >>> "/usr/local/lib/python2.6/dist-packages/numpy/lib/polynomial.py", line
> >>> 11, in <module>
> >>>     import numpy.core.numeric as NX
> >>> AttributeError: 'module' object has no attribute 'core'
> >>>
> >>> ----------------------------------------------------------------------
> >>> Ran 2 tests in 0.001s
> >>>
> >>> FAILED (errors=2)
> >>> root at domU-12-31-38-01-58-B1:/data#
> >>>
> >>> Is there a reason for not using the Ubuntu packages for numpy and
> >>> scipy?
> >>>
> >>> Just curious.
> >>>
> >>> Thanks,
> >>>
> >>> Damian
> >>>
> >>>
> >>>
> >>> -----------------------------------------------------
> >>> Damian Eads                           Ph.D. Candidate
> >>> University of California             Computer Science
> >>> 1156 High Street         Machine Learning Lab, E2-489
> >>> Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads
> >>> _______________________________________________
> >>> Starcluster mailing list
> >>> Starcluster at mit.edu
> >>> http://mailman.mit.edu/mailman/listinfo/starcluster
> 
> _______________________________________________
> Starcluster mailing list
> Starcluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
> 