[Starcluster] NumPy and SciPy tests fail on Ubuntu 9.10 Karmic

[Justin Riley]

Hi Damian, 

I just launched the 32bit version on m1.small and ran the scipy/numpy tests in 
ipython:

~> numpy.test()
......
.......
OK (KNOWNFAIL=1, SKIP=11)
Out[3]: <nose.result.TextTestResult run=2030 errors=0 failures=0>


~> scipy.test()
...
....
OK (KNOWNFAIL=4, SKIP=17)
Out[5]: <nose.result.TextTestResult run=3490 errors=0 failures=0>

However when I run using "nosetests numpy scipy" like you I get a summary 
saying things failed. However, I don't see any issues with ptcblas and the 
likes so I'm not sure what's up.

I've always used numpy/scipy.test() and numpy/scipy.show_config() to validate 
the installation (eg proper detection of atlas and company) and I get a test 
pass when using them as you see above. So now I'm confused at which is the 
preferred method and why they produce different results. 

Regardless, the numpy/scipy that comes with ubuntu did not pass the tests that 
pass above in ipython using our custom compiled atlas/blas/lapack/numpy/scipy 
on the AMI. I compiled atlas on the largest instance in order to optimize for 
the c1.xlarge given that the atlas support from ubuntu is not optimized. Also, 
I may be incorrect here, but I seem to recall that ubuntu's atlas was also not 
threaded (ie missing the libpt*.so's).

BTW, here are the (pseudo) steps I used to create the AMI which includes the 
numpy/scipy/lapack/atlas setup:

http://starcluster.scripts.mit.edu/~starcluster/wiki/index.php?title=StarCluster_AMI_Cookbook

I'll try the numpy/scipy.test()'s with the 64bit ami and c1.xlarge instance.

~Justin

On Friday 16 April 2010 7:13:08 pm Damian Eads wrote:
> Hi Justin,
> 
> I decided to run my usual computer vision regressions but before I got
> there, I ran the NumPy and SciPy regression suite.
> 
> root at domU-12-31-38-01-58-B1:/data# nosetests numpy scipy
> EE
> ======================================================================
> ERROR: Failure: ImportError (/usr/lib/libptcblas.so: undefined symbol:
> ATL_cpttrsm)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.6/dist-packages/nose/loader.py", line
> 368, in loadTestsFromName
>     module = resolve_name(addr.module)
>   File "/usr/local/lib/python2.6/dist-packages/nose/util.py", line
> 334, in resolve_name
>     module = __import__('.'.join(parts_copy))
>   File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py",
> line 130, in <module>
>     import add_newdocs
>   File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py",
> line 9, in <module>
>     from lib import add_newdoc
>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py",
> line 13, in <module>
>     from polynomial import *
>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/polynomial.py",
> line 18, in <module>
>     from numpy.linalg import eigvals, lstsq
>   File "/usr/local/lib/python2.6/dist-packages/numpy/linalg/__init__.py",
> line 47, in <module>
>     from linalg import *
>   File "/usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py",
> line 22, in <module>
>     from numpy.linalg import lapack_lite
> ImportError: /usr/lib/libptcblas.so: undefined symbol: ATL_cpttrsm
> 
> ======================================================================
> ERROR: Failure: AttributeError ('module' object has no attribute 'core')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.6/dist-packages/nose/loader.py", line
> 368, in loadTestsFromName
>     module = resolve_name(addr.module)
>   File "/usr/local/lib/python2.6/dist-packages/nose/util.py", line
> 334, in resolve_name
>     module = __import__('.'.join(parts_copy))
>   File "/usr/local/lib/python2.6/dist-packages/scipy/__init__.py",
> line 11, in <module>
>     from numpy import show_config as show_numpy_config
>   File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py",
> line 130, in <module>
>     import add_newdocs
>   File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py",
> line 9, in <module>
>     from lib import add_newdoc
>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py",
> line 13, in <module>
>     from polynomial import *
>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/polynomial.py",
> line 11, in <module>
>     import numpy.core.numeric as NX
> AttributeError: 'module' object has no attribute 'core'
> 
> ----------------------------------------------------------------------
> Ran 2 tests in 0.001s
> 
> FAILED (errors=2)
> root at domU-12-31-38-01-58-B1:/data#
> 
> Is there a reason for not using the Ubuntu packages for numpy and scipy?
> 
> Just curious.
> 
> Thanks,
> 
> Damian
> 
> 
> 
> -----------------------------------------------------
> Damian Eads                           Ph.D. Candidate
> University of California             Computer Science
> 1156 High Street         Machine Learning Lab, E2-489
> Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads
> _______________________________________________
> Starcluster mailing list
> Starcluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>