[StarCluster] strange root vs. user behavior for GPU nodes

[Charles Cadieu]

Hi,

I'm noticing some unusual behavior between running as a user vs. as root
when running some GPU code. As far as I can tell (PYTHONPATH, PATH) are
pretty much identical between the user and root, and both are using the
same relevant binaries. Is there some obvious step that's necessary to
allow the user code to run on the GPU?

Here's some more info on the problem:
I'm running some tutorial code from theano:
http://deeplearning.net/software/theano/tutorial/using_gpu.html

when I run as root I get output like this:
[root at node002 ~]# THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32
python2.6 thing.py
Setup took 0.873317956924 seconds
Looping 1000 times took 36.3292770386 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

[root at node002 ~]# THEANO_FLAGS=mode=FAST_RUN,device=gpu0,floatX=float32
python2.6 thing.py
Using gpu device 0: Tesla M2050
Setup took 6.41229701042 seconds
Looping 1000 times took 1.73471212387 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu

BUT, as a user, I get:
[render at node002 ~]$ THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32
python2.6 thing.py
Setup took 0.591382980347 seconds
Looping 1000 times took 36.3925471306 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

[render at node002 ~]$ THEANO_FLAGS=mode=FAST_RUN,device=gpu0,floatX=float32
python2.6 thing.py
Using gpu device 0: Tesla M2050
Setup took 186.97786808 seconds
Looping 1000 times took 182.75831604 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu

So, major slowdown, but otherwise no obvious errors.

Let me know if there's more info or something else I should try,
Charles
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20120123/1c99e1b3/attachment.htm