<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">I just found openBLAS is already the default BLAS library in my customized AMI. And the problem is solved by simply setting "export OPENBLAS_NUM_THREADS=1". Thanks Rayson and Chris for your valuable suggestions. Without your prompt helps, I cannot figure out the issue so fast. <br>
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I have submitted 20000 jobs of ~100,000 cpu hours. I am using loadbalancer to gradually increase the size of cluster to ~5000 cores. Let's see how well the sge and starcluster will handler this.<br>
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks!<br>Jin<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 18, 2014 at 12:29 PM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The best way to handle this is to setup a PE in Grid Engine, so that the Grid Engine scheduler knows that your job uses more than 1 CPU core.<div>
<br></div><div>ALso, Grid Engine has the ability to bind a job to a CPU core, but ATLAS would still create 32 threads on a c3.8xlarge, as it still gets the full view of the hardware configuration. So with core binding, the job can only use 1 CPU, but ATLAS still creates 32 threads, and all 32 threads will be fighting for CPU resources. And ATLAS does not have a dynamic way to limit the number of threads it creates. IMO if R works with other BLAS libraries, then switching to use something like OpenBLAS and tell it to use only the number of cores Grid Engine assigns to the job would fix this issue. And if you have a large number of R jobs, then use a serial BLAS library can be a good choice too, because you have 32 jobs that each uses 1 virtual core of the c3.8xlarge.</div>
<div class="gmail_extra"><div class=""><br clear="all"><div>Rayson<br><br>==================================================<br>Open Grid Scheduler - The Official Open Source Grid Engine<br><a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br>
<a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div>
<br><br></div><div><div class="h5"><div class="gmail_quote">On Fri, Jul 18, 2014 at 12:01 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div style="font-family:verdana,sans-serif"><div style="font-family:verdana,sans-serif">Rayson,<br><br></div><div style="font-family:verdana,sans-serif">
Thank you
for pointing out the ATLAS in the starcluster AMI could creates
multiple threads by default in matrix manipulations. I checked my
running jobs again more carefully using htop and I found each job
created exactly 32 threads! That is why I have an overload of ~100. <br><br>I have qalter all the waiting jobs for a hard limit of using only one core. I will report back if it is not working.<br><br></div>Thanks!<span><font color="#888888"><br>
Jin</font></span></div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 18, 2014 at 10:04 AM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:verdana,sans-serif">Ha, I see. Probably this is the cause and I want to validate it. Do you know how to configure the job in SGE to force it using only one core per job?<br>
<br></div><div style="font-family:verdana,sans-serif">Thanks!<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div style="font-family:verdana,sans-serif">
Jin<br></div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Jul 18, 2014 at 12:09 AM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">The BLAS library in the StarCluster AMI is ATLAS, which takes advantage of multicore machines by running each BLAS call <b></b>using multiple threads.<br><div><div class="gmail_extra"><br></div><div class="gmail_extra">
I googled and found that SVD with ALTAS does use more than 1 core: <a href="http://blog.revolutionanalytics.com/2010/06/performance-benefits-of-multithreaded-r.html" target="_blank">http://blog.revolutionanalytics.com/2010/06/performance-benefits-of-multithreaded-r.html</a><br>
<br></div><div class="gmail_extra">This can explain the behavior you are getting...<br></div><div class="gmail_extra"><div><br clear="all"><div>Rayson<br><br>==================================================<br>
Open Grid Scheduler - The Official Open Source Grid Engine<br>
<a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br><a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div>
<br><br></div><div><div><div class="gmail_quote">On Thu, Jul 17, 2014 at 10:03 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div style="font-family:verdana,sans-serif">Hi Rayson,<br><br>My local virtualboxa also has more than one core, but the cpu usage is never more than 100% when I run it locally. I also checked it using htop, it only has one thread. <br>
<br>I don't have any multicores or parallel modules in my R codes, but I did have a couple of svd calls and exception handlers in my R code. <br><br></div><div style="font-family:verdana,sans-serif">
Thanks!<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div style="font-family:verdana,sans-serif">Jin<br></div></font></span></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 5:58 PM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">What R program or module are you using?? As there are 32 virtual cores on the C3.8xlarge, if your code by default creates 1 thread per virtual core, then there will be 3200% CPU usage on the C3.8xlarge. And also, may be your local machine has much fewer cores and that's why this is not happening??<br>
<br clear="all"><div class="gmail_extra"><div>Rayson<br><br>==================================================<br>Open Grid Scheduler - The Official Open Source Grid Engine<br><a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br>
<a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div><div><div>
<br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 6:06 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div style="font-family:verdana,sans-serif">Here is a followup of my investigation of the unusual high CPU/core usage in EC2 instances. <br></div><div style="font-family:verdana,sans-serif">
<br>In the last post, I reported my observations of 1. unusual high CPU/core usage of the R process in EC2 instances, which is designed to use one core on the local machine; And 2. unusual high percentage of kernel time in CPU usage. <br>
<br></div><div style="font-family:verdana,sans-serif">I looked more into the R processes using htop and found a lot of threads were created in each of them. And there are tons of sched_yield() system calls in each thread.<br>
<br></div><div style="font-family:verdana,sans-serif">Do these phenomenons with starcluster at EC2 ring a bell for someone?<br><br>Thanks!<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div style="font-family:verdana,sans-serif">
Jin<br></div></font></span><div><div><div style="font-family:verdana,sans-serif"><br><br></div><br></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div>
</div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div><br></div></div>