<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Ha, I see. Probably this is the cause and I want to validate it. Do you know how to configure the job in SGE to force it using only one core per job?<br>
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks!<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Jin<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Jul 18, 2014 at 12:09 AM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">The BLAS library in the StarCluster AMI is ATLAS, which takes advantage of multicore machines by running each BLAS call <strong></strong>using multiple threads.<br><div><div class="gmail_extra"><br></div><div class="gmail_extra">
I googled and found that SVD with ALTAS does use more than 1 core: <a href="http://blog.revolutionanalytics.com/2010/06/performance-benefits-of-multithreaded-r.html" target="_blank">http://blog.revolutionanalytics.com/2010/06/performance-benefits-of-multithreaded-r.html</a><br>
<br></div><div class="gmail_extra">This can explain the behavior you are getting...<br></div><div class="gmail_extra"><div class=""><br clear="all"><div>Rayson<br><br>==================================================<br>
Open Grid Scheduler - The Official Open Source Grid Engine<br>
<a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br><a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div>
<br><br></div><div><div class="h5"><div class="gmail_quote">On Thu, Jul 17, 2014 at 10:03 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div style="font-family:verdana,sans-serif">Hi Rayson,<br><br>My local virtualboxa also has more than one core, but the cpu usage is never more than 100% when I run it locally. I also checked it using htop, it only has one thread. <br>
<br>I don't have any multicores or parallel modules in my R codes, but I did have a couple of svd calls and exception handlers in my R code. <br><br></div><div style="font-family:verdana,sans-serif">
Thanks!<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div style="font-family:verdana,sans-serif">Jin<br></div></font></span></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 5:58 PM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">What R program or module are you using?? As there are 32 virtual cores on the C3.8xlarge, if your code by default creates 1 thread per virtual core, then there will be 3200% CPU usage on the C3.8xlarge. And also, may be your local machine has much fewer cores and that's why this is not happening??<br>
<br clear="all"><div class="gmail_extra"><div>Rayson<br><br>==================================================<br>Open Grid Scheduler - The Official Open Source Grid Engine<br><a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br>
<a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div><div><div>
<br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 6:06 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div style="font-family:verdana,sans-serif">Here is a followup of my investigation of the unusual high CPU/core usage in EC2 instances. <br></div><div style="font-family:verdana,sans-serif">
<br>In the last post, I reported my observations of 1. unusual high CPU/core usage of the R process in EC2 instances, which is designed to use one core on the local machine; And 2. unusual high percentage of kernel time in CPU usage. <br>
<br></div><div style="font-family:verdana,sans-serif">I looked more into the R processes using htop and found a lot of threads were created in each of them. And there are tons of sched_yield() system calls in each thread.<br>
<br></div><div style="font-family:verdana,sans-serif">Do these phenomenons with starcluster at EC2 ring a bell for someone?<br><br>Thanks!<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div style="font-family:verdana,sans-serif">
Jin<br></div></font></span><div><div><div style="font-family:verdana,sans-serif"><br><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 3:48 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:verdana,sans-serif">Hi Chris,</div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif">
Thanks for your prompt reply and point me to look the unusual high load of the instance! And I found something more mysterious in EC2 instances (C3.8xlarge, to be more specific) :</div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif">1. I found some of my jobs are using CPU as much as 900%, although these job are designed to use only one core and behave so in my local machine, which lead to the unexpected high load of the system. Following is an example snapshot of these process. </div>
<div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">2. While all the 8 running jobs takes 3000% CPU which is close to the full of 32 cores. The kernel time takes up to 70% of the CPU time.</div>
<div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">Are these problem related to the visualization nature of the EC2 instances? Can you give me a hint to investigate them?<br>
</div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">Thanks!</div><div style="font-family:verdana,sans-serif">
Jin</div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif"><img src="cid:ii_147460b67e1cb45a" alt="Inline image 1" style="margin-right:0px" height="18" width="624"><br></div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif"><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Jul 17, 2014 at 1:48 PM, Chris Dagdigian <span dir="ltr"><<a href="mailto:dag@bioteam.net" target="_blank">dag@bioteam.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Hi Jin,<br>
<br>
The cluster is not accepting jobs into those open slots because your<br>
compute nodes are reporting alarm state "a" - your first host has a<br>
reported load average of 148!<br>
<br>
Alarm state 'a' means "load threshold alarm level reached" it basically<br>
means that the server load is high enough that the nodes are refusing<br>
new work until the load average goes down.<br>
<br>
All of those load alarm thresholds are configurable values within SGE so<br>
you can revise them upwards if you want<br>
<br>
Regards,<br>
Chris<br>
<div><div><br>
<br>
Jin Yu wrote:<br>
> Hello,<br>
><br>
> I just started a cluster of 20 c3.8xlarge instances, which have 32<br>
> virtual cores in each. In my understanding, each instance should have<br>
> 32 slots available to run the jobs by default. But after running it<br>
> for a while, I found a lot of nodes are not running at the full speed.<br>
><br>
> Following as an example, you can see node016 has only 13 jobs running<br>
> and node017 has 9 jobs running, while node018 has 32 jobs running. I<br>
> have another ~10000 jobs waiting in the queue, so it is not a matter<br>
> of running out of jobs.<br>
><br>
> Can anyone give me a hint what is going on here?<br>
><br>
> Thanks!<br>
> Jin<br>
><br>
><br>
> all.q@node016 BIP 0/13/32 148.35 linux-x64<br>
> a<br>
> 784 0.55500 job.part.a sgeadmin r 07/17/2014 11:25:59<br>
> 1<br>
> 982 0.55500 job.part.a sgeadmin r 07/17/2014 14:43:59<br>
> 1<br>
> 1056 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:44<br>
> 1<br>
> 1057 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:44<br>
> 1<br>
> 1058 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:59<br>
> 1<br>
> 1121 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1122 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1123 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1124 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1125 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1126 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1127 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1128 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> ---------------------------------------------------------------------------------<br>
> all.q@node017 BIP 0/9/32 83.86 linux-x64<br>
> a<br>
> 568 0.55500 job.part.a sgeadmin r 07/17/2014 04:01:14<br>
> 1<br>
> 1001 0.55500 job.part.a sgeadmin r 07/17/2014 15:07:29<br>
> 1<br>
> 1002 0.55500 job.part.a sgeadmin r 07/17/2014 15:07:29<br>
> 1<br>
> 1072 0.55500 job.part.a sgeadmin r 07/17/2014 16:53:29<br>
> 1<br>
> 1116 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:29<br>
> 1<br>
> 1117 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:29<br>
> 1<br>
> 1118 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:44<br>
> 1<br>
> 1119 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:59<br>
> 1<br>
> 1120 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:59<br>
> 1<br>
> ---------------------------------------------------------------------------------<br>
> all.q@node018 BIP 0/32/32 346.00 linux-x64<br>
> a<br>
><br>
</div></div>> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
<br>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>