<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Here is a followup of my investigation of the unusual high CPU/core usage in EC2 instances. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif">
<br>In the last post, I reported my observations of 1. unusual high CPU/core usage of the R process in EC2 instances, which is designed to use one core on the local machine; And 2. unusual high percentage of kernel time in CPU usage. <br>
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I looked more into the R processes using htop and found a lot of threads were created in each of them. And there are tons of sched_yield() system calls in each thread.<br>
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Do these phenomenons with starcluster at EC2 ring a bell for someone?<br><br>Thanks!<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">
Jin<br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 17, 2014 at 3:48 PM, Jin Yu <span dir="ltr"><<a href="mailto:yujin2004@gmail.com" target="_blank">yujin2004@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:verdana,sans-serif">Hi Chris,</div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif">
Thanks for your prompt reply and point me to look the unusual high load of the instance! And I found something more mysterious in EC2 instances (C3.8xlarge, to be more specific) :</div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif">1. I found some of my jobs are using CPU as much as 900%, although these job are designed to use only one core and behave so in my local machine, which lead to the unexpected high load of the system. Following is an example snapshot of these process. </div>
<div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">2. While all the 8 running jobs takes 3000% CPU which is close to the full of 32 cores. The kernel time takes up to 70% of the CPU time.</div>
<div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">Are these problem related to the visualization nature of the EC2 instances? Can you give me a hint to investigate them?<br>
</div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">Thanks!</div><div style="font-family:verdana,sans-serif">
Jin</div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif"><img src="cid:ii_147460b67e1cb45a" alt="Inline image 1" style="margin-right: 0px;" height="18" width="624"><br></div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif">
<br></div><div style="font-family:verdana,sans-serif"><br></div><div style="font-family:verdana,sans-serif"><br></div></div><div class=""><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Jul 17, 2014 at 1:48 PM, Chris Dagdigian <span dir="ltr"><<a href="mailto:dag@bioteam.net" target="_blank">dag@bioteam.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Hi Jin,<br>
<br>
The cluster is not accepting jobs into those open slots because your<br>
compute nodes are reporting alarm state "a" - your first host has a<br>
reported load average of 148!<br>
<br>
Alarm state 'a' means "load threshold alarm level reached" it basically<br>
means that the server load is high enough that the nodes are refusing<br>
new work until the load average goes down.<br>
<br>
All of those load alarm thresholds are configurable values within SGE so<br>
you can revise them upwards if you want<br>
<br>
Regards,<br>
Chris<br>
<div><div><br>
<br>
Jin Yu wrote:<br>
> Hello,<br>
><br>
> I just started a cluster of 20 c3.8xlarge instances, which have 32<br>
> virtual cores in each. In my understanding, each instance should have<br>
> 32 slots available to run the jobs by default. But after running it<br>
> for a while, I found a lot of nodes are not running at the full speed.<br>
><br>
> Following as an example, you can see node016 has only 13 jobs running<br>
> and node017 has 9 jobs running, while node018 has 32 jobs running. I<br>
> have another ~10000 jobs waiting in the queue, so it is not a matter<br>
> of running out of jobs.<br>
><br>
> Can anyone give me a hint what is going on here?<br>
><br>
> Thanks!<br>
> Jin<br>
><br>
><br>
> all.q@node016 BIP 0/13/32 148.35 linux-x64<br>
> a<br>
> 784 0.55500 job.part.a sgeadmin r 07/17/2014 11:25:59<br>
> 1<br>
> 982 0.55500 job.part.a sgeadmin r 07/17/2014 14:43:59<br>
> 1<br>
> 1056 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:44<br>
> 1<br>
> 1057 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:44<br>
> 1<br>
> 1058 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:59<br>
> 1<br>
> 1121 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1122 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1123 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1124 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1125 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1126 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1127 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1128 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> ---------------------------------------------------------------------------------<br>
> all.q@node017 BIP 0/9/32 83.86 linux-x64<br>
> a<br>
> 568 0.55500 job.part.a sgeadmin r 07/17/2014 04:01:14<br>
> 1<br>
> 1001 0.55500 job.part.a sgeadmin r 07/17/2014 15:07:29<br>
> 1<br>
> 1002 0.55500 job.part.a sgeadmin r 07/17/2014 15:07:29<br>
> 1<br>
> 1072 0.55500 job.part.a sgeadmin r 07/17/2014 16:53:29<br>
> 1<br>
> 1116 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:29<br>
> 1<br>
> 1117 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:29<br>
> 1<br>
> 1118 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:44<br>
> 1<br>
> 1119 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:59<br>
> 1<br>
> 1120 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:59<br>
> 1<br>
> ---------------------------------------------------------------------------------<br>
> all.q@node018 BIP 0/32/32 346.00 linux-x64<br>
> a<br>
><br>
</div></div>> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>