<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Hi Chris,</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">
Thanks for your prompt reply and point me to look the unusual high load of the instance! And I found something more mysterious in EC2 instances (C3.8xlarge, to be more specific) :</div><div class="gmail_default" style="font-family:verdana,sans-serif">
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">1. I found some of my jobs are using CPU as much as 900%, although these job are designed to use only one core and behave so in my local machine, which lead to the unexpected high load of the system. Following is an example snapshot of these process. </div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">2. While all the 8 running jobs takes 3000% CPU which is close to the full of 32 cores. The kernel time takes up to 70% of the CPU time.</div>
<div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Are these problem related to the visualization nature of the EC2 instances? Can you give me a hint to investigate them?<br>
</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks!</div><div class="gmail_default" style="font-family:verdana,sans-serif">
Jin</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><img src="cid:ii_147460b67e1cb45a" alt="Inline image 1" style="margin-right: 0px;" width="624" height="18"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif">
<br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Jul 17, 2014 at 1:48 PM, Chris Dagdigian <span dir="ltr"><<a href="mailto:dag@bioteam.net" target="_blank">dag@bioteam.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hi Jin,<br>
<br>
The cluster is not accepting jobs into those open slots because your<br>
compute nodes are reporting alarm state "a" - your first host has a<br>
reported load average of 148!<br>
<br>
Alarm state 'a' means "load threshold alarm level reached" it basically<br>
means that the server load is high enough that the nodes are refusing<br>
new work until the load average goes down.<br>
<br>
All of those load alarm thresholds are configurable values within SGE so<br>
you can revise them upwards if you want<br>
<br>
Regards,<br>
Chris<br>
<div><div class="h5"><br>
<br>
Jin Yu wrote:<br>
> Hello,<br>
><br>
> I just started a cluster of 20 c3.8xlarge instances, which have 32<br>
> virtual cores in each. In my understanding, each instance should have<br>
> 32 slots available to run the jobs by default. But after running it<br>
> for a while, I found a lot of nodes are not running at the full speed.<br>
><br>
> Following as an example, you can see node016 has only 13 jobs running<br>
> and node017 has 9 jobs running, while node018 has 32 jobs running. I<br>
> have another ~10000 jobs waiting in the queue, so it is not a matter<br>
> of running out of jobs.<br>
><br>
> Can anyone give me a hint what is going on here?<br>
><br>
> Thanks!<br>
> Jin<br>
><br>
><br>
> all.q@node016 BIP 0/13/32 148.35 linux-x64<br>
> a<br>
> 784 0.55500 job.part.a sgeadmin r 07/17/2014 11:25:59<br>
> 1<br>
> 982 0.55500 job.part.a sgeadmin r 07/17/2014 14:43:59<br>
> 1<br>
> 1056 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:44<br>
> 1<br>
> 1057 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:44<br>
> 1<br>
> 1058 0.55500 job.part.a sgeadmin r 07/17/2014 16:34:59<br>
> 1<br>
> 1121 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1122 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1123 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1124 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1125 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1126 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1127 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> 1128 0.55500 job.part.a sgeadmin r 07/17/2014 17:22:44<br>
> 1<br>
> ---------------------------------------------------------------------------------<br>
> all.q@node017 BIP 0/9/32 83.86 linux-x64<br>
> a<br>
> 568 0.55500 job.part.a sgeadmin r 07/17/2014 04:01:14<br>
> 1<br>
> 1001 0.55500 job.part.a sgeadmin r 07/17/2014 15:07:29<br>
> 1<br>
> 1002 0.55500 job.part.a sgeadmin r 07/17/2014 15:07:29<br>
> 1<br>
> 1072 0.55500 job.part.a sgeadmin r 07/17/2014 16:53:29<br>
> 1<br>
> 1116 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:29<br>
> 1<br>
> 1117 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:29<br>
> 1<br>
> 1118 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:44<br>
> 1<br>
> 1119 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:59<br>
> 1<br>
> 1120 0.55500 job.part.a sgeadmin r 07/17/2014 17:19:59<br>
> 1<br>
> ---------------------------------------------------------------------------------<br>
> all.q@node018 BIP 0/32/32 346.00 linux-x64<br>
> a<br>
><br>
</div></div>> _______________________________________________<br>
> StarCluster mailing list<br>
> <a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
> <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br></div>