<div dir="ltr"><div><div>Hi John,<br></div>Could you explain a little more about what a DRMAA job is and what resources it requires? Found something on wikipedia but it doesn't seem relevant.<br><br>I wrote big parts of the load balancer and am guessing that it does not understand your inter-machine dependencies. Sounds like your job is somewhat tolerant of hosts dropping off, but we can probably come up with a better solution.<br>
<br></div>Best,<br>Rajat<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 6, 2014 at 12:11 PM, Lilley, John F. <span dir="ltr"><<a href="mailto:johnbot@caltech.edu" target="_blank">johnbot@caltech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
<br>
I’m running a simple java based DRMAA job that runs a sleep commands on each of the Starcluster compute instances but am having a problem. If I create 10 non-loadbalanced nodes and then submit a job from DRMAA that runs sleep on each of those nodes in parallel everything completes fine.If I submit the same 10 node DRMAA sleep job with 1 non-loadbalanced node available everything works fine and the jobs eventually work their way through the single node serially and the main DRMAA process is happy.<br>
<br>
If I then enable load balancing, submit 10 jobs that sleep 30 minutes each from the main DRMAA process it load balances beautifully by adding 9 nodes, all 10 jobs complete and the main process exits gracefully. However, if I submit 10 jobs that sleep for 70 minutes most of them finish but then the DRMAA process bails before all 10 jobs are complete. My guess is that when the first sleep jobs start to finish up the load balancer removes the nodes they ran on from the available execution hosts throwing the main DRMAA process which is monitoring the jobs for a loop.<br>
<br>
<br>
Perhaps there’s a way I can make the DRMAA process more tolerant of execution hosts being removed from the available pool? Another issue I have is that unless I have 1 execution host running all the time the DRMAA process refuses to start at all. I’d rather not have to keep an execution host running to accept the DRMAA job submissions if possible. I would really appreciate hearing any insights the community has on running DRMAA jobs in Starcluster and if anyone has experienced similar obstacles.<br>
<br>
<br>
Thanks for the help!<br>
John<br>
<br>
<br>
Output received from DRMAA after an hour when load balancing (jobs over 70 minutes)<br>
--------------------------------------------------------------------------------------------------------------------<br>
INFO [2014-03-05 19:59:26,077] [OGSJob.java:211] [main] Waiting for job 61...<br>
Exception in thread "main" java.lang.IllegalStateException<br>
at com.sun.grid.drmaa.JobInfoImpl.getExitStatus(JobInfoImpl.java:75)<br>
at nextgen.core.job.OGSJob.waitFor(OGSJob.java:213)<br>
at nextgen.core.job.JobUtils.waitForAll(JobUtils.java:23)<br>
at tests.DrmaaSleepTest.main(DrmaaSleepTest.java:50)<br>
INFO [2014-03-05 19:59:37,064] [OGSUtils.java:84] [Thread-0] Ending DRMAA session<br>
--------------------------------------------------------------------------------------------------------------------<br>
<br>
Output received from DRMAA when there are no execution hosts available on initial job submission (If I have execution host available it submits OK)<br>
--------------------------------------------------------------------------------------------------------------------<br>
user@master:~/$ java -jar DrmaaSleepTest.jar -m 5 -n 10 -s Sleep.jar<br>
log4j:ERROR Could not find value for key log4j.appender.R<br>
log4j:ERROR Could not instantiate appender named "R".<br>
WARN [2014-03-06 05:09:54,984] [OGSUtils.java:65] [main] Starting a DRMAA session.<br>
WARN [2014-03-06 05:09:54,989] [OGSUtils.java:66] [main] There should only be one active DRMAA session at a time.<br>
INFO [2014-03-06 05:09:55,430] [OGSUtils.java:92] [main] Attached shutdown hook to close DRMAA session upon JVM exit.<br>
Exception in thread "main" org.ggf.drmaa.DeniedByDrmException: warning:user your job is not allowed to run in any queue<br>
error: no suitable queues<br>
at com.sun.grid.drmaa.SessionImpl.nativeRunJob(Native Method)<br>
at com.sun.grid.drmaa.SessionImpl.runJob(SessionImpl.java:349)<br>
at nextgen.core.job.OGSJob.submit(OGSJob.java:188)<br>
at tests.DrmaaSleepTest.main(DrmaaSleepTest.java:46)<br>
INFO [2014-03-06 05:09:55,500] [OGSUtils.java:84] [Thread-0] Ending DRMAA session<br>
--------------------------------------------------------------------------------------------------------------------<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
</blockquote></div><br></div>