<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Alexey,<br>
<br>
The Sun Grid Engine queueing system is useful when you have a lot of
tasks to execute and not just one at a time interactively. For
example, you might need to convert 300 videos from one format to
another. You could either<br>
<br>
1. Write a script that gets the list of nodes from /etc/hosts and
then loops over the jobs and the nodes, ssh'ing commands to be
executed on each node. A big problem with this approach is that the
task execution and management all depends on this script executing
successfully all the way through. What happens if the script fails?
You would then lose all task accounting information. Also, what if
you suddenly discover you need to do another batch of 300 videos
while the previous batch is still processing? Are you going to
re-execute your script and overload the cluster? This would
definitely slow down all of your jobs. How will you write your
script to avoid overloading the cluster in this situation without
losing the fact that you want to submit new jobs *now*?<br>
<br>
OR<br>
<br>
2. Skip needing to get the list of nodes and ssh'ing commands to
them and instead just write a loop that sends 300 jobs to the
queuing system using "qsub". The queuing system will then do the
work to find an available node, execute the job, and store it's
accounting information (status, start time, end time, which node
executed the job, etc) . The queuing system will also handle load
balancing your tasks across the cluster so that any one node doesn't
get significantly overloaded compared to the other nodes in the
cluster. If you suddenly discover you need 300 more videos processed
you could simply "qsub" 300 more jobs. These jobs will be
'queued-up' and executed when a node becomes available. This
approach reduces your concerns to just executing a task on a node
rather than managing multiple jobs and nodes.<br>
<br>
Also it is true that you can create "as many clusters as you want"
with cloud computing. However, in many cases it could get *very*
expensive launching multiple clusters for every single task or set
of tasks. Whether it's more cost effective to launch multiple
clusters or just queue a ton of jobs on a single cluster depends
highly on the sort of tasks you're executing.<br>
<br>
Of course, just because a queueing system is installed doesn't mean
you *have* to use it at all. You can of course run things however
you want on the cluster. Hopefully I've made it clear that there are
significant advantages to using a queuing system to execute jobs on
a cluster rather than a home-brewed script.<br>
<br>
Hope that helps...<br>
<br>
~Justin<br>
<br>
On 10/22/10 5:02 PM, Alexey PETROV wrote:
<blockquote
cite="mid:AANLkTimzvLrLgzMMhO9FO5Vc+4tL4zSYM7FYH27JUdDq@mail.gmail.com"
type="cite">Ye, StartCluster is a great.
<div>But, what for do we need to use whatever "<span
class="Apple-style-span" style="font-family: sans-serif;
font-size: 13px; line-height: 19px;"><i>queuing system".</i></span>
<div><span class="Apple-style-span" style="font-family:
sans-serif; font-size: 13px; line-height: 19px;">Surely, in
cloud computing, user can create as many clusters as he
wants, each for his particular tasks.</span></div>
<div><span class="Apple-style-span" style="font-family:
sans-serif; font-size: 13px; line-height: 19px;">So, why?!</span></div>
</div>
</blockquote>
<br>
</body>
</html>