Hi,<div>I've read several emails on this list about the delay in starting a large cluster.</div><div><br></div><div>As someone who has personally worked on StarCluster development, many if not all of the delays are with EC2. It is difficult to start virtual machines on demand, guarantee that they're up, wait for SSH and the filesystem shares to become available, and to present the machine as usable to the end user. There are probably some improvements that can be made to StarCluster, but the problem lies mainly with EC2.</div>
<div><br></div><div>One suggestion that may work for some would be to use the Elastic Load Balancer in StarCluster to gradually increase the size of your cluster until the desired size is attained. You could start the cluster with 50 nodes and submit all of your jobs to the SGE queue, launch the ELB and let ELB scale up the cluster. This would have several benefits:</div>
<div>- Potential time savings</div><div> - Jobs would start as soon as they're queued, won't wait for 200 nodes to be ready</div><div> - Jobs are already running while cluster adds node 50 through node 200 </div>
<div> - Less upfront, frustrating waiting</div><div>- Potential $ savings</div><div> - Not all nodes started at the same time, would save $ by starting some later</div><div> - ELB would shut down idle nodes when job queue empties out</div>
<div><br></div><div>Here is a sample command to start the ELB once the cluster of 50 (or even less) is up:</div><div><div><br></div><div>$ starcluster loadbalance -n 50 -m 200 -a 10 mycluster</div></div><div><br></div><div>
This would start the load balancer with:</div><div>'-n 50' = a minimum of 50 nodes</div><div>'-m 200' = a maximum of 200 nodes</div><div>'-a 10' = add 10 new nodes whenever ELB detects that jobs are waiting and nodes need to be added to the cluster</div>
<div>'mycluster' = your cluster name</div><div><br></div><div>There are a few other options described on this page:</div><div><a href="http://web.mit.edu/stardev/cluster/docs/latest/manual/load_balancer.html">http://web.mit.edu/stardev/cluster/docs/latest/manual/load_balancer.html</a></div>
<div><br></div><div>That can help, such as changing the stabilization time and polling interval.</div><div><br></div><div>Best,</div><div>Rajat</div><div><br></div>