[StarCluster] A way to automatically terminate an EC2 instance

[Rayson Ho]

BTW, I recall we had discussions related to automatically transferring data back to the local machine...

I think you can add a few lines in starcluster/balancers/sge/__init__.py so that before it shuts down the master
node, it transfers data back to the local machine first.

If you don't want to modify starcluster, just create a separate EBS for the output directory and use the load
balancer to shutdown the nodes, or tell the balancer to not kill the master.


Rayson

=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/



----- Original Message -----
From: "Napolitano, Diane" <dnapolitano at ets.org>
To: Rayson Ho <raysonlogin at yahoo.com>; "starcluster at mit.edu" <starcluster at mit.edu>
Cc: 
Sent: Friday, May 4, 2012 11:53 AM
Subject: RE: [StarCluster] A way to automatically terminate an EC2 instance

Aha!  Perfect, thank you!

- Diane

-----Original Message-----
From: Rayson Ho [mailto:raysonlogin at yahoo.com] 
Sent: Friday, May 04, 2012 11:03 AM
To: Napolitano, Diane; starcluster at mit.edu
Subject: Re: [StarCluster] A way to automatically terminate an EC2 instance

Use the Elastic Load Balancer to shutdown idle instances:


http://web.mit.edu/star/cluster/docs/latest/manual/load_balancer.html

The master instance can be shutdown as well, as long as you define an EBS that is not
the root filesystem of the instance.


Rayson

=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/




----- Original Message -----
From: "Napolitano, Diane" <dnapolitano at ets.org>
To: "starcluster at mit.edu" <starcluster at mit.edu>
Cc: 
Sent: Friday, May 4, 2012 9:41 AM
Subject: [StarCluster] A way to automatically terminate an EC2 instance

Hello, I hope someone will point me to the answer to my question, if it exists; if not, what I'm looking to do is the following:

Let's say I have an EC2 instance running several jobs that take some unknown number of hours (possibly as long as a day).  They generate a whole bunch of output files, which need to be downloaded back to my local servers before I can terminate the instance; I don't have S3 here.

Now, because these jobs could take almost a day to finish, what I'd like to do is NOT sit there watching them run all day. :)  I'd like some automatic way to tell EC2, be it with starcluster commands or otherwise, to logout, download all the data to my local servers, and terminate the instance the minute the jobs finish.  That way I'm not stuck watching the jobs all day, but I'm not wasting money on an idle instance should the jobs finish at a time when I'm doing something crazy like sleeping. :)

Any recommendations on how to do this?

Thanks,
Diane

_______________________________________________
StarCluster mailing list
StarCluster at mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster