[StarCluster] newbie problems

Sat May 19 22:48:35 EDT 2012

If you bid below the current spot price, then your request will take a
long time (or never, if the spot price never goes lower than your bid
price). The common practice is to bid above the current spot price and
below the standard on-demand price to get the best chance of getting
an instance and without over paying for spot instances.

See the "Deciding on Your Spot Bidding Strategy" video tutorial
created by AmazonWebServices:

http://www.youtube.com/watch?v=WD9N73F3Fao

Rayson

================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/

On Sat, May 19, 2012 at 10:30 PM, Manal Helal <manal.helal at gmail.com> wrote:
> Thanks Rayson, I just waited for too long on a one-time request, and
> misunderstood the persistence  feature, though it keeps on requesting during
> waiting, not just after termination,
>
> thanks again,
>
> Manal
>
>
> On 20 May 2012 10:29, Rayson Ho <raysonlogin at gmail.com> wrote:
>>
>> Is there a real use case for persistent spot request?
>>
>> http://aws.amazon.com/ec2/spot-instances/#1
>>
>> "... with a persistent request, after your request has been satisfied
>> and your instance has been terminated—by you or by Amazon EC2—your
>> request will be submitted again automatically with the same parameters
>> as your initial request. A persistent request will continue submitting
>> the request until you cancel it..."
>>
>> Rayson
>>
>> ================================
>> Open Grid Scheduler / Grid Engine
>> http://gridscheduler.sourceforge.net/
>>
>> Scalable Grid Engine Support Program
>> http://www.scalablelogic.com/
>>
>>
>> On Fri, May 18, 2012 at 10:03 PM, Manal Helal <manal.helal at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I also need information about the command line option for making a spot
>> > request persistent. I am currently using:
>> >
>> > starcluster start -o -b mybid mycluster
>> >
>> > and in the AWS online GUI console, I see that this generates a one-time
>> > spot
>> > request not a persistent one. I googled on AWS or starcluster to find
>> > out
>> > how to change this after request or when I make the initial request,
>> >
>> > thank you very much for your support,
>> >
>> > Kind Regards,
>> >
>> > On 19 May 2012 11:17, Manal Helal <manalorama at gmail.com> wrote:
>> >>
>> >> Hi Rayson,
>> >>
>> >> Thank you very much for the extended explanations and for sharing your
>> >> experience. I had to start fast, and read only quick-guides, and first
>> >> two
>> >> pages of everything, and googling what I need only,
>> >>
>> >> thanks again,
>> >>
>> >> Kind Regards,
>> >>
>> >> Manal
>> >>
>> >>
>> >> On 19 May 2012 02:47, Rayson Ho <raysonlogin at gmail.com> wrote:
>> >>>
>> >>> Manal,
>> >>>
>> >>> Here's my spending back in Feb. I am also in the free tier, BTW.
>> >>>
>> >>> In that month, I used 27 hours of t1.micro & 1 hour of m1.small, thus
>> >>> I got charged $ 0.09 ($0.085 rounded up!).
>> >>>
>> >>>        Amazon EC2 running Linux/UNIX
>> >>>                 $0.00 per Micro Instance (t1.micro) instance-hour (or
>> >>> partial
>> >>> hour) under monthly free tier    27 Hrs  0.00
>> >>>                 $0.085 per Small Instance (m1.small) instance-hour (or
>> >>> partial
>> >>> hour)    1 Hr       0.09
>> >>>
>> >>>        Amazon EC2 EBS
>> >>>                 $0.00 per GB-month of provisioned storage under
>> >>> monthly
>> >>> free
>> >>> tier     0.398 GB-Mo     0.00
>> >>>                 $0.00 per 1 million I/O requests under monthly free
>> >>> tier
>> >>>        428,595 IOs     0.00
>> >>>
>> >>>        Elastic IP Addresses
>> >>>                 $0.00 per Elastic IP address remap - first 100 remaps
>> >>> /
>> >>> month   1
>> >>> Count    0.00
>> >>>
>> >>>
>> >>> I am under the free EBS limit (it was then 10GB month, but the limit
>> >>> got bumped up to 30 GB month recently). Note that you can use 60GB for
>> >>> 15 days, then 0 GB for the rest of the month, and Amazon will not
>> >>> charge you anything. I think Amazon has a very straightforward
>> >>> pricing, and you don't need to worry about whether the EBS is used by
>> >>> a t1.micro or a cg1.4xlarge or even just left there for backup...
>> >>>
>> >>> Basically, feel free to mix and match free & non-free EC2 resource.
>> >>>
>> >>> Rayson
>> >>>
>> >>> ================================
>> >>> Open Grid Scheduler / Grid Engine
>> >>> http://gridscheduler.sourceforge.net/
>> >>>
>> >>> Scalable Grid Engine Support Program
>> >>> http://www.scalablelogic.com/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, May 17, 2012 at 9:59 PM, Rayson Ho <raysonlogin at gmail.com>
>> >>> wrote:
>> >>> > So you are in the "free tier" indeed! Keep in mind that you get 30
>> >>> > GB
>> >>> > of free EBS storage, and if you are willing to pay, then you don't
>> >>> > need to limit yourself to micro instances.
>> >>> >
>> >>> > For example, you can still use CG1 instances (which are not free
>> >>> > even
>> >>> > for those in the free tier) and you still get free 30GB of EBS
>> >>> > storage. You basically can mix and match free & non-free stuff in
>> >>> > AWS.
>> >>> >
>> >>> > Rayson
>> >>> >
>> >>> > ================================
>> >>> > Open Grid Scheduler / Grid Engine
>> >>> > http://gridscheduler.sourceforge.net/
>> >>> >
>> >>> > Scalable Grid Engine Support Program
>> >>> > http://www.scalablelogic.com/
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Thu, May 17, 2012 at 9:27 PM, Manal Helal <manalorama at gmail.com>
>> >>> > wrote:
>> >>> >> Thank you very much Justin and Rayson for the explanation,
>> >>> >>
>> >>> >> I actually started the cluster Yesterday and learned a few issues
>> >>> >> by
>> >>> >> try and
>> >>> >> error, realising the already mounted EBS in the starcluster HVM
>> >>> >> AMI,
>> >>> >> but
>> >>> >> terminates with it and needs downloading the data. I also realised
>> >>> >> the
>> >>> >> unfeasibility of the thought of downloading an image, updating it
>> >>> >> on
>> >>> >> my
>> >>> >> machine, and then upload it after testing and everytime I need it.
>> >>> >> Also
>> >>> >> using a spotnode cost less, so I can experiment with less worries,
>> >>> >> but
>> >>> >> the
>> >>> >> IO free-tier is small, and only tiny instances are in the free
>> >>> >> tier,
>> >>> >> and I
>> >>> >> need cg1.4xlarge to use the GPU Cluster,
>> >>> >>
>> >>> >> I tried the following steps:
>> >>> >>
>> >>> >>
>> >>> >> http://instantbadger.blogspot.com.au/2009/09/how-to-create-and-save-ami-image-from.html
>> >>> >>
>> >>> >> to bundle, upload and register the customised running AMI after my
>> >>> >> updates,
>> >>> >> and the bundle took time indeed, but this seams like the
>> >>> >> alternative
>> >>> >> to the
>> >>> >> scenario download/upload I was thinking of,
>> >>> >>
>> >>> >> thanks again for your help,
>> >>> >>
>> >>> >> Kind Regards,
>> >>> >>
>> >>> >>
>> >>> >> On 18 May 2012 07:40, Rayson Ho <raysonlogin at gmail.com> wrote:
>> >>> >>>
>> >>> >>> On Thu, May 17, 2012 at 5:23 PM, Justin Riley <jtriley at mit.edu>
>> >>> >>> wrote:
>> >>> >>> > If you're trying to find a way to store data without having to
>> >>> >>> > keep
>> >>> >>> > the
>> >>> >>> > cluster around there are much better approaches:
>> >>> >>>
>> >>> >>> Manal,
>> >>> >>>
>> >>> >>> Since you mentioned that you are new to AWS, you should be able to
>> >>> >>> get
>> >>> >>> the benefits of the free tier:
>> >>> >>>
>> >>> >>> http://aws.amazon.com/free/
>> >>> >>>
>> >>> >>> Basically, you can use 30 GB of EBS storage free of charge. On the
>> >>> >>> other hand, if you pull data from AWS to your local harddrive
>> >>> >>> everytime you are done with the instance, then you will need to
>> >>> >>> pay
>> >>> >>> for I/O & data transfer costs.
>> >>> >>>
>> >>> >>>
>> >>> >>> Justin,
>> >>> >>>
>> >>> >>> I am trying to find the version of the NVIDIA Driver & CUDA
>> >>> >>> toolkit
>> >>> >>> installed on the CG1 AMIs, but instead of booting up an instance
>> >>> >>> to
>> >>> >>> find out what's there, is there a page that lists the additional
>> >>> >>> software packages installed in the AMIs?? So far, I could only
>> >>> >>> find:
>> >>> >>>
>> >>> >>> https://github.com/jtriley/StarCluster/issues/9
>> >>> >>>
>> >>> >>> Rayson
>> >>> >>>
>> >>> >>> ================================
>> >>> >>> Open Grid Scheduler / Grid Engine
>> >>> >>> http://gridscheduler.sourceforge.net/
>> >>> >>>
>> >>> >>> Scalable Grid Engine Support Program
>> >>> >>> http://www.scalablelogic.com/
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>> >
>> >>> >>> > 1. Use an external EBS volume with StarCluster and store your
>> >>> >>> > data
>> >>> >>> > there. This will allow you to terminate the cluster and still
>> >>> >>> > preserve
>> >>> >>> > your important data on the EBS volume. See:
>> >>> >>> >
>> >>> >>> > http://web.mit.edu/star/cluster/docs/latest/manual/volumes.html
>> >>> >>> >
>> >>> >>> > 2. If you don't wish to use external EBS volumes at all you can
>> >>> >>> > always
>> >>> >>> > use starcluster's "get" command to manually download files from
>> >>> >>> > the
>> >>> >>> > cluster before terminating, e.g.:
>> >>> >>> >
>> >>> >>> > $ starcluster get mycluster /path/to/results.tar.gz
>> >>> >>> >
>> >>> >>> > Similarly you can upload files using the "put" command:
>> >>> >>> >
>> >>> >>> > $ starcluster put mycluster /path/to/local/file /remote/path/
>> >>> >>> >
>> >>> >>> > See the following doc for more details:
>> >>> >>> >
>> >>> >>> > http://web.mit.edu/star/cluster/docs/latest/manual/putget.html
>> >>> >>> >
>> >>> >>> >>    4. After knowing how to download the image, I will need the
>> >>> >>> >> command
>> >>> >>> >> to
>> >>> >>> >>    start a new cluster using the image I have stored on my
>> >>> >>> >> local
>> >>> >>> >> disk.
>> >>> >>> >> I
>> >>> >>> >>    think I have seen something to install from an image to the
>> >>> >>> >> cluster
>> >>> >>> >
>> >>> >>> > Launching a cluster using a locally stored disk image is not
>> >>> >>> > supported.
>> >>> >>> > The latency in doing this would be *insane*. Having to upload
>> >>> >>> > multiple
>> >>> >>> > GBs and register an AMI everytime would make starting a cluster
>> >>> >>> > *extremely* time-consuming and error-prone.
>> >>> >>> >
>> >>> >>> > As mentioned above you're better off creating a new AMI for your
>> >>> >>> > needs
>> >>> >>> > and keeping the new AMI on Amazon. Then simply update your
>> >>> >>> > StarCluster
>> >>> >>> > config to use your new AMI. If you need to save data before
>> >>> >>> > terminating
>> >>> >>> > the cluster then either save your data to an external EBS volume
>> >>> >>> > or
>> >>> >>> > copy
>> >>> >>> > the data using the 'get' command as mentined above.
>> >>> >>> >
>> >>> >>> > Hope that helps,
>> >>> >>> >
>> >>> >>> > ~Justin
>> >>> >>> >
>> >>> >>> > _______________________________________________
>> >>> >>> > StarCluster mailing list
>> >>> >>> > StarCluster at mit.edu
>> >>> >>> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >>> >>> >
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>> --
>> >>> >>> ==================================================
>> >>> >>> Open Grid Scheduler - The Official Open Source Grid Engine
>> >>> >>> http://gridscheduler.sourceforge.net/
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Kind Regards,
>> >>> >>
>> >>> >> Manal Helal
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > ==================================================
>> >>> > Open Grid Scheduler - The Official Open Source Grid Engine
>> >>> > http://gridscheduler.sourceforge.net/
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> ==================================================
>> >>> Open Grid Scheduler - The Official Open Source Grid Engine
>> >>> http://gridscheduler.sourceforge.net/
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>>
>>
>>
>> --
>> ==================================================
>> Open Grid Scheduler - The Official Open Source Grid Engine
>> http://gridscheduler.sourceforge.net/
>
>
>
>

-- 
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/