[StarCluster] newbie problems
Manal Helal
manal.helal at gmail.com
Sun May 20 02:28:17 EDT 2012
Hi Rayson,
Thanks again. Now even away from bidding, and using the same configuration
file that connected successfully 3 days ago,
I am consistently now getting this error:
!!! ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not
supported in your requested Availability Zone (us-east-1c). Please retry
your request by not specifying an Availability Zone or choosing us-east-1a.
I do as it says, and remove any region specifications, or specify the
"us-east-1a" region at says, and it still keep coming back with the same
error, 3 days now without successful connection with bidding and without.
I am not sure how to solve that
I appreciate your support very much.
manal
On 20 May 2012 12:48, Rayson Ho <raysonlogin at gmail.com> wrote:
> If you bid below the current spot price, then your request will take a
> long time (or never, if the spot price never goes lower than your bid
> price). The common practice is to bid above the current spot price and
> below the standard on-demand price to get the best chance of getting
> an instance and without over paying for spot instances.
>
> See the "Deciding on Your Spot Bidding Strategy" video tutorial
> created by AmazonWebServices:
>
> http://www.youtube.com/watch?v=WD9N73F3Fao
>
> Rayson
>
> ================================
> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
>
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
>
>
> On Sat, May 19, 2012 at 10:30 PM, Manal Helal <manal.helal at gmail.com>
> wrote:
> > Thanks Rayson, I just waited for too long on a one-time request, and
> > misunderstood the persistence feature, though it keeps on requesting
> during
> > waiting, not just after termination,
> >
> > thanks again,
> >
> > Manal
> >
> >
> > On 20 May 2012 10:29, Rayson Ho <raysonlogin at gmail.com> wrote:
> >>
> >> Is there a real use case for persistent spot request?
> >>
> >> http://aws.amazon.com/ec2/spot-instances/#1
> >>
> >> "... with a persistent request, after your request has been satisfied
> >> and your instance has been terminated—by you or by Amazon EC2—your
> >> request will be submitted again automatically with the same parameters
> >> as your initial request. A persistent request will continue submitting
> >> the request until you cancel it..."
> >>
> >> Rayson
> >>
> >> ================================
> >> Open Grid Scheduler / Grid Engine
> >> http://gridscheduler.sourceforge.net/
> >>
> >> Scalable Grid Engine Support Program
> >> http://www.scalablelogic.com/
> >>
> >>
> >> On Fri, May 18, 2012 at 10:03 PM, Manal Helal <manal.helal at gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > I also need information about the command line option for making a
> spot
> >> > request persistent. I am currently using:
> >> >
> >> > starcluster start -o -b mybid mycluster
> >> >
> >> > and in the AWS online GUI console, I see that this generates
> a one-time
> >> > spot
> >> > request not a persistent one. I googled on AWS or starcluster to find
> >> > out
> >> > how to change this after request or when I make the initial request,
> >> >
> >> > thank you very much for your support,
> >> >
> >> > Kind Regards,
> >> >
> >> > On 19 May 2012 11:17, Manal Helal <manalorama at gmail.com> wrote:
> >> >>
> >> >> Hi Rayson,
> >> >>
> >> >> Thank you very much for the extended explanations and for sharing
> your
> >> >> experience. I had to start fast, and read only quick-guides, and
> first
> >> >> two
> >> >> pages of everything, and googling what I need only,
> >> >>
> >> >> thanks again,
> >> >>
> >> >> Kind Regards,
> >> >>
> >> >> Manal
> >> >>
> >> >>
> >> >> On 19 May 2012 02:47, Rayson Ho <raysonlogin at gmail.com> wrote:
> >> >>>
> >> >>> Manal,
> >> >>>
> >> >>> Here's my spending back in Feb. I am also in the free tier, BTW.
> >> >>>
> >> >>> In that month, I used 27 hours of t1.micro & 1 hour of m1.small,
> thus
> >> >>> I got charged $ 0.09 ($0.085 rounded up!).
> >> >>>
> >> >>> Amazon EC2 running Linux/UNIX
> >> >>> $0.00 per Micro Instance (t1.micro) instance-hour
> (or
> >> >>> partial
> >> >>> hour) under monthly free tier 27 Hrs 0.00
> >> >>> $0.085 per Small Instance (m1.small) instance-hour
> (or
> >> >>> partial
> >> >>> hour) 1 Hr 0.09
> >> >>>
> >> >>> Amazon EC2 EBS
> >> >>> $0.00 per GB-month of provisioned storage under
> >> >>> monthly
> >> >>> free
> >> >>> tier 0.398 GB-Mo 0.00
> >> >>> $0.00 per 1 million I/O requests under monthly free
> >> >>> tier
> >> >>> 428,595 IOs 0.00
> >> >>>
> >> >>> Elastic IP Addresses
> >> >>> $0.00 per Elastic IP address remap - first 100
> remaps
> >> >>> /
> >> >>> month 1
> >> >>> Count 0.00
> >> >>>
> >> >>>
> >> >>> I am under the free EBS limit (it was then 10GB month, but the limit
> >> >>> got bumped up to 30 GB month recently). Note that you can use 60GB
> for
> >> >>> 15 days, then 0 GB for the rest of the month, and Amazon will not
> >> >>> charge you anything. I think Amazon has a very straightforward
> >> >>> pricing, and you don't need to worry about whether the EBS is used
> by
> >> >>> a t1.micro or a cg1.4xlarge or even just left there for backup...
> >> >>>
> >> >>> Basically, feel free to mix and match free & non-free EC2 resource.
> >> >>>
> >> >>> Rayson
> >> >>>
> >> >>> ================================
> >> >>> Open Grid Scheduler / Grid Engine
> >> >>> http://gridscheduler.sourceforge.net/
> >> >>>
> >> >>> Scalable Grid Engine Support Program
> >> >>> http://www.scalablelogic.com/
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, May 17, 2012 at 9:59 PM, Rayson Ho <raysonlogin at gmail.com>
> >> >>> wrote:
> >> >>> > So you are in the "free tier" indeed! Keep in mind that you get 30
> >> >>> > GB
> >> >>> > of free EBS storage, and if you are willing to pay, then you don't
> >> >>> > need to limit yourself to micro instances.
> >> >>> >
> >> >>> > For example, you can still use CG1 instances (which are not free
> >> >>> > even
> >> >>> > for those in the free tier) and you still get free 30GB of EBS
> >> >>> > storage. You basically can mix and match free & non-free stuff in
> >> >>> > AWS.
> >> >>> >
> >> >>> > Rayson
> >> >>> >
> >> >>> > ================================
> >> >>> > Open Grid Scheduler / Grid Engine
> >> >>> > http://gridscheduler.sourceforge.net/
> >> >>> >
> >> >>> > Scalable Grid Engine Support Program
> >> >>> > http://www.scalablelogic.com/
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > On Thu, May 17, 2012 at 9:27 PM, Manal Helal <
> manalorama at gmail.com>
> >> >>> > wrote:
> >> >>> >> Thank you very much Justin and Rayson for the explanation,
> >> >>> >>
> >> >>> >> I actually started the cluster Yesterday and learned a few issues
> >> >>> >> by
> >> >>> >> try and
> >> >>> >> error, realising the already mounted EBS in the starcluster HVM
> >> >>> >> AMI,
> >> >>> >> but
> >> >>> >> terminates with it and needs downloading the data. I also
> realised
> >> >>> >> the
> >> >>> >> unfeasibility of the thought of downloading an image, updating it
> >> >>> >> on
> >> >>> >> my
> >> >>> >> machine, and then upload it after testing and everytime I need
> it.
> >> >>> >> Also
> >> >>> >> using a spotnode cost less, so I can experiment with less
> worries,
> >> >>> >> but
> >> >>> >> the
> >> >>> >> IO free-tier is small, and only tiny instances are in the free
> >> >>> >> tier,
> >> >>> >> and I
> >> >>> >> need cg1.4xlarge to use the GPU Cluster,
> >> >>> >>
> >> >>> >> I tried the following steps:
> >> >>> >>
> >> >>> >>
> >> >>> >>
> http://instantbadger.blogspot.com.au/2009/09/how-to-create-and-save-ami-image-from.html
> >> >>> >>
> >> >>> >> to bundle, upload and register the customised running AMI after
> my
> >> >>> >> updates,
> >> >>> >> and the bundle took time indeed, but this seams like the
> >> >>> >> alternative
> >> >>> >> to the
> >> >>> >> scenario download/upload I was thinking of,
> >> >>> >>
> >> >>> >> thanks again for your help,
> >> >>> >>
> >> >>> >> Kind Regards,
> >> >>> >>
> >> >>> >>
> >> >>> >> On 18 May 2012 07:40, Rayson Ho <raysonlogin at gmail.com> wrote:
> >> >>> >>>
> >> >>> >>> On Thu, May 17, 2012 at 5:23 PM, Justin Riley <jtriley at mit.edu>
> >> >>> >>> wrote:
> >> >>> >>> > If you're trying to find a way to store data without having to
> >> >>> >>> > keep
> >> >>> >>> > the
> >> >>> >>> > cluster around there are much better approaches:
> >> >>> >>>
> >> >>> >>> Manal,
> >> >>> >>>
> >> >>> >>> Since you mentioned that you are new to AWS, you should be able
> to
> >> >>> >>> get
> >> >>> >>> the benefits of the free tier:
> >> >>> >>>
> >> >>> >>> http://aws.amazon.com/free/
> >> >>> >>>
> >> >>> >>> Basically, you can use 30 GB of EBS storage free of charge. On
> the
> >> >>> >>> other hand, if you pull data from AWS to your local harddrive
> >> >>> >>> everytime you are done with the instance, then you will need to
> >> >>> >>> pay
> >> >>> >>> for I/O & data transfer costs.
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> Justin,
> >> >>> >>>
> >> >>> >>> I am trying to find the version of the NVIDIA Driver & CUDA
> >> >>> >>> toolkit
> >> >>> >>> installed on the CG1 AMIs, but instead of booting up an instance
> >> >>> >>> to
> >> >>> >>> find out what's there, is there a page that lists the additional
> >> >>> >>> software packages installed in the AMIs?? So far, I could only
> >> >>> >>> find:
> >> >>> >>>
> >> >>> >>> https://github.com/jtriley/StarCluster/issues/9
> >> >>> >>>
> >> >>> >>> Rayson
> >> >>> >>>
> >> >>> >>> ================================
> >> >>> >>> Open Grid Scheduler / Grid Engine
> >> >>> >>> http://gridscheduler.sourceforge.net/
> >> >>> >>>
> >> >>> >>> Scalable Grid Engine Support Program
> >> >>> >>> http://www.scalablelogic.com/
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> >
> >> >>> >>> > 1. Use an external EBS volume with StarCluster and store your
> >> >>> >>> > data
> >> >>> >>> > there. This will allow you to terminate the cluster and still
> >> >>> >>> > preserve
> >> >>> >>> > your important data on the EBS volume. See:
> >> >>> >>> >
> >> >>> >>> >
> http://web.mit.edu/star/cluster/docs/latest/manual/volumes.html
> >> >>> >>> >
> >> >>> >>> > 2. If you don't wish to use external EBS volumes at all you
> can
> >> >>> >>> > always
> >> >>> >>> > use starcluster's "get" command to manually download files
> from
> >> >>> >>> > the
> >> >>> >>> > cluster before terminating, e.g.:
> >> >>> >>> >
> >> >>> >>> > $ starcluster get mycluster /path/to/results.tar.gz
> >> >>> >>> >
> >> >>> >>> > Similarly you can upload files using the "put" command:
> >> >>> >>> >
> >> >>> >>> > $ starcluster put mycluster /path/to/local/file /remote/path/
> >> >>> >>> >
> >> >>> >>> > See the following doc for more details:
> >> >>> >>> >
> >> >>> >>> >
> http://web.mit.edu/star/cluster/docs/latest/manual/putget.html
> >> >>> >>> >
> >> >>> >>> >> 4. After knowing how to download the image, I will need
> the
> >> >>> >>> >> command
> >> >>> >>> >> to
> >> >>> >>> >> start a new cluster using the image I have stored on my
> >> >>> >>> >> local
> >> >>> >>> >> disk.
> >> >>> >>> >> I
> >> >>> >>> >> think I have seen something to install from an image to
> the
> >> >>> >>> >> cluster
> >> >>> >>> >
> >> >>> >>> > Launching a cluster using a locally stored disk image is not
> >> >>> >>> > supported.
> >> >>> >>> > The latency in doing this would be *insane*. Having to upload
> >> >>> >>> > multiple
> >> >>> >>> > GBs and register an AMI everytime would make starting a
> cluster
> >> >>> >>> > *extremely* time-consuming and error-prone.
> >> >>> >>> >
> >> >>> >>> > As mentioned above you're better off creating a new AMI for
> your
> >> >>> >>> > needs
> >> >>> >>> > and keeping the new AMI on Amazon. Then simply update your
> >> >>> >>> > StarCluster
> >> >>> >>> > config to use your new AMI. If you need to save data before
> >> >>> >>> > terminating
> >> >>> >>> > the cluster then either save your data to an external EBS
> volume
> >> >>> >>> > or
> >> >>> >>> > copy
> >> >>> >>> > the data using the 'get' command as mentined above.
> >> >>> >>> >
> >> >>> >>> > Hope that helps,
> >> >>> >>> >
> >> >>> >>> > ~Justin
> >> >>> >>> >
> >> >>> >>> > _______________________________________________
> >> >>> >>> > StarCluster mailing list
> >> >>> >>> > StarCluster at mit.edu
> >> >>> >>> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >> >>> >>> >
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> --
> >> >>> >>> ==================================================
> >> >>> >>> Open Grid Scheduler - The Official Open Source Grid Engine
> >> >>> >>> http://gridscheduler.sourceforge.net/
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Kind Regards,
> >> >>> >>
> >> >>> >> Manal Helal
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > ==================================================
> >> >>> > Open Grid Scheduler - The Official Open Source Grid Engine
> >> >>> > http://gridscheduler.sourceforge.net/
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> ==================================================
> >> >>> Open Grid Scheduler - The Official Open Source Grid Engine
> >> >>> http://gridscheduler.sourceforge.net/
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> ==================================================
> >> Open Grid Scheduler - The Official Open Source Grid Engine
> >> http://gridscheduler.sourceforge.net/
> >
> >
> >
> >
>
>
>
> --
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20120520/92d585db/attachment-0001.htm
More information about the StarCluster
mailing list