[StarCluster] help starting up cluster
Archie Russell
archier at gmail.com
Thu Feb 3 12:39:51 EST 2011
Thanks Justin, I got it
The trick is to put that empty " " before AWS_ACCESS_KEY_ID
/AWS_SECRET_ACCCESS_KEY,
and what tripped me up, was no space before the value of AWS_USER_ID.
Not sure why its needed but with it things seem to work and without it they
don't
-A
PS The only other person online I found with the problem attributed it to
character set encoding issues w python 2.7
Hi Archie,
Are you using quotes in your AWS_ACCESS_KEY_ID/AWS_SECRET_ACCCESS_KEY?
Also please be sure to double check you're not missing any characters
at the end or leaving any characters out. You can copy your access key
and secret access key directly from this page:
http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key
You will need to click the "Show" link next to your "Acess Key ID" in
order to access your secret access key.
They should look something like the example keys shown here:
http://s3.amazonaws.com/mturk/tools/pages/aws-access-identifiers/access_key_data.png
When you paste them into your config file the result should look
something like (no quotes!):
[aws info]
AWS_ACCESS_KEY_ID = 2S44JYKSZZ3D44ETWF02
AWS_SECRET_ACCESS_KEY = kl1GhCR9L9m0pw5bYR0+GyC1jsy5Fd0g/B\xjM/A
Hope that helps,
~Justin
On Wed, Feb 2, 2011 at 3:19 AM, <starcluster-request at mit.edu> wrote:
> Send StarCluster mailing list submissions to
> starcluster at mit.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.mit.edu/mailman/listinfo/starcluster
> or, via email, send a message with subject or body 'help' to
> starcluster-request at mit.edu
>
> You can reach the person managing the list at
> starcluster-owner at mit.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of StarCluster digest..."
>
>
> Today's Topics:
>
> 1. Re: help starting up cluster (Justin Riley)
> 2. Re: Privilege level Amazon GPU AMI. (Justin Riley)
> 3. Re: ERROR - "createvolume" failure due to missing AMI in
> eu-west zone (Justin Riley)
> 4. Re: Starcluster and elastic load balancing
> (Kyeong Soo (Joseph) Kim)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 1 Feb 2011 12:36:20 -0500
> From: Justin Riley <justin.t.riley at gmail.com>
> Subject: Re: [StarCluster] help starting up cluster
> To: starcluster at mit.edu
> Message-ID:
> <AANLkTikD5hQ7g4U9iLKWUpii+a3E7wBQLE_WhmwBNFAJ at mail.gmail.com<AANLkTikD5hQ7g4U9iLKWUpii%2Ba3E7wBQLE_WhmwBNFAJ at mail.gmail.com>
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Archie,
>
> Are you using quotes in your AWS_ACCESS_KEY_ID/AWS_SECRET_ACCCESS_KEY?
> Also please be sure to double check you're not missing any characters
> at the end or leaving any characters out. You can copy your access key
> and secret access key directly from this page:
>
>
> http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key
>
> You will need to click the "Show" link next to your "Acess Key ID" in
> order to access your secret access key.
>
> They should look something like the example keys shown here:
>
>
> http://s3.amazonaws.com/mturk/tools/pages/aws-access-identifiers/access_key_data.png
>
> When you paste them into your config file the result should look
> something like (no quotes!):
>
> [aws info]
> AWS_ACCESS_KEY_ID = 2S44JYKSZZ3D44ETWF02
> AWS_SECRET_ACCESS_KEY = kl1GhCR9L9m0pw5bYR0+GyC1jsy5Fd0g/B\xjM/A
>
> Hope that helps,
>
> ~Justin
>
> On Mon, Jan 31, 2011 at 9:37 PM, Archie Russell <archier at gmail.com> wrote:
> >
> > I get this error when I startup
> > % starcluster -d start myfirstcluster
> > StarCluster - (http://web.mit.edu/starcluster)
> > Software Tools for Academics and Researchers (STAR)
> > Please submit bug reports to starcluster at mit.edu
> > config.py:281 - DEBUG - Loading config
> > config.py:99 - DEBUG - Loading file: /users/archie/.starcluster/config
> >>>> Using default cluster template: smallcluster
> > awsutils.py:46 - DEBUG - creating self._conn w/ connection_authenticator
> > kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True}
> >>>> Validating cluster template settings...
> > awsutils.py:46 - DEBUG - creating self._conn w/ connection_authenticator
> > kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True}
> > cluster.py:766 - ERROR - Invalid AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
> > combination.
> > cli.py:243 - ERROR - settings for cluster template "smallcluster" are not
> > valid
> > AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY are in my config file and they
> > appear to be OK.
> >
> > I *can* start using ec2-run-instances on the commandline, ?e.g.
> > ec2-run-instances -K /mumble/privatekey.pem -C /mumble/certificate.pem
> > ami-4bf71e22 -z us-east-1d -k mumble-keypair -t c1.xlarge -n 1 2>&1
> > Has anyone seen this before?
> >
> >
> >
> > _______________________________________________
> > StarCluster mailing list
> > StarCluster at mit.edu
> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >
> >
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 1 Feb 2011 12:39:06 -0500
> From: Justin Riley <justin.t.riley at gmail.com>
> Subject: Re: [StarCluster] Privilege level Amazon GPU AMI.
> To: starcluster at mit.edu
> Message-ID:
> <AANLkTikMGmp73A0JfvLj6DrtRfx_c1XktLyc2JOLL1mz at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi,
>
> What is the exact error you get when trying to run OpenCL as a
> non-root user? Do you have the same issue using CUDA? I'll try to
> launch an instance and see what the deal is in the next couple days.
>
> ~Justin
>
> On Mon, Jan 17, 2011 at 7:37 AM, Progga <proggaprogga at gmail.com> wrote:
> > Hi,
> > I was trying to run some OpenCL code yesterday on Amazon EC2 GPU
> > cluster using the Star cluster AMI. ?Everything works fine if I am
> > logged in as root, but it fails when I am a non-root user. ?I googled
> > and found that /dev/nvidia* needs to have 0666 permission. ?But Star
> > cluster already have these, but still I have to run everything as
> > root. ?Is this a known issue? ?Thanks.
> >
> >
> > [0] http://forums.nvidia.com/index.php?showtopic=167152
> > [1] http://forums.nvidia.com/index.php?showtopic=156836
> > [2] http://forums.nvidia.com/index.php?showtopic=81071
> > _______________________________________________
> > StarCluster mailing list
> > StarCluster at mit.edu
> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 1 Feb 2011 13:25:08 -0500
> From: Justin Riley <justin.t.riley at gmail.com>
> Subject: Re: [StarCluster] ERROR - "createvolume" failure due to
> missing AMI in eu-west zone
> To: "Kyeong Soo (Joseph) Kim" <kyeongsoo.kim at gmail.com>
> Cc: starcluster at mit.edu
> Message-ID:
> <AANLkTinXD6RcnBSW58=67XhzrPHju9j8KjnJE=ZT6M3O at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Kyeong,
>
> Sorry you've run into this issue. This is happening because the
> us-east-1 StarCluster AMI is hard-coded in the createvolume command in
> StarCluster 0.91.2 and cannot be modified via the config or an option
> to the createvolume command. The latest github code, however, allows
> you to pass an --image-id flag that allows you to switch to another
> AMI when creating new volumes. In general any Linux AMI should work in
> theory provided it has the standard sfdisk/mkfs.ext3 commands that are
> commonly available on most Linux distros. You can also change the mkfs
> command used by the createvolume command via the --mkfs-cmd flag if
> needed. If you're interested in testing the latest code please note
> that you will need to shutdown all clusters previously launched using
> the 0.91* release before using the development version. Also any
> volumes created can only be used with the development version. Any
> volumes created with the development version will be compatible with
> the upcoming 0.92 release.
>
> Hope that helps,
>
> ~Justin
>
>
> On Wed, Jan 12, 2011 at 11:41 AM, Kyeong Soo (Joseph) Kim
> <kyeongsoo.kim at gmail.com> wrote:
> > Dear All,
> > This is to report the said failure of "createvolume" in the eu-west zone.
> > Per the starcluster documentation, I have set environment variable and
> > configuration file as follows:
> > * Environment variable
> > EC2_URL=https://ec2.eu-west-1.amazonaws.com
> > * Configuration file
> > =============================================
> > [aws_info]
> > ...
> > AWS_REGION_NAME = eu-west-1
> > AWS_REGION_HOS?=?ec2.eu-west-1.amazonaws.com
> > ...
> >
> > [cluster smallcluster]
> > ...
> > # AMI for cluster nodes.
> > # US region
> > # - The base i386 StarCluster AMI is ami-d1c42db8
> > # - The base x86_64 StarCluster AMI is ami-a5c42dcc
> > # EU region
> > # - The base i386 StarCluster AMI is ami-a4d7fdd0
> > # - The base x86_64 StarCluster AMI is ami-38d5ff4c
> > NODE_IMAGE_ID = ami-a4d7fdd0
> > NODE_INSTANCE_TYPE = m1.small
> > ...
> > =============================================
> > Because the default AMIs suggested in the template (i.e., "ami-d1c42db8"
> and
> > "ami-a5c42dcc")?are not available in the eu-west region, I changed it to
> one
> > available (based on "listpublic" command).?Then, I got the following
> error
> > when trying to create a volume:
> > =============================================
> > ~$ starcluster createvolume 30 eu-west-1a
> > StarCluster - (http://web.mit.edu/starcluster)
> > Software Tools for Academics and Researchers (STAR)
> > Please submit bug reports to starcluster at mit.edu
> > cli.py:1079 - ERROR - AMI ami-d1c42db8 does not exist
> > =============================================
> > It seems that the setting of NODE_IMAGE_ID has no impact on the AMI used
> for
> > volume creation.
> > With Regards,
> > Joseph
> > --
> > Kyeong Soo (Joseph) Kim, Ph.D.
> > Senior Lecturer in Networking
> > Room 112, Digital Technium
> > Multidisciplinary Nanotechnology Centre, College of Engineering
> > Swansea University, Singleton Park, Swansea SA2 8PP, Wales UK
> > TEL: +44 (0)1792 602024
> > EMAIL: k.s.kim_at_swansea.ac.uk
> > HOME: http://iat-hnrl.swan.ac.uk/ (group)
> > ? ? ?? ? ? ?http://iat-hnrl.swan.ac.uk/~kks/ (personal)
> >
> > _______________________________________________
> > StarCluster mailing list
> > StarCluster at mit.edu
> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >
> >
>
>
>
> ------------------------------
>
> Message: 4
> Date: Wed, 2 Feb 2011 11:19:32 +0000
> From: "Kyeong Soo (Joseph) Kim" <kyeongsoo.kim at gmail.com>
> Subject: Re: [StarCluster] Starcluster and elastic load balancing
> To: Rajat Banerjee <rbanerj at fas.harvard.edu>
> Cc: starcluster at mit.edu, archier at gmail.com
> Message-ID:
> <AANLkTikYs2SKvHFxcggbDDs1dd48=w=RKeiPxtBDecvF at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Rajat,
>
> Great thanks for the updates and the new graphs!
>
> You must be quite busy now working on the final copy.
> Of course, I do look forward to seeing either version soon.
>
> In the meanwhile, I will try the ELB myself and get back to the list, if
> there anything to report.
>
> Regards,
> Joseph
> --
> Kyeong Soo (Joseph) Kim, Ph.D.
> Senior Lecturer in Networking
> Room 112, Digital Technium
> Multidisciplinary Nanotechnology Centre, College of Engineering
> Swansea University, Singleton Park, Swansea SA2 8PP, Wales UK
> TEL: +44 (0)1792 602024
> EMAIL: k.s.kim_at_swansea.ac.uk
> HOME: http://iat-hnrl.swan.ac.uk/ (group)
> http://iat-hnrl.swan.ac.uk/~kks/ (personal)
>
>
> On Tue, Feb 1, 2011 at 4:23 AM, Rajat Banerjee <rbanerj at fas.harvard.edu
> >wrote:
>
> > Hello Joseph,
> > I really appreciate your words. I am pleased that the load balancer will
> be
> > put to good use.
> >
> > I have written my masters thesis on this topic and it is very near
> > completion. The final copy is due within the next two weeks, after which
> I
> > will post a PDF to this list for everyone to view. Due to some onerous
> > requirements, the document is over 50 pages. I hope to create an abridged
> > version soon.
> >
> > In the meantime, ELB is fully operational (just like the 2nd death star).
> I
> > hope to clean up the visualizer next week so others can use it. Justin
> has
> > already given me useful suggestions that I simply need to implement.
> >
> > Here are some graphs that I've been toying with for the thesis
> experiment:
> > http://dl.dropbox.com/u/224960/sc/index.html
> >
> > Best,
> > Rajat
> >
> >
> > On Mon, Jan 31, 2011 at 4:09 AM, Kyeong Soo (Joseph) Kim <
> > kyeongsoo.kim at gmail.com> wrote:
> >
> >> Hello Rajat,
> >>
> >> I am very interested in your work on the elastic load balancing; I do
> >> remember that you posted some graphs on early results in the past and
> that
> >> you were working on your MSc thesis.
> >>
> >> In fact, this new feature will be critical for my current research
> >> requiring about 3~400 independent simulation runs and I do highly
> appreciate
> >> your great contribution to the StarCluster.
> >>
> >> By the way, I wonder whether you have published your work in any
> >> conferences/journals yet.
> >>
> >> Regards,
> >> Joseph
> >> --
> >> Kyeong Soo (Joseph) Kim, Ph.D.
> >> Senior Lecturer in Networking
> >> Room 112, Digital Technium
> >> Multidisciplinary Nanotechnology Centre, College of Engineering
> >> Swansea University, Singleton Park, Swansea SA2 8PP, Wales UK
> >> TEL: +44 (0)1792 602024
> >> EMAIL: k.s.kim_at_swansea.ac.uk
> >> HOME: http://iat-hnrl.swan.ac.uk/ (group)
> >> http://iat-hnrl.swan.ac.uk/~kks/<
> http://iat-hnrl.swan.ac.uk/%7Ekks/>(personal)
> >>
> >>
> >>
> >> On Fri, Jan 28, 2011 at 6:31 PM, Rajat Banerjee <
> rbanerj at fas.harvard.edu>wrote:
> >>
> >>> Hi Archie,
> >>> Yes, there is ELB built into the latest releases of StarCluster. I
> wrote
> >>> it, so feel free to write me (+ the list) with any questions.
> >>>
> >>> The docs on
> >>> http://web.mit.edu/stardev/cluster/docs/index.html
> >>>
> >>> haven't been updated in a while. There is a documentation page on
> >>> starcluster in the code base, see
> >>> /starcluster/StarCluster/docs/sphinx/load_balancer.rst
> >>>
> >>> That doc should have all of the information you need, and is readable
> in
> >>> plain text.
> >>>
> >>> Typically, this is how I fire up the load balancer:
> >>> starcluster bal <cluster_tag> -m <MAX_NODES you want> -n <MIN_NODES you
> >>> want>
> >>>
> >>> It will poll the cluster every 60 seconds and make decisions. The
> >>> decisions are described in load_balancer.rst. There is a visualizer
> which
> >>> makes 6 graphs with matplotlib to show you how many nodes are working,
> how
> >>> many jobs are running, queued, avg load, etc, but the visualizer still
> needs
> >>> a little bit of work.
> >>>
> >>> Hope that helps, and feel free to send back questions.
> >>> Rajat Banerjee
> >>>
> >>>
> >>> On Fri, Jan 28, 2011 at 12:29 PM, <starcluster-request at mit.edu> wrote:
> >>>
> >>>> Send StarCluster mailing list submissions to
> >>>> starcluster at mit.edu
> >>>>
> >>>> To subscribe or unsubscribe via the World Wide Web, visit
> >>>> http://mailman.mit.edu/mailman/listinfo/starcluster
> >>>> or, via email, send a message with subject or body 'help' to
> >>>> starcluster-request at mit.edu
> >>>>
> >>>> You can reach the person managing the list at
> >>>> starcluster-owner at mit.edu
> >>>>
> >>>> When replying, please edit your Subject line so it is more specific
> >>>> than "Re: Contents of StarCluster digest..."
> >>>>
> >>>> Today's Topics:
> >>>>
> >>>> 1. Starcluster and elastic load balancing (Archie Russell)
> >>>>
> >>>>
> >>>>
> >>>> ---------- Forwarded message ----------
> >>>> From: Archie Russell <archier at gmail.com>
> >>>> To: starcluster at mit.edu
> >>>> Date: Thu, 27 Jan 2011 11:40:00 -0800
> >>>> Subject: [StarCluster] Starcluster and elastic load balancing
> >>>>
> >>>> Hi,
> >>>>
> >>>> Online it says Starcluster has Elastic Load Balancing built into the
> >>>> latest code
> >>>> version at Github. How would I go about using this? How does
> >>>> it work, e.g.
> >>>> when does it fire up new nodes and when does it shut them down?
> >>>>
> >>>> Thanks,
> >>>> Archie
> >>>>
> >>>> _______________________________________________
> >>>> StarCluster mailing list
> >>>> StarCluster at mit.edu
> >>>>
> >>>> http://mailman.mit.edu/mailman/listinfo/starcluster
> >>>>
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> StarCluster mailing list
> >>> StarCluster at mit.edu
> >>> http://mailman.mit.edu/mailman/listinfo/starcluster
> >>>
> >>>
> >>
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.mit.edu/pipermail/starcluster/attachments/20110202/c754ce73/attachment.htm
>
> ------------------------------
>
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
> End of StarCluster Digest, Vol 18, Issue 2
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20110203/33bec6fa/attachment.htm
More information about the StarCluster
mailing list