[StarCluster] EC2 User data

[Justin Riley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Awesome, thanks! Yea, it may not be universal but at least it allows
standard shell scripts (and apparently YAML) to work as expected where
it previously did not so I think it's worth while including. I'm just
speculating but my guess is scripts are probably the most common
user-data.

Once we have the code in and tested the next step is to document how
to use user-data with StarCluster and add it to the user manual...

~Justin

On 02/07/2012 09:38 PM, Joydeep Sen Sarma wrote:
> ok - i think the patch is pretty simple - let me file a pull
> request. i had added the user_data param as well ..
> 
> i think the '#' strategy works for yaml as well (looking one
> message below). i was just afraid it wasn't universal.
> 
> On Tue, Feb 7, 2012 at 11:03 PM, Justin Riley
> <justin.t.riley at gmail.com <mailto:justin.t.riley at gmail.com>>
> wrote:
> 
> Hi Paolo/Joydeep,
> 
> You're correct that StarCluster currently uses the user-data to
> store node-aliases which at the moment kills the ability to utilize
> user-data scripts. It is possible to run user-scripts on the entire
> cluster fairly easily, however, using a StarCluster plugin. I'm
> happy to provide an example or perhaps include it in a future
> release if you're interested.
> 
> The reason for storing the alias in user-data is that I needed some
> way to 'stick' a name to a node at launch time. The new tagging API
> for EC2 is useful but it's still not possible to assign names at
> launch time. My current approach is to use the user-data at launch
> time, then later fetch it from user-data once the instance is up
> and apply an alias tag. This is useful for example if things crash
> somewhere in the middle - the nodes are guaranteed to be named the
> same thing the next time around because of the user-data.
> 
> Your solution of effectively 'commenting' the alias line in
> user-data so that other scripts can run is not a bad solution
> actually. I'd be willing to accept a pull request for this and
> include the functionality in a future release. We need to add a
> USER_DATA config parameter to the cluster config that would let
> users point to a user-data script.
> 
> Would you be interested in contributing this back? I'm happy to
> help you along with the fork/pull request process if you're
> willing. There's also a contribute guide that gives an overview of
> the process:
> 
> http://web.mit.edu/star/cluster/docs/latest/contribute.html
> 
> ~Justin
> 
> 
> On 01/24/2012 11:41 AM, Joydeep Sen Sarma wrote:
>> It doesn't.
>> 
>> Currently the user data is used by starcluster itself to store
> node aliases.
>> 
>> I had a similar requirement as yours - but in my case - the user
>> data was restricted to shell scripts. So what i did for my use
>> case was
> hack
>> the starcluster code to use the first line of the user data -
> prepended
>> with a '#' for node aliases - and the rest of the user data could
>> be used as a shell script (filled in when launching the cluster
>> from the configuration).
>> 
>> it seemed pretty hacky to me - so didn't try to contribute back.
>> 
>> there may be other ways of achieving this outcome. for example -
>> one could take the cluster-name and the instance alias (node001
>> etc.) and then map that to a location in s3
>> (s3://urbucket/cluster-name/node001) and then store instance
>> specific user data in that location. the boot scripts can get the
>> cluster name and the instance alias from ec2 apis and then use
>> that to locate the instance data and get rolling.
>> 
>> On Tue, Jan 24, 2012 at 9:22 PM, Paolo Di Tommaso 
>> <Paolo.DiTommaso at crg.eu <mailto:Paolo.DiTommaso at crg.eu>
> <mailto:Paolo.DiTommaso at crg.eu <mailto:Paolo.DiTommaso at crg.eu>>>
> wrote:
>> 
>> Dear all,
>> 
>> Does StarCluster provide any support for EC2 user data feature?
>> 
>> http://www.turnkeylinux.org/blog/ec2-userdata
>> 
>> 
>> I would like to use this mechanism to specify some extra
>> instance configuration at node boot time.
>> 
>> If no, is there any other alternative?
>> 
>> 
>> Thank you, Paolo Di Tommaso
>> 
>> Software Engineer Comparative Bioinformatics Group Centre de
>> Regulacio Genomica (CRG) Dr. Aiguader, 88 08003 Barcelona, Spain
>> 
>> 
>> _______________________________________________ StarCluster
>> mailing list StarCluster at mit.edu <mailto:StarCluster at mit.edu>
> <mailto:StarCluster at mit.edu <mailto:StarCluster at mit.edu>>
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>> 
>> 
>> 
>> 
>> _______________________________________________ StarCluster
>> mailing list StarCluster at mit.edu <mailto:StarCluster at mit.edu> 
>> http://mailman.mit.edu/mailman/listinfo/starcluster
> 
> 
> 
> 
> This body part will be downloaded on demand.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8y1HoACgkQ4llAkMfDcrl64wCglpNHQYKfAjfrubiBTVXSxCDN
ersAnRdUZ94ku8OcWvxM7r4IXzIn8jc1
=UZq3
-----END PGP SIGNATURE-----