[StarCluster] @sc-core tag exceeds maximum length of 255 chars (#348)

[Justin Riley]

Just following up to the list: this issue occured because of an
unsupported change in a fork of StarCluster. Nevertheless we're working
on a fix for this but this shouldn't be an issue for folks using the
official repo.

~Justin

On Thu, Dec 12, 2013 at 03:34:22PM -0500, Jennifer Staab wrote:
>    I have a feeling you were about to suggest the fix that I have outlined
>    below. It involves no code change just a change in how you set up your
>    cluster template such that your @sc-core tag squeezes within the 255
>    character limit.
>    Things to note about EC2-VPC and the this issue: 
>    1) EC2-VPC limits their tag lengths to 255 characters (this is not true
>    for EC2-Classic)
>    2) @sc-core tag and (non-truncated) value are needed for starcluster to
>    function properly
>    3) @sc-core tag values are serialized (via json) and compressed to make
>    them usable but reduce their size
>    4) changing zlib.compress() function to use level 9 (highest compression)
>    instead of default 6 doesn't solve the problem
>    SOLUTION
>    Changing your [cluster]/cluster template configuration in the
>    configuration file such that 'master_instance_type' and
>    'node_instance_type' are of same type (like m1.small) AND
>    'master_image_id' and 'node_image_id' use the same ID (like ami-xxxxxxxx)
>    will allow compression such that the @sc-core tag value can be compressed
>    to a size that is under 255 characters.
>    You can use VPC even if your master and worker nodes are of different
>    types or use different AMI's. To allow this type of variation and not
>    violate the 255 character limit, I would suggest creating a cluster
>    template with the AMI id and type you want for the master node for both
>    master and (worker) node. Then use starcluster addnode with -i and -I
>    options to add all the worker nodes you want of the TYPE and AMI ID you
>    want. I have tried it and it seems to work perfectly.
>    Just in case someone runs into a similar problem in the future, below are
>    the dictionary of values that are serialized and compressed to create the
>    @sc-core tag's value:
>    {'disable_cloudinit': False, 'master_instance_type': 'm1.small',
>    'disable_queue': False, 'subnet_id': 'subnet-xxxxxxxx',
>    'availability_zone': 'us-east-1b', 'node_image_id': 'ami-xxxxxxxx',
>    'node_instance_type': 'm1.small', 'use_nfs_crossmnt': True,
>    'cluster_size': 1, 'master_image_id': 'ami-xxxxxxxx', 'dns_prefix': False}
>    Thanks,
>    -Jennifer
> 
>    On Thu, Dec 12, 2013 at 1:04 PM, Justin Riley
>    <[1]notifications at github.com> wrote:
> 
>      [2]@jmstaab Yep, that's the issue. The reason I asked for the config is
>      I'm curious what is making the value that long. I have a more general
>      fix for this but in the mean time would you mind pasting your [cluster]
>      section? Thanks!
> 
>      —
>      Reply to this email directly or [3]view it on GitHub.[IMG]
> 
> References
> 
>    Visible links
>    1. mailto:notifications at github.com
>    2. https://github.com/jmstaab
>    3. https://github.com/jtriley/StarCluster/issues/348#issuecomment-30446236

> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
Url : http://mailman.mit.edu/pipermail/starcluster/attachments/20131212/bd231e08/attachment.bin