[StarCluster] @sc-core tag exceeds maximum length of 255 chars (#348)

[Jennifer Staab]

I have a feeling you were about to suggest the fix that I have outlined
below. It involves no code change just a change in how you set up your
cluster template such that your @sc-core tag squeezes within the 255
character limit.

Things to note about EC2-VPC and the this issue:
1) EC2-VPC limits their tag lengths to 255 characters (this is not true for
EC2-Classic)
2) @sc-core tag and (non-truncated) value are needed for starcluster to
function properly
3) @sc-core tag values are serialized (via json) and compressed to make
them usable but reduce their size
4) changing zlib.compress() function to use level 9 (highest compression)
instead of default 6 doesn't solve the problem

SOLUTION
Changing your [cluster]/cluster template configuration in the configuration
file such that 'master_instance_type' and 'node_instance_type' are of same
type (like m1.small) AND 'master_image_id' and 'node_image_id' use the same
ID (like ami-xxxxxxxx) will allow compression such that the @sc-core tag
value can be compressed to a size that is under 255 characters.

You can use VPC even if your master and worker nodes are of different types
or use different AMI's. To allow this type of variation and not violate the
255 character limit, I would suggest creating a cluster template with the
AMI id and type you want for the master node for both master and (worker)
node. Then use starcluster addnode with -i and -I options to add all the
worker nodes you want of the TYPE and AMI ID you want. I have tried it and
it seems to work perfectly.

Just in case someone runs into a similar problem in the future, below are
the dictionary of values that are serialized and compressed to create the
@sc-core tag's value:

{'disable_cloudinit': False, 'master_instance_type': 'm1.small',
'disable_queue': False, 'subnet_id': 'subnet-xxxxxxxx',
'availability_zone': 'us-east-1b', 'node_image_id': 'ami-xxxxxxxx',
'node_instance_type': 'm1.small', 'use_nfs_crossmnt': True, 'cluster_size':
1, 'master_image_id': 'ami-xxxxxxxx', 'dns_prefix': False}

Thanks,
-Jennifer



On Thu, Dec 12, 2013 at 1:04 PM, Justin Riley <notifications at github.com>wrote:

> @jmstaab <https://github.com/jmstaab> Yep, that's the issue. The reason I
> asked for the config is I'm curious what is making the value that long. I
> have a more general fix for this but in the mean time would you mind
> pasting your [cluster] section? Thanks!
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/jtriley/StarCluster/issues/348#issuecomment-30446236>
> .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20131212/2e6b8896/attachment.htm