[StarCluster] Amazon AMI 64-bit Ubuntu 11 HVM-EBS for eu-west-1 region

[MacMullan, Hugh]

Hi Ed:

Woohoo, great news!

Lincoln Stein, maintainer of the VM::EC2 Perl module, has added HVM copy functionality to his (already pretty awesome) module! The module includes the Perl script migrate-ebs-image.pl, which makes copies a breeze.

I just tested it with a copy of a Scientific Linux 6.3 HVM StarCluster image that I use frequently. Here's the output:

# migrate-ebs-image.pl --from us-east-1 --to eu-west-1 ami-XXXXXXXXX
[info] Scanning for existing staging servers and volumes in http://ec2.us-east-1.amazonaws.com.
[info] Scanning for existing staging servers and volumes in http://ec2.eu-west-1.amazonaws.com.
[info]    Gathering information about image ami-XXXXXXXXX.
[info]    Copying EBS volumes attached to this image (this may take a long time).
[info]       Copying snapshot snap-XXXXXXXX from us-east-1 to eu-west-1...
[info]       ...new snapshot = snap-XXXXXXXXXX; status = pending
[info]    Waiting for all snapshots to complete. This may take a long time.
An image named 'starcluster-base-scientific-linux-6.3-x86_64-hvm-005' already exists in destination region. Renamed to 'starcluster-base-scientific-linux-6.3-x86_64-hvm-005-d88a4dfe' <-- I REALLY LIKE THIS!!!
[info]    Registering snapshot in destination with the equivalent of:
[info]    ec2-register -n 'starcluster-base-scientific-linux-6.3-x86_64-hvm-005-XXXXX' -d 'Image created @ 201301202215' -a x86_64 --kernel '' --ramdisk '' --root-device-name /dev/sda1 -b /dev/sda1=snap-XXXXXXXXXXX:false:standard
[info]       Searching for a suitable HVM image in destination region
[info]       ...Found ami-011b1975 (suse-sles-11-sp2-hvm-v1.01.x86_64)
[info]       Launching an HVM staging server in the target region. Heuristically choosing instance type of 'cc2.8xlarge' for this type of HVM..
[info]       Waiting for instance to become ready.
[info]       Stopping instance temporarily to swap root volumes.
[info]       Detaching original root volume...
[info]       Creating and attaching new root volume..
[info]       Creating image in destination region...
[info]       Terminating the staging server
New snapshot is now located in eu-west-1 under ami-XXXXXXXXX.

No muss, no fuss. As you can see, it has to launch a cc2.8xlarge to do the copy, so it'll cost you to do the transfer. It's currently in the hvm-migrate alpha GIT tree:

https://github.com/lstein/LibVM-EC2-Perl/tree/hvm-migrate

He'll be publishing to CPAN today or tomorrow after Windows AMI testing. It will be VM::EC2 1.23 on CPAN.

So there you go.

-Hugh


From: Ed Morris [mailto:ecm200 at gmail.com]
Sent: Friday, January 18, 2013 12:43 PM
To: MacMullan, Hugh
Cc: starcluster at mit.edu
Subject: Re: [StarCluster] Amazon AMI 64-bit Ubuntu 11 HVM-EBS for eu-west-1 region


Hi Hugh.

Thanks for that, much appreciated.

I tried using the "snapshot" route, which allows you to transfer snapshots between regions through the AWS web console (AWS documentation here<http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-copy-snapshot.html>), but so far I have had no luck with regards to that. I started an instance with the US-EAST-1 Ubuntu 11.10 x86_64 HVM AMI (not through Star Cluster), and then created a snap shot of the root volume. I then transferred the snapshot to EU-WEST-1 region and converted that into an AMI. One issue I had, and have still not solved, was the "kernel ID" specification which was needed during the creation of the AMI from the snap shot. I could not find the kernel ID for the AMI in question, even though other flavours of Star Cluster AMI's did have associated Kernel IDs.  I came unstuck when I tried to execute the AMI that I had transferred from the US-EAST-1 region to EU-WEST-1 region, and I was told by StarCluster that the AMI was not HVM, and therefore could not be executed on the CC2.8xlarge instance type. I am positive that I am using the correct AMI which is HVM supported, as this is currently the only HVM supported AMI that's available, and it's only available in US-EAST-1.

I plan to try something slightly different, by creating a custom AMI using the StarCluster tools, and then creating a snap shot from that AMI, and transferring that. But again I may come up against the Kernel ID issue, which I still don't know how to solve.

However some good news, I have read in a number of places on the web that Amazon are developing a tool for transferring AMI's the same way you can now transfer snap shots, and can be undertaken through the AWS console. I haven't managed to find when this is expected to be implemented though.

There is also this that I found (Ylastic<http://ylastic.com/features.html>), which amongst a lot of things, appears to suggest that it can migrate EBS AMI's from region to region.


On 18 January 2013 18:16, MacMullan, Hugh <hughmac at wharton.upenn.edu<mailto:hughmac at wharton.upenn.edu>> wrote:
Hi Ed:

I have no EU AMIs, but I've been curious about the second question you have: what's the 'easy' way to transfer AMIs? I poked around a bit today, and it looks like 'migrate-ebs-image.pl<http://migrate-ebs-image.pl>', part of the VM::EC2 Perl module, is a good solution ... except that it converts my HVM AMIs to PV. Argh! :)

I've got a query in with the developer (Lincoln Stein) about HVM copies. I'll let you know what I discover.

That said, if others have other easy ways to transfer AMIs ... I'm all ears too!

-Hugh


From: starcluster-bounces at mit.edu<mailto:starcluster-bounces at mit.edu> [mailto:starcluster-bounces at mit.edu<mailto:starcluster-bounces at mit.edu>] On Behalf Of Ed Morris
Sent: Friday, January 18, 2013 5:26 AM
To: starcluster at mit.edu<mailto:starcluster at mit.edu>
Subject: [StarCluster] Amazon AMI 64-bit Ubuntu 11 HVM-EBS for eu-west-1 region

Hi.

Does anyone know if there is an AMI available using HVM version of Ubuntu for the cc8.xlarge compute node for starcluster in the eu-west-1 region?

I have been quite happily using the AMI ami-4583572c us-east-1 starcluster-base-ubuntu-11.10-x86_64-hvm (HVM-EBS) on the us-east region, but couldn't find any HVM versions in the eu region.

Any ideas? Can the HVM-EBS be easily transferred to the eu region?

Ed.

--
Ed Morris
------------------------------------------------
email: ecm200 at gmail.com<mailto:ecm200 at gmail.com>
phone:



--
Ed Morris
------------------------------------------------
email: ecm200 at gmail.com<mailto:ecm200 at gmail.com>
phone:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20130124/7eb30444/attachment-0001.htm