Bureaucrats, cc_docs_admin, cc_staff
2,879
edits
Arbutus Migration Guide: Difference between revisions
standardize formatting, simplify some language
(Arbutus Migration guide first version) |
(standardize formatting, simplify some language) |
||
| Line 1: | Line 1: | ||
This document aims to describe | This document aims to describe how to migrate virtual machine (VM) instances from the legacy West Cloud to the new Arbutus Cloud. | ||
You know your workload best, so we recommend that you migrate your instances according to your own application requirements and schedule. | |||
== Preliminaries == | |||
Note the following URLs for accessing the Horizon Web UI for the 2 Clouds: | Note the following URLs for accessing the Horizon Web UI for the 2 Clouds: | ||
'''West Cloud (legacy):''' [https://west.cloud.computecanada.ca https://west.cloud.computecanada.ca] | |||
'''Arbutus Cloud (new):''' [https://arbutus.cloud.computecanada.ca https://arbutus.cloud.computecanada.ca] | |||
Firefox and Chrome browsers are supported. Safari and Edge may work but are not validated. | Firefox and Chrome browsers are supported. Safari and Edge may work but are not validated. | ||
| Line 15: | Line 14: | ||
Your Project (Tenant), Network, and Router will be pre-created for you in Arbutus Cloud. User access will also be pre-populated. | Your Project (Tenant), Network, and Router will be pre-created for you in Arbutus Cloud. User access will also be pre-populated. | ||
Prior to migrating instances, | Prior to migrating instances, we recommend that you complete the following preliminaries to prepare the necessary environment for migration. | ||
'''IMPORTANT''': Backup any critical data. While the Cloud has redundant storage systems, no backups of any instances are taken. | '''IMPORTANT''': Backup any critical data. While the Cloud has redundant storage systems, no backups of any instances are taken. | ||
# Get RC files (used to set environment variables used by the OpenStack command-line tools) after logging in to the URLs above with your Compute Canada credentials: | # Get RC files (used to set environment variables used by the OpenStack command-line tools) after logging in to the URLs above with your Compute Canada credentials: | ||
| Line 47: | Line 44: | ||
# Plan an outage window. Generally, shutting down services and the instance(s) is the best way to avoid corrupt or inconsistent data after the migration. Smaller volumes should copy over fairly quickly i.e. within minutes, but larger volumes will take longer. Plan accordingly for this. Additionally, floating IP addresses will change, so ensure the TTL of your DNS records is set to a small value so that the changes propagate as quickly as possible. | # Plan an outage window. Generally, shutting down services and the instance(s) is the best way to avoid corrupt or inconsistent data after the migration. Smaller volumes should copy over fairly quickly i.e. within minutes, but larger volumes will take longer. Plan accordingly for this. Additionally, floating IP addresses will change, so ensure the TTL of your DNS records is set to a small value so that the changes propagate as quickly as possible. | ||
There are three general migration scenarios to consider. Depending on your applications and current instance setup, you may use any or all of these scenarios to migrate from the West Cloud to the Arbutus Cloud. | |||
== Manual or orchestrated migration == | |||
== | |||
In essence, this will be a “net new” infrastructure provisioning. Instances and volumes are created in Arbutus with the same specifications as that on West Cloud. The general approach would be: | In essence, this will be a “net new” infrastructure provisioning. Instances and volumes are created in Arbutus with the same specifications as that on West Cloud. The general approach would be: | ||
| Line 61: | Line 55: | ||
# Assign floating IP(s) to the new instance(s) and update DNS. | # Assign floating IP(s) to the new instance(s) and update DNS. | ||
# Decommission old instance(s) and delete old volume(s). | # Decommission old instance(s) and delete old volume(s). | ||
The above steps can be done manually or orchestrated via various configuration management tools. The use of such tools is beyond the scope of this document, but if you were already using orchestration tools on West Cloud, they should work with Arbutus Cloud as well. | The above steps can be done manually or orchestrated via various configuration management tools. The use of such tools is beyond the scope of this document, but if you were already using orchestration tools on West Cloud, they should work with Arbutus Cloud as well. | ||
== Migrating volume-backed instances == | |||
== | |||
Volume-backed instances as their name implies have a persistent volume attached to them containing the operating system and any required data. General best practices would involve a smaller volume for the operating system and a larger volume for data sets. | Volume-backed instances as their name implies have a persistent volume attached to them containing the operating system and any required data. General best practices would involve a smaller volume for the operating system and a larger volume for data sets. | ||
=== Migration using Glance images === | |||
=== | |||
This method is recommended for volumes less than 150GB in size. For volumes larger than that, creating new volumes in Arbutus Cloud and copying the required data across from West Cloud is preferred to creating Glance images and transferring the images between clouds. | This method is recommended for volumes less than 150GB in size. For volumes larger than that, creating new volumes in Arbutus Cloud and copying the required data across from West Cloud is preferred to creating Glance images and transferring the images between clouds. | ||
# SSH to the migration host ''cloudmigration.computecanada.ca'' with your Compute Canada credentials. | # SSH to the migration host ''cloudmigration.computecanada.ca'' with your Compute Canada credentials. | ||
| Line 90: | Line 78: | ||
# Once your instances/volumes are migrated & validated and any associated DNS records updated, please delete your old instances/volumes on the legacy West Cloud. | # Once your instances/volumes are migrated & validated and any associated DNS records updated, please delete your old instances/volumes on the legacy West Cloud. | ||
=== Alternative method: Migrating a volume-backed instance using Linux 'dd' === | |||
=== | |||
# Launch a temporary instance on West Cloud with the smallest flavor possible “p1-1.5gb”. The instructions below assume CentOS 7, but any Linux distribution with Python and Pip available should work. | # Launch a temporary instance on West Cloud with the smallest flavor possible “p1-1.5gb”. The instructions below assume CentOS 7, but any Linux distribution with Python and Pip available should work. | ||
| Line 110: | Line 96: | ||
# Once your instances/volumes are migrated & validated and any associated DNS records updated, please delete your old instances/volumes on the legacy West Cloud. | # Once your instances/volumes are migrated & validated and any associated DNS records updated, please delete your old instances/volumes on the legacy West Cloud. | ||
== Migrating ephemeral instances == | |||
== | === Migration using Glance images and volume snapshots === | ||
This method is recommended for instances with ephemeral storage less than 150GB in size. For instances with storage larger than that, creating new instances in Arbutus Cloud and copying the required data across from West Cloud is preferred to creating Glance images and transferring the images between clouds. You will still need to copy data from any non-boot ephemeral storage (i.e. mounted under <code>/mnt</code>) separately. Consult the section on methods to copy data for this. | This method is recommended for instances with ephemeral storage less than 150GB in size. For instances with storage larger than that, creating new instances in Arbutus Cloud and copying the required data across from West Cloud is preferred to creating Glance images and transferring the images between clouds. You will still need to copy data from any non-boot ephemeral storage (i.e. mounted under <code>/mnt</code>) separately. Consult the section on methods to copy data for this. | ||
# SSH to the migration host ''cloudmigration.computecanada.ca'' with your Compute Canada credentials. | # SSH to the migration host ''cloudmigration.computecanada.ca'' with your Compute Canada credentials. | ||
| Line 133: | Line 115: | ||
# Once your instances/data are migrated & validated and any associated DNS records updated, please delete your old instances on the legacy West Cloud. | # Once your instances/data are migrated & validated and any associated DNS records updated, please delete your old instances on the legacy West Cloud. | ||
=== Alternative method: Migrating an ephemeral instance using Linux 'dd' === | |||
=== | |||
# Login to the instance running on West Cloud via SSH. When migrating an ephemeral instance (an instance without a backing volume as aforementioned), it is important to shut down as many unnecessary services as possible on the instance prior to migration e.g. httpd, databases, etc. Ideally, leave only SSH running. | # Login to the instance running on West Cloud via SSH. When migrating an ephemeral instance (an instance without a backing volume as aforementioned), it is important to shut down as many unnecessary services as possible on the instance prior to migration e.g. httpd, databases, etc. Ideally, leave only SSH running. | ||
| Line 151: | Line 131: | ||
# Once your instances/data are migrated & validated and any associated DNS records updated, please delete your old instances on the legacy West Cloud. | # Once your instances/data are migrated & validated and any associated DNS records updated, please delete your old instances on the legacy West Cloud. | ||
== Methods to copy data == | |||
There are a couple of recommended approaches for copying data between instances running in the two clouds. The most appropriate method would depend upon the size of the data volumes in your tenant. For very large volumes greater than 5TB, Globus is recommended. Please see here for configuration details: [https://computecanada.github.io/DHSI-cloud-course/globus/ https://computecanada.github.io/DHSI-cloud-course/globus/] | |||
There are a couple of recommended approaches for copying data between instances running in the | |||
If you have very large volumes, we recommend submitting a support ticket as well. | If you have very large volumes, we recommend submitting a support ticket as well. | ||
For volumes in the several hundred GB to low TB range, rsync+ssh provides good transfer speeds and can also work in an incremental way. A typical use case would be: | For volumes in the several hundred GB to low TB range, rsync+ssh provides good transfer speeds and can also work in an incremental way. A typical use case would be: | ||
# SSH to the West Cloud instance which has the large volume attached and note the absolute path you want synced over to the instance on Arbutus Cloud. | # SSH to the West Cloud instance which has the large volume attached and note the absolute path you want synced over to the instance on Arbutus Cloud. | ||
| Line 166: | Line 143: | ||
#: <code> rsync -avzP -e 'ssh -i ~/.ssh/key.pem' /local/path/ remoteuser@remotehost:/path/to/files/ </code> | #: <code> rsync -avzP -e 'ssh -i ~/.ssh/key.pem' /local/path/ remoteuser@remotehost:/path/to/files/ </code> | ||
# Validate the data successfully copied over on the instance in Arbutus Cloud. Please delete any data from the legacy West Cloud once you have completed validation on Arbutus Cloud. | # Validate the data successfully copied over on the instance in Arbutus Cloud. Please delete any data from the legacy West Cloud once you have completed validation on Arbutus Cloud. | ||
You may also use any other method you are familiar with for transferring data. | You may also use any other method you are familiar with for transferring data. | ||
== Support == | |||
== | |||
Support requests can be sent to the usual Cloud support address at [mailto:cloud@computecanada.ca cloud@computecanada.ca] | Support requests can be sent to the usual Cloud support address at [mailto:cloud@computecanada.ca cloud@computecanada.ca] | ||