Managing your cloud resources with OpenStack

OpenStack is the software suite used on the Compute Canada Cloud to control hardware resources such as computers, storage and networking. It allows the creation and management of virtual machines ("VMs", or "instances"), which act like separate individual machines, by emulation in software. This allows users complete control over their computing environment, from choosing an operating system to software installation and configuration. Diverse use cases are supported, from hosting websites to creating virtual clusters. More documentation on OpenStack can be found at the OpenStack web site.

This page describes how to perform common tasks encountered while working with OpenStack. It is assumed that you have already read Cloud Quick Start and understand the basic operations of launching and connecting to a VM. Most tasks can be performed using the dashboard, as described there and below. But some require use of the command line tools, for example creating an image. See OpenStack Command Line Clients for more information.

Working with Volumes

A volume provides storage which is not destroyed when a VM is terminated. On the Compute Canada cloud resources, volumes use Ceph storage with a 3-fold replication factor to provide safety against hardware failure. More documentation about OpenStack volumes can be found here.

Creating a Volume

OpenStack Dashboard, Volumes page (Click for larger image)

Create Volume dialog (Click for larger image)

To create a volume click and fill in the following fields:

• Volume Name: data, for example
  
• Description:
  Optional text
• Volume Source: No source, empty volume
  
• Type: No volume type
  
• Size (GB): 40, or some suitable size
  
• Availability Zone: Any Availability Zone
  

Finally click the blue "Create Volume" button.

Accessing a Volume from a VM

To access the volume you must first attach it to a running VM. This is analogous to inserting a USB key or plugging an external drive into your personal computer. You can attach a volume from the Compute->Volumes page.

• At the right-hand end of the line describing the volume will be a drop-down menu. Select "Manage Attachments" or "Edit Attachments".
• In the "Attach To Instance" drop-down box select a VM,
• Click the blue "Attach Volume" button.

The process should complete in a few seconds. Then the Compute->Volumes page will show the newly created volume is attached to your selected VM on /dev/vdb or some similar location.

Formatting and Mounting an Empty Volume

On most Linux distributions the following steps can be used to partition, format, and mount the newly created volume. NOTE: If this is not a newly created volume the partition and format steps should be skipped as they will result in loss of data on that volume, and only the steps to mount the volume should be followed.

1. Create a partition on the volume with
   

   [name@server ~]$ sudo fdisk /dev/vdb
   

   fdisk will prompt you to enter a command. Use this sequence of single-character commands to create a new partition on your volume.
n => new partition p => primary, only one partition on disk 1 => partition number 1 <return> => first sector (use default) <return> => last sector (use default) w => write partition table to disk and exit
2. Format the newly created partition with
   

   [name@server ~]$ sudo mkfs -t ext4 /dev/vdb1
   

3. Create a place to mount the device with
   

   [name@server ~]$ sudo mkdir /media/data
   

4. Finally, mount the volume with
   

   [name@server ~]$ sudo mount /dev/vdb1 /media/data
   

If the VM is rebooted for some reason the volume will need to be remounted. To cause the VM to mount the volume automatically at boot time, edit /etc/fstab and add a line like

/dev/vdb1 /media/data ext4 defaults 0 2

If you are not rebooting, you can mount the device just added to /etc/fstab with

[name@server ~]$ sudo mount -a

Booting From a Volume

If running a persistent machine, booting from a volume is a safer option. When creating a VM booting from an image, rather than a volume, it will store the VM on the local disk of the actual machine running the VM. If something went wrong with that machine or its disk, there is a good chance your VM would be lost. Volume storage has a built in redundancy which will protect your volumes from hardware failure.

There are several ways to boot a VM from a volume, you can either create a volume (or use a pre-existing one) and then select to boot from it, or you can create a volume as part of the process of launching a VM.

If creating a volume first, proceed in the same way as described above, except that you choose a volume source. You can either use an image or another volume as the source for your new volume. When launching a VM from a pre-existing volume OpenStack does not load an operating system onto the volume. This is what you would want if you were booting from a volume which you have already been working with previously, containing a working and configured operating system. However, if you create an empty volume as done above, there will be no operating system loaded to boot from so you will not be able to use your VM.

If creating a volume as part of the process of launching the VM, select Boot from image (creates a new volume) select the image to use and the size of the volume. The default device name of vda should work well. If this volume is something you would like to remain longer than the VM ensure that the "Delete on Terminate" check box is unchecked. If you are unsure about this option, it is better to leave this box unchecked as you can always manually delete the volume later as an extra step.

Creating an Image From a Volume

Creating an image from a volume allows you to download the image either for a backup or to use in virtual box.

[name@server ~]$ openstack image create --disk-format <format> --volume <volume_name> <image_name>

where <volume_name> can be found from the OpenStack dash board by clicking on the volume name and <image_name> is the name given to the image you are creating and <format> is the disk format (two possible values are qcow2 and vmdk). See OpenStack Command Line Clients for more details about working with command line. This image can then be downloaded in the way described in Downloading an image into a local VirtualBox

Accessing a VM with Multiple Users

Cloud init to add multiple users (Click for larger image)

There are a number of ways to approach this, but basically what is needed is to add another public ssh key to the VM (see ssh keys about working with and creating keys). This could be done by adding to the authorized_keys of the primary user account provided for the VM, usually named after the operating system, or it could be added to a newly created user account. In Ubuntu, when logged in with the primary user's account this can be accomplished with the command

[name@server ~]$ sudo adduser USERNAME

Then add the supplied public key to that user's .ssh/authorized_keys file as described in ssh keys and linked pages.

Another alternative would be to do this during the creation of your VM using CloudInit. The following CloudInit script (which uses the YAML format, see YAML Preview):

#cloud-config
users:
  - name: sysadmin
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <Insert public key>
  - name: user
    shell: /bin/bash
    ssh_authorized_keys:
      - <Insert public key>

adds two users sysadmin and user with and without sudo permissions respectively. The <Insert public key> must be replaced with the public key to use for that user. Note that YAML is very picky about white space formatting, so that there must be a space after the "-" before your public key string. Also, this configuration overwrites the default user that is added when no CloudInit script is specified, so that the users listed in this configuration script will be the only users on the newly created VM so it is important to have at least one user with sudo permission. More users can be added by simply including another - name: username section.

After the VM has finished spawning, take a look at the log to ensure that the public keys have been added correctly for those users. The log can be found by click the name of the instance under the "Compute->Instances" panel and then selecting the "log" tab. The log should show something like:

ci-info: ++++++++Authorized keys from /home/sysadmin/.ssh/authorized_keys for user sysadmin++++++++
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | Keytype |                Fingerprint (md5)                | Options |     Comment      |
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | ssh-rsa | ad:a6:35:fc:2a:17:c9:02:cd:59:38:c9:18:dd:15:19 |    -    | rsa-key-20160229 |
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: ++++++++++++Authorized keys from /home/user/.ssh/authorized_keys for user user++++++++++++
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | Keytype |                Fingerprint (md5)                | Options |     Comment      |
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | ssh-rsa | ad:a6:35:fc:2a:17:c9:02:cd:59:38:c9:18:dd:15:19 |    -    | rsa-key-20160229 |
ci-info: +---------+-------------------------------------------------+---------+------------------+

Once this is done, users can log into the VM with their private keys as usual (see ssh keys).

Downloading an image into a local VirtualBox

The first step is to install the OpenStack client and download the OpenStack RC file and source it (see OpenStack Command Line Clients). The OpenStack client can list the available images on your OpenStack project with

[name@server ~]$ openstack image list

producing something like:

+--------------------------------------+---------------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                                  | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------------------------------+-------------+------------------+-------------+--------+
| 982761b2-c77b-4852-8ae3-bf98b32b8894 | Hadoop-2.2.4                          | qcow2       | bare             | 10253107200 | active |
| b7bd3033-9836-406d-a8f2-2e91978026b4 | hadoopmaster                          | qcow2       | bare             | 3493527552  | active |
| 2c751755-854d-49c3-af82-d501e51e7159 | hadoopmaster-active                   | qcow2       | bare             | 13134004224 | active |
| c41012f4-ed82-4478-a81f-5efb96a31b1a | hadoopmaster-old                      | qcow2       | bare             | 3493527552  | active |
| 78e61a3f-b546-441a-b476-a7077b04ca36 | hadoopslave                           | qcow2       | bare             | 3490971648  | active |
| 516845c3-b256-4c6d-a2cb-e31e822c7e34 | hadoopslave1-active                   | qcow2       | bare             | 8345026560  | active |
| 1546bd86-5314-4fce-9576-e2f6930dad30 | hadoopslave1-old                      | qcow2       | bare             | 3490971648  | active |
| baf78e8d-8288-4854-a66b-812cdf3ccbca | TestVM                                | qcow2       | bare             | 13167616    | active |
| 2faf97d7-5b0b-44ce-8024-3bef5a634570 | test_ubuntu_initial                   | qcow2       | bare             | 1799487488  | active |
| 308b6614-396a-4360-9c33-4e86f41ea0ec | trusty                                | qcow2       | bare             | 256180736   | active |
| 9b3c3fda-2aca-43b5-a3e7-662a94f5e7fb | Ubuntu_14.04_Trusty-amd64-20150708    | qcow2       | bare             | 257884672   | active |
| f93e66cf-fec1-4460-8fc7-506e716fbf30 | ucernvm-prod.1.18-10                  | raw         | bare             | 20971520    | active |
+--------------------------------------+---------------------------------------+-------------+------------------+-------------+--------+

You can then download a particular image with

[name@server ~]$ openstack image save --file ./<file-name-for-image>.<format> 2c751755-854d-49c3-af82-d501e51e7159

where <format> matches that given in the table above for your selected image andthe last argument is the ID of the image you wish to download.

Then to use this image in VirtualBox you will need to convert the image in the qcow2 format to the vmdk format. This can be done with the qemu-img tool. This can be installed with something like

[name@server ~]$ sudo apt-get install qemu-img

then do the conversion with

[name@server ~]$ qemu-img convert -f qcow2 vdisk.qcow2 -O vmdk vdisk.vmdk

Then you can create a new virtual machine and attach the vmdk image to it see [1] for detailed instructions on this.