Managing your cloud resources with OpenStack/en

Parent page: CC-Cloud

OpenStack is the software suite used on the Compute Canada Cloud to control hardware resources such as computers, storage and networking. It allows the creation and management of virtual machines ("VMs", or "instances"), which act like separate individual machines, by emulation in software. This allows users complete control over their computing environment, from choosing an operating system to software installation and configuration. Diverse use cases are supported, from hosting websites to creating virtual clusters. More documentation on OpenStack can be found at the OpenStack web site.

This page describes how to perform common tasks encountered while working with OpenStack. It is assumed that you have already read Cloud Quick Start and understand the basic operations of launching and connecting to a VM. Most tasks can be performed using the dashboard, as described there and below. But some require use of the command line tools, for example creating an image. See OpenStack Command Line Clients for more information.

Security Groups

Security groups allow you to control network traffic into and out of your virtual machines. To manage security groups go to Project->Compute->Access & Security and select the Security Groups tab. You will see a list of currently defined security groups. If you have not previously defined any security groups, there will be single default security group.

To add or remove security rules from a security group click Manage Rules for that particular security group. To add a new rule click Add Rule button in the top right, to remove a rule click Delete Rule beside the rule you wish to delete.

The default security group contains a number of rules by default. These rules allow network traffic for any port, from any ip, into (Ingress) a VM originating from another VM in the default security group for internet protocols version 4 and 6. They also allow network traffic out (Egress) of a VM from any port to any IP for both internet protocol versions. In other words these rules allow a VM which belongs to the Default security group access out to the internet, to download content (e.g. operating system upgrades, package installations) but does not allow another machine outside the default security group access to the VM. These default rules allow you to correctly launch a VM, removing them may cause problems when creating new VMs and is not recommended. These rules do not allow access to your VM from outside the default security group which is why to connect to your VM via SSH a security rule was added for port 22 to allow incoming (Ingress) traffic so that you were able to connect to your VM (see Firewall, add rules to allow SSH).

You can define multiple security groups and a VM can belong to multiple security groups with each security group having a number of security rules. When deciding on how to manage your security groups and rules, think carefully about what needs to be accessed and who needs to access it. For example, if you will always be connecting to your VM from the same computer with a static IP via SSH it makes sense to allow SSH access only from that IP. To specify the allowed IP or IP range use the CIDR box. Further, if you are only connecting to one VM via ssh from the outside, and then connecting to other VMs within the default security group from that VM it makes sense to add the SSH security rule to a separate group and add that group to the VM you are SSHing to, however, you will also need to ensure your ssh keys are configured correctly to ssh between VMs (see SSH Keys).

The security group a VM belongs to can be configured when they are created on the Launch Instance screen under the Access & Security tab, or after the VM has been launched by selecting Edit Security Groups form the drop down menu of actions for the VM on the Project->Compute->Instances page.

Working with Volumes

A volume provides storage which is not destroyed when a VM is terminated. On the Compute Canada cloud resources, volumes use Ceph storage with a 3-fold replication factor to provide safety against hardware failure. More documentation about OpenStack volumes can be found here.

Creating a Volume

OpenStack Dashboard, Volumes page (Click for larger image)

Create Volume dialog (Click for larger image)

To create a volume click and fill in the following fields:

• Volume Name: data, for example
  
• Description:
  Optional text
• Volume Source: No source, empty volume
  
• Type: No volume type
  
• Size (GB): 40, or some suitable size
  
• Availability Zone: Any Availability Zone
  

Finally click the blue "Create Volume" button.

Accessing a Volume from a VM

To access the volume you must first attach it to a running VM. This is analogous to inserting a USB key or plugging an external drive into your personal computer. You can attach a volume from the Compute->Volumes page.

• At the right-hand end of the line describing the volume will be a drop-down menu. Select "Manage Attachments" or "Edit Attachments".
• In the "Attach To Instance" drop-down box select a VM,
• Click the blue "Attach Volume" button.

The process should complete in a few seconds. Then the Compute->Volumes page will show the newly created volume is attached to your selected VM on /dev/vdb or some similar location.

Formatting and Mounting an Empty Volume

On most Linux distributions the following steps can be used to partition, format, and mount the newly created volume. NOTE: If this is not a newly created volume the partition and format steps should be skipped as they will result in loss of data on that volume, and only the steps to mount the volume should be followed.

1. Create a partition on the volume with
   

   [name@server ~]$ sudo fdisk /dev/vdb
   

   fdisk will prompt you to enter a command. Use this sequence of single-character commands to create a new partition on your volume.
n => new partition p => primary, only one partition on disk 1 => partition number 1 <return> => first sector (use default) <return> => last sector (use default) w => write partition table to disk and exit
2. Format the newly created partition with
   

   [name@server ~]$ sudo mkfs -t ext4 /dev/vdb1
   

3. Create a place to mount the device with
   

   [name@server ~]$ sudo mkdir /media/data
   

4. Finally, mount the volume with
   

   [name@server ~]$ sudo mount /dev/vdb1 /media/data
   

If the VM is rebooted for some reason the volume will need to be remounted. To cause the VM to mount the volume automatically at boot time, edit /etc/fstab and add a line like

/dev/vdb1 /media/data ext4 defaults 0 2

If you are not rebooting, you can mount the device just added to /etc/fstab with

[name@server ~]$ sudo mount -a

Booting From a Volume

If you want to run a persistent machine, it is safest to boot from a volume. When you boot a VM from an image rather than a volume, the VM is stored on the local disk of the actual machine running the VM. If something goes wrong with that machine or its disk the VM may be lost. Volume storage has redundancy which protects the VM from hardware failure. Typically when booting from a volume VM flavors starting with a 'p' are used (see Virtual machine flavors).

There are several ways to boot a VM from a volume. You can

• boot from an image, creating a new volume, or
• boot from a pre-existing volume, or
• boot from a volume snapshot, creating a new volume.

If you have not done this before, then the first one is your only option. The other two are only possible if you have already created a bootable volume or a volume snapshot.

If creating a volume as part of the process of launching the VM, select Boot from image (creates a new volume), select the image to use, and the size of the volume. If this volume is something you would like to remain longer than the VM ensure that the "Delete on Terminate" box is unchecked. If you are unsure about this option, it is better to leave this box unchecked. You can manually delete the volume later.

Creating an Image From a Volume

Creating an image from a volume allows you to download the image. Do this if you want to save it as a backup, or to spin up a VM somewhere other than the CC Cloud, e.g. with VirtualBox.

Use the command line client to do this:

[name@server ~]$ openstack image create --disk-format <format> --volume <volume_name> <image_name>

where

• <format> is the disk format (two possible values are qcow2 and vmdk),
• <volume_name> can be found from the OpenStack dashboard by clicking on the volume name, and
• <image_name> is a name you choose for the image.

You can then download the image as described below. It is best to detach the volume from the VM before you create an image from the volume. If the volume is a boot volume you will likely need to delete your VM to detach it, however, make sure you have not checked "Delete Volume on Instance Delete" when creating the VM.

Creating an Image From a VM

Using the OpenStack Command Line Clients issue the command

[name@server ~]$  openstack server image create <server-name>

where <server-name> should be replaced with the name of your server. The command only includes the VM's root drive (e.g. /dev/vda) in the image. This means that ephemeral drives and non-boot attached volumes are not included in the image and additional measures should be taken to preserve this data. In addition, a running VM which is writing to disk while the image is being created, may leave the file system in an inconsistent state. It is recommended that a VM be shutdown (not terminated) before an image is created from it.

Accessing a VM with Multiple Users

Cloud init to add multiple users (Click for larger image)

There are a number of ways to approach this, but basically what is needed is to add another public ssh key to the VM (see ssh keys about working with and creating keys). This could be done by adding to the authorized_keys of the primary user account provided for the VM, usually named after the operating system, or it could be added to a newly created user account. In Ubuntu, when logged in with the primary user's account this can be accomplished with the command

[name@server ~]$ sudo adduser USERNAME

Then add the supplied public key to that user's .ssh/authorized_keys file as described in ssh keys and linked pages.

Another alternative would be to do this during the creation of your VM using CloudInit. The following CloudInit script (which uses the YAML format, see YAML Preview):

#cloud-config
users:
  - name: sysadmin
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <Insert public key>
  - name: user
    shell: /bin/bash
    ssh_authorized_keys:
      - <Insert public key>

adds two users sysadmin and user with and without sudo permissions respectively. The <Insert public key> must be replaced with the public key to use for that user. Note that YAML is very picky about white space formatting, so that there must be a space after the "-" before your public key string. Also, this configuration overwrites the default user that is added when no CloudInit script is specified, so that the users listed in this configuration script will be the only users on the newly created VM so it is important to have at least one user with sudo permission. More users can be added by simply including another - name: username section.

If you wish to preserve the default user created by the distribution (users debian, centos, ubuntu, etc.), use the following form:

#cloud-config
users:
  - default
  - name: sysadmin
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - <Insert public key>
  - name: user
    shell: /bin/bash
    ssh_authorized_keys:
      - <Insert public key>

After the VM has finished spawning, take a look at the log to ensure that the public keys have been added correctly for those users. The log can be found by click the name of the instance under the "Compute->Instances" panel and then selecting the "log" tab. The log should show something like:

ci-info: ++++++++Authorized keys from /home/gretzky/.ssh/authorized_keys for user gretzky++++++++
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | Keytype |                Fingerprint (md5)                | Options |     Comment      |
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | ssh-rsa | ad:a6:35:fc:2a:17:c9:02:cd:59:38:c9:18:dd:15:19 |    -    | rsa-key-20160229 |
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: ++++++++++++Authorized keys from /home/lemieux/.ssh/authorized_keys for user lemieux++++++++++++
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | Keytype |                Fingerprint (md5)                | Options |     Comment      |
ci-info: +---------+-------------------------------------------------+---------+------------------+
ci-info: | ssh-rsa | ad:a6:35:fc:2a:17:c9:02:cd:59:38:c9:18:dd:15:19 |    -    | rsa-key-20160229 |
ci-info: +---------+-------------------------------------------------+---------+------------------+

Once this is done, users can log into the VM with their private keys as usual (see Ssh keys).

Downloading an image

The first step is to install the OpenStack client and download the OpenStack RC file and source it (see OpenStack Command Line Clients). The OpenStack client can list the available images on your OpenStack project with

[name@server ~]$ openstack image list

producing something like:

+--------------------------------------+---------------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                                  | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------------------------------+-------------+------------------+-------------+--------+
| 982761b2-c77b-4852-8ae3-bf98b32b8894 | Hadoop-2.2.4                          | qcow2       | bare             | 10253107200 | active |
| b7bd3033-9836-406d-a8f2-2e91978026b4 | hadoopmaster                          | qcow2       | bare             | 3493527552  | active |
| 2c751755-854d-49c3-af82-d501e51e7159 | hadoopmaster-active                   | qcow2       | bare             | 13134004224 | active |
| c41012f4-ed82-4478-a81f-5efb96a31b1a | hadoopmaster-old                      | qcow2       | bare             | 3493527552  | active |
| 78e61a3f-b546-441a-b476-a7077b04ca36 | hadoopslave                           | qcow2       | bare             | 3490971648  | active |
| 516845c3-b256-4c6d-a2cb-e31e822c7e34 | hadoopslave1-active                   | qcow2       | bare             | 8345026560  | active |
| 1546bd86-5314-4fce-9576-e2f6930dad30 | hadoopslave1-old                      | qcow2       | bare             | 3490971648  | active |
| baf78e8d-8288-4854-a66b-812cdf3ccbca | TestVM                                | qcow2       | bare             | 13167616    | active |
| 2faf97d7-5b0b-44ce-8024-3bef5a634570 | test_ubuntu_initial                   | qcow2       | bare             | 1799487488  | active |
| 308b6614-396a-4360-9c33-4e86f41ea0ec | trusty                                | qcow2       | bare             | 256180736   | active |
| 9b3c3fda-2aca-43b5-a3e7-662a94f5e7fb | Ubuntu_14.04_Trusty-amd64-20150708    | qcow2       | bare             | 257884672   | active |
| f93e66cf-fec1-4460-8fc7-506e716fbf30 | ucernvm-prod.1.18-10                  | raw         | bare             | 20971520    | active |
+--------------------------------------+---------------------------------------+-------------+------------------+-------------+--------+

You can then download a particular image with

[name@server ~]$ openstack image save --file ./<file-name-for-image>.<format> 2c751755-854d-49c3-af82-d501e51e7159

where <format> matches that given in the table above for your selected image and the last argument is the ID of the image you wish to download.

Creating VirtualBox VM from a Cloud Image

To use this image in VirtualBox you will need to convert the image in the qcow2 format to the vmdk format. This can be done with the qemu-img tool. This can be installed with something like

[name@server ~]$ sudo apt-get install qemu-img

then do the conversion with

[name@server ~]$ qemu-img convert -f qcow2 vdisk.qcow2 -O vmdk vdisk.vmdk

Then you can create a new virtual machine and attach the vmdk image to it (see how to run a vmdk file in virtualbox for detailed instructions on this).