Automating VM creation

Parent page: Cloud

The below virtual machine (VM) setup files are intended to be used to quickly create VMs and clusters of VMs. The files come in two forms, Heat templates, and CloudInit files; both use the YAML file format. CloudInit files are used to initialize a particular VM and run within that VM. They can be thought of as a way to automate tasks you would perform at the command line while logged into your VM. They can be used to perform tasks such as updating the operating system, installing and configuring applications, creating files, running commands, and create users and groups. Heat templates are even more powerful, they can be used to automate tasks performed in the OpenStack dashboard such as creating multiple VMs at once, configuring security groups, creating and configuring networks, and creating and attaching volumes to VMs. Heat templates can be used in conjunction with CloudInit files, once Heat has created the VM it can pass a cloudInit file to that VM to perform setup tasks and even include information about other resources dynamically in the CloudInit files (e.g. floating IPs of other VMs).

Enabling CVMFS on your VM

CVMFS is a HTTP-based file system that provides a scalable, reliable, and low maintenance research software distribution service. At the client end, users just need to mount CVMFS and then use the software or libraries directly without worrying about compiling, building, or patching. All the software are pre-compiled for common OS flavors and even modularized so that users can simply load a software as a module. CVMFS has already been installed on Compute Canada cluster systems such as Cedar, Graham, and Beluga, while on cloud systems users need to enable it by hand, following these cloud instructions: To enable CVMFS on CC Clouds.

For more information please see the Compute Canada CVMFS documentation and CERN CVMFS documentation

Using CloudInit

The creation of CloudInit YAML files is not covered here, instead see the official CloudInit documentation . When you are creating a new VM a CloudInit file can be used to describe how the VM is to be configured after the selected image has been loaded. CloudInit files can be used with the Horizon dashboard (OpenStack's web GUI), the CLI, or the Python API. Here we describe how to use a CloudIinit file with Horizon.

Specifying a CloudInit File

1. Start as normal when launching an instance, by clicking under Project->Compute->Instances and specifying your VM's configuration as described in Launching a VM.
2. Before clicking Launch, select the Post-Creation tab and specify your Customization Script Source, in this case a CloudInit YAML file, by either copying and pasting into a text box (Direct Input method) or uploading from a file from your desktop computer (File method). Older versions of OpenStack, in particular IceHouse, only provide a text box to copy and past your CloudInit file into.
3. Once the usual selections for your VM, as described in Launching a VM, have been made and the CloudInit YAML file is included, click Launch to create the VM. It may take some time for CloudInit to complete depending on what has been specified in the CloudInit YAML file.

Checking CloudInit Progress

To see the progress of CloudInit on a VM, check the console log of the VM by:

1. Selecting Project->Compute->Instances in the left hand menu.
2. Click on the Instance Name of the VM. This will provide more information about the particular VM.
3. Select the Log tab and look for lines containing 'clout-init' for information about the various phases of CloudInit.
4. When CloudInit has finished running the following line will appear near or at the end of the log:

 Cloud-init v. 0.7.5 finished at Wed, 22 Jun 2016 17:52:29 +0000. Datasource DataSourceOpenStack [net,ver=2].  Up 44.33 seconds

The log must be refreshed manually by clicking the Go button

Using Heat Templates

As with CloudInit the creation of Heat Orchestration Template (HOT) files is not covered here, instead see the official documentation. Heat allows automation of operations performed in the OpenStack dashboard (Horizon) as well as the ability to pass information into the embedded CloudInit files, such as an IP of another server. Before using a Heat template there is usually no need to create any resources in advance. In fact it is often good practice to remove any resources you are not currently using before hand, as using a Heat template consumes resources towards your quota and will fail if it tries to exceed your quota.

To create a stack using a HOT file:

1. Select Project->Orchestration->Stacks and click the Launch Stack button to start creating a new stack.
2. Provide a HOT file by entering the URL, the File name, or by Direct Input. Here, we will use a HOT file from one of the links in section Available Setups below.
3. In the Template Source box, select URL from the drop-down list.
4. Paste the selected URL into the Template URL box.
5. Click Next to begin setting stack parameters; these vary depending on the template, however all stacks have the following parameters by default:
   • Stack Name; choose a name which is meaningful.
   • Creation Timeout; indicates how long after stack creation before OpenStack will give up trying to create the stack if it hasn't finished; the default value is usually sufficient.
   • Password for user ; sets the password required for later stack changes. This is seldom used as many of the stacks mentioned in the next section are not designed to be updated.
6. Click Launch to begin creating your stack.

To graphically see the progress of your stack creation click on the Stack Name and select the Topology tab. Gray nodes indicate that creation is in progress, green nodes have finished being created, and red nodes indicate failures. Once the stack has completed successfully click the Overview tab to see any information that the stack may provide (e.g. a URL to access a service or website).

Available Setups

Hadoop+Spark Heat Template

Creates a Hadoop cluster with Spark configured to run ontop of HDFS and submit jobs using YARN.

Apache2 CloudInit

Sets up an apache2 webserver with the document root in users home dir under the public_html folder.

Mediawiki Heat Template

Sets up a mediawiki webserver.

CernVM File System

HyperText Transfer Protocol