Arbutus object storage

Introduction

Object storage is a storage facility that is simpler than a normal hierarchical filesystem, but benefits by avoiding some performance bottlenecks.

An object is a fixed file in a flat namespace: you can create/upload an object as a whole, but cannot modify bytes within it. Objects are named as bucket:tag with no further nesting. Since bucket operations are basically whole-file, the provider can use a simpler internal representation. The flat namespace allows the provider to avoid metadata bottlenecks; it's basically a key-value store.

The best use of object storage is to store and export items which do not need hierarchical naming; are accessed mostly atomically and mostly read-only; and with simplified access-control rules.

All Arbutus projects are allocated a default 1TB of Object Store. If more is required, you can either apply for a RAS allocation or a RAC allocation.

Unlike a cluster computing environment, system administration for a user's Object Storage buckets are managed solely by that user. This means that operations like backups must be managed by the user. For more information about differences between Object Storage and other cloud storage types, see Cloud storage options.

We offer access to the Object Store via two different protocols: Swift or S3.

These protocols are very similar and in most situations you can use whichever you like. You don't have to commit to one, as buckets and objects created with Swift or S3 can be accessed using both protocols. There are a few key differences in the context of Arbutus Object Store.

Swift is given by default and is simpler since you do not have to manage credentials yourself. Access is governed using your Arbutus account. However, Swift does not replicate all the functionality of S3. The main use case here is when you want to manage your buckets using bucket policies you must use S3 as Swift does not support bucket policies. You can also create and manage your own keys using S3, which could be useful if you for example want to create a read-only user for a specific application. A full list of Swift/S3 compatibility can be found here:

https://docs.openstack.org/swift/latest/s3_compat.html

Accessing and managing Object Store

You can manage your object storage using the Object Store tab for your project at https://arbutus.cloud.computecanada.ca/. This interface refers to buckets as containers (not to be confused with containers based on namespace functionality of the Linux kernel). You can create containers (AKA buckets) in this interface, upload files, and create directories. Containers can also be created using S3-compatible CLI clients. Please note that if you create a new container as Public, any object placed within this container can be freely accessed (read-only) by anyone on the internet simply by navigating to https://object-arbutus.cloud.computecanada.ca/<YOUR CONTAINER NAME HERE>/<YOUR OBJECT NAME HERE> with your container and object names inserted in place.

You can also use the OpenStack command line client. For instructions on how to install and operate the OpenStack command line clients, see OpenStack Command Line Clients.

To generate your own S3 access ID and secret key for the S3 protocol, use the OpenStack command line client:

openstack ec2 credentials create

The s3cmd tool which is available in Linux is the preferred way to access our S3 gateway; however there are other tools out there that will also work.

The users are responsible for operations inside the tenant. As such, the buckets and management of those buckets are up to the user.

General information

• Buckets are owned by the user who creates them, and no other user can manipulate them.
• You can make a bucket accessible to the world, which then gives you a URL to share that will serve content from the bucket.
• Bucket names must be unique across all users in the Object Store, so you may benefit by prefixing each bucket with your project name to maintain uniqueness. In other words, don't bother trying to create a bucket named test, but def-myname-test is probably OK.
• Bucket policies are managed via json files.

Connection details and s3cmd Configuration

Object storage is accessible via an HTTPS endpoint:

object-arbutus.cloud.computecanada.ca:443

The following is an example of a minimal s3cmd configuration file. You will need these values, but are free to explore additional s3cmd configuration options to fit your use case. Note that in the example the keys are redacted and you will need to replace them with your provided key values:

[default]
access_key = <redacted>
check_ssl_certificate = True
check_ssl_hostname = True
host_base = object-arbutus.cloud.computecanada.ca
host_bucket = object-arbutus.cloud.computecanada.ca
secret_key = <redacted>
use_https = True

Using s3cmd's --configure feature is described here.

Example operations on a bucket

• Make a bucket public so that it is web accessible:

  s3cmd setacl s3://testbucket --acl-public

• Make the bucket private again:

  s3cmd setacl s3://testbucket --acl-private

Bucket policies

Currently Arbutus Object Storage only implements a subset of Amazon's specification for [polices].

Example bucket policy:

You need to first create a policy json file:

{
    "Version": "2012-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::testbucket",
                "arn:aws:s3:::testbucket/*"
            ],
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": "206.12.0.0/16"
                    "aws:SourceIp": "142.104.0.0/16"
                }
            }
        }
    ]
}

This file allows you to limit users of that bucket from certain source IP addresses. In this example the s3://testbucket is limited to the public IP address range (206.12.0.0/16) used by the Arbutus Cloud and the public IP address ranged (142.104.0.0/16) used by the University of Victoria.

Once you have your policy file, you can implement that policy on the bucket:

s3cmd setpolicy testbucket.policy s3://testbucket

To view the policy you can use the following command:

s3cmd info s3://testbucket

Rapid Access Service

Resource Allocation Competition