Arbutus object storage

Introduction

Object storage is a service that manages data as objects. This is different from other storage architectures where data is managed in a file hierarchy. Objects can be created, replaced, or deleted, but unlike traditional storage, they cannot be edited in place. Object storage has become popular due to its ability to handle large files and large numbers of files, and due to the prevalence of compatible tools.

Unlike other storage types, a unit of data or object is managed as a whole, and the information within it cannot be modified in place. Objects are stored in containers in the object store. The containers are stored in a way that makes them easier and often faster to access than in a traditional filesystem.

The best use of object storage is to store and export items which do not need hierarchical naming; are accessed mostly as a whole and mostly read-only; and have simplified access-control rules. We recommend using it with software or platforms that are designed to work with data living in an object store.

All Arbutus projects are allocated a default 1TB of object storage. If more is required, you can either request an additional 9 TB available through our Rapid Access Service. More than 10TB must be requested and allocated under the annual Resource Allocation Competition.

Unlike a cluster computing environment, system administration for a project's containers are managed by that user, which includes operations like backups. For more information about differences between object storage and other cloud storage types, see Cloud storage options.

We offer access to the OpenStack Object Store via two different protocols: Swift or Amazon Simple Storage Service (S3).

These protocols are very similar and in most situations you can use whichever you like. You don't have to commit to one, as object storage containers and objects created with Swift or S3 can be accessed using both protocols. There are a few key differences in the context of the Arbutus Object Store.

Swift is the default and is simpler since you do not have to manage credentials yourself. Access is governed using your Arbutus account. However, Swift does not replicate all the functionalities of S3. The main use case here is that when you want to manage your object storage containers using access policies, you must use S3, as Swift does not support access policies. You can also create and manage your own keys using S3, which could be useful if you for example want to create a read-only user for a specific application. A full list of Swift/S3 compatibility can be found here:

https://docs.openstack.org/swift/latest/s3_compat.html

Establishing access to your Arbutus object store

In order to manage your Arbutus Object store, you will need your own storage access ID and secret key. To generate these, use the OpenStack command line client:

openstack ec2 credentials create

Accessing your Arbutus Object Store

Setting access policies cannot be done via web browser but must be done with a SWIFT or S3-compatible client. There are two ways to access your data containers:

1. if your data container policies are set to private (default), object storage is accessible via an S3-compatible client (e.g. s3cmd).
2. if your object storage policies are set to public (not default), object storage is accessible using a browser via an HTTPS endpoint:

https://object-arbutus.cloud.computecanada.ca:443/DATA_CONTAINER/FILENAME

Managing your Arbutus object store

The recommended way to manage buckets and objects in the Arbutus Object Store is by using the s3cmd tool, which is available in Linux. Our documentation provides specific instructions on configuring and managing access with the s3cmd client We can also use other S3-compatible clients that are also compatible with Arbutus Object Store.

In addition, we can perform certain management tasks for our object storage using the Containers section under the Object Store tab in the Arbutus OpenStack Dashboard.

This interface refers to "data containers", which are also known as "buckets" in other object storage systems.

Using the dashboard, we can create new data containers, upload files, and create directories. Alternatively, we can also create data containers using S3-compatible clients.

Please note that data containers are owned by the user who creates them and cannot be manipulated by others.
Therefore, you are responsible for managing your data containers and their contents within your cloud project.

If you create a new container as Public, anyone on the Internet can read its contents by simply navigating to

https://object-arbutus.cloud.computecanada.ca/<YOUR CONTAINER NAME HERE>/<YOUR OBJECT NAME HERE>

with your container and object names inserted in place.

It's important to keep in mind that each data container on the Arbutus Object Store must have a unique name across all users. To ensure uniqueness, we may want to prefix our data container names with our project name to avoid conflicts with other users. One useful rule of thumb is to refrain from using generic names like test for data containers. Instead, consider using more specific and unique names like def-myname-test.

To make a data container accessible to the public, we can change its policy to allow public access. This can come in handy if we need to share files to a wider audience. We can manage container policies using JSON files, allowing us to specify various access controls for our containers and objects.

Managing data container (bucket) policies for your Arbutus Object Store

Attention

Be careful with policies because an ill-conceived policy can lock you out of your data container.

Currently, Arbutus Object Storage only implements a subset of Amazon's specification for data container polices. The following example shows how to create, apply, and view a policy. The first step is create a policy json file:

{
    "Version": "2012-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::testbucket",
                "arn:aws:s3:::testbucket/*"
            ],
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": "206.12.0.0/16",
                    "aws:SourceIp": "142.104.0.0/16"
                }
            }
        }
    ]
}

This example denies access except from the specified source IP address ranges in Classless Inter-Domain Routing (CIDR) notation. In this example the s3://testbucket is limited to the public IP address range (206.12.0.0/16) used by the Arbutus cloud and the public IP address range (142.104.0.0/16) used by the University of Victoria.

Once you have your policy file, you can implement that policy on the data container:

s3cmd setpolicy testbucket.policy s3://testbucket

To view the policy you can use the following command:

s3cmd info s3://testbucket