Arbutus object storage

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Arbutus Object storage

Object storage at Arbutus can be requested via cloud@computecanada.ca.

You can either apply for a RAS allocation or a RAC allocation.

We offer access to the Object Store via three different protocols:

1. S3
2. Swift
3. Radosgw

Access Request Information

When requesting access we will ask you for the following:

• Project code (e.g. rrg_piUserName)
• CC username of the user(s) to add
• Actual name of user(s) (e.g. Mike Cave)
• Expiry date of user(s) - This is used if the user is a temp member of the group (e.g. grad student, lab assistant, temporary group member)
• Permission type (read, write, both), per user
• Do you need a Swift key?

Once we have the basic users and account setup on the object storage service, we will let you know how to collect the keys for access.

Bucket Management

Admins will create the project for new requests and users are generated for the project at the time.

The users are responsible for operations inside of the 'tenant'. As such, the buckets and management of those buckets are up to the user.

The tool "s3cmd" which is available in Linux is the perferred way to access our S3 gateway, however ther are other tools out there that will also work.

Some General Information

• Buckets are owned to the user that creates them and no other users can manipulate them
• You can make a bucket world accessible which then gives you a URL to post that will serve content in the bucket
• Bucket policies are managed via json files

Connection Details and s3cmd Configuration

The object storage is accessible via an HTTP endpoint:

object-arbutus.cloud.computecanada.ca:443

The following is an example of a bare minimum s3cmd configuration file. You will need these values, but are free to explore additional s3cmd configuration options to fit your use case. Note that in the example the keys are redacted and you will need to replace them with your provided key values:

[default]
access_key = <redacted>
check_ssl_certificate = True
check_ssl_hostname = True
host_base = object-arbutus.cloud.computecanada.ca:443
host_bucket = object-arbutus.cloud.computecanada.ca:443
secret_key = <redacted>
use_https = True

Using s3cmd's --configuration feature is described here.

Example Bucket operations

• Making a bucket public so that it is web accessible:

  s3cmd setacl s3://testbucket --acl-public

• Make the bucket private again:

  s3cmd setacl s3://testbucket --acl-private

• Example bucket policy:

  You need to first create a policy json file:

  "testbucket.policy": 
  {
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Principal": {"AWS": [
      "arn:aws:iam::rrg_cjhuofw:user/parsa7",
      "arn:aws:iam::rrg_cjhuofw:user/dilbar"
      ]},
      "Action": [
      "s3:ListBucket",
      "s3:PutObject",
      "s3:DeleteObject",
      "s3:GetObject"
      ],
      "Resource": [
      "arn:aws:s3:::testbucket/*",
      "arn:aws:s3:::testbucket"
      ]
  }]
  }
  

  This file allows you to set specific permissions for any number of users of that bucket.

  You can even specify users from another tenant if there is a user from another project working with you.

  Now that you have your policy file, you can implement that policy on the bucket:

  s3cmd setpolicy testbucket.policy s3://testbucket

  More extensive examples and actions can be found here: https://www.linode.com/docs/platform/object-storage/how-to-use-object-storage-acls-and-bucket-policies/

Rapid Access Service

Resource Allocation Competition

HyperText Transfer Protocol