Accessing object storage with AWS CLI

From Alliance Doc
Jump to navigation Jump to search
This site replaces the former Compute Canada documentation site, and is now being managed by the Digital Research Alliance of Canada.

Ce site remplace l'ancien site de documentation de Calcul Canada et est maintenant géré par l'Alliance de recherche numérique du Canada.

Other languages:

This page contains instructions on how to set up and access Arbutus object storage with the AWS Command Line Interface (CLI), one of the object storage clients available for this storage type.

Compared to other object storage clients, AWS CLI has better support for large (>5GB) files and the helpful sync command. However, not all features have not been tested.

Installing AWS CLI

pip install awscli awscli-plugin-endpoint

Configuring AWS CLI

Generate an access key ID and secret key 

openstack ec2 credentials create

Edit or create ~/.aws/credentials and add the credentials generated above 

[default]
aws_access_key_id = <access_key>
aws_secret_access_key = <secret_key>

Edit ~/.aws/config and add the following configuration 

[plugins]
endpoint = awscli_plugin_endpoint

[profile default]
s3 =
  endpoint_url = https://object-arbutus.cloud.computecanada.ca
  signature_version = s3v4
s3api =
  endpoint_url = https://object-arbutus.cloud.computecanada.ca

Using AWS CLI

export AWS_PROFILE=default
aws s3 ls <container-name>
aws s3 sync local_directory s3://container-name/prefix

More examples of using the AWS CLI can be found on this external site.

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Accessing_object_storage_with_AWS_CLI&oldid=133765"