Arbutus object storage clients: Difference between revisions
(Inital version) |
No edit summary |
||
| Line 13: | Line 13: | ||
Depending on your Linux distribution, the <code>s3cmd</code> command can be installed using the appropriate yum or apt command: | Depending on your Linux distribution, the <code>s3cmd</code> command can be installed using the appropriate yum or apt command: | ||
<code>$ sudo yum install s3cmd</code> | <code>$ sudo yum install s3cmd</code><br/> | ||
<code>$ sudo apt-get install s3cmd </code> | <code>$ sudo apt-get install s3cmd </code> | ||
| Line 20: | Line 20: | ||
And make the following configurations: | And make the following configurations: | ||
<code> | |||
Enter new values or accept defaults in brackets with Enter. | Enter new values or accept defaults in brackets with Enter. | ||
Refer to user manual for detailed description of all options. | Refer to user manual for detailed description of all options. | ||
| Line 49: | Line 49: | ||
Try setting it here if you can't connect to S3 directly | Try setting it here if you can't connect to S3 directly | ||
HTTP Proxy server name: | HTTP Proxy server name: | ||
</code> | |||
The next task is to make a bucket. Buckets contain files. Bucket names must be globally unique across the Arbutus object storage solution. Therefore, you will need to create a uniquely named bucket which will not conflict with other users. For example, the buckets "s3://test/" and "s3://data" are likely already taken. Consider creating buckets reflective of your project, for example "s3://def-test-bucket1" or "s3://atlas_project_bucket". Bucket names can only use the characters A-Z, a-z, 0-9, ., - and _. | The next task is to make a bucket. Buckets contain files. Bucket names must be globally unique across the Arbutus object storage solution. Therefore, you will need to create a uniquely named bucket which will not conflict with other users. For example, the buckets "s3://test/" and "s3://data" are likely already taken. Consider creating buckets reflective of your project, for example "s3://def-test-bucket1" or "s3://atlas_project_bucket". Bucket names can only use the characters A-Z, a-z, 0-9, ., - and _. | ||
| Line 69: | Line 69: | ||
<code>$ s3cmd setacl --acl-public s3://BUCKET_NAME</code> | <code>$ s3cmd setacl --acl-public s3://BUCKET_NAME</code> | ||
The result of this command is that anyone can access the bucket and the files in the bucket. Files can be accessed via URLs such as https://object-arbutus.cloud.computecanada.ca/BUCKET_NAME/FILE_NAME. | The result of this command is that anyone can access the bucket and the files in the bucket. Files can be accessed via URLs such as https://object-arbutus.cloud.computecanada.ca/BUCKET_NAME/FILE_NAME.dat. | ||
The second ACL example limits access to the bucket by the owner: | The second ACL example limits access to the bucket by the owner: | ||
Revision as of 17:09, 9 January 2022
This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.
Arbutus Object Storage Clients[edit]
For information on obtaining Arbutus Object Storage, please see the page on Arbutus Object Storage
This page describes how to configure and use two common object storage clients
- s3cmd
- WinSCP
s3cmd[edit]
Depending on your Linux distribution, the s3cmd command can be installed using the appropriate yum or apt command:
$ sudo yum install s3cmd
$ sudo apt-get install s3cmd
To configure the s3cmd tool use the command
$ s3cmd --configure
And make the following configurations:
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: 20_DIGIT_ACCESS_KEY
Secret Key []: 40_DIGIT_SECRET_KEY
Default Region [US]:
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint []: object-arbutus.cloud.computecanada.ca
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: object-arbutus.cloud.computecanada.ca
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password []: PASSWORD
Path to GPG program []: /usr/bin/gpg
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol []: Yes
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:
The next task is to make a bucket. Buckets contain files. Bucket names must be globally unique across the Arbutus object storage solution. Therefore, you will need to create a uniquely named bucket which will not conflict with other users. For example, the buckets "s3://test/" and "s3://data" are likely already taken. Consider creating buckets reflective of your project, for example "s3://def-test-bucket1" or "s3://atlas_project_bucket". Bucket names can only use the characters A-Z, a-z, 0-9, ., - and _.
To create a bucket, use the tool's mb (make bucket) command:
$ s3cmd mb s3://BUCKET_NAME/
To see the status of a bucket, use the command:
$ s3cmd info s3://BUCKET_NAME/
To upload a file to the bucket, use the command:
$ s3cmd put --guess-mime-type FILE_NAME.dat s3://BUCKET_NAME/FILE_NAME.dat
Buckets can have Access Control Lists (ACLs) and policies which govern who can access what resources in the object store. These features are quite sophisticated. Here are two simple examples of using ACLs using the tool's setacl command.
$ s3cmd setacl --acl-public s3://BUCKET_NAME
The result of this command is that anyone can access the bucket and the files in the bucket. Files can be accessed via URLs such as https://object-arbutus.cloud.computecanada.ca/BUCKET_NAME/FILE_NAME.dat.
The second ACL example limits access to the bucket by the owner:
$ s3cmd setacl --acl-privte s3://BUCKET_NAME
Other more sophisticated examples can be found in the s3cmd man page.
WinSCP[edit]
WinSCP can be installed https://winscp.net/