Arbutus object storage clients: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
(fix internal link broken by name change)
 
(5 intermediate revisions by 2 users not shown)
Line 4: Line 4:
<!--T:1-->
<!--T:1-->
For information on obtaining Arbutus Object Storage, please see [[Arbutus object storage|this page]]. For information on how to use an object storage client to manage your Arbutus object store, choose a client and follow instructions from these pages:
For information on obtaining Arbutus Object Storage, please see [[Arbutus object storage|this page]]. For information on how to use an object storage client to manage your Arbutus object store, choose a client and follow instructions from these pages:
# [[ Accessing object_storage with s3cmd ]]
* [[ Accessing object storage with s3cmd ]]
# WinSCP
* [[ Accessing object storage with WinSCP ]]
# awscli
* [[Accessing the Arbutus object storage with AWS CLI ]]


<!--T:2-->
<!--T:2-->
It is important to note that Arbutus' Object Storage solution does not use Amazon's [https://documentation.help/s3-dg-20060301/VirtualHosting.html S3 Virtual Hosting] (i.e. DNS-based bucket) approach which these clients assume by default. They need to be configured not to use that approach, as described in the pages linked above.  
It is important to note that Arbutus' Object Storage solution does not use Amazon's [https://documentation.help/s3-dg-20060301/VirtualHosting.html S3 Virtual Hosting] (i.e. DNS-based bucket) approach which these clients assume by default. They need to be configured not to use that approach, as described in the pages linked above.  
== WinSCP == <!--T:30-->
=== Installing WinSCP === <!--T:31-->
WinSCP can be installed from https://winscp.net/.
=== Configuring WinSCP === <!--T:32-->
Under "New Session", make the following configurations:
<ul>
<li>File protocol: Amazon S3</li>
<li>Host name: object-arbutus.cloud.computecanada.ca</li>
<li>Port number: 443</li>
<li>Access key ID: 20_DIGIT_ACCESS_KEY</li>
</ul>
and "Save" these settings as shown below
<!--T:33-->
[[File:WinSCP Configuration.png|600px|thumb|center|WinSCP configuration screen]]
<!--T:34-->
Next, click on the "Edit" button and then click on "Advanced..." and navigate to "Environment" to "S3" to "Protocol options" to "URL style:" which <b>must</b> changed from "Virtual Host" to "Path" as shown below:
<!--T:35-->
[[File:WinSCP Path Configuration.png|600px|thumb|center|WinSCP Path Configuration]]
<!--T:36-->
This "Path" setting is important, otherwise WinSCP will not work and you will see hostname resolution errors, like this:
[[File:WinSCP resolve error.png|400px|thumb|center|WinSCP resolve error]]
=== Using WinSCP === <!--T:37-->
Click on the "Login" button and use the WinSCP GUI to create buckets and to transfer files:
<!--T:38-->
[[File:WinSCP transfers.png|800px|thumb|center|WinSCP file transfer screen]]
=== Access Control Lists (ACLs) and Policies === <!--T:41-->
Right-clicking on a file will allow you to set a file's ACL, like this:
[[File:WinSCP ACL.png|400px|thumb|center|WinSCP ACL screen]]
== AWS CLI == <!--T:43-->
<!--T:44-->
The <code>awscli</code> client also works with the Object Store service with better support for large (>5GB) files and the helpful <code>sync</code> command. However, not all features have not been tested.
=== Installing awscli === <!--T:45-->
<!--T:46-->
<pre>
pip install awscli awscli-plugin-endpoint
</pre>
=== Configuring awscli === <!--T:47-->
<!--T:48-->
Generate an access key ID & secret key
<!--T:49-->
<pre>
openstack ec2 credentials create
</pre>
<!--T:50-->
Edit or create <code>~/.aws/credentials</code> and add the credentials generated above
<!--T:51-->
<pre>
[default]
aws_access_key_id = <access_key>
aws_secret_access_key = <secret_key>
</pre>
<!--T:52-->
Edit <code>~/.aws/config</code> and add the following configuration
<!--T:53-->
<pre>
[plugins]
endpoint = awscli_plugin_endpoint
<!--T:54-->
[profile default]
s3 =
  endpoint_url = https://object-arbutus.cloud.computecanada.ca
  signature_version = s3v4
s3api =
  endpoint_url = https://object-arbutus.cloud.computecanada.ca
</pre>
=== Using awscli === <!--T:55-->
<!--T:56-->
<pre>
export AWS_PROFILE=default
aws s3 ls <container-name>
aws s3 sync local_directory s3://container-name/prefix
</pre>
<!--T:57-->
More examples can be found here: https://docs.ovh.com/us/en/storage/getting_started_with_the_swift_S3_API/


<!--T:42-->
<!--T:42-->
[[Category:Cloud]]
[[Category:Cloud]]
</translate>
</translate>

Latest revision as of 18:32, 20 June 2024

Other languages:

For information on obtaining Arbutus Object Storage, please see this page. For information on how to use an object storage client to manage your Arbutus object store, choose a client and follow instructions from these pages:

It is important to note that Arbutus' Object Storage solution does not use Amazon's S3 Virtual Hosting (i.e. DNS-based bucket) approach which these clients assume by default. They need to be configured not to use that approach, as described in the pages linked above.

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Arbutus_object_storage_clients&oldid=156001"