rsnt_translations
56,426
edits
Arbutus object storage: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 12: | Line 12: | ||
<!--T:29--> | <!--T:29--> | ||
The best use of object storage is to store and export items which do not need hierarchical naming; are accessed mostly as a whole and mostly read-only; and have simplified access-control rules. We recommend using it with software or platforms that are designed to work with data living in an object store. | The best use of object storage is to store and export items which do not need hierarchical naming; are accessed mostly as a whole and mostly read-only; and have simplified access-control rules. We recommend using it with software or platforms that are designed to work with data living in an object store. | ||
<!--T:2--> | <!--T:2--> | ||
All Arbutus projects are allocated a default 1TB of object storage. If more is required, you can either request an additional | All Arbutus projects are allocated a default 1TB of object storage. If more is required, you can either request an additional 9 TB available through our [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service Rapid Access Service]. More than 10TB must be requested and allocated under the annual [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition Resource Allocation Competition]. | ||
<!--T:30--> | <!--T:30--> | ||
| Line 26: | Line 23: | ||
<!--T:5--> | <!--T:5--> | ||
These protocols are very similar and in most situations you can use whichever you like. You don't have to commit to one, as object storage containers and objects created with Swift or S3 can be accessed using both protocols. There are a few key differences in the context of the Arbutus | These protocols are very similar and in most situations you can use whichever you like. You don't have to commit to one, as object storage containers and objects created with Swift or S3 can be accessed using both protocols. There are a few key differences in the context of the Arbutus Object Store. | ||
<!--T:6--> | <!--T:6--> | ||
| Line 34: | Line 31: | ||
https://docs.openstack.org/swift/latest/s3_compat.html | https://docs.openstack.org/swift/latest/s3_compat.html | ||
= Establishing access to your Arbutus | = Establishing access to your Arbutus Object Store = <!--T:8--> | ||
<!--T:13--> | <!--T:13--> | ||
In order to manage your Arbutus Object | In order to manage your Arbutus Object Store, you will need your own storage access ID and secret key. To generate these, use the [[OpenStack command line clients|OpenStack command line client]]: | ||
<!--T:14--> | <!--T:14--> | ||
<code>openstack ec2 credentials create</code> | <code>openstack ec2 credentials create</code> | ||
= Accessing your Arbutus | = Accessing your Arbutus Object Store = <!--T:35--> | ||
Setting access policies cannot be done via | Setting access policies cannot be done via web browser but must be done with a [[Arbutus object storage clients|SWIFT or S3-compatible client]]. There are two ways to access your data containers: | ||
<!--T:21--> | <!--T:21--> | ||
| Line 54: | Line 51: | ||
<!--T:15--> | <!--T:15--> | ||
The recommended way to manage buckets and objects in the Arbutus Object Store is by using the <code>s3cmd</code> tool, which is available in Linux. | The recommended way to manage buckets and objects in the Arbutus Object Store is by using the <code>s3cmd</code> tool, which is available in Linux. | ||
Our documentation provides specific instructions on [[Accessing_object_storage_with_s3cmd|configuring and managing access]] with the <code>s3cmd</code> client | Our documentation provides specific instructions on [[Accessing_object_storage_with_s3cmd|configuring and managing access]] with the <code>s3cmd</code> client | ||
We can also use other [[Arbutus object storage clients|S3-compatible clients]] that are also compatible with Arbutus Object Store. | We can also use other [[Arbutus object storage clients|S3-compatible clients]] that are also compatible with Arbutus Object Store. | ||
| Line 64: | Line 61: | ||
<!--T:38--> | <!--T:38--> | ||
Using the dashboard, | Using the dashboard, we can create new data containers, upload files, and create directories. Alternatively, we can also create data containers using [[Arbutus object storage clients|S3-compatible clients]]. | ||
<!--T:39--> | <!--T:39--> | ||
| Line 70: | Line 67: | ||
<!--T:40--> | <!--T:40--> | ||
If you create a new container as | If you create a new container as '''Public''', anyone on the Internet can read its contents by simply navigating to | ||
<!--T:41--> | <!--T:41--> | ||
| Line 81: | Line 78: | ||
<!--T:43--> | <!--T:43--> | ||
{{quote|It's important to keep in mind that each data container on the Arbutus Object Store must have a | {{quote|It's important to keep in mind that each data container on the '''Arbutus Object Store''' must have a '''unique name across all users'''. To ensure uniqueness, we may want to prefix our data container names with our project name to avoid conflicts with other users. One useful rule of thumb is to refrain from using generic names like <code>test</code> for data containers. Instead, consider using more specific and unique names like <code>def-myname-test</code>.}} | ||
<!--T:44--> | <!--T:44--> | ||
To make a data container accessible to the public, we can change its policy to allow public access. This can come in handy if we need to share files to a wider audience. We can manage container policies using JSON files, allowing us to specify various access controls for our containers and objects. | To make a data container accessible to the public, we can change its policy to allow public access. This can come in handy if we need to share files to a wider audience. We can manage container policies using JSON files, allowing us to specify various access controls for our containers and objects. | ||
== Managing data container (bucket) policies for your Arbutus | == Managing data container (bucket) policies for your Arbutus Object Store == <!--T:31--> | ||
<br/> | <br/> | ||
{{Warning|title=Attention|content=Be careful with policies because an ill-conceived policy can lock you out of your data container.}} | {{Warning|title=Attention|content=Be careful with policies because an ill-conceived policy can lock you out of your data container.}} | ||
<!--T:34--> | <!--T:34--> | ||
Currently, Arbutus | Currently, Arbutus Object Storage only supports a [[Arbutus_object_storage#Policy_subset|subset]] of the AWS specification for [https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-iam-policies.html data container polices]. The following example shows how to create, apply, and view a policy. The first step is create a policy json file: | ||
<!--T:45--> | <!--T:45--> | ||