CephFS

CephFS provides a common filesystem that can be shared amongst multiple OpenStack VM hosts. Access to the service is granted via requests to cloud@tech.alliancecan.ca.

This is a fairly technical procedure that assumes basic Linux skills for creating/editing files, setting permissions, and creating mount points. For assistance in setting up this service, write to cloud@tech.alliancecan.ca.

Procedure

Request Access to Shares

• If you do not already have a quota for the service you will need to request this through ccloud@tech.alliancecan.ca.
  • In your request please provide the following:
    • OpenStack Project name
    • Amount of quota required in GB.
    • If more than one share is required, how many are required?

Create Share

1. Create a share in "Shares" under the "Share" menu:
   • Give this a name that identifies your project: project-name-shareName
     • e.g. def-project-shareName
   • Share Protocol = cephfs
   • Size = size you need for this share
   • Share Type = cephfs
   • Availability Zone = nova
   • Do not check "Make visible for all", otherwise the share would be accessible by anyone in every project.
2. Create an Access Rule which generates an Access Key
   • On the "Shares" pane, click on the drop down menu under "Actions" and select "Manage Rules".
   • Create a new rule using the "+Add Rule" button.
     • Access Type = cephx
     • Select "read-write" or "read-only" under "Access Level". You can create multiple rules for either access level if required.
     • Choose a key name in the "Access To" field that describes the key (e.g. def-project-shareName-read-write).
3. Note the Share details:
   • Click on the share.
   • Under "Overview", note the "Path" which you will need later.
   • Under "Access Control", note the "Access Key" which you will need later.
   • Access Keys are approximately 40 characters and end with an "=" sign.
   • If you do not see an Access Key, you probably didn't add an Access Rule with an Access Type = cephx

Configure Host

1. Install required packages

   • Red Hat Family (RHEL, CentOS, Fedora, Scientific Linux, SUSE, etc.):

     1. Install relevant repos for access to ceph client packages:

        ceph-stable (nautilus is current as of this writting)
            https://docs.ceph.com/en/nautilus/install/get-packages/
        epel (sudo yum install epel-release)
        
        

     2. Install packages to enable the ceph client on all the VMs you plan on mounting the share:

        libcephfs2
        python-cephfs
        ceph-common
        python-ceph-argparse
        ceph-fuse (only if you intend a fuse mount)
        

     • Debian Family (Debian, Ubuntu, Mint, etc.):

         https://docs.ceph.com/en/nautilus/install/get-packages/
     

2. Configure Keys:

   • Create two files in your VM each containing the "Access Key". This key can be found in the rule definition, or in the "Access Rules" section of your share definition.

   • File 1: /etc/ceph/client.fullkey.shareName (e.g. client.fullkey.def-project-shareName-read-write)

     • contents:

       [client.shareName]
           key = AccessKey
       

   • File 2: /etc/ceph/client.keyonly.shareName (e.g client.keyonly.def-project-shareName-read-write)

     • contents:

       AccessKey
       

     • This file only contains the Access Key

   • Own these files correctly to protect the key information:

     • Each file should be own to root

     sudo chown root.root filename
     

     • Each file should be only readable by root

     sudo chmod 600 filename
     

3. Create /etc/ceph/ceph.conf with contents:

   [client]
       client quota = true
       mon host = 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789
   

   • Note: these are the monitors for the Arbutus cluster - if connecting to a different cluster you will need the monitor information specific to that cluster.
     • You can find the monitor information in the Share Details for your share in the "Path" field.

4. Retrieve the connection information from the share page for your connection:

   • Open up the share details by clicking the name of the share in the Shares page.
   • Copy the entire path of the share for mounting the filesystem.

5. Mount the filesystem

   • Create mount point directory somewhere in your host (likely under /mnt/ - e.g. /mnt/ShareName)
   • Via kernel mount using the ceph driver:
     • Syntax: sudo mount -t ceph <path information> <mountPoint> -o name=<shareKeyName>, secretfile=</path/to/keyringfileOnlyFile>
     • sudo mount -t ceph mon1:6789,mon2:6789,mon3:6789:/volumes/_nogroup/share_instance_id
       • e.g sudo mount -t ceph 192.168.17.13:6789,192.168.17.14:6789,192.168.17.15:6789:/volumes/_nogroup/a87b5ef3-b266-4664-a5ed-026cddfdcdb7 /mnt/WebServerShare -o name=def-project-shareName-read-write,secretfile=/etc/ceph/client.keyonly.def-project-sharename-read-write
   • Via ceph-fuse
     • Need to install ceph-fuse
     • Syntax: sudo ceph-fuse <mountPoint> --id=<shareKeyName> --conf=<pathtoCeph.conf> --keyring=<fullKeyringLocation> --client-mountpoint=pathFromShareDetails
       • e.g. sudo ceph-fuse /mnt/WebServerShare --id=def-project-shareName-read-write --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/client.fullkey.def-project-shareName-read-write --client-mountpoint=/volumes/_nogroup/a87b5ef3-b266-4664-a5ed-026cddfdcdb7

Notes

• A particular share can have more than one user key provisioned for it.
  • This allows a more granular access to the filesystem.
  • For example, if you needed some hosts to only access the filesystem in a read only capacity.
  • If you have multiple keys for a share you can add the extra keys to your host and modify the above mounting procedure.
• This service is not available to hosts outside of the Openstack cluster.