CephFS

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

User Guide to Provisioning and Deploying CephFS

Introduction

CephFS provides a common filesystem that can be shared amongst multiple openstack vm hosts. Access to the service is granted via requests to cloud@computecanada.ca.

This is a fairly technical procedure that assumes basic linux skills for creating/editing files, setting permissions and creating mount points. Contact your technical resource for your project for assistance in setting up this service.

Technical Procedure

Request Access to Shares

• If you do not already have a quota for the service you will need to request this through cloud@computecanada.ca.
  • In your request provide the following:
    • Project name
    • Expected file system share name.
      • This should be the name of the project followed by a unique name for the share (e.g. def-smith-WebServerShare).
    • Approximate size of share in GB.

Create Share

1. Create a share in "Shares" under the "Share" menu:
   • Give this a name that identifies your project: project-name-shareName
     • e.g. def-smith-WebServerShare
   • Share Protocol = cephfs
   • Size = size you need for this share
   • Share Type = cephfs
2. Provide the name of your share to the cloud team so they can generate a key for you. Do so by replying to the initial request ticket you opened in the first steps.
   • You will need to provide the following in your request:
     • Share name (e.g. def-smith-WebServerShare).

Configure Host

1. Retrieve Key:

   • View the "Share Details" by clicking on the share name in the "Shares" pane.

     • Key is located under the "Access Rules" section labeled "Access Key".

   • Create two files each containing the key provided.

   • File 1: /etc/ceph/client.fullkey.shareName (e.g. client.fullkey.WebServerShare)

     • contents:

       [client.shareName]
           key = KeyProvidedByAdmin
       

   • File 2: /etc/ceph/client.keyonly.shareName (e.g client.keyonly.WebServerShare)

     • contents:

       KeyProvidedByAdmin
       

     • This file only contains the key provided by the admin

   • Own these files correctly to protect the key information:

     • Each file should be own to root (sudo chown root.root filename).
     • Each file should be only readable by root (sudo chmod 600 filename).

2. Create /etc/ceph/ceph.conf with contents:

   [client]
       client quota = true
       mon host = 10.30.201.3:6789,10.30.202.3:6789,10.30.203.3:6789
   

   • Note: these are the monitors for the Arbutus cluster - if connecting to a different cluster you will need the monitor information specific to that cluster.
     • You can find the monitor information in the Share Details for your share in the "Path" field.

3. Install required packages

• Red Hat Family (RHEL, CentOS, Fedora, Scientific Linux, SUSE, etc.): 1. Install relevant repos for access to ceph client packages: ceph-stable (nautilus is current as of this writting) epel 1. Install packages to enable the ceph client on all the VMs you plan on mounting the share: libcephfs2 python-cephfs ceph-common python-ceph-argparse ceph-fuse (only if you intend a fuse mount)

• Debian Family (Debian, Ubuntu, Mint, etc.):

  sudo apt install ceph-common
  

1. Retrieve the connection information from the share page for your connection:
   • Open up the share details by clicking the name of the share in the Shares page.
   • Copy the entire path of the share for mounting the filesystem.
2. Mount the filesystem
   • Create mount point directory somewhere in your host (likely under /mnt/ - e.g. /mnt/ShareName)
   • Via kernel mount using the ceph driver:
     • Syntax: sudo mount -t ceph < path information> < mountPoint > -o name=< shareKeyName >, secretfile=< /path/to/keyringfileOnlyFile >
     • sudo mount -t ceph mon1:6789,mon2:6789,mon3:6789:/volumes/_nogroup/share_instance_id
       • e.g sudo mount -t ceph 192.168.17.13:6789,192.168.17.14:6789,192.168.17.15:6789:/volumes/_nogroup/a87b5ef3-b266-4664-a5ed-026cddfdcdb7 /mnt/WebServerShare -o name=def-smith,secretfile=/etc/ceph/client.keyonly.WebServerShare
   • Via ceph-fuse
     • Need to install ceph-fuse
     • Syntax: sudo ceph-fuse < mountPoint > --id=< projectName > --conf=< pathtoCeph.conf > --keyring=< fullKeyringLocation > --client-mountpoint=pathFromShareDetails
       • e.g. sudo ceph-fuse /mnt/WebServerShare --id=def-smith --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/client.fullkey.WebServerShare --client-mountpoint=/volumes/_nogroup/a87b5ef3-b266-4664-a5ed-026cddfdcdb7

Notes

• A particular share can have more than one user key provisioned for it.
  • This allows a more granular access to the filesystem.
  • For example, if you needed some hosts to only access the filesystem in a read only capacity.
  • If you have multiple keys for a share you can add the extra keys to your host and modify the above mounting procedure.
• This service is not available to hosts outside of the Openstack cluster.