Arbutus CephFS

From Alliance Doc
Jump to navigation Jump to search
This site replaces the former Compute Canada documentation site, and is now being managed by the Digital Research Alliance of Canada.

Ce site remplace l'ancien site de documentation de Calcul Canada et est maintenant géré par l'Alliance de recherche numérique du Canada.

Other languages:

CephFS provides a common filesystem that can be shared amongst multiple OpenStack VM hosts. Access to the service is granted via requests to

This is a fairly technical procedure that assumes basic Linux skills for creating/editing files, setting permissions, and creating mount points. For assistance in setting up this service, write to


Request access to shares

If you do not already have a quota for the service, you will need to request this through In your request please provide the following:

  • OpenStack project name
  • amount of quota required in GB
  • number of shares required

Create share

  1. Create the share.
    • In Project --> Share --> Shares, click on +Create Share.
    • Share Name = enter a name that identifies your project (e.g. project-name-shareName)
    • Share Protocol = CephFS
    • Size = size you need for this share
    • Share Type = cephfs
    • Availability Zone = nova
    • Do not check Make visible for all, otherwise the share will be accessible by all users in all projects.
    • Click on the Create button.
  2. Create an access rule to generate an access key.
    • In Project --> Share --> Shares --> Actions column, select Manage Rules from the drop-down menu.
    • Click on the +Add Rule button (right of page).
    • Access Type = cephx
    • Access Level = select read-write or read-only (you can create multiple rules for either access level if required)
    • Access To = select a key name that describes the key (e.g. def-project-shareName-read-write)
  3. Note the share details which you will need later.
    • In Project --> Share --> Shares, click on the name of the share.
    • In the Share Overview, note the Path.
    • Under Access Rules, note the Access Key (the access key is approximately 40 characters and ends with the = sign; if you do not see an access key, you probably didn't add an access rule of type cephx.

Configure host

  1. Install the required packages.

    • Red Hat family (RHEL, CentOS, Fedora, Scientific Linux, SUSE, etc.):

      1. Install relevant repositories for access to ceph client packages:
        ceph-stable (nautilus is current as of this writting)
        epel (sudo yum install epel-release)
      2. Install packages to enable the ceph client on all the instances where you plan on mounting the share:
        ceph-fuse (only if you intend a fuse mount)
      • Debian family (Debian, Ubuntu, Mint, etc.):

  2. Configure Keys:

    • Create two files in your instance, each containing the access key. This key can be found in the rule definition, or in the Access Rules section of your share definition.

    • File 1: /etc/ceph/client.fullkey.shareName (e.g. client.fullkey.def-project-shareName-read-write)

      • contents:
            key = AccessKey
    • File 2: /etc/ceph/client.keyonly.shareName (e.g client.keyonly.def-project-shareName-read-write)

      • contents:
      • This file only contains the access key.
    • Own these files correctly to protect the key information:

      • Each file should be own to root
      sudo chown root.root filename
      • Each file should be only readable by root
      sudo chmod 600 filename
  3. Create /etc/ceph/ceph.conf with contents:

        client quota = true
        mon host =,,
        keyring = /etc/ceph/client.keyonly.def-project-sharename-read-write
    • Note: these are the monitors for the Arbutus cluster. If connecting to a different cluster, you will need the monitor information specific to that cluster.
      • You can find the monitor information in the share details for your share in the Path field.
  4. Retrieve the connection information from the share page for your connection:

    • Open up the share details by clicking the name of the share in the Shares page.
    • Copy the entire path of the share for mounting the filesystem.
  5. Mount the filesystem

    • Create a mount point directory somewhere in your host (likely under /mnt/ - e.g. /mnt/ShareName)
    • Via kernel mount using the ceph driver:
      • Syntax: sudo mount -t ceph <path information> <mountPoint> -o name=<shareKeyName>, secretfile=</path/to/keyringfileOnlyFile>
      • sudo mount -t ceph mon1:6789,mon2:6789,mon3:6789:/volumes/_nogroup/share_instance_id
        • e.g sudo mount -t ceph,, /mnt/WebServerShare -o name=def-project-shareName-read-write,secretfile=/etc/ceph/client.keyonly.def-project-sharename-read-write
    • Via ceph-fuse
      • Need to install ceph-fuse
      • Syntax: sudo ceph-fuse <mountPoint> --id=<shareKeyName> --conf=<pathtoCeph.conf> --keyring=<fullKeyringLocation> --client-mountpoint=pathFromShareDetails
        • e.g. sudo ceph-fuse /mnt/WebServerShare --id=def-project-shareName-read-write --conf=/etc/ceph/ceph.conf --keyring=/etc/ceph/client.fullkey.def-project-shareName-read-write --client-mountpoint=/volumes/_nogroup/a87b5ef3-b266-4664-a5ed-026cddfdcdb7


  • A particular share can have more than one user key provisioned for it.
    • This allows a more granular access to the filesystem, for example if you needed some hosts to only access the filesystem in a read-only capacity.
    • If you have multiple keys for a share, you can add the extra keys to your host and modify the above mounting procedure.
  • This service is not available to hosts outside of the OpenStack cluster.