Storage and file management: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
No edit summary
Tag: Manual revert
 
(91 intermediate revisions by 17 users not shown)
Line 4: Line 4:


<!--T:2-->
<!--T:2-->
Compute Canada provides a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. In most cases the [https://en.wikipedia.org/wiki/File_system filesystems] on Compute Canada systems are a ''shared'' resource and for this reason should be used responsibly - unwise behaviour can negatively affect dozens or hundreds of other users. These filesystems are also designed to store a limited number of very large files, typically binary rather than text files, i.e. they are not directly human-readable. You should therefore avoid storing thousands of small files, where small means less than a few megabytes, particularly in the same directory. A better approach is to use commands like [[Archiving and compressing files|<tt>tar</tt>]] or <tt>zip</tt> to convert a directory containing many small files into a single very large archive file.  
We provide a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. In most cases the [https://en.wikipedia.org/wiki/File_system filesystems] on our systems are a <i>shared</i> resource and for this reason should be used responsibly because unwise behaviour can negatively affect dozens or hundreds of other users. These filesystems are also designed to store a limited number of very large files, which are typically binary since very large (hundreds of MB or more) text files lose most of their interest in being readable by humans. You should therefore avoid storing tens of thousands of small files, where small means less than a few megabytes, particularly in the same directory. A better approach is to use commands like [[Archiving and compressing files|<code>tar</code>]] or <code>zip</code> to convert a directory containing many small files into a single very large archive file.  


<!--T:3-->
<!--T:3-->
It is also your responsibility to manage the age of your stored data: most of the filesystems are not intended to provide an indefinite archiving service so when a given file or directory is no longer needed, you need to move it to a more appropriate filesystem which may well mean your personal workstation or some other storage system under your control. Moving significant amounts of data between your workstation and a Compute Canada system or between two Compute Canada systems should generally be done using [[Globus]].  
It is also your responsibility to manage the age of your stored data: most of the filesystems are not intended to provide an indefinite archiving service so when a given file or directory is no longer needed, you need to move it to a more appropriate filesystem which may well mean your personal workstation or some other storage system under your control. Moving significant amounts of data between your workstation and one of our systems or between two of our systems should generally be done using [[Globus]].  


<!--T:4-->
<!--T:4-->
Note that Compute Canada storage systems are not for personal use and should only be used to store research data.
Note that our storage systems are not for personal use and should only be used to store research data.


<!--T:17-->
<!--T:17-->
When your account is created on Cedar and Graham, your home directory will not be entirely empty. It will contain references to your scratch and [[Project layout|project]] spaces through the mechanism of a [https://en.wikipedia.org/wiki/Symbolic_link symbolic link], a kind of shortcut that allows easy access to these other filesystems from your home directory. Note that these symbolic links may appear up to a few hours after you first connect to the cluster. While your home and scratch spaces are unique to you as an individual user, the project space is a shared by a research group. This group may consist of those individuals with a Compute Canada account sponsored by a particular faculty member or members of a [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/ RAC allocation]. A given individual may thus have access to several different project spaces, associated with one or more faculty members, with symbolic links to these different project spaces in the directory projects of your home. Every account has one or many projects. In the folder <tt>projects</tt> within their home directory, each user has a link to each of the projects they have access to. For users with a single active sponsored role is the default project of your sponsor while users with more than one active sponsored role will have a default project that corresponds to the default project of the faculty member with the most sponsored accounts.
When your account is created on a cluster, your home directory will not be entirely empty. It will contain references to your scratch and [[Project layout|project]] spaces through the mechanism of a [https://en.wikipedia.org/wiki/Symbolic_link symbolic link], a kind of shortcut that allows easy access to these other filesystems from your home directory. Note that these symbolic links may appear up to a few hours after you first connect to the cluster. While your home and scratch spaces are unique to you as an individual user, the project space is shared by a research group. This group may consist of those individuals with an account sponsored by a particular faculty member or members of an [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition/ RAC allocation]. A given individual may thus have access to several different project spaces, associated with one or more faculty members, with symbolic links to these different project spaces in the directory projects of your home. Every account has one or many projects. In the folder <code>projects</code> within their home directory, each user has a link to each of the projects they have access to. For users with a single active sponsored role, it is the default project of your sponsor while users with more than one active sponsored role will have a default project that corresponds to the default project of the faculty member with the most sponsored accounts.


<!--T:16-->
<!--T:16-->
All users can check the available disk space and the current disk utilization for the ''project'', ''home'' and ''scratch'' file systems with the command line utility '''''diskusage_report''''', available on both '''Cedar''' and '''Graham'''. To use this utility, log into Cedar or Graham using SSH, at the command prompt type diskusage_report, and press the Enter key. Following is a typical output of this utility:
All users can check the available disk space and the current disk utilization for the <i>project</i>, <i>home</i> and <i>scratch</i> filesystems with the command line utility <b><i>diskusage_report</b></i>, available on our clusters. To use this utility, log into the cluster using SSH, at the command prompt type <i>diskusage_report</i>, and press the Enter key. Below is a typical output of this utility:
<pre>
<pre>
# diskusage_report
# diskusage_report
Line 24: Line 24:
       Project (def-username-ab)      4096 B/9536 GB              2/500k
       Project (def-username-ab)      4096 B/9536 GB              2/500k
           Project (def-username)      4096 B/9536 GB              2/500k
           Project (def-username)      4096 B/9536 GB              2/500k
</pre>
</pre>
More detailed output is available using the [[Diskusage Explorer]] tool.


== Storage Types == <!--T:5-->
== Storage types == <!--T:5-->
Unlike your personal computer, a Compute Canada system will typically have several storage spaces or filesystems and you should ensure that you are using the right space for the right task. In this section we will discuss the principal filesystems available on most Compute Canada systems and the intended use of each one along with its characteristics. Storage options are distinguished by the available hardware, access mode and write system. Typically, most Compute Canada systems offer the following storage types:
Unlike your personal computer, our systems will typically have several storage spaces or filesystems and you should ensure that you are using the right space for the right task. In this section we will discuss the principal filesystems available on most of our systems and the intended use of each one along with some of its characteristics.  
* <b>HOME:</b> While your home directory may seem like the logical place to store all your files and do all your work, in general this isn't the case; your home normally has a relatively small quota and doesn't have especially good performance for writing and reading large amounts of data. The most logical use of your home directory is typically source code, small parameter files and job submission scripts.
* <b>PROJECT:</b> The project space has a significantly larger quota and is well adapted to [[Sharing data | sharing data]] among members of a research group since it, unlike the home or scratch, is linked to a professor's account rather than an individual user. The data stored in the project space should be fairly static, that is to say the data are not likely to be changed many times in a month. Otherwise, frequently changing data, including just moving and renaming directories, in project can become a heavy burden on the tape-based backup system.  
* <b>SCRATCH</b>: For intensive read/write operations on large files (> 100 MB per file), scratch is the best choice. However, remember that important files must be copied off scratch since they are not backed up there, and older files are subject to [[Scratch purging policy|purging]]. The scratch storage should therefore be used for temporary files: checkpoint files, output from jobs and other data that can easily be recreated. <b>Do not regard SCRATCH as your normal storage!  It is for transient files that you can afford to lose.</b>
* <b>SLURM_TMPDIR</b>: While a job is running, the environment variable <code>$SLURM_TMPDIR</code> holds a unique path to a temporary folder on a fast, local filesystem on each compute node allocated to the job. When the job ends, the directory and its contents are deleted, so <code>$SLURM_TMPDIR</code> should be used for temporary files that are only needed for the duration of the job. Its advantage, compared to the other networked filesystem types above, is increased performance due to the filesystem being local to the compute node. It is especially well-suited for large collections of small files (for example, smaller than a few megabytes per file). Note that this filesystem is shared between all jobs running on the node, and that the available space depends on the compute node type. A more detailed discussion of using <code>$SLURM_TMPDIR</code> is available at [[Using_$SLURM_TMPDIR | this page]].


<!--T:6-->
==Project space consumption per user== <!--T:23-->                                                          
;Network Filesystem (NFS)
: This type of storage is generally equally visible on both login and compute nodes. This is the appropriate place to put small but important files that are regularly used: source code, programs, job scripts and parameter files. This type of storage offers performance comparable to a conventional hard disk.
;Parallel Filesystem (Lustre, GPFS)
: This type of storage is generally equally visible on both login and compute nodes. Combining multiple disk arrays and fast servers, it offers excellent performance for large files and large input/output operations. Often two types of storage are distinguished on such systems: long term storage and temporary storage (scratch). Performance is subject to variations caused by other users.
;Local Filesystem
: This type of storage consists of a local hard drive attached to each compute node. Its advantage is that its performance is high because it is very rarely shared --- typically, only one user will access a local drive at a time. However, you must copy your files back to another storage medium like the scratch space or project space before your job ends because everything will be cleaned after each job.
;RAM (memory) Filesystem
: This is a filesystem that exists within a compute node's RAM, so its use reduces available memory for computations. Such filesystems are very fast for small files and particularly faster than other systems when file access is random. A RAM disk is always cleaned at the end of a job.


<!--T:7-->
<!--T:24-->
The following table summarizes the properties of these storage types.
While the command <b>diskusage_report</b> gives the space and file count usage per user on <i>home</i> and <i>scratch</i>, it shows the total quota of the group on project. It includes all the files from each member of the group. Since the files that belong to a user could however be anywhere in the project space, it is difficult to obtain correct figures per user and per given project in case a user has access to more than one project. However, users can obtain an estimate of their space and file count use on the entire project space by running the command


<!--T:8-->
<!--T:26-->
{| class="wikitable" style="font-size: 95%; text-align: center;"
<code>lfs quota -u $USER /project</code>
|+align="bottom" style="color:#e76700;"|''Description of storage type''
! scope="col" width="120px" | Type
! scope="col" width="120px" | Accessibility
! scope="col" width="120px" | Throughput
! scope="col" width="120px" | Latency
! scope="col" width="120px" | Longevity
|-
|Network Filesystem (NFS)
|style="background-color:#00c000;"|All nodes
|style="background-color:#ff0000;"|Poor
|style="background-color:#ff0000;"|High
|style="background-color:#00c000;"|Long term
|-
|Long-Term Parallel Filesystem
|style="background-color:#00c000;"|All nodes
|style="background-color:#ffff00;"|Fair
|style="background-color:#ff0000;"|High
|style="background-color:#00c000;"|Long term
|-
|Short-Term Parallel Filesystem
|style="background-color:#00c000;"|All nodes
|style="background-color:#ffff00;"|Fair
|style="background-color:#ff0000;"|High
|style="background-color:#ffff00;"|Short term (periodically cleaned)
|-
|Local Filesystem
|style="background-color:#ff0000;"|Local to the node
|style="background-color:#ffff00;"|Fair
|style="background-color:#ffff00;"|Medium
|style="background-color:#ff0000;"|Very short term
|-
|Memory (RAM) Filesystem
|style="background-color:#ff0000;"|Local to the node
|style="background-color:#00c000;"|Good
|style="background-color:#00c000;"|Very low
|style="background-color:#ff0000;"|Very short term, cleaned after every job
|}
'''Throughput''' describes the efficiency of the file system for large operations, such as those involving a megabyte or more per read or write.


<!--T:15-->
<!--T:25-->
'''Latency''' describes the efficiency of the file system for multiple small operations. Low latency is good; however, if one has a choice between a small number of large operations and a large number of small ones, it is almost always better to use a small number of large operations.
In addition to that, users can obtain an estimate for the number of files in a given directory (and its subdirectories) using the command <code>lfs find</code>, e.g.
<source lang="console">
lfs find <path to the directory> -type f | wc -l
</source>


== Best practices == <!--T:9-->
== Best practices == <!--T:9-->
* Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
* Only use text format for files that are smaller than a few megabytes.
* Only use text format for files that are smaller than a few megabytes.
* As far as possible, use local storage for temporary files. It is best to use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>.
* As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>.
* If your program must search within a file, it is fastest to do it by first reading it completely before searching, or to use a RAM disk.
* If your program must search within a file, it is fastest to do it by first reading it completely before searching.
* Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible move them to an alternative location like [[Using nearline storage|nearline]].
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible copy them elsewhere.
* For more on managing many files, see [[Handling large collections of files]], especially if you are limited by a quota on the number of files.
* Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO].  
* If your needs are not well served by the available storage options please contact [[technical support]].
* If your needs are not well served by the available storage options please contact [[technical support]].


==Filesystem Quotas and Policies== <!--T:10-->
==Filesystem quotas and policies== <!--T:10-->


<!--T:11-->
<!--T:11-->
In order to ensure that there is adequate space for all Compute Canada users, there are a variety of quotas and policy restrictions concerning back-ups and automatic purging of certain filesystems.  
In order to ensure that there is adequate space for all users, there are a variety of quotas and policy restrictions concerning backups and automatic purging of certain filesystems.  
By default on our clusters each user has access to the home and scratch spaces, and each group has access to 1 TB of project space. The nearline filesystem is made up of medium to low performance storage in very high capacity. This filesystem should be used for storage of data that is infrequently accessed that needs to be kept for long periods of time. Small increases in project and scratch spaces are available through our Rapid Access Service ([https://www.computecanada.ca/research-portal/accessing-resources/rapid-access-service/ RAS]). Larger increases in project spaces are available through the annual Resource Allocation Competitions ([https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions RAC]). You can see your current quota usage for various filesystems on Cedar and Graham using the command [[Storage and file management#Overview|<tt>diskusage_report</tt>]].
By default on our clusters, each user has access to the home and scratch spaces, and each group has access to 1 TB of project space. Small increases in project and scratch spaces are available through our Rapid Access Service ([https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service RAS]). Larger increases in project spaces are available through the annual Resource Allocation Competition ([https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition RAC]). You can see your current quota usage for various filesystems on Cedar and Graham using the command [[Storage and file management#Overview|<code>diskusage_report</code>]].


<!--T:12-->
<!--T:12-->
<tabs>
<tab name="Cedar">
{| class="wikitable" style="font-size: 95%; text-align: center;"
|+Filesystem Characteristics
! Filesystem
! Default Quota
! Lustre-based
! Backed up
! Purged
! Available by Default
! Mounted on Compute Nodes
|-
|Home Space
|50 GB and 500K files per user<ref>This quota is fixed and cannot be changed.</ref>
|Yes
|Yes
|No
|Yes
|Yes
|-
|Scratch Space
|20 TB and 1M files per user
|Yes
|No
|Files older than 60 days are purged.<ref>See [[Scratch purging policy]] for more information.</ref>
|Yes
|Yes
|-
|Project Space
|1 TB and 500K files per group<ref>Project space can be increased to 40 TB per group by a RAS request, subject to the limitations that the minimum project space per quota cannot be less than 1 TB and the sum over all four general-purpose clusters cannot exceed 43 TB. The group's sponsoring PI should write to [[technical support]] to make the request.</ref>
|Yes
|Yes
|No
|Yes
|Yes
|-
|Nearline Space
|2 TB and 5000 files per group
|Yes
|Yes
|No
|Yes
|No
|}
<references />
Starting April 1, 2024, new Rapid Access Service (RAS) policies will allow larger quotas for the project and nearline spaces. For more details, see the "Storage" section at [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service Rapid Access Service].  Quota changes larger than those permitted by RAS will require an application to the annual [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition Resource Allocation Competition (RAC)].
</tab>
<tab name="Graham">
{| class="wikitable" style="font-size: 95%; text-align: center;"
|+Filesystem Characteristics
! Filesystem
! Default Quota
! Lustre-based
! Backed up
! Purged
! Available by Default
! Mounted on Compute Nodes
|-
|Home Space
|50 GB and 500K files per user<ref>This quota is fixed and cannot be changed.</ref>
|No
|Yes
|No
|Yes
|Yes
|-
|Scratch Space
|20 TB and 1M files per user
|Yes
|No
|Files older than 60 days are purged.<ref>See [[Scratch purging policy]] for more information.</ref>
|Yes
|Yes
|-
|Project Space
|1 TB and 500K files per group<ref>Project space can be increased to 40 TB per group by a RAS request. The group's sponsoring PI should write to [[technical support]] to make the request.</ref>
|Yes
|Yes
|No
|Yes
|Yes
|-
|Nearline Space
|10 TB and 5000 files per group
|Yes
|Yes
|No
|Yes
|No
|}
<references />
Starting April 1, 2024, new Rapid Access Service (RAS) policies will allow larger quotas for project and nearline spaces. For more details, see the "Storage" section at [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service Rapid Access Service].  Quota changes larger than those permitted by RAS will require an application to the annual [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition Resource Allocation Competition (RAC)]. 
</tab>
<tab name="Béluga and Narval">
{| class="wikitable" style="font-size: 95%; text-align: center;"
{| class="wikitable" style="font-size: 95%; text-align: center;"
|+Filesystem Characteristics  
|+Filesystem Characteristics  
! Filesystem
! Filesystem
! Default Quota
! Default Quota
! Lustre-based?
! Lustre-based
! Backed up?
! Backed up
! Purged?
! Purged
! Available by Default?
! Available by Default
! Mounted on Compute Nodes?
! Mounted on Compute Nodes
|-
|-
|Home Space
|Home Space
|50 GB and 500K files per user<ref>This quota is fixed and cannot be changed.</ref>
|50 GB and 500K files per user<ref>This quota is fixed and cannot be changed.</ref>
|Yes for Cedar, No for Graham (NFS)
|Yes
|Yes
|Yes
|No
|No
Line 120: Line 178:
|-
|-
|Scratch Space
|Scratch Space
|20 TB and 1M files per user<ref>Currently scratch space cannot be increased, and an increased quota of 100 TB per user is temporarily applied on Graham. We plan to move it back down to 20TB when a technical solution has been deployed.</ref>
|20 TB and 1M files per user
|Yes
|Yes
|No
|No
Line 128: Line 186:
|-
|-
|Project Space
|Project Space
|1 TB and 500k files per group<ref>Project space can be increased to 10 TB per group by a RAS request. The group's sponsoring PI should write to [[technical support]] to make the request.</ref>
|1 TB and 500K files per group<ref>Project space can be increased to 40 TB per group by a RAS request. The group's sponsoring PI should write to [[technical support]] to make the request.</ref>
|Yes
|Yes
|Yes
|Yes
Line 134: Line 192:
|Yes
|Yes
|Yes
|Yes
 
|-
|Nearline Space
|1 TB and 5000 files per group
|Yes
|Yes
|No
|Yes
|No
|}
|}
The backup policy on the home and project space is nightly backups which are retained for 30 days, while deleted files are retained for a further 60 days - note that is entirely distinct from the age limit for purging files from the scratch space. If you wish to recover a previous version of a file or directory, you should contact [[technical support]] with the full path for the file(s) and desired version (by date).
<references />
<references />
Starting April 1, 2024, new Rapid Access Service (RAS) policies will allow larger quotas for project and nearline spaces. For more details, see the "Storage" section at [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service Rapid Access Service].  Quota changes larger than those permitted by RAS will require an application to the annual [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition Resource Allocation Competition (RAC)]. 
</tab>
<tab name="Niagara">
{| class="wikitable"
! location
!colspan="2"| quota
!align="right"| block size
! expiration time
! backed up
! on login nodes
! on compute nodes
|-
| $HOME
|colspan="2"| 100 GB per user
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
|rowspan="6"| $SCRATCH
|colspan="2"| 25 TB per user (dynamic per group)
|align="right" rowspan="6" | 16 MB
|rowspan="6"| 2 months
|rowspan="6"| no
|rowspan="6"| yes
|rowspan="6"| yes
|-
|align="right"|up to 4 users per group
|align="right"|50TB
|-
|align="right"|up to 11 users per group
|align="right"|125TB
|-
|align="right"|up to 28 users per group
|align="right"|250TB
|-
|align="right"|up to 60 users per group
|align="right"|400TB
|-
|align="right"|above 60 users per group
|align="right"|500TB
|-
| $PROJECT
|colspan="2"| by group allocation (RRG or RPP)
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
|colspan="2"| by group allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
|colspan="2"| 10 TB per user
|align="right"| 1 MB
| very short
| no
| yes
| yes
|}
<ul>
<li>[https://docs.scinet.utoronto.ca/images/9/9a/Inode_vs._Space_quota_-_v2x.pdf Inode vs. Space quota (PROJECT and SCRATCH)]</li>
<li>[https://docs.scinet.utoronto.ca/images/0/0e/Scratch-quota.pdf dynamic quota per group (SCRATCH)]</li>
<li>Compute nodes do not have local storage.</li>
<li>Archive (a.k.a. nearline) space is on [https://docs.scinet.utoronto.ca/index.php/HPSS HPSS]</li>
<li>Backup means a recent snapshot, not an archive of all data that ever was.</li>
<li><code>$BBUFFER</code> stands for [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer], a faster parallel storage tier for temporary data.</li></ul>
<!--T:21-->
</tab>
</tabs>
<!--T:22-->
The backup policy on the home and project space is nightly backups which are retained for 30 days, while deleted files are retained for a further 60 days; note that is entirely distinct from the age limit for purging files from the scratch space. If you wish to recover a previous version of a file or directory, you should contact [[technical support]] with the full path for the file(s) and desired version (by date).


== See also == <!--T:13-->
== See also == <!--T:13-->


<!--T:14-->
<!--T:14-->
* [[Diskusage Explorer]]
* [[Project layout]]
* [[Project layout]]
* [[Sharing data]]
* [[Sharing data]]
* [[National Data Cyberinfrastructure]]
* [[Transferring data]]
* [[Tuning Lustre]]
* [[Tuning Lustre]]
* [[Archiving and compressing files]]
* [[Archiving and compressing files]]
* [[Handling large collections of files]]
* [[Parallel I/O introductory tutorial]]
</translate>
</translate>

Latest revision as of 14:46, 24 October 2024

Other languages:

Overview[edit]

We provide a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. In most cases the filesystems on our systems are a shared resource and for this reason should be used responsibly because unwise behaviour can negatively affect dozens or hundreds of other users. These filesystems are also designed to store a limited number of very large files, which are typically binary since very large (hundreds of MB or more) text files lose most of their interest in being readable by humans. You should therefore avoid storing tens of thousands of small files, where small means less than a few megabytes, particularly in the same directory. A better approach is to use commands like tar or zip to convert a directory containing many small files into a single very large archive file.

It is also your responsibility to manage the age of your stored data: most of the filesystems are not intended to provide an indefinite archiving service so when a given file or directory is no longer needed, you need to move it to a more appropriate filesystem which may well mean your personal workstation or some other storage system under your control. Moving significant amounts of data between your workstation and one of our systems or between two of our systems should generally be done using Globus.

Note that our storage systems are not for personal use and should only be used to store research data.

When your account is created on a cluster, your home directory will not be entirely empty. It will contain references to your scratch and project spaces through the mechanism of a symbolic link, a kind of shortcut that allows easy access to these other filesystems from your home directory. Note that these symbolic links may appear up to a few hours after you first connect to the cluster. While your home and scratch spaces are unique to you as an individual user, the project space is shared by a research group. This group may consist of those individuals with an account sponsored by a particular faculty member or members of an RAC allocation. A given individual may thus have access to several different project spaces, associated with one or more faculty members, with symbolic links to these different project spaces in the directory projects of your home. Every account has one or many projects. In the folder projects within their home directory, each user has a link to each of the projects they have access to. For users with a single active sponsored role, it is the default project of your sponsor while users with more than one active sponsored role will have a default project that corresponds to the default project of the faculty member with the most sponsored accounts.

All users can check the available disk space and the current disk utilization for the project, home and scratch filesystems with the command line utility diskusage_report, available on our clusters. To use this utility, log into the cluster using SSH, at the command prompt type diskusage_report, and press the Enter key. Below is a typical output of this utility: 

# diskusage_report
                   Description                Space           # of files
                 Home (username)         280 kB/47 GB              25/500k
              Scratch (username)         4096 B/18 TB              1/1000k
       Project (def-username-ab)       4096 B/9536 GB              2/500k
          Project (def-username)       4096 B/9536 GB              2/500k

More detailed output is available using the Diskusage Explorer tool.

Storage types[edit]

Unlike your personal computer, our systems will typically have several storage spaces or filesystems and you should ensure that you are using the right space for the right task. In this section we will discuss the principal filesystems available on most of our systems and the intended use of each one along with some of its characteristics.

  • HOME: While your home directory may seem like the logical place to store all your files and do all your work, in general this isn't the case; your home normally has a relatively small quota and doesn't have especially good performance for writing and reading large amounts of data. The most logical use of your home directory is typically source code, small parameter files and job submission scripts.
  • PROJECT: The project space has a significantly larger quota and is well adapted to sharing data among members of a research group since it, unlike the home or scratch, is linked to a professor's account rather than an individual user. The data stored in the project space should be fairly static, that is to say the data are not likely to be changed many times in a month. Otherwise, frequently changing data, including just moving and renaming directories, in project can become a heavy burden on the tape-based backup system.
  • SCRATCH: For intensive read/write operations on large files (> 100 MB per file), scratch is the best choice. However, remember that important files must be copied off scratch since they are not backed up there, and older files are subject to purging. The scratch storage should therefore be used for temporary files: checkpoint files, output from jobs and other data that can easily be recreated. Do not regard SCRATCH as your normal storage! It is for transient files that you can afford to lose.
  • SLURM_TMPDIR: While a job is running, the environment variable $SLURM_TMPDIR holds a unique path to a temporary folder on a fast, local filesystem on each compute node allocated to the job. When the job ends, the directory and its contents are deleted, so $SLURM_TMPDIR should be used for temporary files that are only needed for the duration of the job. Its advantage, compared to the other networked filesystem types above, is increased performance due to the filesystem being local to the compute node. It is especially well-suited for large collections of small files (for example, smaller than a few megabytes per file). Note that this filesystem is shared between all jobs running on the node, and that the available space depends on the compute node type. A more detailed discussion of using $SLURM_TMPDIR is available at this page.

Project space consumption per user[edit]

While the command diskusage_report gives the space and file count usage per user on home and scratch, it shows the total quota of the group on project. It includes all the files from each member of the group. Since the files that belong to a user could however be anywhere in the project space, it is difficult to obtain correct figures per user and per given project in case a user has access to more than one project. However, users can obtain an estimate of their space and file count use on the entire project space by running the command

lfs quota -u $USER /project

In addition to that, users can obtain an estimate for the number of files in a given directory (and its subdirectories) using the command lfs find, e.g. 

lfs find <path to the directory> -type f | wc -l

Best practices[edit]

  • Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
  • Only use text format for files that are smaller than a few megabytes.
  • As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the job scheduler for this, named $SLURM_TMPDIR.
  • If your program must search within a file, it is fastest to do it by first reading it completely before searching.
  • If you no longer use certain files but they must be retained, archive and compress them, and if possible move them to an alternative location like nearline.
  • For more on managing many files, see Handling large collections of files, especially if you are limited by a quota on the number of files.
  • Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as MPI-IO.
  • If your needs are not well served by the available storage options please contact technical support.

Filesystem quotas and policies[edit]

In order to ensure that there is adequate space for all users, there are a variety of quotas and policy restrictions concerning backups and automatic purging of certain filesystems. By default on our clusters, each user has access to the home and scratch spaces, and each group has access to 1 TB of project space. Small increases in project and scratch spaces are available through our Rapid Access Service (RAS). Larger increases in project spaces are available through the annual Resource Allocation Competition (RAC). You can see your current quota usage for various filesystems on Cedar and Graham using the command diskusage_report.

Filesystem Characteristics
Filesystem Default Quota Lustre-based Backed up Purged Available by Default Mounted on Compute Nodes
Home Space 50 GB and 500K files per user[1] Yes Yes No Yes Yes
Scratch Space 20 TB and 1M files per user Yes No Files older than 60 days are purged.[2] Yes Yes
Project Space 1 TB and 500K files per group[3] Yes Yes No Yes Yes
Nearline Space 2 TB and 5000 files per group Yes Yes No Yes No
  1. ↑ This quota is fixed and cannot be changed.
  2. ↑ See Scratch purging policy for more information.
  3. ↑ Project space can be increased to 40 TB per group by a RAS request, subject to the limitations that the minimum project space per quota cannot be less than 1 TB and the sum over all four general-purpose clusters cannot exceed 43 TB. The group's sponsoring PI should write to technical support to make the request.

Starting April 1, 2024, new Rapid Access Service (RAS) policies will allow larger quotas for the project and nearline spaces. For more details, see the "Storage" section at Rapid Access Service. Quota changes larger than those permitted by RAS will require an application to the annual Resource Allocation Competition (RAC).

Filesystem Characteristics
Filesystem Default Quota Lustre-based Backed up Purged Available by Default Mounted on Compute Nodes
Home Space 50 GB and 500K files per user[1] No Yes No Yes Yes
Scratch Space 20 TB and 1M files per user Yes No Files older than 60 days are purged.[2] Yes Yes
Project Space 1 TB and 500K files per group[3] Yes Yes No Yes Yes
Nearline Space 10 TB and 5000 files per group Yes Yes No Yes No
  1. ↑ This quota is fixed and cannot be changed.
  2. ↑ See Scratch purging policy for more information.
  3. ↑ Project space can be increased to 40 TB per group by a RAS request. The group's sponsoring PI should write to technical support to make the request.

Starting April 1, 2024, new Rapid Access Service (RAS) policies will allow larger quotas for project and nearline spaces. For more details, see the "Storage" section at Rapid Access Service. Quota changes larger than those permitted by RAS will require an application to the annual Resource Allocation Competition (RAC).

Filesystem Characteristics
Filesystem Default Quota Lustre-based Backed up Purged Available by Default Mounted on Compute Nodes
Home Space 50 GB and 500K files per user[1] Yes Yes No Yes Yes
Scratch Space 20 TB and 1M files per user Yes No Files older than 60 days are purged.[2] Yes Yes
Project Space 1 TB and 500K files per group[3] Yes Yes No Yes Yes
Nearline Space 1 TB and 5000 files per group Yes Yes No Yes No
  1. ↑ This quota is fixed and cannot be changed.
  2. ↑ See Scratch purging policy for more information.
  3. ↑ Project space can be increased to 40 TB per group by a RAS request. The group's sponsoring PI should write to technical support to make the request.

Starting April 1, 2024, new Rapid Access Service (RAS) policies will allow larger quotas for project and nearline spaces. For more details, see the "Storage" section at Rapid Access Service. Quota changes larger than those permitted by RAS will require an application to the annual Resource Allocation Competition (RAC).

location quota block size expiration time backed up on login nodes on compute nodes
$HOME 100 GB per user 1 MB yes yes read-only
$SCRATCH 25 TB per user (dynamic per group) 16 MB 2 months no yes yes
up to 4 users per group 50TB
up to 11 users per group 125TB
up to 28 users per group 250TB
up to 60 users per group 400TB
above 60 users per group 500TB
$PROJECT by group allocation (RRG or RPP) 16 MB yes yes yes
$ARCHIVE by group allocation dual-copy no no
$BBUFFER 10 TB per user 1 MB very short no yes yes

The backup policy on the home and project space is nightly backups which are retained for 30 days, while deleted files are retained for a further 60 days; note that is entirely distinct from the age limit for purging files from the scratch space. If you wish to recover a previous version of a file or directory, you should contact technical support with the full path for the file(s) and desired version (by date).

See also[edit]

Rapid Access Service

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Storage_and_file_management&oldid=163099"