Storage and file management

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Overview

Compute Canada provides a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. In most cases the filesystems on Compute Canada systems are a shared resource and for this reason should be used responsibly - unwise behaviour can negatively affect dozens or hundreds of other users. These filesystems are also designed to store a limited number of very large files, typically binary rather than text files, i.e. they are not directly human-readable. You should therefore avoid storing thousands of small files, where small means less than a few megabytes, particularly in the same directory. A better approach is to use commands like tar or zip to convert a directory containing many small files into a single very large archive file.

It is also your responsibility to manage the age of your stored data: most of the filesystems are not intended to provide an indefinite archiving service so when a given file or directory is no longer needed, you need to move it to a more appropriate filesystem which may well mean your personal workstation or some other storage system under your control. Moving significant amounts of data between your workstation and a Compute Canada system or between two Compute Canada systems should generally be done using Globus.

Note that Compute Canada storage systems are not for personal use and should only be used to store research data.

Storage Types

Unlike your personal computer, a Compute Canada system will typically have several filesystems and you should ensure that you are using the right filesystem for the right task. In this section we will discuss the principal filesystems available on most Compute Canada systems and the intended use of each one along with its characteristics. Storage options are distinguished by the available hardware, access mode and write system. Typically, most Compute Canada systems offer the following storage types:

Network Filesystem (NFS)

This type of storage is generally equally visible on both login and compute nodes. This is the appropriate place to put small but important files that are regularly used: source code, programs, job scripts and parameter files. This type of storage offers performance comparable to a conventional hard disk.

Parallel Filesystem (Lustre, GPFS)

This type of storage is generally equally visible on both login and compute nodes. Combining multiple disk arrays and fast servers, it offers excellent performance for large files and large input/output operations. Often two types of storage are distinguished on such systems: long term storage and temporary storage (scratch). Performance is subject to variations caused by other users.

Local Filesystem

This type of storage consists of a local hard drive attached to each compute node. Its advantage is that its performance is high because it is very rarely shared --- typically, only one user will access a local drive at a time. However, you must copy your files back to another storage medium like $SCRATCH or $PROJECT before your job ends because everything will be cleaned after each job.

RAM (memory) file system

This is a file system that exists within a compute node's RAM, so its use reduces available memory for computations. Such file systems are very fast for small files and particularly faster than other systems when file access is random. A RAM disk is always cleaned at the end of a job.

The following table summarizes the properties of these storage types.

Description of storage type

Storage Type	Name	Accessibility	Throughput (large operations, > 1 MB per operation)	Latency (small operations)	Longevity
Network Filesystem (NFS)	$HOME	All nodes	100 MB/s shared	High	Long term
Long-Term Parallel Filesystem	$PROJECT	All nodes	1-10 GB/s shared	High	Long term
Short-Term Parallel Filesystem	$SCRATCH	All nodes	1-10 GB/s shared	High	Short term (periodically cleaned)
Local Filesystem	$LSCRATCH	Local to the node	100 MB/s	Medium	Very short term
Memory (RAM) Filesystem	$RAMDISK, /dev/shm	Local to the node	1-10 GB/s	Very low	Very short term, cleaned after every job

Best practices

• Only use text format for files that are smaller than a few megabytes.
• As far as possible, use local storage for temporary files.
• If your program must search within a file, it is fastest to do it by first reading it completely before searching, or to use a RAM disk.
• Regularly clean up your data in the $SCRATCH and $PROJECT filesystems, because those systems are used for huge data collections.
• If you no longer use certain files but they must be retained, archive and compress them, and if possible copy them elsewhere.
• If your needs are not well served by the available storage options please contact us by sending an e-mail to Compute Canada support.

Filesystem Quotas and Policies

In order to ensure that there is adequate space for all Compute Canada users, there are a variety of quotas and policy restrictions concerning back-ups and automatic purging of certain filesystems. Every user has access to the $HOME and $SCRATCH filesystems by default as well as a certain amount of space on $PROJECT. To have access to the full 10 TB quota on $PROJECT users must submit a request while $NEARLINE is allocated using the annual RAC (resource allocation) process.

Filesystem Characteristics

Filesystem	Quotas	Backed up?	Purged?	Available by Default?	Mounted on Compute Nodes?
$HOME	50 GB, 500K files	Yes	No	Yes	Yes
$SCRATCH	20 TB and 1000K files per user, 100 TB and 10M files per group	No	Yes, all files older than 90 days	Yes	Yes
$PROJECT	10 TB and 5M files per group, 500K files per user	Yes	No	Yes	Yes
$NEARLINE	5 TB per group	No	No	No	No

See also

• Sharing data
• Tuning Lustre
• Archiving and compressing files

Resource Allocation Competition