Storage and file management
This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.
Overview
Compute Canada provides a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. In most cases the filesystems on Compute Canada systems are a shared resource and for this reason should be used responsibly - unwise behaviour can negatively affect dozens or hundreds of other users. These filesystems are also designed to store a limited number of very large files, typically binary rather than text files, i.e. they are not directly human-readable. You should therefore avoid storing thousands of small files, where small means less than a few megabytes, particularly in the same directory. A better approach is to use commands like tar or zip to convert a directory containing many small files into a single very large archive file.
It is also your responsibility to manage the age of your stored data: most of the filesystems are not intended to provide an indefinite archiving service so when a given file or directory is no longer needed, you need to move it to a more appropriate filesystem which may well mean your personal workstation or some other storage system under your control. Moving significant amounts of data between your workstation and a Compute Canada system or between two Compute Canada systems should generally be done using Globus.
Note that Compute Canada storage systems are not for personal use and should only be used to store research data.
Best practices
- Only use text format for files that are smaller than a few megabytes.
- As far as possible, use local storage for temporary files.
- If your program must search within a file, it is fastest to do it by first reading it completely before searching, or to use a RAM disk.
- Regularly clean up your data in the $SCRATCH and $PROJECT filesystems, because those systems are used for huge data collections.
- If you no longer use certain files, compress them (you should group them before) and back them up (if possible).
- If your needs are not well served by the available storage options please contact Compute Canada by sending an e-mail to support@computecanada.ca.
Filesystem Layout
Unlike your personal computer, a Compute Canada system will typically have several filesystems and you should ensure that you are using the right filesystem for the right task. In this section we will discuss the principal filesystems available on most Compute Canada systems and the intended use of each one along with its characteristics and restrictions.
When you login to a Compute Canada system you will normally begin in your home directory ($HOME) and which is intended to store relatively small text files such as job submission scripts, parameter files and source code. The scratch filesystem ($SCRATCH) is as the name suggests intended for temporary storage of intermediate results, checkpoint files and so on. The other filesystems available on Compute Canada servers are allocated through the annual resource competition.
| Filesystem | Quotas | Backed up? | Purged? | Technology |
|---|---|---|---|---|
| $HOME | 50 GB, 500K files | Yes | No | NFS |
| $SCRATCH | 20 TB and 1000K files per user, 100 TB and 10M files per group | No | Yes, all files older than 90 days | Lustre |
| $PROJECT | 10 TB and 5M files per group, 500K files per user | Yes | No | Lustre |
| $NEARLINE | TBD | No | No | Tape |