Storage and file management: Difference between revisions

We provide a wide range of storage options to cover the needs of our very diverse users. These storage solutions range from high-speed temporary local storage to different kinds of long-term storage, so you can choose the storage medium that best corresponds to your needs and usage patterns. In most cases the [https://en.wikipedia.org/wiki/File_system filesystems] on our systems are a ''shared'' resource and for this reason should be used responsibly because unwise behaviour can negatively affect dozens or hundreds of other users. These filesystems are also designed to store a limited number of very large files, which are typically binary since very large (hundreds of MB or more) text files lose most of their interest in being readable by humans. You should therefore avoid storing tens of thousands of small files, where small means less than a few megabytes, particularly in the same directory. A better approach is to use commands like [[Archiving and compressing files|<code>tar</code>]] or <code>zip</code> to convert a directory containing many small files into a single very large archive file.  

All users can check the available disk space and the current disk utilization for the ''project'', ''home'' and ''scratch'' file systems with the command line utility '''''diskusage_report''''', available on our clusters. To use this utility, log into the cluster using SSH, at the command prompt type diskusage_report, and press the Enter key. Below is a typical output of this utility:

While the command <b>diskusage_report</b> gives the space and file count usage per user on ''home'' and ''scratch'', it shows the total quota of the group on project. It includes all the files from each member of the group. Since the files that belong to a user could however be anywhere in the project space, it is difficult to obtain correct figures per user and per given project in case a user has access to more than one project. However, users can obtain an estimate of their space and file count use on the entire project space by running the command