rsnt_translations
56,420
edits
Storage and file management: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 36: | Line 36: | ||
<!--T:24--> | <!--T:24--> | ||
While the command '''diskusage_report''' gives the space and inode usage per user on ''home'' and ''scratch'', it shows the total quota of the group on project. It includes all the files from each member of the group. Since the files that belong to a user could however be anywhere in the project space, it is difficult to obtain correct figures per user and per given project in case a user has access to more than one project. However, users can obtain an estimate of their space and inode use on the entire project space by running the command | While the command '''diskusage_report''' gives the space and inode usage per user on ''home'' and ''scratch'', it shows the total quota of the group on project. It includes all the files from each member of the group. Since the files that belong to a user could however be anywhere in the project space, it is difficult to obtain correct figures per user and per given project in case a user has access to more than one project. However, users can obtain an estimate of their space and inode use on the entire project space by running the command | ||
<!--T:26--> | <!--T:26--> | ||
| Line 53: | Line 53: | ||
* If your program must search within a file, it is fastest to do it by first reading it completely before searching. | * If your program must search within a file, it is fastest to do it by first reading it completely before searching. | ||
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible move them to an alternative location like [[Using nearline storage|nearline]]. | * If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible move them to an alternative location like [[Using nearline storage|nearline]]. | ||
* For more | * For more on managing many files, see [[Handling large collections of files]], especially if you are limited by a quota on the number of files. | ||
* Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO]. | * Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO]. | ||
* If your needs are not well served by the available storage options please contact [[technical support]]. | * If your needs are not well served by the available storage options please contact [[technical support]]. | ||
| Line 60: | Line 60: | ||
<!--T:11--> | <!--T:11--> | ||
In order to ensure that there is adequate space for all | In order to ensure that there is adequate space for all users, there are a variety of quotas and policy restrictions concerning backups and automatic purging of certain filesystems. | ||
By default on our clusters each user has access to the home and scratch spaces, and each group has access to 1 TB of project space. Small increases in project and scratch spaces are available through our Rapid Access Service ([https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service RAS]). Larger increases in project spaces are available through the annual Resource Allocation | By default on our clusters each user has access to the home and scratch spaces, and each group has access to 1 TB of project space. Small increases in project and scratch spaces are available through our Rapid Access Service ([https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/rapid-access-service RAS]). Larger increases in project spaces are available through the annual Resource Allocation Competition ([https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition RAC]). You can see your current quota usage for various filesystems on Cedar and Graham using the command [[Storage and file management#Overview|<tt>diskusage_report</tt>]]. | ||
<!--T:12--> | <!--T:12--> | ||
| Line 70: | Line 70: | ||
! Filesystem | ! Filesystem | ||
! Default Quota | ! Default Quota | ||
! Lustre-based | ! Lustre-based | ||
! Backed up | ! Backed up | ||
! Purged | ! Purged | ||
! Available by Default | ! Available by Default | ||
! Mounted on Compute Nodes | ! Mounted on Compute Nodes | ||
|- | |- | ||
|Home Space | |Home Space | ||
| Line 115: | Line 115: | ||
! Filesystem | ! Filesystem | ||
! Default Quota | ! Default Quota | ||
! Lustre-based | ! Lustre-based | ||
! Backed up | ! Backed up | ||
! Purged | ! Purged | ||
! Available by Default | ! Available by Default | ||
! Mounted on Compute Nodes | ! Mounted on Compute Nodes | ||
|- | |- | ||
|Home Space | |Home Space | ||
| Line 160: | Line 160: | ||
! Filesystem | ! Filesystem | ||
! Default Quota | ! Default Quota | ||
! Lustre-based | ! Lustre-based | ||
! Backed up | ! Backed up | ||
! Purged | ! Purged | ||
! Available by Default | ! Available by Default | ||
! Mounted on Compute Nodes | ! Mounted on Compute Nodes | ||
|- | |- | ||
|Home Space | |Home Space | ||
| Line 278: | Line 278: | ||
<!--T:22--> | <!--T:22--> | ||
The backup policy on the home and project space is nightly backups which are retained for 30 days, while deleted files are retained for a further 60 days | The backup policy on the home and project space is nightly backups which are retained for 30 days, while deleted files are retained for a further 60 days; note that is entirely distinct from the age limit for purging files from the scratch space. If you wish to recover a previous version of a file or directory, you should contact [[technical support]] with the full path for the file(s) and desired version (by date). | ||
== See also == <!--T:13--> | == See also == <!--T:13--> | ||