cc_staff
290
edits
Data management at Niagara: Difference between revisions
Jump to navigation
Jump to search
→Quotas and purging
No edit summary |
|||
| Line 28: | Line 28: | ||
/archive is a nearline storage pool, if you want to temporarily offload semi-active material from any of the above file systems. In practice users will offload/recall material as part of their regular workflow, or when they hit their quotas on scratch or project. That material can remain on HPSS for a few months to a few years. Note that on niagara /archive is only available to groups with RAC allocation. | /archive is a nearline storage pool, if you want to temporarily offload semi-active material from any of the above file systems. In practice users will offload/recall material as part of their regular workflow, or when they hit their quotas on scratch or project. That material can remain on HPSS for a few months to a few years. Note that on niagara /archive is only available to groups with RAC allocation. | ||
= Quotas and purging = | |||
You should familiarize yourself with the [[Data_Management#Purpose_of_each_file_system | various file systems]], what purpose they serve, and how to properly use them. This table summarizes the various file systems. | You should familiarize yourself with the [[Data_Management#Purpose_of_each_file_system | various file systems]], what purpose they serve, and how to properly use them. This table summarizes the various file systems. | ||
| Line 105: | Line 105: | ||
<li><p><code>$BBUFFER</code> stands for [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer], a faster parallel storage tier for temporary data.</p></li></ul> | <li><p><code>$BBUFFER</code> stands for [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer], a faster parallel storage tier for temporary data.</p></li></ul> | ||
==How much Disk Space Do I have left?== <!--T:151--> | |||
<!--T:152--> | |||
The <tt>'''/scinet/niagara/bin/diskUsage'''</tt> command, available on the login nodes and datamovers, provides information in a number of ways on the home, scratch, project and archive file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details. | |||
<pre> | |||
Usage: diskUsage [-h|-?| [-a] [-u <user>] | |||
-h|-?: help | |||
-a: list usages of all members on the group | |||
-u <user>: as another user on your group | |||
</pre> | |||
<!--T:153--> | |||
Did you know that you can check which of your directories have more than 1000 files with the <tt>'''/scinet/niagara/bin/topUserDirOver1000list'''</tt> command and which have more than 1GB of material with the <tt>'''/scinet/niagara/bin/topUserDirOver1GBlist'''</tt> command? | |||
<!--T:154--> | |||
Note: | |||
* information on usage and quota is only updated every 3 hours! | |||
== Moving data == <!--T:42--> | == Moving data == <!--T:42--> | ||