cc_staff
290
edits
Data management at Niagara: Difference between revisions
Jump to navigation
Jump to search
no edit summary
(Created page with " = Data Management = <!--T:110--> == Purpose of each file system == === /home === /home is intended primarily for individual user files, common software or small datasets used...") |
No edit summary |
||
| Line 23: | Line 23: | ||
For instance, if you run multi-process jobs, having each process write to a file of its own is not an scalable I/O solution. A directory gets locked by the first process accessing it, so all other processes have to wait for it. Not only has the code just become considerably less parallel, chances are the file system will have a time-out while waiting for your other processes, leading your program to crash mysteriously. | For instance, if you run multi-process jobs, having each process write to a file of its own is not an scalable I/O solution. A directory gets locked by the first process accessing it, so all other processes have to wait for it. Not only has the code just become considerably less parallel, chances are the file system will have a time-out while waiting for your other processes, leading your program to crash mysteriously. | ||
Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file. | Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file. | ||
<!--T:41--> | |||
<ul> | |||
<li>[https://docs.scinet.utoronto.ca/images/9/9a/Inode_vs._Space_quota_-_v2x.pdf Inode vs. Space quota (PROJECT and SCRATCH)]</li> | |||
<li>[https://docs.scinet.utoronto.ca/images/0/0e/Scratch-quota.pdf dynamic quota per group (SCRATCH)]</li> | |||
<li>Compute nodes do not have local storage.</li> | |||
<li>Archive space is on [https://docs.scinet.utoronto.ca/index.php/HPSS HPSS]</li> | |||
<li>Backup means a recent snapshot, not an achive of all data that ever was.</li> | |||
<li><p><code>$BBUFFER</code> stands for [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer], a faster parallel storage tier for temporary data.</p></li></ul> | |||
== Moving data == <!--T:42--> | == Moving data == <!--T:42--> | ||