Data management at Niagara: Difference between revisions

Data management at Niagara (view source)

777 bytes removed , 6 years ago

no edit summary

cc_staff

290

edits

@@ Line 276: / Line 276: @@
 Note:
 * information on usage and quota is only updated every 3 hours!
-== I/O Tips == <!--T:95-->
-<!--T:96-->
-* Your files can be seen on all Niagara login and compute nodes.
-* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
-* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes.
-* Accessing data sets which consist of many, small files leads to poor performance on GPFS.
-* Avoid reading and writing lots of small amounts of data to disk.<br />
-<!--T:97-->
-* Many small files on the system would waste space and would be slower to access, read and write.
-* Write data out in binary. Faster and takes less space.
-* The [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer] is better for i/o heavy jobs and to speed up checkpoints