Niagara Quickstart: Difference between revisions

Jump to navigation Jump to search
no edit summary
No edit summary
Line 375: Line 375:
* information on usage and quota is only updated every 3 hours!
* information on usage and quota is only updated every 3 hours!


== I/O Tips ==
== I/O Tips == <!--T:95-->


<!--T:96-->
* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* Your files can be seen on all Niagara login and compute nodes.
* Your files can be seen on all Niagara login and compute nodes.
Line 383: Line 384:
* Avoid reading and writing lots of small amounts of data to disk.<br />
* Avoid reading and writing lots of small amounts of data to disk.<br />


<!--T:97-->
* Many small files on the system would waste space and would be slower to access, read and write.
* Many small files on the system would waste space and would be slower to access, read and write.
* Write data out in binary. Faster and takes less space.
* Write data out in binary. Faster and takes less space.
* The [[Burst Buffer]] is better for i/o heavy jobs and to speed up checkpoints.
* The [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer] is better for i/o heavy jobs and to speed up checkpoints


= Loading software modules = <!--T:48-->
= Loading software modules = <!--T:48-->
Line 759: Line 761:
<!--T:103-->
<!--T:103-->
For more information, check out the wiki page devoted to [[Running jobs]].
For more information, check out the wiki page devoted to [[Running jobs]].
= Data management and I/O tips = <!--T:95-->
<!--T:96-->
* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* Your files can be seen on all Niagara login and compute nodes.
* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes.
* But accessing data sets which consist of many, small files leads to poor performance.
* Avoid reading and writing lots of small amounts of data to disk.<br />
<!--T:97-->
* Many small files on the system would waste space and would be slower to access, read and write.
* Write data out in binary. Faster and takes less space.
* Burst buffer (to come) is better for i/o heavy jobs and to speed up checkpoints.
</translate>
</translate>
cc_staff
290

edits

Navigation menu