cc_staff
290
edits
Niagara Quickstart: Difference between revisions
→File Input/Output Tips
| Line 375: | Line 375: | ||
* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes. | * GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes. | ||
* Accessing data sets which consist of many, small files leads to poor performance on GPFS. | * Accessing data sets which consist of many, small files leads to poor performance on GPFS. | ||
* Avoid reading and writing lots of small amounts of data to disk. Many small files on the system waste space and are slower to access, read and write. If you must write many small files, use [ | * Avoid reading and writing lots of small amounts of data to disk. Many small files on the system waste space and are slower to access, read and write. If you must write many small files, use [https://docs.scinet.utoronto.ca/index.php/User_Ramdisk ramdisk]. | ||
* Write data out in a binary format. This is faster and takes less space. | * Write data out in a binary format. This is faster and takes less space. | ||
* The [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer] is better for i/o heavy jobs and to speed up [https://docs.scinet.utoronto.ca/index.php/Checkpoints checkpoints]. | * The [https://docs.scinet.utoronto.ca/index.php/Burst_Buffer Burst Buffer] is better for i/o heavy jobs and to speed up [https://docs.scinet.utoronto.ca/index.php/Checkpoints checkpoints]. | ||