cc_staff
318
edits
Handling large collections of files: Difference between revisions
Jump to navigation
Jump to search
Handling large collections of files (view source)
Revision as of 16:29, 21 February 2020
, 4 years agono edit summary
No edit summary |
No edit summary |
||
| Line 28: | Line 28: | ||
Local disks attached to compute nodes are at least SATA SSD or better, and, in general, will have a performance that is considerably better than the project or scratch filesystems. Note that local disk is shared by all running jobs on that node without being allocated by the scheduler. The actual amount of local disk space varies from one cluster to another (and might also vary within a given cluster). For example, | Local disks attached to compute nodes are at least SATA SSD or better, and, in general, will have a performance that is considerably better than the project or scratch filesystems. Note that local disk is shared by all running jobs on that node without being allocated by the scheduler. The actual amount of local disk space varies from one cluster to another (and might also vary within a given cluster). For example, | ||
* [[Béluga]] offers roughly 370GB of local disk for the CPU nodes, the GPU nodes have a 1.6TB NVMe disk (to help with the AI image datasets with their millions of small files). | * [[Béluga/en | Béluga]] offers roughly 370GB of local disk for the CPU nodes, the GPU nodes have a 1.6TB NVMe disk (to help with the AI image datasets with their millions of small files). | ||
* [[Niagara]] does not have local storage on the compute nodes | * [[Niagara]] does not have local storage on the compute nodes | ||
* for other clusters you can assume the available disk size to be at least 190GB | * for other clusters you can assume the available disk size to be at least 190GB | ||