Bureaucrats, cc_docs_admin, cc_staff
2,879
edits
Using node-local storage: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 6: | Line 6: | ||
Because this directory resides on local disk, input and output (I/O) to it | Because this directory resides on local disk, input and output (I/O) to it | ||
is almost always faster than I/O to a network file system (/project, /scratch, or /home). | is almost always faster than I/O to a network file system (/project, /scratch, or /home). | ||
Any job doing | Specifically, local disk is better for frequent small I/O transactions than network storage. | ||
Any job doing a lot of input and output (which is most jobs!) may expect | |||
to run more quickly if it uses <code>$SLURM_TMPDIR</code> instead of network disk. | to run more quickly if it uses <code>$SLURM_TMPDIR</code> instead of network disk. | ||
The temporary character of <code>$SLURM_TMPDIR</code> makes | The temporary character of <code>$SLURM_TMPDIR</code> makes more trouble to use than | ||
network storage. | network storage. | ||
Input must be copied from network storage to <code>$SLURM_TMPDIR</code> before it can be read, | Input must be copied from network storage to <code>$SLURM_TMPDIR</code> before it can be read, | ||
| Line 36: | Line 36: | ||
We particularly find that using an application in a [[Python]] virtual environment | We particularly find that using an application in a [[Python]] virtual environment | ||
generates a large number of small I/O transactions--- | generates a large number of small I/O transactions--- More than it takes | ||
to create the virtual environment in the first place. This is why we recommend | to create the virtual environment in the first place. This is why we recommend | ||
[[Python#Creating virtual environments inside of your jobs|creating virtual environments inside your jobs]] | [[Python#Creating virtual environments inside of your jobs|creating virtual environments inside your jobs]] | ||
| Line 46: | Line 46: | ||
job ends. If a job times out, then the last few lines of the job script might not | job ends. If a job times out, then the last few lines of the job script might not | ||
be executed. | be executed. | ||
This should be addressed two ways: | |||
* ... | |||
* ... | |||
=== Multi-node jobs === | === Multi-node jobs === | ||