Bureaucrats, cc_docs_admin, cc_staff
2,306
edits
GNU Parallel: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 51: | Line 51: | ||
You can also use Gnu Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on a Compute Canada server. An example of this use is the following: | You can also use Gnu Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on a Compute Canada server. An example of this use is the following: | ||
{{Command | {{Command | ||
|parallel --jobs | |parallel --jobs 32 --sshloginfile $SLURM_JOB_NODELIST --env MY_VARIABLE --workdir $PWD ./my_program | ||
}} | }} | ||
In this case, we suppose that each node has | In this case, we suppose that each node has 32 CPU cores and we will use the <tt>$SLURM_JOB_NODELIST</tt> file created automatically by the job scheduler to tell Gnu Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the Gnu Parallel tasks will start in same directory as the main node. | ||
==Keeping Track of Completed and Failed Commands, and Restart Capabilities== <!--T:11--> | ==Keeping Track of Completed and Failed Commands, and Restart Capabilities== <!--T:11--> | ||