cc_staff
2
edits
GNU Parallel: Difference between revisions
m
$SLURM_JOB_NODELIST is a condensed line and can not be used by gnu-parallel
(Marked this version for translation) |
m ($SLURM_JOB_NODELIST is a condensed line and can not be used by gnu-parallel) |
||
| Line 51: | Line 51: | ||
You can also use Gnu Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on a Compute Canada server. An example of this use is the following: | You can also use Gnu Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on a Compute Canada server. An example of this use is the following: | ||
{{Command | {{Command | ||
|parallel --jobs 32 --sshloginfile $ | |scontrol show hostname ${SLURM_JOB_NODELIST} > ./node_list_${SLURM_JOB_ID} | ||
}} | |||
{{Command | |||
|parallel --jobs 32 --sshloginfile ./node_list_${SLURM_JOB_ID} --env MY_VARIABLE --workdir $PWD ./my_program | |||
}} | }} | ||
In this case, we suppose that each node has 32 CPU cores and we will use the <tt>$SLURM_JOB_NODELIST</tt> file created automatically by the job scheduler to tell Gnu Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the Gnu Parallel tasks will start in same directory as the main node. | In this case, we suppose that each node has 32 CPU cores and we will use the <tt>$SLURM_JOB_NODELIST</tt> file created automatically by the job scheduler to tell Gnu Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the Gnu Parallel tasks will start in same directory as the main node. | ||