cc_staff
2
edits
GNU Parallel: Difference between revisions
Jump to navigation
Jump to search
creating file containing list of nodes
m ($SLURM_JOB_NODELIST is a condensed line and can not be used by gnu-parallel) |
(creating file containing list of nodes) |
||
| Line 56: | Line 56: | ||
|parallel --jobs 32 --sshloginfile ./node_list_${SLURM_JOB_ID} --env MY_VARIABLE --workdir $PWD ./my_program | |parallel --jobs 32 --sshloginfile ./node_list_${SLURM_JOB_ID} --env MY_VARIABLE --workdir $PWD ./my_program | ||
}} | }} | ||
In this case, we suppose that each node has 32 CPU cores and we | In this case, we suppose that each node has 32 CPU cores and we create a file containing the list of nodes from <tt>$SLURM_JOB_NODELIST</tt> (which is created automatically by the job scheduler), and we use the this file to tell Gnu Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the Gnu Parallel tasks will start in same directory as the main node. | ||
==Keeping Track of Completed and Failed Commands, and Restart Capabilities== <!--T:11--> | ==Keeping Track of Completed and Failed Commands, and Restart Capabilities== <!--T:11--> | ||