Bureaucrats, cc_docs_admin, cc_staff
2,306
edits
GNU Parallel: Difference between revisions
Jump to navigation
Jump to search
no edit summary
m (Mboisson moved page Gnu Parallel to GNU Parallel without leaving a redirect) |
No edit summary |
||
| Line 51: | Line 51: | ||
You can also use Gnu Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on a Compute Canada server. An example of this use is the following: | You can also use Gnu Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on a Compute Canada server. An example of this use is the following: | ||
{{Command | {{Command | ||
|parallel --jobs 12 --sshloginfile $ | |parallel --jobs 12 --sshloginfile $SLURM_JOB_NODELIST --env MY_VARIABLE --workdir $PWD ./my_program | ||
}} | }} | ||
In this case, we suppose that each node has 12 CPU cores and we will use the <tt>$PBS_NODEFILE</tt> file created automatically by the job scheduler to tell Gnu Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the Gnu Parallel tasks will start in same directory as the main node. | In this case, we suppose that each node has 12 CPU cores and we will use the <tt>$PBS_NODEFILE</tt> file created automatically by the job scheduler to tell Gnu Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the Gnu Parallel tasks will start in same directory as the main node. | ||