cc_staff
782
edits
GNU Parallel: Difference between revisions
Two more details.
(A better multi-node example with OpenMP threads.) |
(Two more details.) |
||
| Line 100: | Line 100: | ||
In this case, we create a file containing the list of nodes, and we use this file to tell GNU Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> option allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the GNU Parallel tasks will start in the same directory as the main node. | In this case, we create a file containing the list of nodes, and we use this file to tell GNU Parallel which nodes to use for the distribution of tasks. The <tt>--env</tt> option allows us to transfer a named environment variable to all the nodes while the <tt>--workdir</tt> option ensures that the GNU Parallel tasks will start in the same directory as the main node. | ||
For example, when a long list of [[OpenMP]] tasks are executed as a single job submitted with <tt>--nodes=N</tt>, <tt>--ntasks-per-node=5</tt> and <tt>--cpus-per-task=8</tt>, the following command will take into account all processes to be started on all nodes and the number of OpenMP threads: | For example, when a long list of [[OpenMP]] tasks are executed as a single job submitted with <tt>--nodes=N</tt>, <tt>--ntasks-per-node=5</tt> and <tt>--cpus-per-task=8</tt>, the following command will take into account all processes to be started on all reserved nodes and the number of OpenMP threads per process: | ||
{{Command | {{Command | ||
|parallel --jobs $SLURM_NTASKS_PER_NODE --sshloginfile ./node_list_${SLURM_JOB_ID} --workdir $PWD OMP_NUM_THREADS{{=}}$SLURM_CPUS_PER_TASK ./my_program | |parallel --jobs $SLURM_NTASKS_PER_NODE --sshloginfile ./node_list_${SLURM_JOB_ID} --workdir $PWD OMP_NUM_THREADS{{=}}$SLURM_CPUS_PER_TASK ./my_program | ||