Translations:GNU Parallel/38/en

You can also use GNU Parallel to distribute a workload across multiple nodes in a cluster, such as in the context of a job on our servers. An example of this use is the following:

[name@server ~]$ scontrol show hostname > ./node_list_${SLURM_JOB_ID}

[name@server ~]$ parallel --jobs $SLURM_NTASKS_PER_NODE --sshloginfile ./node_list_${SLURM_JOB_ID} --env MY_VARIABLE --workdir $PWD --sshdelay 30 ./my_program

In this case, we create a file containing the list of nodes, and we use this file to tell GNU Parallel which nodes to use for the distribution of tasks. The --env option allows us to transfer a named environment variable to all the nodes while the --workdir option ensures that the GNU Parallel tasks will start in the same directory as the main node.