cc_staff
284
edits
GNU Parallel: Difference between revisions
Jump to navigation
Jump to search
Added section on multiples simulations in one job
No edit summary |
(Added section on multiples simulations in one job) |
||
| Line 164: | Line 164: | ||
The third line shows that the block size is too big and that we are only using 1 core out of 8, therefore inefficiently processing chunks. | The third line shows that the block size is too big and that we are only using 1 core out of 8, therefore inefficiently processing chunks. | ||
Finally, the last line shows that in many cases, letting GNU Parallel adapt and decide on the block size is often faster. | Finally, the last line shows that in many cases, letting GNU Parallel adapt and decide on the block size is often faster. | ||
== Running hundreds or thousands of simulations == | |||
First you must determine how many resources is required by one simulation, then you can estimate the total resources required in your job. | |||
The submission scripts given in example below are based on: 1 serial simulation requiring 2 Gb of memory, 1 core and 5 minutes, and 1000 simulations. It would take 83.3 hours or 3.472 days to run with 1 core. | |||
On one node, using 32 cores, it can be completed in 2.6 hours. | |||
One could also use more than one node (see [[#Running on Multiple Nodes]]). | |||
=== Arguments List === | |||
As shown in the section [[#File Content as Argument List]], you can use a file containing all parameters. In this case, the parameters are delimitered by a tab character (<tt>\t</tt>) and each line corresponds to one simulation. | |||
<tabs> | |||
<tab name="Parameters"> | |||
{{File | |||
|name=my_parameters.txt | |||
|lang="txt" | |||
|contents= | |||
1 1 | |||
1 2 | |||
1 3 | |||
... | |||
}} | |||
</tab> | |||
<tab name="Script"> | |||
{{File | |||
|name=sim_submit.sh | |||
|lang="bash" | |||
|contents= | |||
#!/bin/bash | |||
#SBATCH --account=def-someuser | |||
#SBATCH --nodes=1 | |||
#SBATCH --cpus-per-task=32 | |||
#SBATCH --time=03:00:00 | |||
#SBATCH --mem-per-cpu=2G | |||
# Read the parameters, placing each column to their respective argument | |||
parallel -j $SLURM_CPUS_PER_TASK --colsep '\t' my_simulator --alpha {1} --beta {2} :::: ./my_parameters.txt | |||
}} | |||
</tab> | |||
</tabs> | |||
=== Commands List === | |||
As shown in the section [[#File Content as Command List]], you can use a file containing all the commands and their parameters. | |||
<tabs> | |||
<tab name="Commands"> | |||
{{File | |||
|name=my_commands.txt | |||
|lang="txt" | |||
|contents= | |||
my_simulator --alpha 1 --beta 1 | |||
my_simulator --alpha 1 --beta 2 | |||
my_simulator --alpha 1 --beta 3 | |||
... | |||
}} | |||
</tab> | |||
<tab name="Script"> | |||
{{File | |||
|name=sim_submit.sh | |||
|lang="bash" | |||
|contents= | |||
#!/bin/bash | |||
#SBATCH --account=def-someuser | |||
#SBATCH --nodes=1 | |||
#SBATCH --cpus-per-task=32 | |||
#SBATCH --time=03:00:00 | |||
#SBATCH --mem-per-cpu=2G | |||
parallel -j $SLURM_CPUS_PER_TASK < ./my_commands.txt | |||
}} | |||
</tab> | |||
</tabs> | |||
=== Multiple Arguments === | |||
You can use GNU Parallel to generate the parameters and feed them to the command. | |||
{{File | |||
|name=sim_submit.sh | |||
|lang="bash" | |||
|contents= | |||
#!/bin/bash | |||
#SBATCH --account=def-someuser | |||
#SBATCH --nodes=1 | |||
#SBATCH --cpus-per-task=32 | |||
#SBATCH --time=03:00:00 | |||
#SBATCH --mem-per-cpu=2G | |||
# Generates 1000 simulations where the alpha argument ranges from 1-10, and beta from 1-100 | |||
# placing each source to their respective argument | |||
parallel -j $SLURM_CPUS_PER_TASK my_simulator --alpha {1} --beta {2} ::: {1..10} ::: {1..100} | |||
}} | |||
==Related topics== <!--T:14--> | ==Related topics== <!--T:14--> | ||