Job arrays: Difference between revisions

Job arrays (view source)

201 bytes added , 6 years ago

Bureaucrats, cc_docs_admin, cc_staff

2,879

edits

@@ Line 23: / Line 23: @@
 == Running the same script in multiple directories ==
-This example assumes that you have multiple directories, each with the same structure, and you want to run the same script in each of these directories. You could use the following bash script :
+This example assumes that you have multiple directories, each with the same structure and you want to run the same script in each directory. If the directories can be named with sequential numbers then the example above can be easily adapted. If they are not so systematic, then create a file with the names of the directories, like so:
+ $ cat case_list
+ pacific2016
+ pacific2017
+ atlantic2016
+ atlantic2017
 {{File
-|name=script.sh
+|name=directories_array.sh
 |language=bash
 |contents=
 #!/bin/bash
-#SBATCH --ntasks=1
+#SBATCH --time=0:15:00
-#SBATCH --time=2:00:00
+#SBATCH --array=1-4
-echo "Starting run at: `date`"
-DIRS=($(find . -maxdepth 1 -type d \( ! -name . \)))
-DIR=${DIRS[$SLURM_ARRAY_TASK_ID]}
+echo "Starting task $SLURM_ARRAY_TASK_ID at $(date)"
+DIR=$(sed -n "${SLURM_ARRAY_TASK_ID}p" case_list)
 cd $DIR
-pwd
 # Place the code to execute here
+pwd
+ls
 }}
-and submit it using the following command :
+The file <code>case_list</code> should not be changed until all the tasks in the array have run, since it will be read each time a new task starts.
-{{Commands
-|N{{=}}$(find . -maxdepth 1 -type d \( ! -name . \) {{!}} wc -l)
-|sbatch --array{{=}}1-$N script.sh
-}}