META-Farm

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Overview

META package is a suite of scripts designed in-house to fully automate throughput computing (running a large number of related serial/parallel/GPU jobs - so called "farming"). They will work on all National systems, and other clusters which use the same setup (most importantly, SLURM scheduler). The same set of scripts can be used with little or no modifications to organize almost any type of farming workflow, including

• either SIMPLE (one case per job) mode or META (many cases per job) mode, with dynamic workload balancing for the latter (here a "case" is one independent computation, could be serial or parallel or GPU-based);
• capturing the exit status of all individual cases;
• the ability to automatically resubmit all the cases which failed or never ran;
• the ability to submit and independently operate multiple serial farms (groups of cases) on the same cluster;
• the ability to automatically run a post-processing job once all the cases have been processed successfully.

Some technical points:

• All "cases" to be computed have to be described as separate lines (one line per case) in the file table.dat in the farm-specific directory. (One can run multiple farms independently; each farm has to have its own directory.)
• In the META mode, the number of actual jobs (so-called "meta-jobs") submitted by the package is usually much smaller than the number of cases to process. Each meta-job can process multiple lines from table.dat (multiple cases). Running meta-jobs read lines from table.dat, starting from the first one, in a serialized manner (using the lockfile mechanism to prevent a race condition). This ensures a good dynamic workload balance between meta-jobs, as meta-jobs which happen to handle shorter cases would process more of them.
• Not all meta-jobs need to ever run in the META mode. The first meta-job to run will start processing lines from table.dat; if/when the second job starts, it joins the first one, and so on. If runtime of individual meta-jobs is long enough, it is possible to process all the cases with just a single running meta-job.

META vs. GLOST

There are three important advantages of the META package over the other approaches (like GLOST) where farm processing is done by bundling up all the jobs into a large parallel (MPI) job:

1. As the scheduler has a full flexibility to start individual meta jobs when it wants, the queue wait time with the META package can be dramatically shorter than with GLOST. Consider a large farm where 1000 cpu cores need to be used for 3 days. With META, some meta-jobs start to run - producing first results - within minutes. With GLOST, with a 1000-way MPI job, queue wait time can be weeks, so it'll be weeks before you see your very first result.
2. With GLOST, at the end of the farm computations, some MPI ranks will finish earlier and will sit idle until the very last - the slowest - MPI rank ends. In META package there is no such waste at the end of the farm - individual meta-jobs exit earlier if they have no more workload to process.
3. Finally, GLOST and other similar packages do not support an automated resubmission of the cases which failed or never ran. META package does have this feature, and it is very easy to use.

The META webinar

This webinar (recorded on October 6th, 2021) describes the META package: https://youtu.be/GcYbaPClwGE

Quick start guide

If you are impatient to start using the package, just follow the steps listed below. But it is highly recommended to also read the rest of the page.

Using module command

• Login to the cluster.
• Load the meta-farm module:

$ module load meta-farm

• To create a farm directory with the name Farm_name, one can run the command

$ farm_init.run  Farm_name

• The above command will also copy a few important files inside the farm directory, some of which will have to be customized.
• Copy your code and initial conditions files to the META directory (optional; you can use full paths instead).
• Create table.dat inside the META directory (text file, one case per line; see #single_case.sh script, #FORTRAN code example: using standard input, #FORTRAN code example: copying files for each case, #Using all the columns in the cases table explicitly for some examples). Each line of this file describes how to run one case (one independent computation).
• Modify “single_case.sh” script if needed. In many cases you don't have to make any changes. See #single_case.sh script, #Handling code exit status, #FORTRAN code example: copying files for each case, #Using all the columns in the cases table explicitly for more information.
• Modify "job_script.sh" file to suit your needs (#job_script.sh script). In particular, use a correct account name, and set an appropriate job runtime. (See #Estimating the runtime and number of meta-jobs.)
• Inside the farm directory, execute

$ submit.run -1

for the one case per job (SIMPLE) mode, or

$ submit.run N

for the many cases per job (META) mode (N is the number of meta-jobs to use; it should be significantly smaller than the total number of cases).

To run another farm concurrently with the first one, run "farm_init.run" again (providing a different farm name) and customize the files single_case.sh and job_script.sh inside the new farm directory, then create a new table.dat file there. Also copy the code executable and all the input files as needed. Now you can execute the "submit.run" command inside the second farm directory to submit the second farm.

To use any of the provided *.run utilities, one has first to cd to the corresponding farm directory. (Farm_init.run is the only exception.)

Using the git repository

Instead of loading the module, you can also clone the meta-farm package from our git repository:

$ git clone https://git.computecanada.ca/syam/meta-farm.git

Then you'll have to modify your $PATH variable, to point it to the bin subdirectory of the newly created meta-farm directory. Assuming you executed "git clone" inside your home directory, you'll have to execute

$ export PATH=~/meta-farm/bin:$PATH

Full list of commands

• farm_init.run : Initiates a farm. See #Using module command.
• submit.run : Submits the farm to the scheduler. See #submit.run script.
• resubmit.run : Resubmits all computations which failed or never ran as a new farm. See #Resubmitting failed/never-run jobs.
• list.run will list all the jobs with their current state for the serial farm.
• query.run will provide a short summary (number of queued / running / done jobs) in the farm, which is more convenient than using “list.run” when the number of jobs is large. It will also print the progress - number of processed cases vs. the total number of cases - both for the current run, and globally.
• kill.run: will kill all the running/queued jobs in the farm.
• prune.run: will only kill (remove) queued jobs.
• Status.run (capital “S”!) will list statuses of all processed cases. With the optional "-f" switch, the non-zero status lines (if any) will be listed at the end.
• clean.run: will delete all the files in the farm directory (including subdirectories if any present), except for *.sh files (job_script.sh, single_case.sh, final.sh, and resubmit_script.sh ), config.h, and table.dat. It will also delete all files associated with this farm in the /home/$USER/tmp directory. Be very careful with this script!

All of these commands (except for farm_init.run) have to be executed inside a root farm directory.

Small number of cases (SIMPLE mode)

Overview

Let's call a single execution of the code in a serial/parallel farm a “case”. When the total number of cases, N_cases, is fairly small (say, <500) it is convenient to dedicate a separate job to each case. (You should make sure that each case runs for at least 10 minutes. If this is not the case, you should consider the "many cases per job" - META - mode; see #Large number of cases (META mode).)

The three essential scripts are the command “submit.run”, and two customizable scripts “single_case.sh” and "job_script.sh".

submit.run script

“submit.run” has one obligatory command line argument - number of jobs to submit N:

   $ submit.run N [-auto] [optional_sbatch_arguments]

If N=-1, you are requesting the SIMPLE mode ("submit as many jobs as there are lines in table.dat"). Otherwise (if N is a positive integer), you are requesting the META mode (multiple cases per job), with N being the number of meta-jobs requested.

If the optional switch "-auto" is present, the farm will resubmit itself automatically at the end, more than once if necessary, until all the cases from table.dat have been processed. This advanced feature is described in #Running resubmit.run automatically.

If a file named "final.sh" is present in the farm directory, the submit.run command will treat it as a job script for a post-processing job (which will be launched automatically once all the cases from table.dat have been successfully processed). See #Running a post-processing job automatically for more details.

All optional_sbatch_arguments (there could more than one) will be passed to the job submitting command, sbatch. They will be used with all meta-jobs submitted for this farm.

single_case.sh script

Another principal script, “single_case.sh”, is only one of the two scripts (the other one being job_script.sh) which might need customization. Its task is to read one line from table.dat, parse it, and use these data to launch your code for this particular case. The version of the file provided literally executes one full line from table.dat file in a separate subdirectory, RUNyyy (yyy being the case number).

“single_case.sh”:

...
# ++++++++++++++++++++++  This part can be customized:  ++++++++++++++++++++++++
#  Here:
#  $ID contains the case id from the original table (can be used to provide a unique seed to the code etc)
#  $COMM is the line corresponding to the case $ID in the original table, without the ID field
#  $METAJOB_ID is the jobid for the current meta-job (convenient for creating per-job files)

mkdir -p RUN$ID
cd RUN$ID

echo "Case $ID:"

# Executing the command (a line from table.dat)
# It's allowed to use more than one shell command (separated by semicolons) on a single line
eval "$COMM"

# Exit status of the code:
STATUS=$?

cd ..
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...

Your table.dat can look like this:

 /home/user/bin/code1  1.0  10  2.1
 cp -f /input_dir/input1 .; /code_dir/code 
 ./code2 < IC.2
 sleep 10m
 ...

In other words, any executable statement(s) which can be written on one line can go there.

No explicit code names are used in the original version of “single_case.sh”. Instead, explicit code names (with the path if needed) should be listed on each line of table.dat file.

Alternatively, you can edit the single_case.sh file to use there the code name explicitly, in which case your table.dat file will only contain command line switch(es) for your code and/or redirects. For example:

• single_case.sh:

# ++++++++++++++++++++++  This part can be customized:  ++++++++++++++++++++++++
...
# Here we use $ID (case number) as a unique seed for Monte-Carlo type serial farming:
/path/to/your/code -par $COMM  -seed $ID
...
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

• table.dat:

 12.56
 21.35
 ...

A note: if your code doesn't need any switches to be read from table.dat, you still have to generate table.dat, with the number of lines being equal to the number of cases you want to compute. In this case, it doesn't matter what you put inside table.dat - all that matters is the total number of the lines. The above example would then look like

/path/to/your/code -seed $ID

(no references to $COMM).

Another note: “submit.run” will modify your table.dat once (will add the line number at the beginning of each line, if you didn't do it yourself). The file table.dat can be used with submit.run either way (with or without the first case_ID column) - this is handled automatically.

Handling code exit status

What is “$STATUS” for in “single_case.sh”? It is a shell variable which should be set to “0” if your case was computed correctly, and >0 otherwise. It is very important: it is used by “resubmit.run” to figure out which cases failed, so they can be re-computed. In the provided version of “single_case.sh”, $STATUS only reflects the exit code of your program. This likely won't cover all potential problems. (Some codes do not exit with non-zero status even if something went wrong.) You can always change or augment $STATUS derivation in “single_case.sh”. E.g., if your code is supposed to create a new non-empty file (say, “out.dat”) at the very end of each case run, the existence of such a non-empty file can be used to judge if the case failed or not:

  STATUS=$?
  if test ! -s out.dat
     then
     STATUS=1
     fi

In the above example, $STATUS will be positive if either code exit status is positive, or "out.dat" file doesn't exist or is empty.

job_script.sh script

The file job_script.sh is the SLURM job script which will be used by all meta-jobs in your serial farm. It can look like this:

#!/bin/bash
# Here you should provide the sbatch arguments to be used in all jobs in this serial farm
# It has to contain the runtime switch (either -t or --time):
#SBATCH -t 0-00:10
#SBATCH --mem=4G
#SBATCH -A Your_account_name

# Don't change this line:
task.run

At the very least you'd have to change the account name (the "-A" switch), and the meta-job runtime ("-t" switch). In the SIMPLE mode, you should request the job's runtime to be somewhat larger than the longest expected individual case run.

Important: your job_script.sh file must include the runtime switch (either -t or --time). This cannot be passed to sbatch as an optional argument to submit.run.

Sometimes the following problem happens: one of the meta-jobs is allocated on a node which has an issue causing your code to fail instantly (e.g., no GPU is available, and your code needs a GPU; or project file space is not mounted). This is definitely not normal, and issues like this need to be reported to Compute Canada. But if it does happen, then your single bad meta-job can churn quickly through table.dat, so your whole farm fails. As a precaution, one can add a testing routine in job_scrip.sh, before the "task.run" line. For example, the following code will test for the presence of a GPU, and forces the meta-job to exit if none are present - before it started failing your serial farm cases:

gpu_test
retVal=$?
if [ $retVal -ne 0 ]; then
    exit 1
fi

task.run

You can copy the utility "gpu_test" to your ~/bin directory (only on graham, cedar, and beluga):

cp ~syam/bin/gpu_test ~/bin

The META package also has a built-in mechanism to detect early problems of this kind, and kill meta-job(s) which churn through the cases too quickly. The two relevant parameters from config.h file are N_failed_max and dt_failed. The protection mechanism is triggered when the first $N_failed_max (5 by default) cases are very short - less than $dt_failed (5 by default) seconds in duration. If you get false triggering of the protective mechanism because some of your normal cases have runtime shorter than $dt_failed, you have to reduce the value of $dt_failed in config.h .

Output files

Once one or more meta-jobs in your farm are running, the following files will be created in the farm directory:

• OUTPUT/slurm-jobid.out files (one file per meta-job): standard output from meta-jobs;
• STATUSES/status.jobid files (one file per meta-job): files containing the statuses of the processed cases.

In both cases, jobid stands for the jobid of the corresponding meta-job.

One more directory - MISC - will also be created inside the root farm directory. It contains some auxiliary data.

Also, every "submit.run" script execution will create a unique subdirectory inside "/home/$USER/tmp". Inside that subdirectory, some small scratch files (like files used by "lockfile" command, to serialize certain operations inside the jobs) will be created. These subdirectories have names "NODE.PID", where "NODE" is the name of the current node (typically a login node), and "PID" is the unique process ID for the script. Once the farm execution is done, one can safely erase this subdirectory. This will happen automatically if you run "clean.run" command (be careful - clean.run also deletes all the results produced by your farm!).

Resubmitting failed/never-run jobs

The command “resubmit.run” takes the same arguments as “submit.run”:

   $  resubmit.run N [-auto] [optional_sbatch_arguments]

“resubmit.run”:

• will analyze all those status.* files (#Output files);
• figure out which cases failed and which never ran for whatever reason (e.g. because of the meta-jobs' runtime limit);
• create a new case table (adding “_” at the end of the original table name), which lists only the cases which still need to be run;
• uses “submit.run” internally to launch a new farm, for the unfinished/failed jobs.

Notes: You won't be able to run “resubmit.run” until all the jobs from the original run are done or killed. If some cases still fail or do not run, one can resubmit the farm as many times as needed.

Of course, if certain cases persistently fail, then there must a be a problem with either your initial conditions parameters or with your code (a code bug). It is convenient to use here the command "Status.run" (capital S!) which displays the statuses for all computed cases. With the optional argument "-f", the Status.run command will sort the output according to the exit status, showing non-zero status lines (if any) at the bottom, to make them easier to spot.

Similarly to "submit.run", if the optional switch "-auto" is present, the farm will resubmit itself automatically at the end, more than once if necessary. This advanced feature is described in #Running resubmit.run automatically.

Large number of cases (META mode)

META mode overview

The SIMPLE (one case per job) mode works fine when the number of cases is fairly small (<500). When N_cases >> 500, the following problems arise:

• Each cluster has a limit on how many jobs a user can submit (for Graham, it is 1000).
• With a very large number of cases, each case computation is typically short. If one case runs for <20 min, you start wasting cpu cycles due to scheduling overheads.

The solution: instead of submitting a separate job for each case, one should submit a smaller number of "meta-jobs", each of which would process multiple cases (so-called META mode). As cases can take different time to process, it is highly desirable to utilize a dynamic workload balancing scheme here.

This is how it is implemented:

As the above diagram shows, "submit.run N" script in the META mode will submit N jobs, with N being a fairly small number (much smaller than the number of cases to process). Each job would execute the same script - "task.run". Inside that script, there is a "while" loop, for different cases. Each iteration of the loop has to go through a serialized (only one job at a time can do that) portion of the code, where it figures out which next case (if any) to process. Then the already familiar script "single_case.sh" (see section #single_case.sh script) is executed - once per each case, which in turn calls the user code.

This approach results in dynamic workload balancing achieved across all the running "meta-jobs" belonging to the same farm. The algorithm is illustrated by the diagram below:

This can be seen more clearly in this animation (part of the META webinar).

The dynamic workload balancing results in all meta-jobs finishing around the same time, regardless of how different the runtimes are for individual cases, regardless of how fast CPUs are on different nodes, and regardless of when individual "meta-jobs" start. In addition, this approach is very robust: not all meta-jobs need to start running for all the cases to be processed; if a meta-job dies (e.g. due to a node crash), at most one case will be lost. (The latter can be easily rectified by running the "resubmit.run" script; see #Resubmitting failed/never-run jobs.)

To enable the META mode (with dynamic workload balancing), the first argument to “submit.run” script should be the desired number of meta-jobs, e.g.:

   $  submit.run  32

Not all of the requested meta-jobs will necessarily run (this depends on how busy the cluster is). But as described above, in the META mode you will eventually get all your results regardless of how many meta-jobs will run. (You might need to run "resubmit.run", sometimes more than once, to complete particularly large serial farms).

Estimating the runtime and number of meta-jobs

How to figure out the optimum number of meta-jobs, and the runtime (to be used in job_script.sh)?

First you need to figure out what is the average runtime for an individual case (a single line in table.dat). One way to do it is to allocate a cpu with salloc command, cd to the farm directory, and execute the single_case.sh script there multiple times, for different cases, measuring the total runtime, and then dividing that by the number of cases to get an estimate of the average case runtime. This can be conveniently achieved with a bash "for" loop:

   $  N=10; time for ((i=1; i<=$N; i++)); do  ./single_case.sh table.dat $i  ; done

The "real" time obtained with the above command should be divided by $N (10 in this example) to get the average case runtime estimate. Let's call it dt_case (in seconds).

You can estimate the amount of cpu cycles needed to process the whole farm, by multiplying dt_case by the number of cases (number of lines in table.dat). This will be in cpu-seconds. Dividing that by 3600 gives you the amount of compute resources in cpu-hours. Multiply that by something like 1.1 - 1.3 to have a bit of a safety margin.

Now you can make a sensible choice for the runtime of meta-jobs, and that will also give you the number of meta-jobs needed to finish the whole farm.

The runtime you choose should be significantly larger (ideally by a factor of 100 or more) than the average runtime of individual cases. In any case, it should definitely be larger than the expected longest individual case runtime. On the other hand, it should not be too large (say, no more than 3 days) to avoid very long queue wait time (the longer the job's runtime is, the smaller is the number of cluster's nodes available for such jobs). A good choice would be either 12h or 1d. Once you settled on the runtime, you can divide the farm's cpu cycles amount (in cpu-hours) by the meta-job's runtime (in hours) to get the required number of meta-jobs (should be rounded up to the next larger integer number).

With the above choices, the queue wait time should be fairly small, and the throughput and efficiency of the farm should be fairly high.

Let's consider a specific example. You ran the above "for loop" command on a dedicated (obtained via salloc command) CPU-core, and estimated that the average individual case runtime is 95 seconds. The total number of cases to process (number of lines in table.dat) is say 1000. The total amount of resources required to compute all your cases is then 95s x 1000 = 95,000 core-seconds = 26.4 core-hours. You choose a short runtime of 3 hours for your meta-jobs to ensure the shortest possible queue wait time (use it as the "#SBATCH -t " argument in your job_script.sh file). Now you can estimate how many meta-jobs you'll need to process all the cases: N = 26.4 core-hours / 3 hours = 8.8. You multiply that by a factor of 1.2 as a safety measure (10.56), and then round it up to the next larger integer - 11. That means you can process the whole farm by executing a single command "submit.run 11".

For particularly large farms, if the number of jobs in the above analysis is larger than 1000 (the maximum number of jobs which can be submitted on Graham and Beluga), the workaround would be to go through the sequence of commands (each command can only be executed after the previous farm has finished running):

   $  submit.run 1000
   $  resubmit.run 1000
   $  resubmit.run 1000
   ...

As this can get rather tedious, one should instead consider using an advanced feature of the META package - #Running resubmit.run automatically - for such large farms. This will fully automate the farm resubmission steps.

Reducing the waste

Here is one potential problem when one is running multiple cases per job, utilizing dynamic workload balancing: what if the number of running meta-jobs times the requested runtime per meta-job (say, 3 days) is not enough to process all your cases? E.g., you managed to start the maximum allowed 1000 meta-jobs, each of which has a 3 day runtime limit. That means that your serial farm can only process all the cases in a single run if the average_case_runtime x N_cases < 1000 x 3d = 3000 cpu days. Once your meta-jobs start hitting the 3d runtime limit, they will start dying in the middle of processing one of your cases. This will result in up to 1000 interrupted cases calculations. This is not a big deal in terms of accounting (the "resubmit.run" will find all the cases which failed or never ran, and will resubmit them automatically). But this can become a waste of cpu cycles. On average, you will be wasting 0.5 x N_jobs x average_case_runtime_hours cpu-hours. E.g. if your cases have an average runtime of 1 hour, and you have 1000 meta-jobs running, you will waste ~20 cpu days, which is not acceptable.

Fortunately, the scripts we are providing have some built-in intelligence to mitigate this problem. This is implemented in the "task.run" script as follows:

• The script measures runtime of each case, and adds the value as one line in a scratch file "times" created inside /home/$USER/tmp/NODE.PID directory (see #Output files). This is done by all running meta-jobs.
• Once the first 8 cases were computed, one of the meta-jobs will read the contents of the file "times" and compute the larger 12.5% quantile for the current distribution of case runtimes. This will serve as a conservative estimate of the runtime for your individual cases, dt_cutoff. (The current estimate is stored in dt_cutoff file inside the /home/$USER/tmp/NODE.PID directory.)
• From now on, each meta-job will estimate if it has the time to finish the case it is about to start computing, by ensuring that t_finish - t_now > dt_cutoff. (Here t_finish is the time when the job will die because of the job's runtime limit; t_now is the current time.) If it thinks it doesn't have the time, it will exit early, which will minimize the chance of a case computation aborted half-way due to the job's runtime limit.
• At every subsequent power of two number of computed cases (8, then 16, then 32 and so on) dt_cutoff is recomputed using the above algorithm. This will make the dt_cutoff estimate more and more accurate. Power of two is used to minimize the overheads related to computing dt_cutoff; the algorithm will be equally efficient for both very small (tens) and very large (many thousands) number of cases.
• The above algorithm reduces the amount of cpu cycles wasted due to jobs hitting the runtime limit by a factor of 8.

As a useful side effect, every time you run a farm, you get individual runtimes for all of your cases (stored in /home/$USER/tmp/NODE.PID/times file). You can analyze that file to fine-tune your farm setup, for profiling your code etc.

Advanced features

Running resubmit.run automatically

If your farm is particularly large (needs more resources than NJOBS_MAX * job_runtime, where NJOBS_MAX is the maximum number of jobs one is allowed to submit), you will have to run resubmit.run after the original farm finished running (perhaps more than once). You can do it by hand, but with META you can also fully automate this process. To enable this feature, you have to add "-auto" switch to your submit.run (or resubmit.run) command:

$ submit.run N -auto

This can be used in both SIMPLE and META modes. If your original submit.run command did not have the "-auto" switch, you can add it to the resubmit.run command, after the original farm finished running, to the same effect.

When you add the "-auto" switch, the (re)submit.run script submits one more (serial) job, in addition to the farm jobs. (This should be accommodated for in the value of the NJOBS_MAX parameter defined in config.h file. E.g. if the largest number of jobs one can submit on the cluster is 999, set this parameter to 998 when using the "-auto" feature.) The purpose of this job is to run the resubmit.run command automatically right after the current farm finished running. The job script for this additional job is resubmit_script.sh (should be present in the root farm directory; a sample file is automatically copied there when you run farm_init.run command). The only customization you need to do to this file is to put the correct account name as the "#SBATCH -A" argument.

To prevent a risk of creating an infinite loop when using the "-auto" feature, if at some point the only cases left to be processed are the ones which failed earlier, auto-resubmission will stop, and farm computations will end. (You can see the relevant messages in the file farm.log created in the root farm directory.) If this happens, you will have to address the reasons for these cases failing before attempting to resubmit the farm.

Running a post-processing job automatically

Another advanced feature is the ability to run a post-processing job automatically once all the cases from table.dat have been successfully processed. (If there are any cases which failed - had a non-zero exit status - the post-processing job will not run.) To enable this feature, simply create a job script file for the post-processing job with the name "final.sh" inside the root farm directory. This job can be of any kind - serial, parallel, or an array job.

This feature, similarly to #Running resubmit.run automatically feature, uses one additional serial job, described by the file resubmit_script.sh in the root farm directory (make sure it uses the correct account name). Adjust the parameter NJOBS_MAX defined in config.h file accordingly (e.g. if the cluster has a job limit of 999, set it to 998).

You can use both auto-resubmit and auto-post-processing features together, and that will still only use one serial job (in addition to your farm jobs).

The system messages from the auto-resubmission feature are logged into farm.log file, in the root farm directory.

Additional information

Passing additional sbatch arguments

What if you need to use additional sbatch arguments (like --mem 4G, --gres=gpu:1 etc.)? Simple: just add all those arguments at the end of “submit.run” and “resubmit.run” command line, and they will be passed to sbatch, e.g.:

   $  submit.run  -1  --mem 4G

Alternatively, you can supply these arguments as separate "#SBATCH" lines in your job_script.sh file.

Multi-threaded farming

For “multi-threaded farming” (OpenMP etc.), add "--cpus-per-task=N" and "--mem=XXX" sbatch arguments to “(re)submit.run” (or add the corresponding #SBATCH lines to your job_script.sh file). Here “N” is the number of cpu cores/threads to use. Also, add the following line inside your “job_script.sh” file right before the task.run line:

   export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

MPI farming

For “MPI farming”, use these sbatch arguments with “(re)submit.run” (or add the corresponding #SBATCH lines to your job_script.sh file):

   --ntasks=N  --mem-per-cpu=XXX

Also add “srun” before the path to your code inside “single_case.sh”, e.g.:

   srun  $COMM

Alternatively, you can prepend “srun” on each line of your table.dat:

   srun /path/to/mpi_code arg1 arg2
   srun /path/to/mpi_code arg1 arg2
   ...
   srun /path/to/mpi_code arg1 arg2

GPU farming

For GPU farming, you only need to modify your job_script.sh file accordingly. For example, for farming where the code uses one GPU add one extra line:

#SBATCH --gres=gpu:1

It is also a good idea to copy my utility ~syam/bin/gpu_test to your ~/bin directory (only on graham, cedar, and beluga), and put the following lines in your job_script.sh file right before the "task.run" line:

gpu_test
retVal=$?
if [ $retVal -ne 0 ]; then
    echo "No GPU found - exiting..."
    exit 1
fi

This will catch those quite rare situations when there is a technical issue with the node rendering the GPU being not available. If that happens to one of your meta-jobs, and you don't have the above lines in your script, the rogue meta-job will churn through (and fail) all your cases from table.dat.

FORTRAN code example: using standard input

You have a FORTRAN (or C/C++) serial code, “fcode”; each case needs to read a separate file from standard input – say “data.xxx” (in /home/user/IC directory), where xxx goes from 1 to N_cases. Place “fcode” on your $PATH (e.g., in ~/bin, make sure /home/$USER/bin is added to $PATH in .bashrc; alternatively, use a full path to your code in the cases table). Create table.dat (inside META directory) like this:

  fcode < /home/user/IC/data.1
  fcode < /home/user/IC/data.2
  ...
  fcode < /home/user/IC/data.N_cases

The task of creating the table can be greatly simplified if you use a BASH loop command, e.g.:

   $  for ((i=1; i<=10; i++)); do echo "fcode < /home/user/IC/data.$i"; done >table.dat

FORTRAN code example: copying files for each case

Another typical FORTRAN code situation: you need to copy a file (say, /path/to/data.xxx) to each case subdirectory, before executing the code, and rename it to some standard input file name. Your table.dat can look like this:

  /path/to/code
  /path/to/code
  ...

Add one line (first line in the example below) to your “single_case.sh”:

   \cp /path/to/data.$ID standard_name
   $COMM
   STATUS=$?

Using all the columns in the cases table explicitly

The examples shown so far presume that each line in the cases table is an executable statement, starting with either the code binary name (when the binary is on your $PATH) or full path to the binary, and then listing the code's command line arguments (if any) particular to that case, or something like " < input.$ID" if your code expects the initial conditions via standard input.

In the most general case, one wants to have the ultimate flexibility in being able to access all the columns in the table individually. That is easy to achieve by slightly modifying the "single_case.sh" script:

...
# ++++++++++++  This part can be customized:  ++++++++++++++++
#  $ID contains the case id from the original table
#  $COMM is the line corresponding to the case $ID in the original table, without the ID field
mkdir RUN$ID
cd RUN$ID

# Converting $COMM to an array:
COMM=( $COMM )
# Number of columns in COMM:
Ncol=${#COMM[@]}
# Now one can access the columns individually, as ${COMM[i]} , where i=0...$Ncol-1
# A range of columns can be accessed as ${COMM[@]:i:n} , where i is the first column
# to display, and n is the number of columns to display
# Use the ${COMM[@]:i} syntax to display all the columns starting from the i-th column
# (use for codes with a variable number of command line arguments).

# Call the user code here.
...

# Exit status of the code:
STATUS=$?
cd ..
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...

For example, you need to provide to your code both an initial conditions file (to be used via standard input), and a variable number of command line arguments. Your cases table will look like this:

  /path/to/IC.1 0.1
  /path/to/IC.2 0.2 10
  ...

The way to implement this in "single_case.sh" is as follows:

# Call the user code here.
/path/to/code ${COMM[@]:1} < ${COMM[0]}

Troubleshooting

Here we explain typical error messages you might get when using this package.

Problems affecting multiple commands

"Non-farm directory, or no farm has been submitted; exiting"

Either the current directory is not a farm directory, or you never ran the "submit.run" command for this farm.

Problems with submit.run

Wrong first argument: XXX (should be a positive integer or -1) ; exiting

Use the correct first argument: -1 for the SIMPLE mode, or a positive integer N (number of requested meta-jobs) for the META mode.

"lockfile is not on path; exiting"

Make sure the utility lockfile is on your $PATH.

"Non-farm directory (config.h, job_script.sh, single_case.sh, and/or table.dat are missing); exiting"

Either the current directory is not a farm directory, or some important files are missing. Cd to the correct (farm) directory, or create the missing files.

"-auto option requires resubmit_script.sh file in the root farm directory; exiting"

You used the "-auto" option, but you forgot to create the resubmit_script.sh file inside the root farm directory. A sample resubmit_script.sh file is created automatically when you use farm_init.run.

"File table.dat doesn't exist. Exiting"

You forgot to create the table.dat file in the current directory, or perhaps you are running submit.run not inside one of your farm sub-directories.

"Job runtime sbatch argument (-t or --time) is missing in job_script.sh. Exiting"

Make sure you provide the runtime for all meta-jobs as an #SBATCH argument inside your job_script.sh file. This is a requirement - the runtime sbatch argument is the only one which cannot be passed as an optional argument for submit.run.

"Wrong job runtime in job_script.sh - nnn . Exiting"

You didn't format properly the runtime argument inside your job_script.sh file.

"Something wrong with a auto-resubmit job submission; jobid=XXX; aborting"

With either of the two messages, there was an issue with submitting jobs with sbatch. The cluster's scheduler might be misbehaving, or simply too busy. Try again a bit later.

"Couldn't create a file inside XXX ; exiting"

With any of the above messages, something is wrong with a file system - either permissions got messed up, or you hit the file system quota. Fix the issue(s), then try again.

Problems with resubmit.run

"Jobs are still running/queued; cannot resubmit"

You cannot use resubmit.run until all meta-jobs from this farm finished running. Use list.run or queue.run to check the status of the farm.

"No failed/unfinished jobs; nothing to resubmit"

Not an error - simply tells you that your farm was 100% processed, and that there are no more (failed or never-ran) cases to compute.

Problems with running jobs

"Too many failed (very short) cases - exiting"

This happens if the first $N_failed_max (5 by default) cases are very short - less than $dt_failed (5 by default) seconds in duration. The two variables, $N_failed_max and $dt_failed, can be adjusted by editing the config.h file (in the root farm directory). This is a protection mechanism, in case anything is amiss - a problem with the node (file system not mounted, GPU is missing etc), with the job parameters (not enough of RAM etc), or with the code (the binary is missing or instantly crashing, input files are missing etc.). This protection will prevent the bad meta-job churning through (and failing) all the cases in table.dat.

"lockfile is not on path on node XXX"

As the error message suggests, somehow the utility lockfile is not on your $PATH - either you forgot to modify your $PATH variable accordingly, to copy lockfile into your ~/bin directory, or perhaps something is wrong on that particular compute node (home file system not mounted). The lockfile utility is critical for this package (it ensures serialized access of meta-jobs to the table.dat file), and it won't work if the utility is not accessible.

"Exiting after processing one case (-1 option)"

This is actually not an error - it simply tells you that you submitted the farm with via "submit.run -1" (one case per job mode), so each meta-job is exiting after processing a single case.

"Not enough runtime left; exiting."

This message tells you that the meta-job would likely not have enough time left to process the next case (based on the analysis of runtimes for all the cases processed so far), so it is exiting earlier.

"No cases left; exiting."

This is not an error message - this is how each meta-job normally finishes, when all cases have already been computed.

"Only failed cases left; cannot auto-resubmit; exiting"

This can only happen if you used the "-auto" switch when submitting the farm. Find the failed cases ("Status.run -f"), fix the issue(s) causing the cases to fail, then run resubmit.run command to process all those cases.

Words of caution

Always start with a much smaller test farm run, to make sure everything works, before submitting a large production run farm. You can test individual cases by reserving an interactive node with "salloc" command, cd'ing to the farm directory, and executing commands like "./single_case.sh table.dat 1", "./single_case.sh table.dat 2" etc.

If your farm is particularly large (say > 10,000 cases), extra efforts have to be spent to make sure it runs as efficiently as possible. In particular, you have to minimize the number of files and/or directories created during the jobs execution. If possible, instruct your code to append to the existing files (one per meta-job; do not mix results from different meta-jobs in a single output file!) instead of creating a separate file for each case. Avoid creating a separate subdirectory for each case (which is the default setup of this package).

The following example (optimized for a very large number of cases) assumes that your code accepts the output file name via "-o" command line switch, that the output file is used in "append" mode (multiple code runs will keep appending to the existing file), and that each line of table.dat provides the rest of the command line arguments for your code. It is also assumed that multiple instances of your code can safely run concurrently inside the same directory (so no need to create subdirectories for each case), and that each code run will not produce any other files (beside the output file). With this setup, even very large farms (hundreds of thousands or even millions of cases) should run efficiently, as there will be very few files created.

...
# ++++++++++++++++++++++  This part can be customized:  ++++++++++++++++++++++++
#  Here:
#  $ID contains the case id from the original table (can be used to provide a unique seed to the code etc)
#  $COMM is the line corresponding to the case $ID in the original table, without the ID field
#  $METAJOB_ID is the jobid for the current meta-job (convenient for creating per-job files)

# Executing the command (a line from table.dat)
/path/to/your/code  $COMM  -o output.$METAJOB_ID

# Exit status of the code:
STATUS=$?
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...

Glossary

• Case : one independent computation. The file table.dat should list one "case" per line.
• Farm / Farming : running many jobs on a cluster which carry out independent (but related) computations, of the same kind. "Farms" can be serial, multi-threaded, MPI, GPU etc.
• Meta-job : a job which can process multiple cases (independent computations) from table.dat.
• META mode : the mode of operating the META package when each job is a meta-job (can process multiple cases from table.dat).
• SIMPLE mode : the mode of operating the META package when each job will process only one case from table.dat.

If more help is needed

Submit a ticket to Compute Canada ticketing system (by sending an email to support@computecanada.ca), mentioning the name of the package (META), and the name of the staff who wrote the software (Sergey Mashchenko).

Message Passing Interface