Advanced MPI scheduling

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

Most users should submit MPI or distributed memory parallel jobs as illustrated at Running jobs: MPI job. Simply request a number of processes with --ntasks or -n and trust the scheduler to allocate those processes in a way that balances the efficiency of your job with the overall efficiency of the cluster.

If you need more detailed control over how your job is allocated, then read on to learn about SLURM's sbatch command and how its numerous options constrain the placement of processes.

sbatch options affecting process placement

TO DO: This unorganized list of options should be replaced by an organized discussion. If it makes sense to explain the terminology, "node", "task", "code", "socket", "cpu", "thread", etc., it should do so.

• -N, --nodes=
• -n, --ntasks=
• --ntasks-per-core=
• --ntasks-per-node=
• --ntasks-per-socket=
• --tasks-per-node=
• --threads-per-core=
• -c, --cpus-per-task=
• --mincpus=
• --cores-per-socket=
• --exclusive[=user|mcs]
• --spread-job
• --hint=[compute_bound|memory_bound|multithread|nomultithread]
• -m, --distribution=[arbitrary|<block|cyclic|plane]
• --mem_bind=

You should also understand how memory is controlled:

• --mem= and --mem-per-cpu=

Hybrid jobs: MPI and OpenMP, or MPI and threads

To come

MPI and GPUs

To come

Why srun instead of mpiexec or mpirun?

To come

External links

• sbatch documentation
• srun documentation
• Open MPI and SLURM

Message Passing Interface