Alliance Doc

GROMACS

Revision as of 17:02, 2 December 2022 by Stuekero (talk | contribs) (→‎Using Python)
Other languages:

General

GROMACS is a versatile package to perform molecular dynamics for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.

Strengths

  • GROMACS provides extremely high performance compared to all other programs.
  • Since GROMACS 4.6, we have excellent CUDA-based GPU acceleration on GPUs that have Nvidia compute capability >= 2.0 (e.g. Fermi or later).
  • GROMACS comes with a large selection of flexible tools for trajectory analysis.
  • GROMACS can be run in parallel, using either the standard MPI communication protocol, or via our own "Thread MPI" library for single-node workstations.
  • GROMACS is Free Software, available under the GNU Lesser General Public License (LGPL), version 2.1.

Weak points

  • To get very high simulation speed, GROMACS does not do much additional analysis and / or data collection on the fly. It may be a challenge to obtain somewhat non-standard information about the simulated system from a GROMACS simulation.
  • Different versions may have significant differences in simulation methods and default parameters. Reproducing results of older versions with a newer version may not be straightforward.
  • Additional tools and utilities that come with GROMACS are not always of the highest quality, may contain bugs and may implement poorly documented methods. Reconfirming the results of such tools with independent methods is always a good idea.

GPU support

The top part of any log file will describe the configuration, and in particular whether your version has GPU support compiled in. GROMACS will automatically use any GPUs it finds.

GROMACS uses both CPUs and GPUs; it relies on a reasonable balance between CPU and GPU performance.

The new neighbor structure required the introduction of a new variable called "cutoff-scheme" in the mdp file. The behaviour of older GROMACS versions (before 4.6) corresponds to cutoff-scheme = group, while in order to use GPU acceleration you must change it to cutoff-scheme = verlet, which has become the new default in version 5.0.

Quickstart guide

This section summarizes configuration details.

Environment modules

The following versions have been installed:

GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2022.3 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2022.3 GCC, FlexiBLAS & FFTW
gromacs/2022.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.2 GCC, FlexiBLAS & FFTW
gromacs/2021.6 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2021.6 GCC, FlexiBLAS & FFTW
gromacs/2021.4 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.4 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2021.4 warning GCC, FlexiBLAS & FFTW
gromacs/2021.2 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2021.2 warning GCC, FlexiBLAS & FFTW
gromacs/2021.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.2 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs/2021.2 warning GCC & MKL
gromacs/2020.6 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2020.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2020.6 warning GCC, FlexiBLAS & FFTW
gromacs/2020.4 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2020.4 warning GCC, FlexiBLAS & FFTW
gromacs/2020.4 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2020.4 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs/2020.4 warning GCC & MKL
GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2020.2 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2020.2 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2020.2 warning GCC & MKL
gromacs/2019.6 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2019.6 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2019.6 GCC & MKL
gromacs/2019.3 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2019.3 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2019.3 GCC & MKL  ‡
gromacs/2018.7 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2018.7 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2018.7 GCC & MKL
GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2018.3 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2018.3 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2018.3 GCC & FFTW
gromacs/2018.2 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2018.2 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2018.2 GCC & FFTW
gromacs/2018.1 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2018.1 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2018.1 GCC & FFTW
gromacs/2018 StdEnv/2016.4 gromacs/2018 StdEnv/2016.4 cuda/9.0.176 gromacs/2018 Intel & MKL
gromacs/2016.5 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2016.5 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2016.5 GCC & FFTW
gromacs/2016.3 StdEnv/2016.4 gromacs/2016.3 StdEnv/2016.4 cuda/8.0.44 gromacs/2016.3 Intel & MKL
gromacs/5.1.5 StdEnv/2016.4 gromacs/5.1.5 StdEnv/2016.4 cuda/8.0.44 gromacs/5.1.5 Intel & MKL
gromacs/5.1.4 StdEnv/2016.4 gromacs/5.1.4 StdEnv/2016.4 cuda/8.0.44 gromacs/5.1.4 Intel & MKL
gromacs/5.0.7 StdEnv/2016.4 gromacs/5.0.7 StdEnv/2016.4 cuda/8.0.44 gromacs/5.0.7 Intel & MKL
gromacs/4.6.7 StdEnv/2016.4 gromacs/4.6.7 StdEnv/2016.4 cuda/8.0.44 gromacs/4.6.7 Intel & MKL
gromacs/4.6.7 StdEnv/2016.4 gcc/5.4.0 openmpi/2.1.1 gromacs/4.6.7 StdEnv/2016.4 gcc/5.4.0 cuda/8.0 openmpi/2.1.1 gromacs/4.6.7 GCC & MKL & ThreadMPI

Notes:

  • warning GROMACS versions 2020.0 up to and including 2021.5 contain a bug when used on GPUs of Volta or newer generations (i.e. V100, T4 and A100) with mdrun option -update gpu that could have perturbed the virial calculation and, in turn, led to incorrect pressure coupling. The GROMACS developers state in the 2021.6 Release Notes:[1]

    The GPU update is not enabled by default, so the error can only appear in simulations where it [the -update gpu option] was manually selected, and even in this case the error might be rare since we have not observed it in practice in the testing we have performed.

    Further discussion of this bug can be found in the GitLab issue #4393 of the GROMACS project.[2]
  • Version 2020.4 and newer have been compiled for the new Standard software environment StdEnv/2020.
  • Version 2018.7 and newer have been compiled with GCC compilers and the MKL-library, as they run a bit faster.
  • Older versions have been compiled with either with GCC compilers and FFTW or Intel compilers, using Intel MKL and Open MPI 2.1.1 libraries from the default environment as indicated in the table above.
  • CPU (non-GPU) versions are available in both single- and double precision, with the exception of 2019.3 (‡), where double precision not available for AVX512.

These modules can be loaded by using a module load command with the modules as stated in the second column in above table. For example: 

$ module load  StdEnv/2020  gcc/9.3.0  openmpi/4.0.3  gromacs/2021.2
or 
$ module load  StdEnv/2018.3  gcc/7.3.0 openmpi/3.1.2 gromacs/2020.2

These versions are also available with GPU support, albeit only with single precision. In order to load the GPU enabled version, the cuda module needs to be loaded first. The modules needed are listed in the third column of above table, e.g.: 

$ module load  StdEnv/2020  gcc/9.3.0  cuda/11.0  openmpi/4.0.3  gromacs/2021.2 
or
$ module load  StdEnv/2018.3  gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2  gromacs/2020.2

For more information on environment modules, please refer to the Using modules page.

Suffixes

GROMACS 5.x, 2016.x and newer

GROMACS 5 and newer releases consist of only four binaries that contain the full functionality. All GROMACS tools from previous versions have been implemented as sub-commands of the gmx binaries. Please refer to GROMACS 5.0 Tool Changes and the GROMACS documentation manuals for your version.

  • gmx - mixed ("single") precision GROMACS with OpenMP (threading) but without MPI.
  • gmx_mpi - mixed ("single") precision GROMACS with OpenMP and MPI.
  • gmx_d - double precision GROMACS with OpenMP but without MPI.
  • gmx_mpi_d - double precision GROMACS with OpenMP and MPI.

GROMACS 4.6.7

  • The double precision binaries have the suffix _d.
  • The parallel single and double precision mdrun binaries are:
  • mdrun_mpi
  • mdrun_mpi_d


Submission scripts

Please refer to the page Running jobs for help on using the SLURM workload manager.

Serial jobs

Here's a simple job script for serial mdrun:


File : serial_gromacs_job.sh

#!/bin/bash
#SBATCH --time=0-0:30         # time limit (D-HH:MM)
#SBATCH --mem-per-cpu=1000M   # memory per CPU (in MB)
module purge  
module load  StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
gmx mdrun -nt 1 -deffnm em


This will run the simulation of the molecular system in the file em.tpr.

Whole nodes

Commonly the systems which are being simulated with GROMACS are so large, that you want to use a number of whole nodes for the simulation.

Generally the product of --ntasks-per-node= and --cpus-per-task has to match the number of CPU-cores in the compute-nodes of the cluster. Please see section Performance Considerations below.

File : gromacs_whole_node_graham.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=8      # request 8 MPI tasks per node
#SBATCH --cpus-per-task=4        # 4 OpenMP threads per MPI task => total: 8 x 4 = 32 CPUs/node
#SBATCH --mem=0                  # request all available memory on the node
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge  
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_cedar.sh

#!/bin/bash
#SBATCH --nodes=1                        # number of nodes
#SBATCH --ntasks-per-node=12             # request 12 MPI tasks per node
#SBATCH --cpus-per-task=4                # 4 OpenMP threads per MPI task => total: 12 x 4 = 48 CPUs/node
#SBATCH --constraint="[skylake|cascade]" # restrict to AVX512 capable nodes.
#SBATCH --mem=0                          # request all available memory on the node
#SBATCH --time=0-01:00                   # time limit (D-HH:MM)
module purge
module load arch/avx512   # switch architecture for up to 30% speedup
module load StdEnv/2020  gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_beluga.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=10     # request 10 MPI tasks per node
#SBATCH --cpus-per-task=4        # 4 OpenMP threads per MPI task => total: 10 x 4 = 40 CPUs/node
#SBATCH --mem=0                  # request all available memory on the node
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge  
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_niagara.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=10     # request 10 MPI tasks per node
#SBATCH --cpus-per-task=4        # 4 OpenMP threads per MPI task => total: 10 x 4 = 40 CPUs/node
#SBATCH --mem=0                  # request all available memory on the node
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge --force
module load CCEnv
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun gmx_mpi mdrun -deffnm md

GPU job

This is a job script for mdrun using 4 OpenMP threads and one GPU:

File : gpu_gromacs_job.sh

#!/bin/bash
#SBATCH --gres=gpu:p100:1        # request 1 GPU as "generic resource"
#SBATCH --cpus-per-task=4        # number of OpenMP threads per MPI process
#SBATCH --mem-per-cpu=1000       # memory limit per CPU core (megabytes)
#SBATCH --time=0:30:00           # time limit (D-HH:MM:ss)
module purge  
module load StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

gmx mdrun -ntomp ${SLURM_CPUS_PER_TASK:-1} -deffnm md


GPU job - whole node

These are job scripts for mdrun using all GPUs and CPUs within a GPU node.

Note that using more than a single GPU usually leads to poor efficiency. Carefully test and compare multi-GPU and single-GPU performance before deciding to use more than one GPU for your simulations.

File : gromacs_job_GPU_MPI_Graham.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --gres=gpu:p100:2        # request 2 GPUs per node (Graham)
#SBATCH --ntasks-per-node=4      # request 4 MPI tasks per node
#SBATCH --cpus-per-task=8        # 8 OpenMP threads per MPI process
#SBATCH --mem=0                  # Request all available memory in the node
#SBATCH --time=1:00:00           # time limit (D-HH:MM:ss)
module purge  
module load StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

mpiexec gmx_mpi mdrun -deffnm md
File : gromacs_job_GPU_MPI_Cedar.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --gres=gpu:p100:4        # request 4 GPUs per node (Cedar)
#SBATCH --ntasks-per-node=4      # request 4 MPI tasks per node
#SBATCH --cpus-per-task=6        # 6 OpenMP threads per MPI process
#SBATCH --mem=0                  # Request all available memory in the node
#SBATCH --time=1:00:00           # time limit (D-HH:MM:ss)
module purge  
module load StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

mpiexec gmx_mpi mdrun -deffnm md
File : gromacs_job_GPU_MPI_Beluga.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --gres=gpu:v100:4        # request 4 GPUs per node (Beluga)
#SBATCH --ntasks-per-node=8      # request 8 MPI tasks per node
#SBATCH --cpus-per-task=5        # 5 OpenMP threads per MPI process
#SBATCH --mem=0                  # Request all available memory in the node
#SBATCH --time=1:00:00           # time limit (D-HH:MM:ss)
module purge  
module load StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2022.3
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun gmx_mpi mdrun -deffnm md
Notes on running GROMACS in GPUs
  • warning GROMACS versions 2020.0 up to and including 2021.5 contain a bug when used on GPUs of Volta or newer generations (i.e. V100, T4 and A100) with mdrun option -update gpu that could have perturbed the virial calculation and, in turn, led to incorrect pressure coupling. The GROMACS developers state in the 2021.6 Release Notes:[3]

    The GPU update is not enabled by default, so the error can only appear in simulations where it was manually selected, and even in this case the error might be rare since we have not observed it in practice in the testing we have performed.

    Further discussion of this bug can be found in the GitLab issue #4393 of the GROMACS project.[4]
  • Our clusters (Beluga, Cedar, Graham and Narval) have differently configured GPU nodes:
  • Cedar has 4 GPUs and 24 CPU cores per node
  • Graham has 2 GPUs and 32 CPU cores per node
Therefore one needs to use different settings to make use of all GPUs and CPU-cores in a node.
  • Cedar: --gres=gpu:p100:4 --ntasks-per-node=4 --cpus-per-task=6
  • Graham: --gres=gpu:p100:2 --ntasks-per-node=4 --cpus-per-task=8
Of course the simulated system needs to be large enough to utilize the resources.
  • GROMACS imposes a number of constraints for choosing number of GPUs, tasks (MPI ranks) and OpenMP threads.
    For GROMACS 2018.2 the constraints are:
  • The number of --tasks-per-node always needs to be a multiple of the number of GPUs (--gres=gpu:)
  • GROMACS will not run GPU runs with only 1 OpenMP thread, unless forced by setting the -ntomp option.
    According to GROMACS developers, the optimum number of --cpus-per-task is between 2 and 6.
  • Avoid using a larger fraction of CPUs and memory than the fraction of GPUs you have requested in a node.
  • While according to the developers of the SLURM scheduler using srun as a replacement for mpiexec/mpirun is the preferred way to start MPI jobs, we have seen evidence of jobs failing on startup, when two jobs using srun are started on the same compute node.
    At this time we therefore recommend to use mpiexec, especially when utilizing only partial nodes.

Usage

More content for this section will be added at a later time.

System preparation

In order to run a simulation, one needs to create a tpr file (portable binary run input file). This file contains the starting structure of the simulation, the molecular topology and all the simulation parameters.

Tpr files are created with the gmx grompp command (or simply grompp for versions older than 5.0). Therefore one needs the following files:

  • The coordinate file with the starting structure. GROMACS can read the starting structure from various file-formats, such as .gro, .pdb or .cpt (checkpoint).
  • The (system) topology (.top)) file. It defines which force-field is used and how the force-field parameters are applied to the simulated system. Often the topologies for individual parts of the simulated system (e.g. molecules) are placed in separate .itp files and included in the .top file using a #include directive.
  • The run-parameter (.mdp) file. See the GROMACSuser guide for a detailed description of the options.

Tpr files are portable, that is they can be grompp'ed on one machine, copied over to a different machine and used as an input file for mdrun. One should always use the same version for both grompp and mdrun. Although mdrun is able to use tpr files that have been created with an older version of grompp, this can lead to unexpected simulation results.

Running simulations

MD Simulations often take much longer than the maximum walltime for a job to complete and therefore need to be restarted. To minimize the time a job needs to wait before it starts, you should maximise the number of nodes you have access to by choosing a shorter running time for your job. Requesting a walltime of 24 hours or 72 hours (three days) is often a good trade-off between waiting- and running-time.

You should use the mdrun parameter -maxh to tell the program the requested walltime so that it gracefully finishes the current timestep when reaching 99% of this walltime. This causes mdrun to create a new checkpoint file at this final timestep and gives it the chance to properly close all output-files (trajectories, energy- and log-files, etc.).

For example use #SBATCH --time=24:00 along with gmx mdrun -maxh 24 ... or #SBATCH --time=3-00:00 along with gmx mdrun -maxh 72 ....


File : gromacs_job.sh

#!/bin/bash
#SBATCH --nodes=1                # number of Nodes
#SBATCH --tasks-per-node=32      # number of MPI processes per node
#SBATCH --mem-per-cpu=4000       # memory limit per CPU (megabytes)
#SBATCH --time=24:00:00          # time limit (D-HH:MM:ss)
module purge
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2020.4
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun  gmx_mpi  mdrun  -deffnm md  -maxh 24



Restarting simulations

You can restart a simulation by using the same mdrun command as the original simulation and adding the -cpi state.cpt parameter where state.cpt is the filename of the most recent checkpoint file. Mdrun will by default (since version 4.5) try to append to the existing files (trajectories, energy- and log-files, etc.). GROMACS will check the consistency of the output files and - if needed - discard timesteps that are newer than that of the checkpoint file.

Using the -maxh parameter ensures that the checkpoint and output files are written in a consistent state when the simulation reaches the time limit.

The GROMACS manual contains more detailed information [5] [6].


File : gromacs_job_restart.sh

#!/bin/bash
#SBATCH --nodes=1                # number of Nodes
#SBATCH --tasks-per-node=32      # number of MPI processes per node
#SBATCH --mem-per-cpu=4000       # memory limit per CPU (megabytes)
#SBATCH --time=24:00:00          # time limit (D-HH:MM:ss)
module purge
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2020.4
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun  gmx_mpi  mdrun  -deffnm md  -maxh 24.0  -cpi md.cpt


Performance considerations

Getting the best mdrun performance with GROMACS is not a straightforward task. The GROMACS developers are maintaining a long section in their user-guide dedicated to mdrun-performance[7] which explains all relevant options/parameters and strategies.

There is no "One size fits all", but the best parameters to choose highly depend on the size of the system (number of particles as well as size and shape of the simulation box) and the simulation parameters (cut-offs, use of Particle-Mesh-Ewald[8] (PME) method for long-range electrostatics).

GROMACS prints performance information and statistics at the end of the md.log file, which is helpful in identifying bottlenecks. This section often contains notes on how to further improve the performance.

The simulation performance is typically quantified by the number of nanoseconds of MD-trajectory that can be simulated within a day (ns/day).

Parallel scaling is a measure how effectively the compute resources are used. It is defined as:

S = pN / ( N * p1 )

Where pN is the performance using N CPU cores.

Ideally, the performance increases linearly with the number of CPU cores ("linear scaling"; S = 1).


MPI processes / Slurm tasks / Domain decomposition

The most straight-forward way to increase the number of MPI processes (called MPI-ranks in the GROMACS documentation), which is done by using Slurm's --ntasks or --ntasks-per-node in the job script.

GROMACS uses Domain Decomposition[8] (DD) to distribute the work of solving the non-bonded Particle-Particle (PP) interactions across multiple CPU cores. This is done by effectively cutting the simulation box along the X, Y and/or Z axes into domains and assigning each domain to one MPI process.

This works well until the time needed for communication becomes large in respect to the size (in respect of number of particles as well as volume) of the domain. In that case the parallel scaling will drop significantly below 1 and in extreme cases the performance drops when increasing the number of domains.

GROMACS can use Dynamic Load Balancing to shift the boundaries between domains to some extent, in order to avoid certain domains taking significantly longer to solve than others. The mdrun parameter -dlb auto is the default.

Domains cannot be smaller in any direction, than the longest cut-off radius.


Long-range interactions with PME

The Particle-Mesh-Ewald method (PME) is often used to calculate the long-range non-bonded interactions (interactions beyond the cut-off radius). As PME requires global communication, the performance can degrade quickly when many MPI processes are involved that are calculating both the short-range (PP) as well as the long-range (PME) interactons. This is avoided by having dedicated MPI processes that only perform PME (PME-ranks).

GROMACS mdrun by default uses heuristics to dedicate a number of MPI processes to PME when the total number of MPI processes 12 or greater. The mdrun parameter -npme can be used to select the number of PME ranks manually.

In case there is a significant "Load Imbalance" between the PP and PME ranks (e.g. the PP ranks have more work per timestep than the PME ranks), one can shift work from the PP ranks to the PME ranks by increasing the cut-off radius. This will not effect the result, as the sum of short-range + long-range forces (or energies) will be the same for a given timestep. Mdrun will attemtp to do that automatically since version 4.6 unless the mdrun parameter -notunepme is used.

Since version 2018, PME can be offloaded to the GPU (see below) however the implementation as of version 2018.1 has still several limitations [9] among them that only a single GPU rank can be dedicated to PME.


OpenMP threads / CPUs-per-task

Once Domain Decomposition with MPI processes reaches the scaling limit (parallel scaling starts dropping), performance can be further improved by using OpenMP threads to spread the work of an MPI process (rank) over more than one CPU core. To use OpenMP threads, use Slurm's --cpus-per-task parameter in the job script and either set the OMP_NUM_THREADS variable with: export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}" (recommended) or the mdrun parameter -ntomp ${SLURM_CPUS_PER_TASK:-1}.

According to GROMACS developers, the optimum is usually between 2 and 6 OpenMP threads per MPI process (cpus-per-task). However for jobs running on a very large number of nodes it might be worth trying even larger number of cpus-per-task.

Especially for systems that don't use PME, we don't have to worry about a "PP-PME Load Imbalance". In those cases we can choose 2 or 4 ntasks-per-node and set cpus-per-task to a value that ntasks-per-node * cpus-per-task matches the number of CPU cores in a compute node.

CPU architecture

GROMACS uses optimised kernel functions to compute the real-space portion of short-range, non-bonded interactions. Kernel functions are available for a variety of SIMD instruction sets, such as AVX, AVX2, and AVX512. Kernel functions are chosen when compiling GROMACS, and should match the capabilities of the CPUs that will be used to run the simulations. This is done for you by the Compute Canada team: when you load a GROMACS module into your environment, an appropriate AVX/AVX2/AVX512 version is chosen depending on the architecture of the cluster. GROMACS reports what SIMD instruction set it supports in its log file, and will warn you if the selected kernel function is suboptimal.

However, certain clusters contain a mix of CPUs that have different levels of SIMD support. When that is the case, the smallest common denominator is used. For instance, if the cluster has Skylake (AVX/AVX2/AVX512) and Broadwell (AVX/AVX2) CPUs, as Cedar currently (May 2020) does, a version of GROMACS compiled for the AVX2 instruction set will be used. This means that you may end up with a suboptimal choice of kernel function, depending on which compute nodes the scheduler allocates for your job.

You can explicitly request nodes that support AVX512 with the --constraint="[cascade|skylake]" SLURM option on clusters that offer these node types. This will make sure that your job will be assigned to nodes based on either the "Cascade Lake" or the "Skylake" architecture (but not a mix of both types). If working on the command-line, make sure to not forget the quotation marks (") to protect the special characters [, | and ]. You can then explicitly request AVX512 software using module load arch/avx512 before loading any other module.

For example, a simple job script could look like the following:


File : gromacs_job_cedar_avx512.sh

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=48
#SBATCH --constraint="[skylake|cascade]"
#SBATCH --time=24:00:00
module load arch/avx512
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.2
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
srun gmx_mpi mdrun


In our measurements, going from AVX2 to AVX512 on Skylake or Cascade nodes resulted in a 20−30% performance increase. However, you should also consider that restricting yourself to only AVX512-capable nodes will result in longer wait times in the queue.

GPUs

Tips how to use GPUs efficiently will be added soon.

Analyzing results

GROMACS Tools

GROMACS contains a large number of tools that can be used for common tasks of post-processing and analysis. The GROMACS manual contains a list of available commands organized by topic as well as organized by name that give a short description and link to the corresponding command reference.

These commands will typically read the trajectory (in the XTC, TNG or TRR format) as well as a coordinate file (GRO, PDB, TPR, etc.) and write plots in the XVG format which can be used for inputs for the plotting tool Grace (module: grace/5.99.0; command xmgrace; Grace User Guide). As XVG files are simple text files, they can also be processed with scripts or imported into other spreadsheet programs.

VMD

VMD is a molecular visualization program for displaying, animating, and analyzing large biomolecular systems using 3-D graphics and built-in scripting. It can be used to visually inspect GROMACS trajectories and also offers a large number of built-in and external plugins for analysis. It can also be used in command-line mode.

Using Python

MDAnalysis and MDTraj are two Python packages that we provide as pre-compiled Python wheels. They can read and write trajectory- and coordinate-files of GROMACS (TRR and XTC) and many other MD-packages and also include a variety of commonly used analysis functions. MDAnalysis can also read topology information from GROMACS TPR files, though often not those created by the latest versions of GROMACS.

Both packages feature a versatile atom-selection language and expose the coordinates of the trajectories, which makes it very easy to write custom analysis tools that can tailored to a specific problem and integrate well with Python's data-science packages like NumPy, SciPy and Pandas, as well as plotting libraries like Matplotlib/Pyplot and Seaborn.

Common pitfalls

Related Modules

Gromacs-Plumed

PLUMED[10] is an open source library for free energy calculations in molecular systems which works together with some of the most popular molecular dynamics engines.

The gromacs-plumed modules are versions of GROMACS that have been patched with PLUMED's modifications, so that they can run meta-dynamics simulations.

GROMACS PLUMED modules for running on CPUs modules for running on GPUs (CUDA) Notes
v2022.3 v2.8.1 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2022.3 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2022.3 GCC, FlexiBLAS & FFTW
v2021.6 v2.7.4 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2021.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2021.6 GCC, FlexiBLAS & FFTW
v2021.4 v2.7.3 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2021.4 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2021.4 GCC, FlexiBLAS & FFTW
v2021.2 v2.7.1 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2021.2 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs-plumed/2021.2 GCC & MKL
v2019.6 v2.6.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2019.6 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs-plumed/2019.6 GCC & MKL
v2019.6 v2.5.4 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs-plumed/2019.6 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs-plumed/2019.6 GCC & MKL
v2019.5 v2.5.3 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs-plumed/2019.5 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs-plumed/2019.5 GCC & MKL
v2018.1 v2.4.2 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs-plumed/2018.1 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs-plumed/2018.1 GCC & FFTW
v2016.3 v2.3.2 StdEnv/2016.4 intel/2016.4 openmpi/2.1.1 gromacs-plumed/2016.3 StdEnv/2016.4 intel/2016.4 cuda/8.0.44 openmpi/2.1.1 gromacs-plumed/2016.3 Intel & MKL

Gromacs-Colvars

Colvars[11] is an software module for molecular simulation programs, which enhances the programs with enhanced capabilities of collective variables to apply biasing potentials, calculate potentials-of-mean-force (PMFs) along any set of variables, use enhanced sampling methods, such as Adaptive Biasing Force (ABF), metadynamics, steered MD and umbrella sampling.

Please refer to the Colvars Reference manual for GROMACS[12] and their publication[13] for more information.

The gromacs-colvars modules are versions of GROMACS that have been patched with Colvars' modifications, so that the enhanced collective variables can be used in simulations.

GROMACS Colvars modules for running on CPUs modules for running on GPUs (CUDA) Notes
v2020.6 2021-12-20 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-colvars/2020.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-colvars/2020.6 GCC, FlexiBLAS & FFTW

Gromacs-CP2K

CP2K[14] is a quantum chemistry and solid state physics software package. Since version 2022 GROMACS can be compiled with CP2K-support[15] to enable Hybrid Quantum-Classical simulations (QM/MM)[16].

The gromacs-cp2k modules are versions of GROMACS that have been compiled with CP2K QM/MM support.

Different from other GROMACS modules, these modules are only available for CPU calculations and not for GPUs (CUDA). Also the modules contain only MPI-enabled executables:

  • gmx_mpi - mixed precision GROMACS with OpenMP and MPI.
  • gmx_mpi_d - double precision GROMACS with OpenMP and MPI.
GROMACS CP2K modules for running on CPUs Notes
v2022.2 9.1 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-cp2k/2022.2 GCC, FlexiBLAS & FFTW

Here are links to various resources for running QM/MM simulations with this combination of GROMACS and CP2K:

  • Hybrid Quantum-Classical simulations (QM/MM) with CP2K interface in the GROMACS manual.
  • CP2K QM/MM Best Practices Guide by BioExcel.
  • QM/MM with GROMACS + CP2K Workshop material from BioExcel.
    This contains tutorial material for setting up and running QM/MM simulations as well as links to YouTube videos with theory lectures. This material was written to be used with HPC resources from the European Centre of Excellence for Computational Biomolecular Research (BioExcel), however only small adjustments are needed to use our HPC systems instead.
    Most notably the command gmx_cp2k needs to be replaced with either gmx_mpi (mixed precision) or gmx_mpi_d (double precision) and the jobscripts (which are also using Slurm), need to be adjusted as well.

G_MMPBSA

G_MMPBSA[17] is a tool that calculates components of binding energy using MM-PBSA method except the entropic term and energetic contribution of each residue to the binding using energy decomposition scheme.

Development of that tool seems to have stalled in April 2016 and no changes have been made since then. Therefore it is only compatible with Gromacs 5.1.x. For newer version of GROMACS consider using gmx_MMPBSA[18] instead (see below).

The version installed can be loaded with module load StdEnv/2016.4 gcc/5.4.0 g_mmpbsa/2016-04-19 which represent the most up-to-date version and consists of version 1.6 plus the change to make it compatible with Gromacs 5.1.x. The installed version has been compiled with gromacs/5.1.5 and apbs/1.3.

Please be aware that G_MMPBSA uses implicit solvents and there have been studies[19] that conclude that there are issues with the accuracy of these methods for calculating binding free energies.

gmx_MMPBSA

gmx_MMPBSA[18] is a tool based on AMBER's MMPBSA.py aiming to perform end-state free energy calculations with GROMACS files.

Other than the older G_MMPBSA[17], which is only compatible with older versions of GROMACS, gmx_MMPBSA can be used with current versions of GROMACS and AmberTools.

Please be aware that gmx_MMPBSA uses implicit solvents and there have been studies[19] that conclude that there are issues with the accuracy of these methods for calculating binding free energies.

Installing gmx_MMPBSA into a virtualenv

The following has been tested with a combination of gmx_MMPBSA 1.5.0.3, gromacs/2021.4 and ambertools/21. While this should work with other recent versions of GROMACS, currently AmberTools 21 is the only version that is expected to work.

1. Load required modules 

$ module purge
$ module load gcc/9.3.0 python/3.8  gromacs/2021.4
$ module load ambertools/21
$ virtualenv venv_gmxMMPBSA
$ source venv_gmxMMPBSA/bin/activate

2. install needed Python packages: 

$ pip install --no-index "numpy~=1.22.0" gmx_MMPBSA
$ python -m pip install git+https://github.com/ParmEd/ParmEd.git@16fb236

Please note that ParmEd version up to 3.4.3 contain a bug that was fixed in commit 16fb236 Until version greater than 3.4.3 has been released we need to use this unreleased version.

3. The Qt/PyQt module needs to be loaded after the virtualenv is ready: 

$ module load qt/5.15.2

4. Test if the main application works: 

$ gmx_MMPBSA -h
$ gmx_MMPBSA_test -ng -n 4

Fortunately, running the self-test is very quick, therefore it's permissible to run them on the login node.

Later when using gmx_MMPBSA in a job you need to load the modules and activate the virtualenv as follows: 

module purge
module load gcc/9.3.0 python/3.8 ambertools/21 gromacs/2021.4 qt/5.15.2
source venv_gmxMMPBSA/bin/activate

Links

Biomolecular simulation

References

  1. ↑ "Fix missing synchronization in CUDA update kernels" in GROMACS 2021.6 Release Notes [1]
  2. ↑ Issue #4393 in GROMACS Project on GitLab.com [2]
  3. ↑ "Fix missing synchronization in CUDA update kernels" in GROMACS 2021.6 Release Notes [3]
  4. ↑ Issue #4393 in GROMACS Project on GitLab.com [4]
  5. ↑ GROMACS User-Guide: Managing long simulations.
  6. ↑ GROMACS Manual page: gmx mdrun
  7. ↑ GROMACS User-Guide: Getting good performance from mdrun
  8. ↑ 8.0 8.1 GROMACS User-Guide: Performance background information
  9. ↑ GROMACS User-Guide: GPU accelerated calculation of PME
  10. ↑ PLUMED Home
  11. ↑ Colvars Home
  12. ↑ Colvars Reference manual for GROMACS
  13. ↑ Fiorin et al. 2013, Using collective variables to drive molecular dynamics simulations.
  14. ↑ CP2K Home
  15. ↑ Building GROMACS with CP2K QM/MM support
  16. ↑ QM/MM with CP2K in the GROMACS Reference manual
  17. ↑ 17.0 17.1 G_MMPBSA Homepage
  18. ↑ 18.0 18.1 gmx_MMPBSA Homepage
  19. ↑ 19.0 19.1 Comparison of Implicit and Explicit Solvent Models for the Calculation of Solvation Free Energy in Organic Solvents

Message Passing Interface

GNU Compiler Collection, an open source compiler collection

Intel Math Kernel Library, a software library of optimized math routines

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=GROMACS&oldid=124556"