GROMACS: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
(add GROMACS-2024.1-RAMD-2.1)
(Add new section GROMACS-LS)
Line 750: Line 750:
:* [https://github.com/bioexcel/2022-06-16-gromacs-cp2k-tutorial GitHub Repository] with example file for BioExcel Tutorial.
:* [https://github.com/bioexcel/2022-06-16-gromacs-cp2k-tutorial GitHub Repository] with example file for BioExcel Tutorial.
* [https://www.cp2k.org/tools:gromacs GROMACS-CP2K integration] on CP2K homepage.
* [https://www.cp2k.org/tools:gromacs GROMACS-CP2K integration] on CP2K homepage.
== Gromacs-LS ==
GROMACS-LS<ref name="Local_stress">[https://vanegaslab.org/software GROMACS-LS and MDStress library]</ref> and the MDStress library enable the calculation of local stress fields from molecular dynamics simulations.
The MDStress library is included in the GROMACS-LS module.
Please refer to manual for GROMACS-LS at: [https://vanegaslab.org/files/Local_stress.pdf Local_stress.pdf] and the publications listed therein for information about the method and how to use it.
Notes:
* Because the manual was written for the older GROMACS-LS v4.5.5 and that the core gromacs commands have changed in version 5, you need to use commands like <code>gmx_LS mdrun</code> and <code>gmx_LS trjconv</code> instead of <code>mdrun_LS</code> and <code>trjconv_LS</code>.
* GROMACS-LS requires to be compiled in double-precision does not support MPI, SIMD hardware acceleration nor GPUs and is therefore much slower that normal GROMACS. It can only use a single CPU core.
* Unlike other patched versions of GROMACS, the modules <code>gromacs-ls/2016.3</code> and <code>gromacs/2016.6</code> can be loaded at the same time.
{| class="wikitable sortable"
|-
! module || modules for running on CPUs !!  Notes
|-
| gromacs-ls/2016.3 || <code>StdEnv/2023  gcc/12.3  gromacs-ls/2016.3</code>  || GROMACS-LS is a serial application and does not support MPI, OpenMP or GPUs/CUDA.
|-
| gromacs/2016.6    || <code>StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2016.6</code>  || This Gromacs module can be used to prepare TPR input files for GROMACS-LS.
|}


== Gromacs-RAMD == <!--T:149-->
== Gromacs-RAMD == <!--T:149-->

Revision as of 18:44, 2 August 2024

Other languages:

General[edit]

GROMACS is a versatile package to perform molecular dynamics for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.

Strengths[edit]

  • GROMACS provides extremely high performance compared to all other programs.
  • Since GROMACS 4.6, we have excellent CUDA-based GPU acceleration on GPUs that have Nvidia compute capability >= 2.0 (e.g. Fermi or later).
  • GROMACS comes with a large selection of flexible tools for trajectory analysis.
  • GROMACS can be run in parallel, using either the standard MPI communication protocol, or via our own "Thread MPI" library for single-node workstations.
  • GROMACS is Free Software, available under the GNU Lesser General Public License (LGPL), version 2.1.

Weak points[edit]

  • To get very high simulation speed, GROMACS does not do much additional analysis and / or data collection on the fly. It may be a challenge to obtain somewhat non-standard information about the simulated system from a GROMACS simulation.
  • Different versions may have significant differences in simulation methods and default parameters. Reproducing results of older versions with a newer version may not be straightforward.
  • Additional tools and utilities that come with GROMACS are not always of the highest quality, may contain bugs and may implement poorly documented methods. Reconfirming the results of such tools with independent methods is always a good idea.

GPU support[edit]

The top part of any log file will describe the configuration, and in particular whether your version has GPU support compiled in. GROMACS will automatically use any GPUs it finds.

GROMACS uses both CPUs and GPUs; it relies on a reasonable balance between CPU and GPU performance.

The new neighbor structure required the introduction of a new variable called "cutoff-scheme" in the mdp file. The behaviour of older GROMACS versions (before 4.6) corresponds to cutoff-scheme = group, while in order to use GPU acceleration you must change it to cutoff-scheme = verlet, which has become the new default in version 5.0.

Quickstart guide[edit]

This section summarizes configuration details.

Environment modules[edit]

The following versions have been installed:

GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2024.1 StdEnv/2023 gcc/12.3 openmpi/4.1.5 gromacs/2024.1 StdEnv/2023 gcc/12.3 openmpi/4.1.5 cuda/12.2 gromacs/2024.1 GCC, FlexiBLAS & FFTW
gromacs/2023.3 StdEnv/2023 gcc/12.3 openmpi/4.1.5 gromacs/2023.3 StdEnv/2023 gcc/12.3 openmpi/4.1.5 cuda/12.2 gromacs/2023.3 GCC, FlexiBLAS & FFTW
GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2023.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2023.2 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2023.2 GCC, FlexiBLAS & FFTW
gromacs/2023 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2023 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2023 GCC, FlexiBLAS & FFTW
gromacs/2022.3 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.3 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2022.3 GCC, FlexiBLAS & FFTW
gromacs/2022.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2022.2 GCC, FlexiBLAS & FFTW
gromacs/2021.6 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2021.6 GCC, FlexiBLAS & FFTW
gromacs/2021.4 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.4 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2021.4 warning GCC, FlexiBLAS & FFTW
gromacs/2021.2 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2021.2 warning GCC, FlexiBLAS & FFTW
gromacs/2021.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2021.2 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs/2021.2 warning GCC & MKL
gromacs/2020.6 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2020.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2020.6 warning GCC, FlexiBLAS & FFTW
gromacs/2020.4 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs/2020.4 warning GCC, FlexiBLAS & FFTW
gromacs/2020.4 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2020.4 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs/2020.4 warning GCC & MKL


Deprecated

This software environment is no longer supported.



GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2020.2 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2020.2 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2020.2 warning GCC & MKL
gromacs/2019.6 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2019.6 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2019.6 GCC & MKL
gromacs/2019.3 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2019.3 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2019.3 GCC & MKL  ‡
gromacs/2018.7 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs/2018.7 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs/2018.7 GCC & MKL


Deprecated

This software environment is no longer supported.



GROMACS version modules for running on CPUs modules for running on GPUs (CUDA) Notes
gromacs/2018.3 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2018.3 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2018.3 GCC & FFTW
gromacs/2018.2 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2018.2 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2018.2 GCC & FFTW
gromacs/2018.1 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2018.1 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2018.1 GCC & FFTW
gromacs/2018 StdEnv/2016.4 gromacs/2018 StdEnv/2016.4 cuda/9.0.176 gromacs/2018 Intel & MKL
gromacs/2016.5 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs/2016.5 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs/2016.5 GCC & FFTW
gromacs/2016.3 StdEnv/2016.4 gromacs/2016.3 StdEnv/2016.4 cuda/8.0.44 gromacs/2016.3 Intel & MKL
gromacs/5.1.5 StdEnv/2016.4 gromacs/5.1.5 StdEnv/2016.4 cuda/8.0.44 gromacs/5.1.5 Intel & MKL
gromacs/5.1.4 StdEnv/2016.4 gromacs/5.1.4 StdEnv/2016.4 cuda/8.0.44 gromacs/5.1.4 Intel & MKL
gromacs/5.0.7 StdEnv/2016.4 gromacs/5.0.7 StdEnv/2016.4 cuda/8.0.44 gromacs/5.0.7 Intel & MKL
gromacs/4.6.7 StdEnv/2016.4 gromacs/4.6.7 StdEnv/2016.4 cuda/8.0.44 gromacs/4.6.7 Intel & MKL
gromacs/4.6.7 StdEnv/2016.4 gcc/5.4.0 openmpi/2.1.1 gromacs/4.6.7 StdEnv/2016.4 gcc/5.4.0 cuda/8.0 openmpi/2.1.1 gromacs/4.6.7 GCC & MKL & ThreadMPI

Notes:

  • warning GROMACS versions 2020.0 up to and including 2021.5 contain a bug when used on GPUs of Volta or newer generations (i.e. V100, T4 and A100) with mdrun option -update gpu that could have perturbed the virial calculation and, in turn, led to incorrect pressure coupling. The GROMACS developers state in the 2021.6 Release Notes:[1]

    The GPU update is not enabled by default, so the error can only appear in simulations where it [the -update gpu option] was manually selected, and even in this case the error might be rare since we have not observed it in practice in the testing we have performed.

    Further discussion of this bug can be found in the GitLab issue #4393 of the GROMACS project.[2]
  • Version 2020.4 and newer have been compiled for the new Standard software environment StdEnv/2020.
  • Version 2018.7 and newer have been compiled with GCC compilers and the MKL-library, as they run a bit faster.
  • Older versions have been compiled with either with GCC compilers and FFTW or Intel compilers, using Intel MKL and Open MPI 2.1.1 libraries from the default environment as indicated in the table above.
  • CPU (non-GPU) versions are available in both single- and double precision, with the exception of 2019.3 (‡), where double precision not available for AVX512.

These modules can be loaded by using a module load command with the modules as stated in the second column in above table. For example: 

$ module load  StdEnv/2023  gcc/12.3   openmpi/4.1.5  gromacs/2024.1
or 
$ module load  StdEnv/2020  gcc/9.3.0  openmpi/4.0.3  gromacs/2023.2

These versions are also available with GPU support, albeit only with single precision. In order to load the GPU enabled version, the cuda module needs to be loaded first. The modules needed are listed in the third column of above table, e.g.: 

$ module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  cuda/12.2  gromacs/2024.1
or
$ module load  StdEnv/2020  gcc/9.3.0  cuda/11.4  openmpi/4.0.3  gromacs/2023.2

For more information on environment modules, please refer to the Using modules page.

Suffixes[edit]

GROMACS 5.x, 2016.x and newer[edit]

GROMACS 5 and newer releases consist of only four binaries that contain the full functionality. All GROMACS tools from previous versions have been implemented as sub-commands of the gmx binaries. Please refer to GROMACS 5.0 Tool Changes and the GROMACS documentation manuals for your version.

  • gmx - mixed ("single") precision GROMACS with OpenMP (threading) but without MPI.
  • gmx_mpi - mixed ("single") precision GROMACS with OpenMP and MPI.
  • gmx_d - double precision GROMACS with OpenMP but without MPI.
  • gmx_mpi_d - double precision GROMACS with OpenMP and MPI.

GROMACS 4.6.7[edit]

  • The double precision binaries have the suffix _d.
  • The parallel single and double precision mdrun binaries are:
  • mdrun_mpi
  • mdrun_mpi_d


Submission scripts[edit]

Please refer to the page Running jobs for help on using the SLURM workload manager.

Serial jobs[edit]

Here's a simple job script for serial mdrun:


File : serial_gromacs_job.sh

#!/bin/bash
#SBATCH --time=0-0:30         # time limit (D-HH:MM)
#SBATCH --mem-per-cpu=1000M   # memory per CPU (in MB)
module purge  
module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
gmx mdrun -nt 1 -deffnm em


This will run the simulation of the molecular system in the file em.tpr.

Whole nodes[edit]

Commonly the systems which are being simulated with GROMACS are so large, that you want to use a number of whole nodes for the simulation.

Generally the product of --ntasks-per-node= and --cpus-per-task has to match the number of CPU-cores in the compute-nodes of the cluster. Please see section Performance Considerations below.

File : gromacs_whole_node_graham.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=16     # request 16 MPI tasks per node
#SBATCH --cpus-per-task=2        # 2 OpenMP threads per MPI task => total: 16 x 2 = 32 CPUs/node
#SBATCH --mem-per-cpu=2000M      # memory per CPU (in MB)
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge  
module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun --cpus-per-task=$OMP_NUM_THREADS gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_cedar.sh

#!/bin/bash
#SBATCH --nodes=1                        # number of nodes
#SBATCH --ntasks-per-node=24             # request 24 MPI tasks per node
#SBATCH --cpus-per-task=2                # 2 OpenMP threads per MPI task => total: 24 x 2 = 48 CPUs/node
#SBATCH --constraint="[skylake|cascade]" # restrict to AVX512 capable nodes.
#SBATCH --mem-per-cpu=2000M              # memory per CPU (in MB)
#SBATCH --time=0-01:00                   # time limit (D-HH:MM)
module purge
module load arch/avx512   # switch architecture for up to 30% speedup
module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun --cpus-per-task=$OMP_NUM_THREADS gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_beluga.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=20     # request 20 MPI tasks per node
#SBATCH --cpus-per-task=2        # 2 OpenMP threads per MPI task => total: 20 x 2 = 40 CPUs/node
#SBATCH --mem-per-cpu=2000M      # memory per CPU (in MB)
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge  
module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun --cpus-per-task=$OMP_NUM_THREADS gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_beluga.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=32     # request 32 MPI tasks per node
#SBATCH --cpus-per-task=2        # 2 OpenMP threads per MPI task => total: 20 x 2 = 40 CPUs/node
#SBATCH --mem-per-cpu=2000M      # memory per CPU (in MB)
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge  
module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun --cpus-per-task=$OMP_NUM_THREADS gmx_mpi mdrun -deffnm md
File : gromacs_whole_node_niagara.sh

#!/bin/bash
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=10     # request 10 MPI tasks per node
#SBATCH --cpus-per-task=4        # 4 OpenMP threads per MPI task => total: 10 x 4 = 40 CPUs/node
#SBATCH --mem-per-cpu=2000M      # memory per CPU (in MB)
#SBATCH --time=0-01:00           # time limit (D-HH:MM)
module purge --force
module load CCEnv
module load  StdEnv/2023  gcc/12.3  openmpi/4.1.5  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
 
srun --cpus-per-task=$OMP_NUM_THREADS gmx_mpi mdrun -deffnm md

GPU job[edit]

Please read Using GPUs with Slurm for general information on using GPUs on our systems.

This is a job script for mdrun using 4 OpenMP threads and one GPU:

File : gpu_gromacs_job.sh

#!/bin/bash
#SBATCH --gpus-per-node=1        # request 1 GPU per node
#SBATCH --cpus-per-task=4        # number of OpenMP threads per MPI process
#SBATCH --mem-per-cpu=2000M      # memory limit per CPU core (megabytes)
#SBATCH --time=0:30:00           # time limit (D-HH:MM:ss)
module purge  
module load StdEnv/2023  gcc/12.3  openmpi/4.1.5  cuda/12.2  gromacs/2024.1
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

gmx mdrun -ntomp ${SLURM_CPUS_PER_TASK:-1} -deffnm md


Notes on running GROMACS in GPUs[edit]

Note that using more than a single GPU usually leads to poor efficiency. Carefully test and compare multi-GPU and single-GPU performance before deciding to use more than one GPU for your simulations.

  • warning GROMACS versions 2020.0 up to and including 2021.5 contain a bug when used on GPUs of Volta or newer generations (i.e. V100, T4 and A100) with mdrun option -update gpu that could have perturbed the virial calculation and, in turn, led to incorrect pressure coupling. The GROMACS developers state in the 2021.6 Release Notes:[3]

    The GPU update is not enabled by default, so the error can only appear in simulations where it was manually selected, and even in this case the error might be rare since we have not observed it in practice in the testing we have performed.

    Further discussion of this bug can be found in the GitLab issue #4393 of the GROMACS project.[4]
  • Our clusters (Beluga, Cedar, Graham and Narval) have differently configured GPU nodes.
    On the page Using GPUs with Slurm#Available GPUs you can find more information about the different node configurations (GPU models and number of GPUs and CPUs per node).
  • GROMACS imposes a number of constraints for choosing number of GPUs, tasks (MPI ranks) and OpenMP threads.
    For GROMACS 2018.2 the constraints are:
  • The number of --tasks-per-node always needs to be the same as, or a multiple of the number of GPUs (--gpus-per-node).
  • GROMACS will not run GPU runs with only 1 OpenMP thread, unless forced by setting the -ntomp option.
    According to GROMACS developers, the optimum number of --cpus-per-task is between 2 and 6.
  • Avoid using a larger fraction of CPUs and memory than the fraction of GPUs you have requested in a node.

You can explore some benchmark results on our MDBench portal.

Usage[edit]

More content for this section will be added at a later time.

System preparation[edit]

In order to run a simulation, one needs to create a tpr file (portable binary run input file). This file contains the starting structure of the simulation, the molecular topology and all the simulation parameters.

Tpr files are created with the gmx grompp command (or simply grompp for versions older than 5.0). Therefore one needs the following files:

  • The coordinate file with the starting structure. GROMACS can read the starting structure from various file-formats, such as .gro, .pdb or .cpt (checkpoint).
  • The (system) topology (.top)) file. It defines which force-field is used and how the force-field parameters are applied to the simulated system. Often the topologies for individual parts of the simulated system (e.g. molecules) are placed in separate .itp files and included in the .top file using a #include directive.
  • The run-parameter (.mdp) file. See the GROMACS user guide for a detailed description of the options.

Tpr files are portable, that is they can be grompp'ed on one machine, copied over to a different machine and used as an input file for mdrun. One should always use the same version for both grompp and mdrun. Although mdrun is able to use tpr files that have been created with an older version of grompp, this can lead to unexpected simulation results.

Running simulations[edit]

MD Simulations often take much longer than the maximum walltime for a job to complete and therefore need to be restarted. To minimize the time a job needs to wait before it starts, you should maximise the number of nodes you have access to by choosing a shorter running time for your job. Requesting a walltime of 24 hours or 72 hours (three days) is often a good trade-off between waiting- and running-time.

You should use the mdrun parameter -maxh to tell the program the requested walltime so that it gracefully finishes the current timestep when reaching 99% of this walltime. This causes mdrun to create a new checkpoint file at this final timestep and gives it the chance to properly close all output-files (trajectories, energy- and log-files, etc.).

For example use #SBATCH --time=24:00 along with gmx mdrun -maxh 24 ... or #SBATCH --time=3-00:00 along with gmx mdrun -maxh 72 ....


File : gromacs_job.sh

#!/bin/bash
#SBATCH --nodes=1                # number of Nodes
#SBATCH --tasks-per-node=32      # number of MPI processes per node
#SBATCH --mem-per-cpu=4000       # memory limit per CPU (megabytes)
#SBATCH --time=24:00:00          # time limit (D-HH:MM:ss)
module purge
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2023.2
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun  gmx_mpi  mdrun  -deffnm md  -maxh 24



Restarting simulations[edit]

You can restart a simulation by using the same mdrun command as the original simulation and adding the -cpi state.cpt parameter where state.cpt is the filename of the most recent checkpoint file. Mdrun will by default (since version 4.5) try to append to the existing files (trajectories, energy- and log-files, etc.). GROMACS will check the consistency of the output files and - if needed - discard timesteps that are newer than that of the checkpoint file.

Using the -maxh parameter ensures that the checkpoint and output files are written in a consistent state when the simulation reaches the time limit.

The GROMACS manual contains more detailed information [5] [6].


File : gromacs_job_restart.sh

#!/bin/bash
#SBATCH --nodes=1                # number of Nodes
#SBATCH --tasks-per-node=32      # number of MPI processes per node
#SBATCH --mem-per-cpu=4000       # memory limit per CPU (megabytes)
#SBATCH --time=24:00:00          # time limit (D-HH:MM:ss)
module purge
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2023.2
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"

srun  gmx_mpi  mdrun  -deffnm md  -maxh 24.0  -cpi md.cpt


Performance considerations[edit]

Getting the best mdrun performance with GROMACS is not a straightforward task. The GROMACS developers are maintaining a long section in their user-guide dedicated to mdrun-performance[7] which explains all relevant options/parameters and strategies.

There is no "One size fits all", but the best parameters to choose highly depend on the size of the system (number of particles as well as size and shape of the simulation box) and the simulation parameters (cut-offs, use of Particle-Mesh-Ewald[8] (PME) method for long-range electrostatics).

GROMACS prints performance information and statistics at the end of the md.log file, which is helpful in identifying bottlenecks. This section often contains notes on how to further improve the performance.

The simulation performance is typically quantified by the number of nanoseconds of MD-trajectory that can be simulated within a day (ns/day).

Parallel scaling is a measure how effectively the compute resources are used. It is defined as:

S = pN / ( N * p1 )

Where pN is the performance using N CPU cores.

Ideally, the performance increases linearly with the number of CPU cores ("linear scaling"; S = 1).


MPI processes / Slurm tasks / Domain decomposition[edit]

The most straight-forward way to increase the number of MPI processes (called MPI-ranks in the GROMACS documentation), which is done by using Slurm's --ntasks or --ntasks-per-node in the job script.

GROMACS uses Domain Decomposition[8] (DD) to distribute the work of solving the non-bonded Particle-Particle (PP) interactions across multiple CPU cores. This is done by effectively cutting the simulation box along the X, Y and/or Z axes into domains and assigning each domain to one MPI process.

This works well until the time needed for communication becomes large in respect to the size (in respect of number of particles as well as volume) of the domain. In that case the parallel scaling will drop significantly below 1 and in extreme cases the performance drops when increasing the number of domains.

GROMACS can use Dynamic Load Balancing to shift the boundaries between domains to some extent, in order to avoid certain domains taking significantly longer to solve than others. The mdrun parameter -dlb auto is the default.

Domains cannot be smaller in any direction, than the longest cut-off radius.


Long-range interactions with PME[edit]

The Particle-Mesh-Ewald method (PME) is often used to calculate the long-range non-bonded interactions (interactions beyond the cut-off radius). As PME requires global communication, the performance can degrade quickly when many MPI processes are involved that are calculating both the short-range (PP) as well as the long-range (PME) interactons. This is avoided by having dedicated MPI processes that only perform PME (PME-ranks).

GROMACS mdrun by default uses heuristics to dedicate a number of MPI processes to PME when the total number of MPI processes 12 or greater. The mdrun parameter -npme can be used to select the number of PME ranks manually.

In case there is a significant "Load Imbalance" between the PP and PME ranks (e.g. the PP ranks have more work per timestep than the PME ranks), one can shift work from the PP ranks to the PME ranks by increasing the cut-off radius. This will not effect the result, as the sum of short-range + long-range forces (or energies) will be the same for a given timestep. Mdrun will attemtp to do that automatically since version 4.6 unless the mdrun parameter -notunepme is used.

Since version 2018, PME can be offloaded to the GPU (see below) however the implementation as of version 2018.1 has still several limitations [9] among them that only a single GPU rank can be dedicated to PME.


OpenMP threads / CPUs-per-task[edit]

Once Domain Decomposition with MPI processes reaches the scaling limit (parallel scaling starts dropping), performance can be further improved by using OpenMP threads to spread the work of an MPI process (rank) over more than one CPU core. To use OpenMP threads, use Slurm's --cpus-per-task parameter in the job script (both for #SBATCH> and srun) and either set the OMP_NUM_THREADS variable with: export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}" (recommended) or the mdrun parameter -ntomp ${SLURM_CPUS_PER_TASK:-1}.

According to GROMACS developers, the optimum is usually between 2 and 6 OpenMP threads per MPI process (cpus-per-task). However for jobs running on a very large number of nodes it might be worth trying even larger number of cpus-per-task.

Especially for systems that don't use PME, we don't have to worry about a "PP-PME Load Imbalance". In those cases we can choose 2 or 4 ntasks-per-node and set cpus-per-task to a value that ntasks-per-node * cpus-per-task matches the number of CPU cores in a compute node.

CPU architecture[edit]

GROMACS uses optimised kernel functions to compute the real-space portion of short-range, non-bonded interactions. Kernel functions are available for a variety of SIMD instruction sets, such as AVX, AVX2, and AVX512. Kernel functions are chosen when compiling GROMACS, and should match the capabilities of the CPUs that will be used to run the simulations. This is done for you by the Compute Canada team: when you load a GROMACS module into your environment, an appropriate AVX/AVX2/AVX512 version is chosen depending on the architecture of the cluster. GROMACS reports what SIMD instruction set it supports in its log file, and will warn you if the selected kernel function is suboptimal.

However, certain clusters contain a mix of CPUs that have different levels of SIMD support. When that is the case, the smallest common denominator is used. For instance, if the cluster has Skylake (AVX/AVX2/AVX512) and Broadwell (AVX/AVX2) CPUs, as Cedar currently (May 2020) does, a version of GROMACS compiled for the AVX2 instruction set will be used. This means that you may end up with a suboptimal choice of kernel function, depending on which compute nodes the scheduler allocates for your job.

You can explicitly request nodes that support AVX512 with the --constraint="[cascade|skylake]" SLURM option on clusters that offer these node types. This will make sure that your job will be assigned to nodes based on either the "Cascade Lake" or the "Skylake" architecture (but not a mix of both types). If working on the command-line, make sure to not forget the quotation marks (") to protect the special characters [, | and ]. You can then explicitly request AVX512 software using module load arch/avx512 before loading any other module.

For example, a simple job script could look like the following:


File : gromacs_job_cedar_avx512.sh

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=48
#SBATCH --constraint="[skylake|cascade]"
#SBATCH --time=24:00:00
module load arch/avx512
module load StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs/2023.2
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
srun gmx_mpi mdrun


In our measurements, going from AVX2 to AVX512 on Skylake or Cascade nodes resulted in a 20−30% performance increase. However, you should also consider that restricting yourself to only AVX512-capable nodes will result in longer wait times in the queue.

GPUs[edit]

Tips how to use GPUs efficiently will be added soon.

Analyzing results[edit]

GROMACS tools[edit]

GROMACS contains a large number of tools that can be used for common tasks of post-processing and analysis. The GROMACS manual contains a list of available commands organized by topic as well as organized by name that give a short description and link to the corresponding command reference.

These commands will typically read the trajectory (in the XTC, TNG or TRR format) as well as a coordinate file (GRO, PDB, TPR, etc.) and write plots in the XVG format which can be used for inputs for the plotting tool Grace (module: grace/5.99.0; command xmgrace; Grace User Guide). As XVG files are simple text files, they can also be processed with scripts or imported into other spreadsheet programs.

VMD[edit]

VMD is a molecular visualization program for displaying, animating, and analyzing large biomolecular systems using 3-D graphics and built-in scripting. It can be used to visually inspect GROMACS trajectories and also offers a large number of built-in and external plugins for analysis. It can also be used in command-line mode.

Using Python[edit]

MDAnalysis and MDTraj are two Python packages that we provide as pre-compiled Python wheels. They can read and write trajectory- and coordinate-files of GROMACS (TRR and XTC) and many other MD-packages and also include a variety of commonly used analysis functions. MDAnalysis can also read topology information from GROMACS TPR files, though often not those created by the latest versions of GROMACS.

Both packages feature a versatile atom-selection language and expose the coordinates of the trajectories, which makes it very easy to write custom analysis tools that can tailored to a specific problem and integrate well with Python's data-science packages like NumPy, SciPy and Pandas, as well as plotting libraries like Matplotlib/Pyplot and Seaborn.

Common pitfalls[edit]

Related modules[edit]

Gromacs-Plumed[edit]

PLUMED[10] is an open source library for free energy calculations in molecular systems which works together with some of the most popular molecular dynamics engines.

The gromacs-plumed modules are versions of GROMACS that have been patched with PLUMED's modifications, so that they can run meta-dynamics simulations.

GROMACS PLUMED modules for running on CPUs modules for running on GPUs (CUDA) Notes
v2022.6 v2.8.3 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2022.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2022.6 GCC, FlexiBLAS & FFTW
v2022.3 v2.8.1 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2022.3 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2022.3 GCC, FlexiBLAS & FFTW
v2021.6 v2.7.4 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2021.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2021.6 GCC, FlexiBLAS & FFTW
v2021.4 v2.7.3 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2021.4 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-plumed/2021.4 GCC, FlexiBLAS & FFTW
v2021.2 v2.7.1 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2021.2 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs-plumed/2021.2 GCC & MKL
v2019.6 v2.6.2 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-plumed/2019.6 StdEnv/2020 gcc/9.3.0 cuda/11.0 openmpi/4.0.3 gromacs-plumed/2019.6 GCC & MKL
v2019.6 v2.5.4 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs-plumed/2019.6 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs-plumed/2019.6 GCC & MKL
v2019.5 v2.5.3 StdEnv/2018.3 gcc/7.3.0 openmpi/3.1.2 gromacs-plumed/2019.5 StdEnv/2018.3 gcc/7.3.0 cuda/10.0.130 openmpi/3.1.2 gromacs-plumed/2019.5 GCC & MKL
v2018.1 v2.4.2 StdEnv/2016.4 gcc/6.4.0 openmpi/2.1.1 gromacs-plumed/2018.1 StdEnv/2016.4 gcc/6.4.0 cuda/9.0.176 openmpi/2.1.1 gromacs-plumed/2018.1 GCC & FFTW
v2016.3 v2.3.2 StdEnv/2016.4 intel/2016.4 openmpi/2.1.1 gromacs-plumed/2016.3 StdEnv/2016.4 intel/2016.4 cuda/8.0.44 openmpi/2.1.1 gromacs-plumed/2016.3 Intel & MKL

Gromacs-Colvars[edit]

Colvars[11] is an software module for molecular simulation programs, which add additional capabilities of collective variables to apply biasing potentials, calculate potentials-of-mean-force (PMFs) along any set of variables, use enhanced sampling methods, such as Adaptive Biasing Force (ABF), metadynamics, steered MD and umbrella sampling.

As of GROMACS v2024[12], the Colvars library has been added to the official GROMACS releases and can be used without the need of a patched version.

Documentation on how to use Colvars with GROMACS:

  • Collective Variable simulations with the Colvars module[13] in the GROMACS Reference manual,
  • Molecular dynamics parameters (.mdp options) for the Colvars module[14],
  • the Colvars Reference manual for GROMACS[15],
  • the publication: Fiorin et al. 2013, Using collective variables to drive molecular dynamics simulations.[16]

GROMACS versions prior to v2024 required that they have been patched with Colvars modifications, so that the collective variables can be used in simulations.
The gromacs-colvars/2020.6 module is such a modified version of GROMACS that includes Colvars 2021-12-20.

GROMACS Colvars modules for running on CPUs modules for running on GPUs (CUDA) Notes
v2020.6 2021-12-20 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-colvars/2020.6 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-colvars/2020.6 GCC, FlexiBLAS & FFTW

Gromacs-CP2K[edit]

CP2K[17] is a quantum chemistry and solid state physics software package. Since version 2022 GROMACS can be compiled with CP2K-support[18] to enable Hybrid Quantum-Classical simulations (QM/MM)[19].

The gromacs-cp2k modules are versions of GROMACS that have been compiled with CP2K QM/MM support.

Different from other GROMACS modules, these modules are only available for CPU calculations and not for GPUs (CUDA). Also the modules contain only MPI-enabled executables:

  • gmx_mpi - mixed precision GROMACS with OpenMP and MPI.
  • gmx_mpi_d - double precision GROMACS with OpenMP and MPI.
GROMACS CP2K modules for running on CPUs Notes
v2022.2 9.1 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-cp2k/2022.2 GCC, FlexiBLAS & FFTW

Here are links to various resources for running QM/MM simulations with this combination of GROMACS and CP2K:

  • Hybrid Quantum-Classical simulations (QM/MM) with CP2K interface in the GROMACS manual.
  • CP2K QM/MM Best Practices Guide by BioExcel.
  • QM/MM with GROMACS + CP2K Workshop material from BioExcel.
    This contains tutorial material for setting up and running QM/MM simulations as well as links to YouTube videos with theory lectures. This material was written to be used with HPC resources from the European Centre of Excellence for Computational Biomolecular Research (BioExcel), however only small adjustments are needed to use our HPC systems instead.
    Most notably the command gmx_cp2k needs to be replaced with either gmx_mpi (mixed precision) or gmx_mpi_d (double precision) and the jobscripts (which are also using Slurm), need to be adjusted as well.

Gromacs-LS[edit]

GROMACS-LS[20] and the MDStress library enable the calculation of local stress fields from molecular dynamics simulations. The MDStress library is included in the GROMACS-LS module.

Please refer to manual for GROMACS-LS at: Local_stress.pdf and the publications listed therein for information about the method and how to use it.

Notes:

  • Because the manual was written for the older GROMACS-LS v4.5.5 and that the core gromacs commands have changed in version 5, you need to use commands like gmx_LS mdrun and gmx_LS trjconv instead of mdrun_LS and trjconv_LS.
  • GROMACS-LS requires to be compiled in double-precision does not support MPI, SIMD hardware acceleration nor GPUs and is therefore much slower that normal GROMACS. It can only use a single CPU core.
  • Unlike other patched versions of GROMACS, the modules gromacs-ls/2016.3 and gromacs/2016.6 can be loaded at the same time.
module modules for running on CPUs Notes
gromacs-ls/2016.3 StdEnv/2023 gcc/12.3 gromacs-ls/2016.3 GROMACS-LS is a serial application and does not support MPI, OpenMP or GPUs/CUDA.
gromacs/2016.6 StdEnv/2023 gcc/12.3 openmpi/4.1.5 gromacs/2016.6 This Gromacs module can be used to prepare TPR input files for GROMACS-LS.


Gromacs-RAMD[edit]

GROMACS-RAMD is a fork of GROMACS that implements the Random Acceleration Molecular Dynamics (RAMD) method.[21] This method can be used to identify ligand exit routes from the buried binding pockets of receptors and investigate the mechanism of ligand dissociation by running molecular dynamics simulations with an additional randomly oriented force applied to a molecule in the system.

Information on RAMD-specific MDP options[22] can be found on the GROMACS-RAMD GitHub-page.

GROMACS RAMD modules for running on CPUs modules for running on GPUs (CUDA) Notes
v2024.1 2.1 StdEnv/2023 gcc/12.3 openmpi/4.1.5 gromacs-ramd/2024.1-RAMD-2.1 StdEnv/2023 gcc/12.3 openmpi/4.1.5 cuda/12.2 gromacs-ramd/2024.1-RAMD-2.1 GCC, FlexiBLAS & FFTW
v2020.5 2.0 StdEnv/2020 gcc/9.3.0 openmpi/4.0.3 gromacs-ramd/2020.5-RAMD-2.0 StdEnv/2020 gcc/9.3.0 cuda/11.4 openmpi/4.0.3 gromacs-ramd/2020.5-RAMD-2.0 GCC, FlexiBLAS & FFTW

Gromacs-SWAXS[edit]

GROMACS-SWAXS[23] is a modified version of GROMACS for computing small- and wide-angle X-ray or neutron scattering curves (SAXS/SANS) and for doing SAXS/SANS-driven molecular dynamics simulations.

Please refer to the GROMACS-SWAXS Documentation for a description of the features (mdrun input- and output-options, mpd-options, use of gmx genscatt and gmx genenv commands), that have been added in addition to normal GROMACS features, and for a number of tutorials.

GROMACS SWAXS modules for running on CPUs modules for running on GPUs (CUDA) Notes
v2021.7 0.5.1 StdEnv/2023 gcc/12.3 openmpi/4.1.5 gromacs-swaxs/2021.7-0.5.1 StdEnv/2023 gcc/12.3 openmpi/4.1.5 cuda/12.2 gromacs-swaxs/2021.7-0.5.1 GCC, FlexiBLAS & FFTW

G_MMPBSA[edit]

G_MMPBSA[24] is a tool that calculates components of binding energy using MM-PBSA method except the entropic term and energetic contribution of each residue to the binding using energy decomposition scheme.

Development of that tool seems to have stalled in April 2016 and no changes have been made since then. Therefore it is only compatible with Gromacs 5.1.x. For newer version of GROMACS consider using gmx_MMPBSA[25] instead (see below).

The version installed can be loaded with module load StdEnv/2016.4 gcc/5.4.0 g_mmpbsa/2016-04-19 which represent the most up-to-date version and consists of version 1.6 plus the change to make it compatible with Gromacs 5.1.x. The installed version has been compiled with gromacs/5.1.5 and apbs/1.3.

Please be aware that G_MMPBSA uses implicit solvents and there have been studies[26] that conclude that there are issues with the accuracy of these methods for calculating binding free energies.

gmx_MMPBSA[edit]

gmx_MMPBSA[25] is a tool based on AMBER's MMPBSA.py aiming to perform end-state free energy calculations with GROMACS files.

Other than the older G_MMPBSA[24], which is only compatible with older versions of GROMACS, gmx_MMPBSA can be used with current versions of GROMACS and AmberTools.

Please be aware that gmx_MMPBSA uses implicit solvents and there have been studies[26] that conclude that there are issues with the accuracy of these methods for calculating binding free energies.

Installing gmx_MMPBSA into a virtualenv[edit]

The following has been tested with a combination of gmx_MMPBSA 1.5.0.3, gromacs/2021.4 and ambertools/21. While this should work with other recent versions of GROMACS, currently AmberTools 21 is the only version that is expected to work.

1. Load required modules 

$ module purge
$ module load StdEnv/2020 gcc/9.3.0 python/3.8  gromacs/2021.4
$ module load ambertools/21
$ virtualenv venv_gmxMMPBSA
$ source venv_gmxMMPBSA/bin/activate

2. install needed Python packages: 

$ pip install --no-index "numpy~=1.22.0" gmx_MMPBSA
$ python -m pip install git+https://github.com/ParmEd/ParmEd.git@16fb236

Please note that ParmEd version up to 3.4.3 contain a bug that was fixed in commit 16fb236 Until version greater than 3.4.3 has been released we need to use this unreleased version.

3. The Qt/PyQt module needs to be loaded after the virtualenv is ready: 

$ module load qt/5.15.2

4. Test if the main application works: 

$ gmx_MMPBSA -h
$ gmx_MMPBSA_test -ng -n 4

Fortunately, running the self-test is very quick, therefore it's permissible to run them on the login node.

Later when using gmx_MMPBSA in a job you need to load the modules and activate the virtualenv as follows: 

 module purge
module load StdEnv/2020 gcc/9.3.0 python/3.8 ambertools/21 gromacs/2021.4 qt/5.15.2
source venv_gmxMMPBSA/bin/activate

Links[edit]

Biomolecular simulation

References[edit]

  1. ↑ "Fix missing synchronization in CUDA update kernels" in GROMACS 2021.6 Release Notes [1]
  2. ↑ Issue #4393 in GROMACS Project on GitLab.com [2]
  3. ↑ "Fix missing synchronization in CUDA update kernels" in GROMACS 2021.6 Release Notes [3]
  4. ↑ Issue #4393 in GROMACS Project on GitLab.com [4]
  5. ↑ GROMACS User-Guide: Managing long simulations.
  6. ↑ GROMACS Manual page: gmx mdrun
  7. ↑ GROMACS User-Guide: Getting good performance from mdrun
  8. ↑ 8.0 8.1 GROMACS User-Guide: Performance background information
  9. ↑ GROMACS User-Guide: GPU accelerated calculation of PME
  10. ↑ PLUMED Home
  11. ↑ Colvars Home
  12. ↑ GROMACS 2024 Major Release Highlights
  13. ↑ Collective Variable simulations with the Colvars module (GROMACS Reference manual)
  14. ↑ Colvars .mdp Options (GROMACS User guide)
  15. ↑ Colvars Reference manual for GROMACS
  16. ↑ Fiorin et al. 2013, Using collective variables to drive molecular dynamics simulations.
  17. ↑ CP2K Home
  18. ↑ Building GROMACS with CP2K QM/MM support
  19. ↑ QM/MM with CP2K in the GROMACS Reference manual
  20. ↑ GROMACS-LS and MDStress library
  21. ↑ Information on the RAMD method
  22. ↑ RAMD-specific MDP options
  23. ↑ GROMACS-SWAXS Home
  24. ↑ 24.0 24.1 G_MMPBSA Homepage
  25. ↑ 25.0 25.1 gmx_MMPBSA Homepage
  26. ↑ 26.0 26.1 Comparison of Implicit and Explicit Solvent Models for the Calculation of Solvation Free Energy in Organic Solvents

Message Passing Interface

GNU Compiler Collection, an open source compiler collection

Intel Math Kernel Library, a software library of optimized math routines

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=GROMACS&oldid=158656"