GAMESS-US

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

The General Atomic and Molecular Electronic Structure System (GAMESS) is a general ab initio quantum chemistry package.

Running GAMESS

Job Submission

Compute Canada clusters use the Slurm scheduler; for details about submitting jobs, see Running jobs.

First step is to prepare a GAMESS input file containing the molecular geometry and run parameters. Please refer to the GAMESS Documentation and particularly Chapter 2 "Input Description" for a description the file format and keywords.

Besides your input file (in our example, "name.inp"), you have to prepare a job script to define the compute resources for the job. Input file and job script must be in the same directory.

File : gamess_job.sh

#!/bin/bash
#SBATCH --cpus-per-task=1       # Number of CPUs
#SBATCH --mem-per-cpu=4000M     # memory per CPU in MB
#SBATCH --time=0-00:30          # time (DD-HH:MM)

export SLURM_CPUS_PER_TASK
## uncomment the following two lines to use network $SCRATCH
#export USRSCR="$SCRATCH/gamess_${SLURM_JOB_ID}/"
#mkdir -p $USRSCR

module load gamess-us/20170420-R1

rungms name.inp  &>  name.out

Use the following command to submit the job to the scheduler:

 sbatch gamess_job.sh

Running GAMESS on multiple CPUs

GAMESS calculations can make use of more than one CPU. The number of CPUs available for a calculation is determined by the --cpus-per-task setting in the job script.

As GAMESS has been built using sockets for parallelization, it can only use CPU cores that are located on the same compute node. Therefore the maximum number of CPU cores that can be used for a job is dictated by the size of the nodes in the cluster, e.g. 32 CPU cores per node on Graham.

Quantum chemistry calculations are known to not scale well to large numbers of CPUs as compared to e.g. classical molecular mechanics, which means that they can't use large numbers of CPUs efficiently. Exactly how many CPUs can be used efficiently depends on the number of atoms, the number of basis functions, and the level of theory.

To determine a reasonable number of CPUs to use, one needs to run a scaling test--- That is, run the same input file using different numbers of CPUs and compare the execution times. Ideally the execution time should be half as long when using twice as many CPUs. Obviously it is not a good use of resources if a calculation runs (for example) only 30% faster when the number of CPUs is doubled. It is even possible for certain calculations to run slower when increasing the number of CPUs.

Memory

Quantum chemistry calculations are often "memory bound" - that means that larger molecules at high level of theory need a lot of memory (RAM) and in fact often much more than is available in a typical computer. Therefore QM packages like GAMESS will use disk-storage (SCRATCH) to store intermediate results to free up memory and load them back at a later time.

As even our fastes SCRATCH storage is several oders of magnitues slower than the memory, one should make sure to assign sufficient memory to GAMESS. This is a two-step process:

First one needs to request memory for the job via the Slurm submission script. Using --mem-per-cpu=4000M is a reasonable value, as it's compatible with the memory to CPU core ratio on the base nodes. Reqesting more than that will either cause the jobs to wait for being started on a large-memory node or being charged for CPUs it didn't actually used.

Next, in the $SYSTEM group of the input file one needs to define the MWORDS and MEMDDI options. This will tell GAMESS how much memory it is allowed to use. MWORDS is the maximum replicated memory which a job can use, on every core. This is given in units of 1,000,000 words (as opposed to 1024*1024 words), where a word is defined as 64 bits. MEMDDI is the grand total memory needed for the distributed data interface (DDI) storage, given in units of 1,000,000 words. The memory required on each processor core for a run using p CPU-cores is therefore MEMDDI/p + MWORDS. Please refer to the $SYSTEM group section in the GAMESS documentation.

It is important to leave a few hundred MB of memory between the memory requested from the scheduler and the memory that GAMESS is allowed to use, as a safety margin. If the slurm-{JOBID}.out file contains a message like "slurmstepd: error: Exceeded step/job memory limit at some point", then Slurm has terminated the job for trying to use more memory than was requested for the job. In that case one needs to either reduce the MWORDS or MEMDDI in the input file or increase the --mem-per-cpu in the submission script.