GPAW

General

GPAW is a density-functional theory (DFT) Python code based on the projector-augmented wave (PAW) method and the atomic simulation environment (ASE).

Creating a GPAW virtual environment

We provide precompiled Python wheels for GPAW that can be installed into a virtual python environment.

1. Check which versions of gpaw are available:

[name@server ~] avail_wheels gpaw
name    version    python    arch
------  ---------  --------  ------
gpaw    22.8.0     cp39      avx2
gpaw    22.8.0     cp38      avx2
gpaw    22.8.0     cp310     avx2

2. Load a Python module (e.g. python/3.10)

(ENV) [name@server ~] module load python/3.10

3. Create a new virtualenv

[name@server ~] virtualenv --no-download venv_gpaw
created virtual environment CPython3.10.2.final.0-64 in 514ms
[...]

4. Activate the virtualenv (venv)

[name@server ~] source venv_gpaw/bin/activate

5. Install gpaw into venv

(venv_gpaw) [name@server ~] pip install --no-index gpaw
[...]
Successfully installed ... gpaw-22.8.0+computecanada ...

6. Download the data and install it into the SCRATCH filesystem

(venv_gpaw) [name@server ~] gpaw install-data $SCRATCH 
Available setups and pseudopotentials
  [*] https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
[...]
Setups installed into /scratch/name/gpaw-setups-0.9.20000.
Register this setup path in /home/name/.gpaw/rc.py? [y/n] n
As you wish.
[...]
Installation complete.

7. Now set GPAW_SETUP_PATH to point to the data directory

(venv_gpaw) [name@server ~] export GPAW_SETUP_PATH=$SCRATCH/gpaw-setups-0.9.20000

8. We can run the tests, which are very fast:

(venv_gpaw) [name@server ~] gpaw test
------------------------------------------------------------------------------------------------------------
| python-3.10.2     /home/name/venv_gpaw/bin/python                                                         |
| gpaw-22.8.0       /home/name/venv_gpaw/lib/python3.10/site-packages/gpaw/                                 |
| ase-3.22.1        /home/name/venv_gpaw/lib/python3.10/site-packages/ase/                                  |
| numpy-1.23.0      /home/name/venv_gpaw/lib/python3.10/site-packages/numpy/                                |
| scipy-1.9.3       /home/name/venv_gpaw/lib/python3.10/site-packages/scipy/                                |
| libxc-5.2.3       yes                                                                                     |
| _gpaw             /home/name/venv_gpaw/lib/python3.10/site-packages/_gpaw.cpython-310-x86_64-linux-gnu.so |
| MPI enabled       yes                                                                                     |
| OpenMP enabled    yes                                                                                     |
| scalapack         yes                                                                                     |
| Elpa              no                                                                                      |
| FFTW              yes                                                                                     |
| libvdwxc          no                                                                                      |
| PAW-datasets (1)  /scratch/name/gpaw-setups-0.9.20000                                                     |
 -----------------------------------------------------------------------------------------------------------
Doing a test calculation (cores: 1): ... Done
Test parallel calculation with "gpaw -P 4 test".

(venv_gpaw) [name@server ~] gpaw -P 4 test
------------------------------------------------------------------------------------------------------------
| python-3.10.2     /home/name/venv_gpaw/bin/python                                                         |
| gpaw-22.8.0       /home/name/venv_gpaw/lib/python3.10/site-packages/gpaw/                                 |
| ase-3.22.1        /home/name/venv_gpaw/lib/python3.10/site-packages/ase/                                  |
| numpy-1.23.0      /home/name/venv_gpaw/lib/python3.10/site-packages/numpy/                                |
| scipy-1.9.3       /home/name/venv_gpaw/lib/python3.10/site-packages/scipy/                                |
| libxc-5.2.3       yes                                                                                     |
| _gpaw             /home/name/venv_gpaw/lib/python3.10/site-packages/_gpaw.cpython-310-x86_64-linux-gnu.so |
| MPI enabled       yes                                                                                     |
| OpenMP enabled    yes                                                                                     |
| scalapack         yes                                                                                     |
| Elpa              no                                                                                      |
| FFTW              yes                                                                                     |
| libvdwxc          no                                                                                      |
| PAW-datasets (1)  /scratch/name/gpaw-setups-0.9.20000                                                     |
 -----------------------------------------------------------------------------------------------------------
Doing a test calculation (cores: 4): ... Done

Results of the last test can be found in the file test.txt that will be created in the current directory.

Example Jobscript

A jobscript may look something like this for hybrid (OpenMP and MPI) parallelization. This assumes that the virtualenv is in your $HOME directory and the PAW-datasets in $SCRATCH as shown above.

File : job_gpaw.sh

#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=0-01:00
module load gcc/9.3.0 openmpi/4.0.3
source ~/venv_gpaw/bin/activate

export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
export GPAW_SETUP_PATH=/scratch/$USER/gpaw-setups-0.9.20000

srun --cpus-per-task=$OMP_NUM_THREADS gpaw python my_gpaw_script.py

This would use a single node with 8 MPI-ranks (ntasks) and 4 OpenMP threads per MPI rank (cpus-per-task) so a total of 32 CPUs. You probably want to adjust those numbers so that the product matches the number of cores of a whole node (i.e. 32 at Graham, 40 at Béluga and Niagara, 48 at Cedar or 64 at Narval).

Setting OMP_NUM_THREADS as shown above, makes sure it is always set to the same value as cpus-per-task or 1 in case cpus-per-task is not set. Loading the modules gcc/9.3.0 and openmpi/4.0.3 ensures that the exact MPI library is used for the job, as was used for building the wheels.

Message Passing Interface