GPAW
| This site replaces the former Compute Canada documentation site, and is now being managed by the Digital Research Alliance of Canada. Ce site remplace l'ancien site de documentation de Calcul Canada et est maintenant géré par l'Alliance de recherche numérique du Canada. |
General[edit]
GPAW is a density-functional theory (DFT) Python code based on the projector-augmented wave (PAW) method and the atomic simulation environment (ASE).
Creating a GPAW virtual environment[edit]
We provide precompiled Python wheels for GPAW that can be installed into a virtual python environment.
1. Check which versions of gpaw are available:
[name@server ~] avail_wheels gpaw
name version python arch
------ --------- -------- ------
gpaw 22.8.0 cp39 avx2
gpaw 22.8.0 cp38 avx2
gpaw 22.8.0 cp310 avx2
2. Load a Python module (e.g. python/3.10)
(ENV) [name@server ~] module load python/3.10
3. Create a new virtualenv
[name@server ~] virtualenv --no-download venv_gpaw
created virtual environment CPython3.10.2.final.0-64 in 514ms
[...]
4. Activate the virtualenv (venv)
[name@server ~] source venv_gpaw/bin/activate
5. Install gpaw into venv
(venv_gpaw) [name@server ~] pip install --no-index gpaw
[...]
Successfully installed ... gpaw-22.8.0+computecanada ...
6. Download the data and install it into the SCRATCH filesystem
(venv_gpaw) [name@server ~] gpaw install-data $SCRATCH
Available setups and pseudopotentials
[*] https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
[...]
Setups installed into /scratch/name/gpaw-setups-0.9.20000.
Register this setup path in /home/name/.gpaw/rc.py? [y/n] n
As you wish.
[...]
Installation complete.
7. Now set GPAW_SETUP_PATH to point to the data directory
(venv_gpaw) [name@server ~] export GPAW_SETUP_PATH=$SCRATCH/gpaw-setups-0.9.20000
8. We can run the tests, which are very fast:
(venv_gpaw) [name@server ~] gpaw test
------------------------------------------------------------------------------------------------------------
| python-3.10.2 /home/name/venv_gpaw/bin/python |
| gpaw-22.8.0 /home/name/venv_gpaw/lib/python3.10/site-packages/gpaw/ |
| ase-3.22.1 /home/name/venv_gpaw/lib/python3.10/site-packages/ase/ |
| numpy-1.23.0 /home/name/venv_gpaw/lib/python3.10/site-packages/numpy/ |
| scipy-1.9.3 /home/name/venv_gpaw/lib/python3.10/site-packages/scipy/ |
| libxc-5.2.3 yes |
| _gpaw /home/name/venv_gpaw/lib/python3.10/site-packages/_gpaw.cpython-310-x86_64-linux-gnu.so |
| MPI enabled yes |
| OpenMP enabled yes |
| scalapack yes |
| Elpa no |
| FFTW yes |
| libvdwxc no |
| PAW-datasets (1) /scratch/name/gpaw-setups-0.9.20000 |
-----------------------------------------------------------------------------------------------------------
Doing a test calculation (cores: 1): ... Done
Test parallel calculation with "gpaw -P 4 test".
(venv_gpaw) [name@server ~] gpaw -P 4 test
------------------------------------------------------------------------------------------------------------
| python-3.10.2 /home/name/venv_gpaw/bin/python |
| gpaw-22.8.0 /home/name/venv_gpaw/lib/python3.10/site-packages/gpaw/ |
| ase-3.22.1 /home/name/venv_gpaw/lib/python3.10/site-packages/ase/ |
| numpy-1.23.0 /home/name/venv_gpaw/lib/python3.10/site-packages/numpy/ |
| scipy-1.9.3 /home/name/venv_gpaw/lib/python3.10/site-packages/scipy/ |
| libxc-5.2.3 yes |
| _gpaw /home/name/venv_gpaw/lib/python3.10/site-packages/_gpaw.cpython-310-x86_64-linux-gnu.so |
| MPI enabled yes |
| OpenMP enabled yes |
| scalapack yes |
| Elpa no |
| FFTW yes |
| libvdwxc no |
| PAW-datasets (1) /scratch/name/gpaw-setups-0.9.20000 |
-----------------------------------------------------------------------------------------------------------
Doing a test calculation (cores: 4): ... Done
Results of the last test can be found in the file test.txt that will be created in the current directory.
Example Jobscript[edit]
A jobscript may look something like this for hybrid (OpenMP and MPI) parallelization. This assumes that the virtualenv is in your $HOME directory and the PAW-datasets in $SCRATCH as shown above.
#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=0-01:00
module load gcc/9.3.0 openmpi/4.0.3
source ~/venv_gpaw/bin/activate
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK:-1}"
export GPAW_SETUP_PATH=/scratch/$USER/gpaw-setups-0.9.20000
srun --cpus-per-task=$OMP_NUM_THREADS gpaw python my_gpaw_script.py
This would use a single node with 8 MPI-ranks (ntasks) and 4 OpenMP threads per MPI rank (cpus-per-task) so a total of 32 CPUs.
You probably want to adjust those numbers so that the product matches the number of cores of a whole node
(i.e. 32 at Graham, 40 at Béluga and Niagara, 48 at Cedar or 64 at Narval).
Setting OMP_NUM_THREADS as shown above, makes sure it is always set to the same value as cpus-per-task or 1 in case cpus-per-task is not set.
Loading the modules gcc/9.3.0 and openmpi/4.0.3 ensures that the exact MPI library is used for the job, as was used for building the wheels.