cc_staff
284
edits
Dedalus: Difference between revisions
Marked this version for translation
(Added Dedalus page) |
(Marked this version for translation) |
||
| Line 1: | Line 1: | ||
<languages /> | <languages /> | ||
<translate> | <translate> | ||
<!--T:1--> | |||
[https://dedalus-project.org/ Dedalus] is a flexible framework for solving partial differential equations using modern spectral methods. | [https://dedalus-project.org/ Dedalus] is a flexible framework for solving partial differential equations using modern spectral methods. | ||
= Available versions = | = Available versions = <!--T:2--> | ||
Dedalus is available on our clusters as prebuilt Python packages (wheels). You can list available versions with <code>avail_wheels</code>. | Dedalus is available on our clusters as prebuilt Python packages (wheels). You can list available versions with <code>avail_wheels</code>. | ||
{{Command | {{Command | ||
| Line 15: | Line 16: | ||
}} | }} | ||
= Installing Dedalus in a Python virtual environment = | = Installing Dedalus in a Python virtual environment = <!--T:3--> | ||
1. Load Dedalus runtime dependencies. | 1. Load Dedalus runtime dependencies. | ||
{{Command|module load StdEnv/2023 gcc openmpi mpi4py/3.1.4 fftw-mpi/3.3.10 hdf5-mpi/1.14.2 python/3.11}} | {{Command|module load StdEnv/2023 gcc openmpi mpi4py/3.1.4 fftw-mpi/3.3.10 hdf5-mpi/1.14.2 python/3.11}} | ||
<!--T:4--> | |||
2. Create and activate a Python virtual environment. | 2. Create and activate a Python virtual environment. | ||
{{Commands | {{Commands | ||
| Line 25: | Line 27: | ||
}} | }} | ||
<!--T:5--> | |||
3. Install a specific version of Dedalus and its Python dependencies. | 3. Install a specific version of Dedalus and its Python dependencies. | ||
{{Commands | {{Commands | ||
| Line 34: | Line 37: | ||
You can omit to specify the version in order to install the latest one available from the wheelhouse. | You can omit to specify the version in order to install the latest one available from the wheelhouse. | ||
<!--T:6--> | |||
4. Validate it. | 4. Validate it. | ||
{{Command | {{Command | ||
| Line 40: | Line 44: | ||
}} | }} | ||
<!--T:7--> | |||
5. Freeze the environment and requirements set. | 5. Freeze the environment and requirements set. | ||
{{Command | {{Command | ||
| Line 46: | Line 51: | ||
}} | }} | ||
<!--T:8--> | |||
6. Remove the local virtual environment. | 6. Remove the local virtual environment. | ||
{{Command | {{Command | ||
| Line 52: | Line 58: | ||
}} | }} | ||
= Running Dedalus = | = Running Dedalus = <!--T:9--> | ||
You can run dedalus distributed accross multiple nodes or cores. | You can run dedalus distributed accross multiple nodes or cores. | ||
For efficient MPI scheduling, please see: | For efficient MPI scheduling, please see: | ||
| Line 58: | Line 64: | ||
* [[Advanced_MPI_scheduling]] | * [[Advanced_MPI_scheduling]] | ||
<!--T:10--> | |||
1. Write your job submission script. | 1. Write your job submission script. | ||
<tabs> | <tabs> | ||
| Line 67: | Line 74: | ||
#!/bin/bash | #!/bin/bash | ||
<!--T:11--> | |||
#SBATCH --account=def-someprof # adjust this to match the accounting group you are using to submit jobs | #SBATCH --account=def-someprof # adjust this to match the accounting group you are using to submit jobs | ||
#SBATCH --time=08:00:00 # adjust this to match the walltime of your job | #SBATCH --time=08:00:00 # adjust this to match the walltime of your job | ||
| Line 72: | Line 80: | ||
#SBATCH --mem-per-cpu=4G # adjust this according to the memory you need per process | #SBATCH --mem-per-cpu=4G # adjust this according to the memory you need per process | ||
<!--T:12--> | |||
# Run on cores accross the system : https://docs.alliancecan.ca/wiki/Advanced_MPI_scheduling#Few_cores,_any_number_of_nodes | # Run on cores accross the system : https://docs.alliancecan.ca/wiki/Advanced_MPI_scheduling#Few_cores,_any_number_of_nodes | ||
<!--T:13--> | |||
# Load modules dependencies. | # Load modules dependencies. | ||
module load StdEnv/2023 gcc openmpi mpi4py/3.1.4 fftw-mpi/3.3.10 hdf5-mpi/1.14.2 python/3.11 | module load StdEnv/2023 gcc openmpi mpi4py/3.1.4 fftw-mpi/3.3.10 hdf5-mpi/1.14.2 python/3.11 | ||
<!--T:14--> | |||
# create the virtual environment on each allocated node: | # create the virtual environment on each allocated node: | ||
srun --ntasks $SLURM_NNODES --tasks-per-node=1 bash << EOF | srun --ntasks $SLURM_NNODES --tasks-per-node=1 bash << EOF | ||
| Line 82: | Line 93: | ||
source $SLURM_TMPDIR/env/bin/activate | source $SLURM_TMPDIR/env/bin/activate | ||
<!--T:15--> | |||
pip install --no-index --upgrade pip | pip install --no-index --upgrade pip | ||
pip install --no-index -r dedalus-3.0.2-requirements.txt | pip install --no-index -r dedalus-3.0.2-requirements.txt | ||
EOF | EOF | ||
<!--T:16--> | |||
# activate only on main node | # activate only on main node | ||
source $SLURM_TMPDIR/env/bin/activate; | source $SLURM_TMPDIR/env/bin/activate; | ||
<!--T:17--> | |||
export OMP_NUM_THREADS=1 | export OMP_NUM_THREADS=1 | ||
<!--T:18--> | |||
# srun exports the current env, which contains $VIRTUAL_ENV and $PATH variables | # srun exports the current env, which contains $VIRTUAL_ENV and $PATH variables | ||
srun python $SCRATCH/myscript.py; | srun python $SCRATCH/myscript.py; | ||
| Line 96: | Line 111: | ||
</tab> | </tab> | ||
<!--T:19--> | |||
<tab name="Whole nodes"> | <tab name="Whole nodes"> | ||
{{File | {{File | ||
| Line 103: | Line 119: | ||
#!/bin/bash | #!/bin/bash | ||
<!--T:20--> | |||
#SBATCH --account=def-someprof # adjust this to match the accounting group you are using to submit jobs | #SBATCH --account=def-someprof # adjust this to match the accounting group you are using to submit jobs | ||
#SBATCH --time=08:00:00 # adjust this to match the walltime of your job | #SBATCH --time=08:00:00 # adjust this to match the walltime of your job | ||
| Line 109: | Line 126: | ||
#SBATCH --mem-per-cpu=4G # adjust this according to the memory you need per process | #SBATCH --mem-per-cpu=4G # adjust this according to the memory you need per process | ||
<!--T:21--> | |||
# Run on N whole nodes : https://docs.alliancecan.ca/wiki/Advanced_MPI_scheduling#Whole_nodes | # Run on N whole nodes : https://docs.alliancecan.ca/wiki/Advanced_MPI_scheduling#Whole_nodes | ||
<!--T:22--> | |||
# Load modules dependencies. | # Load modules dependencies. | ||
module load StdEnv/2023 gcc openmpi mpi4py/3.1.4 fftw-mpi/3.3.10 hdf5-mpi/1.14.2 python/3.11 | module load StdEnv/2023 gcc openmpi mpi4py/3.1.4 fftw-mpi/3.3.10 hdf5-mpi/1.14.2 python/3.11 | ||
<!--T:23--> | |||
# create the virtual environment on each allocated node: | # create the virtual environment on each allocated node: | ||
srun --ntasks $SLURM_NNODES --tasks-per-node=1 bash << EOF | srun --ntasks $SLURM_NNODES --tasks-per-node=1 bash << EOF | ||
| Line 119: | Line 139: | ||
source $SLURM_TMPDIR/env/bin/activate | source $SLURM_TMPDIR/env/bin/activate | ||
<!--T:24--> | |||
pip install --no-index --upgrade pip | pip install --no-index --upgrade pip | ||
pip install --no-index -r dedalus-3.0.2-requirements.txt | pip install --no-index -r dedalus-3.0.2-requirements.txt | ||
EOF | EOF | ||
<!--T:25--> | |||
# activate only on main node | # activate only on main node | ||
source $SLURM_TMPDIR/env/bin/activate; | source $SLURM_TMPDIR/env/bin/activate; | ||
<!--T:26--> | |||
export OMP_NUM_THREADS=1 | export OMP_NUM_THREADS=1 | ||
<!--T:27--> | |||
# srun exports the current env, which contains $VIRTUAL_ENV and $PATH variables | # srun exports the current env, which contains $VIRTUAL_ENV and $PATH variables | ||
srun python $SCRATCH/myscript.py; | srun python $SCRATCH/myscript.py; | ||
| Line 134: | Line 158: | ||
</tabs> | </tabs> | ||
<!--T:28--> | |||
2. Submit your job to the scheduler. | 2. Submit your job to the scheduler. | ||
{{Command | {{Command | ||