Ansys

ANSYS is a software suite for engineering simulation and 3-D design. It includes packages such as ANSYS Fluent and ANSYS CFX.

Licensing

Compute Canada is a hosting provider for ANSYS . This means that we have ANSYS software installed on our clusters, but we do not provide a generic license accessible to everyone. However, many institutions, faculties, and departments already have licenses that can be used on our cluster. Once the legal aspects are worked out for licensing, there will be remaining technical aspects. The license server on your end will need to be reachable by our compute nodes. This will require our technical team to get in touch with the technical people managing your license software. In some cases, this has already been done. You should then be able to load the ANSYS modules, and it should find its license automatically. If this is not the case, please contact our Technical support, so that we can arrange this for you.

Configuring your own license file

Our module for ANSYS is designed to look for license information in a few places. One of those places is your home folder. If you have your own license server, write the information to access into file $HOME/.licenses/ansys.lic using the following format:

File : ansys.lic

setenv("ANSYSLMD_LICENSE_FILE", "port@hostname")
setenv("ANSYSLI_SERVERS", "port@hostname")

Cluster specific settings for port@hostname are given in the following table:

License	Cluster	ANSYSLMD_LICENSE_FILE	ANSYSLI_SERVERS	Notices
CMC	beluga	6624@10.20.73.21	2325@10.20.73.21	None
CMC	cedar	6624@206.12.126.25	2325@206.12.126.25	None
CMC	graham	6624@199.241.167.222	2325@199.241.167.222	None
CMC	narval	6624@10.100.64.10	2325@10.100.64.10	None
SHARCNET	beluga/cedar/graham/gra-vdi/narval	1055@license3.sharcnet.ca	2325@license3.sharcnet.ca	None

Researchers who purchase a CMC license subscription must send their Compute Canada username to <cmcsupport@cmc.ca> otherwise license checkouts will fail. The number of cores that can be used with a CMC license is described in the Other Tricks and Tips section found here.

Local License Servers

Before a local institutional ANSYS license server can be reached from Compute Canada systems firewall configuration changes will need to be made on both the institution side and the Compute Canada side. To start this process, contact your local ANSYS license server administrator and obtain the following information 1) fully qualified hostname of the local ANSYS license server 2) ANSYS flex port (commonly 1055) 3) ANSYS licensing interconnect port (commonly 2325) and 4) ANSYS static vendor port (site specific). Ensure the administrator is willing to open the firewall on these three ports to accept license checkout requests from your ANSYS jobs running on Compute Canada systems. Contact our Technical support and send us the four pieces of information and indicate which systems(s) you want to run ANSYS on for example Cedar, Beluga, Graham/Gra-vdi or Niagara.

Checking License Usage

ANSYS comes with an lmutil tool that can be used to check the detailed usage of your licenses. Before using this tool as shown below, make sure the ansys.lic file is configured, and an ansys module is loaded:

[name@server ~]$ module load ansys
[name@server ~]$ $EBROOTANSYS/shared_files/licensing/linx64/lmutil lmstat -c $ANSYSLMD_LICENSE_FILE -S ansyslmd

Version Compatibility

As explained in ANSYS Platform Support the current release (2021R2) was tested to read and open databases from the five previous releases. In addition some products can read and open databases from releases before Ansys 18.1.

Cluster Batch Job Submission

The ANSYS software suite comes with multiple implementations of MPI to support parallel computation. Unfortunately, none of them supports our Slurm scheduler. For this reason, we need special instructions for each ANSYS package on how to start a parallel job. In the sections below, we give examples of submission scripts for some of the packages. If one is not covered and you want us to investigate and help you start it, please contact our Technical support.

ANSYS Fluent

Typically you would use the following procedure for running Fluent on one of the Compute Canada clusters:

• Prepare your Fluent job using Fluent from the "ANSYS Workbench" on your Desktop machine up to the point where you would run the calculation.
• Export the "case" file "File > Export > Case..." or find the folder where Fluent saves your project's files. The "case" file will often have a name like FFF-1.cas.gz.
• If you already have data from a previous calculation, which you want to continue, export a "data" file as well (File > Export > Data...) or find it the same project folder (FFF-1.dat.gz).
• Transfer the "case" file (and if needed the "data" file) to a directory on the project or scratch filesystem on the cluster. When exporting, you save the file(s) under a more instructive name than FFF-1.* or rename them when uploading them.
• Now you need to create a "journal" file. It's purpose is to load the case- (and optionally the data-) file, run the solver and finally write the results. See examples below and remember to adjust the filenames and desired number of iterations.
• If jobs frequently fail to start due to license shortages (and manual resubmission of failed jobs is not convenient) consider modifying your slurm script to requeue your job (upto to 4 times) as shown in the following "Fluent Slurm Script (by node + requeue)" tab. Be aware doing this will also requeue simulations that fail due to non-license related issues (such as divergence) resulting lost compute time. Therefore it is strongly recommended to monitor and inspect each slurm output file to confirm each requeue attempt is license related. When it is determined a job requeued due to a simulation issue then immediately manually kill the job progression with scancel jobid and correct the problem.
• After running the job you can download the "data" file and import it back into Fluent with File > import > Data....

Slurm Scripts

Fluent Slurm Script (by node)Fluent Slurm Script (by node + requeue)Fluent Slurm Script (by core)

File : script-flu-bynode.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number compute nodes (1 or more)
#SBATCH --cpus-per-task=32    # Specify number cores per node (graham 32 or 44, cedar 32 or 48, beluga 40, narval 64)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --ntasks-per-node=1   # Do not change

#module load StdEnv/2016      # Applies to graham, cedar, niagara
#module load ansys/2020R2     # Or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # Or ansys/2020R2 
#module load intel/2021 intelmpi
#export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
#unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
#unset I_MPI_ROOT

module load StdEnv/2020       # Applies to all clusters
module load ansys/2021R2      # Or newer module versions

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

fluent 3d -t $NCORES -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -affinity=0 -ssh -g -i sample.jou

File : script-flu-bynode+requeue.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number compute nodes (1 or more)
#SBATCH --cpus-per-task=32    # Specify number cores per node (graham 32 or 44, cedar 32 or 48, beluga 40, narval 64)
#SBATCH --array=1-5%1         # Specify number requeue attempts (2 or more, 5 is shown)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --ntasks-per-node=1   # Do not change

#module load StdEnv/2016      # Applies to graham, cedar, niagara
#module load ansys/2020R2     # Or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # Or ansys/2020R2 
#module load intel/2021 intelmpi
#export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
#unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
#unset I_MPI_ROOT

module load StdEnv/2020       # Applies to all clusters
module load ansys/2021R2      # Or newer module versions

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

fluent 3d -t $NCORES -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -affinity=0 -ssh -g -i sample.jou
if [ $? -eq 0 ]; then
    echo "Job completed successfully! Exiting now."
    scancel $SLURM_ARRAY_JOB_ID
else
    echo "Job failed due to license or simulation issue!"
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
       echo "Resubmitting now ..."
    else
       echo "Exiting now."
    fi
fi

File : script-flu-bycore.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=1            # Uncomment to specify (1 or more)
#SBATCH --ntasks=16           # Specify total number cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change

#module load StdEnv/2016      # Applies to graham, cedar, niagara
#module load ansys/2020R2     # Or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # Or ansys/2020R2 
#module load intel/2021 intelmpi
#export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
#unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
#unset I_MPI_ROOT

module load StdEnv/2020       # Applies to all clusters
module load ansys/2021R2      # Or newer module versions

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

fluent 3d -t $NCORES -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -affinity=0 -g -i sample.jou

Journal Files

Fluent Journal files can include basically any command from Fluent's Text-User-Interface (TUI); commands can be used to change simulation parameters like temperature, pressure and flow speed. With this you can run a series of simulations under different conditions with a single case file, by only changing the parameters in the Journal file. Refer to the Fluent User's Guide for more information and a list of all commands that can be used.

Journal File (steady,cas)Journal File (steady,cas+dat)Journal File (transient)

File : sample1.jou

; SAMPLE FLUENT JOURNAL FILE - STEADY SIMULATION
; ----------------------------------------------
; lines beginning with a semicolon are comments

; Write case and data files in legacy format:
/file/cff-files no

; Read input files (FFF-in.cas and FFF-in.dat):
/file/read-case-data FFF-in

; Run the solver for this many iterations:
/solve/iterate 1000

; Overwrite output files by default:
/file/confirm-overwrite n

; Write final output file (FFF-out.dat):
/file/write-data FFF-out

; Write simulation report to file (optional):
/report/summary y "My_Simulation_Report.txt"

; Exit fluent:
exit

File : sample2.jou

; SAMPLE FLUENT JOURNAL FILE - STEADY SIMULATION
; ----------------------------------------------
; lines beginning with a semicolon are comments

; Write case and data files in legacy format:
/file/cff-files no

; Read compressed input files (FFF-in.cas.gz & FFF-in.dat.gz):
/file/read-case-data FFF-in.gz

; Write a compressed data file every 100 iterations:
/file/auto-save/data-frequency 100

; Retain data files from 5 most recent iterations:
/file/auto-save/retain-most-recent-files y

; Write data files to output sub-directory (appends iteration)
/file/auto-save/root-name output/FFF-out.gz

; Run the solver for this many iterations:
/solve/iterate 1000

; Write final compressed output files (FFF-out.cas.gz & FFF-out.dat.gz):
/file/write-case-data FFF-out.gz

; Write simulation report to file (optional):
/report/summary y "My_Simulation_Report.txt"

; Exit fluent:
exit

File : sample3.jou

; SAMPLE FLUENT JOURNAL FILE - TRANSIENT SIMULATION
; -------------------------------------------------
; lines beginning with a semicolon are comments

; Write case and data files in legacy format:
/file/cff-files no

; Read only the input case file:
/file/read-case FFF-transient-inp.gz

; For continuation (restart) read in both case and data input files:
;/file/read-case-data FFF-transient-inp.gz

; Write a data (and maybe case) file every 100 time steps:
/file/auto-save/data-frequency 100
/file/auto-save/case-frequency if-case-is-modified

; Retain only the most recent 5 data (and maybe case) files:
; [saves disk space if only a recent continuation file is needed]
/file/auto-save/retain-most-recent-files y

; Write to output sub-directory (appends flowtime and timestep)
/file/auto-save/root-name output/FFF-transient-out-%10.6f.gz

; ##### Settings for Transient simulation :  #####

; Set the magnitude of the (physical) time step (delta-t)
/solve/set/time-step 0.0001

; Set the max number of iterations per time step:
/solve/set/max-iterations-per-time-step 20

; Set the number of iterations for which convergence monitors are reported:
/solve/set/reporting-interval 1

; ##### End of settings for Transient simulation #####

; Initialize using the hybrid initialization method:
/solve/initialize/hyb-initialization

; Perform unsteady iterations for a specified number of time steps:
/solve/dual-time-iterate 1000 ,

; Write final case and data output files:
/file/write-case-data FFF-transient-out.gz

; Write simulation report to file (optional):
/report/summary y Report_Transient_Simulation.txt

; Exit fluent:
exit

ANSYS CFX

CFX Slurm Script (multi-node, stable)CFX Slurm Script (single-node, testing)

File : script-cfx.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number compute nodes (1 or more)
#SBATCH --cpus-per-task=32    # Specify number cores per node (graham 32 or 44, cedar 32 or 48, beluga 40)
#SBATCH --mem=0               # Do not change (allocate all memory per compute node)
#SBATCH --ntasks-per-node=1   # Do not change

#module load StdEnv/2016      # Applies to: graham, cedar, beluga
#module load ansys/2020R2     # Or older module versions

module load StdEnv/2020       # Applies to: graham, cedar, beluga, narval
module load ansys/2021R1      # Or newer module versions

NNODES=$(slurm_hl2hl.py --format ANSYS-CFX)

# other options maybe appended to the following command line as needed
cfx5solve -def YOURFILE.def -start-method "Intel MPI Distributed Parallel" -par-dist $NNODES

File : script-cfx.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --ntasks-per-node=4   # Specify number cores (narval upto 64)
#SBATCH --mem=8G              # Specify 0 when using all cores
#SBATCH --nodes=1             # Do not change

module load StdEnv/2020       # Applies to: narval, graham, cedar
module load ansys/2020R2      # Or version 2019R3

# other options maybe appended to the following command line as needed
cfx5solve -def YOURFILE.def -start-method "Open MPI Local Parallel" -part $SLURM_CPUS_ON_NODE

Note: You may get the following errors in your output file : /etc/tmi.conf: No such file or directory. They do not seem to affect the computation.

WORKBENCH

Before submitting a job (sbatch script-wbpj.sh) to the queue 1) specify the name of YOURPROJECT.wbpj file in the slurm script below and 2) initialize the project by opening it in the ANSYS workbench as described in the Graphical_Use then click File -> Open to select then load your project, start Mechanical by double clicking Setup or Solution for your Analysis System located in the main display window, click File -> Clear Generated Data -> Yes, click File -> Save Project, click File -> Close Mechanical and finally click File -> exit.

To avoid writing the solution when a job successfully completes on a cluster, remove ;Save(Overwrite=True) from the last line of your slurm script. Doing this will make it easier to run multiple test jobs (for scaling purposes when changing ntasks) since the initialized solution will not be overwritten each time and therefore not need to be re-initialized between jobs. Alternatively a copy of the initialized YOURPROJECT.wbpj file and YOURPROJECT_files sub-directory could be saved prior to submitting each job then restored after the solution is written. For APDL based simulations submitted under the legacy StdEnv/2016 environment, nodes=1 maybe either removed from the slurm script or changed to be greater than 1 to permit computations across multiple nodes.

Single-Node Slurm Script (StdEnv/2020)Single-Node Slurm Script (StdEnv/2016)

File : script-wbpj-2020.sh

#!/bin/bash

#SBATCH --account=def-account
#SBATCH --time=00-03:00                # Time (DD-HH:MM)
#SBATCH --mem=16G                      # Total Memory (set to 0 for all node memory)
#SBATCH --ntasks=4                     # Number of cores
#SBATCH --nodes=1                      # Do not change

# This script assumes the project has been opened with Workbench Mechanical at least
# once with the same ansys module version on the same cluster where it will be used

module load StdEnv/2020 ansys/2021R2   # OR newer ansys modules (narval supported)
MEMPAR=0                               # Set to 0 for SMP or 1 for DMP

rm -fv *_files/.lock
MWFILE=~/.mw/Application\ Data/Ansys/`basename $(find $EBROOTANSYS/v* -maxdepth 0 -type d)`/SolveHandlers.xml
sed -re "s/(.AnsysSolution>+)[a-zA-Z0-9]*(<\/Distribute.)/\1$MEMPAR\2/" -i "$MWFILE"
sed -re "s/(.Processors>+)[a-zA-Z0-9]*(<\/MaxNumber.)/\1$SLURM_NTASKS\2/" -i "$MWFILE"
sed -i "s!UserConfigured=\"0\"!UserConfigured=\"1\"!g" "$MWFILE"
export KMP_AFFINITY=disabled

runwb2 -B -E "Update();Save(Overwrite=True)" -F YOURPROJECT.wbpj

File : script-wbpj-2016.sh

#!/bin/bash

#SBATCH --account=def-account
#SBATCH --time=00-03:00                # Time (DD-HH:MM)
#SBATCH --mem=16G                      # Total Memory (set to 0 for all node memory)
#SBATCH --ntasks=4                     # Number of cores
#SBATCH --nodes=1                      # Do not change

# This script assumes the project has been opened with Workbench Mechanical at least
# once with the same ansys module version on the same cluster where it will be used

module load StdEnv/2016 ansys/2019R3   # Do not change (narval NOT supported)
MEMPAR=1                               # Do not change

rm -fv *_files/.lock
MWFILE=~/.mw/Application\ Data/Ansys/`basename $(find $EBROOTANSYS/v* -maxdepth 0 -type d)`/SolveHandlers.xml
sed -re "s/(.AnsysSolution>+)[a-zA-Z0-9]*(<\/Distribute.)/\1$MEMPAR\2/" -i "$MWFILE"
sed -re "s/(.Processors>+)[a-zA-Z0-9]*(<\/MaxNumber.)/\1$((SLURM_NTASKS-1))\2/" -i "$MWFILE"
export PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin:$PATH

runwb2 -B -E "Update();Save(Overwrite=True)" -F YOURPROJECT.wbpj

MECHANICAL

The input file can be generated from within your interactive Workbench Mechanical session by clicking Solution -> Tools -> Write Input Files then specify File name: YOURAPDLFILE.inp and Save as type: APDL Input Files (*.inp). APDL jobs can then be submitted to the queue by running the sbatch script-name.sh command. The ANSYS modules given in each script were tested on graham and should work without issue (uncomment one). Once the scripts are tested on other clusters they will be updated below if required.

SMP Script (stdenv/2016)SMP Script (stdenv/2020)DIS Script (stdenv/2016)DIS Script (stdenv/2020)

File : script-smp-2016.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory for all cores
#SBATCH --ntasks=8             # Specify number of cores (1 or more)
#SBATCH --nodes=1              # Specify one node (do not change)

unset SLURM_GTIDS

module load StdEnv/2016

#module load ansys/19.1
#module load ansys/19.2
#module load ansys/2019R2
#module load ansys/2019R3
#module load ansys/2020R1
module load ansys/2020R2

mapdl -smp -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -i YOURAPDLFILE.inp

File : script-smp-2020.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory for all cores
#SBATCH --ntasks=8             # Specify number of cores (1 or more)
#SBATCH --nodes=1              # Specify one node (do not change)

unset SLURM_GTIDS

module load StdEnv/2020

#module load ansys/2021R1
module load ansys/2021R2

mapdl -smp -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -i YOURAPDLFILE.inp

File : script-dis-2016.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem-per-cpu=2G       # Specify memory per core
#SBATCH --ntasks=8             # Specify number of cores (2 or more)
##SBATCH --nodes=2             # Specify number of nodes (optional)
##SBATCH --ntasks-per-node=4   # Specify cores per node (optional)

unset SLURM_GTIDS

module load StdEnv/2016

#module load ansys/2019R3
module load ansys/2020R1

export I_MPI_HYDRA_BOOTSTRAP=ssh; export KMP_AFFINITY=compact
mapdl -dis -mpi intelmpi -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -i YOURAPDLFILE.inp

File : script-dis-2020.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem-per-cpu=2G       # Specify memory per core
#SBATCH --ntasks=8             # Specify number of cores (2 or more)
##SBATCH --nodes=2             # Specify number of nodes (optional)
##SBATCH --ntasks-per-node=4   # Specify cores per node (optional)

unset SLURM_GTIDS

module load StdEnv/2020

#module load ansys/2021R1
module load ansys/2021R2

mapdl -dis -mpi openmpi -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -i YOURAPDLFILE.inp

ANSYS allocates 1024 MB total memory and 1024 MB database memory by default for APDL jobs. These values can be manually specified (or changed) by adding arguments -m 1024 and/or -db 1024 to the last maple command line in the above slurm scripts. When using a remote institutional license server with multiple ANSYS licenses it may be necessary to add arguments such as -p aa_r or -ppf anshpc. As always perform detailed scaling tests before running production jobs to ensure the optimal number of cores and minimum amount memory is specified in your slurm scripts. The single node SMP (Shared Memory Parallel) script will perform better than the multiple node DIS (Distributed Memory Parallel) script and therefore should be used whenever possible. To help avoid compatibility issues the ansys module loaded in your slurm script should ideally match the version used to to generate the input file:

 [gra-login2:~/ansys/mechanical/demo] cat YOURAPDLFILE.inp | grep version
! ANSYS input file written by Workbench version 2019 R3

ANSYS EDT

Ansys Electronic Desktop jobs can be submitted to the cluster queue by running the sbatch script-name.sh command. The following script allows running a job with upto all cores and memory on a single node and was tested on graham. To use it specify the simulation time, memory, number of cores and replace YOUR_AEDT_FILE with your input file name. A full listing of ansysedt command line options can be obtained by starting ansysedt in [Graphical Mode | https://docs.computecanada.ca/wiki/ANSYS#Graphical_Use] with commands ansysedt -help or ansysedt -Batchoptionhelp to obtain scrollable graphical popups. Ansysedt can be run interactively in batch (non-gui) mode by first starting an salloc session with options salloc --time=3:00:00 --tasks=8 --mem=16G --account=def-account and then copy pasting the full ansysedt command given in the last line of script-local-cmd.sh being sure to manually specify $YOUR_AEDT_FILE.

Single Node Script - Command LineSingle Node Script - Options File

File : script-local-cmd.sh

#!/bin/bash

#SBATCH --account=account      # Specify your account (def or rrg)
#SBATCH --time=00-01:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory (set to 0 to use all compute node memory)
#SBATCH --ntasks=8             # Specify cores (beluga 40, cedar 32 or 48, graham 32 or 44, narval 64)
#SBATCH --nodes=1              # Request one node (Do Not Change)

module load StdEnv/2020
module load ansysedt/2021R2

# Uncomment next line to run a test example:
cp -f $EBROOTANSYSEDT/AnsysEM21.2/Linux64/Examples/HFSS/Antennas/TransientGeoRadar.aedt .

# Specify input file such as:
YOUR_AEDT_FILE="TransientGeoRadar.aedt"

# Remove previous output:
rm -rf $YOUR_AEDT_FILE.* ${YOUR_AEDT_FILE}results

# ---- do not change anything below this line ---- #

echo -e "\nANSYSLI_SERVERS= $ANSYSLI_SERVERS"
echo "ANSYSLMD_LICENSE_FILE= $ANSYSLMD_LICENSE_FILE"
echo -e "SLURM_TMPDIR= $SLURM_TMPDIR on $SLURMD_NODENAME\n"

export KMP_AFFINITY=disabled
ansysedt -monitor -UseElectronicsPPE -ng -distributed -machinelist list=localhost:1:$SLURM_NTASKS \
-batchoptions "TempDirectory=$SLURM_TMPDIR HPCLicenseType=pool HFSS/EnableGPU=0" -batchsolve $YOUR_AEDT_FILE

File : script-local-opt.sh

#!/bin/bash

#SBATCH --account=account      # Specify your account (def or rrg)
#SBATCH --time=00-01:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory (set to 0 to allocate all compute node memory)
#SBATCH --ntasks=8             # Specify cores (beluga 40, cedar 32 or 48, graham 32 or 44, narval 64)
#SBATCH --nodes=1              # Request one node (Do Not Change)

module load StdEnv/2020
module load ansysedt/2021R2

# Uncomment next line to run a test example:
cp -f $EBROOTANSYSEDT/AnsysEM21.2/Linux64/Examples/HFSS/Antennas/TransientGeoRadar.aedt .

# Specify input filename such as:
YOUR_AEDT_FILE="TransientGeoRadar.aedt"

# Remove previous output:
rm -rf $YOUR_AEDT_FILE.* ${YOUR_AEDT_FILE}results

# Specify options filename:
OPTIONS_TXT="Options.txt"

# Write sample options file
rm -f $OPTIONS_TXT
cat > $OPTIONS_TXT <<EOF
\$begin 'Config'
'TempDirectory'='$SLURM_TMPDIR'
'HPCLicenseType'='pool'
'HFSS/EnableGPU'=0
\$end 'Config'
EOF

# ---- do not change anything below this line ---- #

echo -e "\nANSYSLI_SERVERS= $ANSYSLI_SERVERS"
echo "ANSYSLMD_LICENSE_FILE= $ANSYSLMD_LICENSE_FILE"
echo -e "SLURM_TMPDIR= $SLURM_TMPDIR on $SLURMD_NODENAME\n"

export KMP_AFFINITY=disabled
ansysedt -monitor -UseElectronicsPPE -ng -distributed -machinelist list=localhost:1:$SLURM_NTASKS \
-batchoptions $OPTIONS_TXT -batchsolve $YOUR_AEDT_FILE

Graphical Use

ANSYS programs maybe run interactively in gui mode on cluster Compute Nodes or graham VDI Nodes.

Compute Nodes

ANSYS can be run interactively on single cluster compute node for upto 24hours. This approach is ideal for testing large simulations since all cores and memory can be requested with salloc as described in TigerVNC. Once connected with vncviewer, any of the following program versions can be started after loading the required modules as shown below. The vertical bar | notation is used to separate the various ANSYS commands.

FLUIDS

module load StdEnv/2020 ansys/2022R1, or

module load StdEnv/2020 ansys/2021R2, or

module load StdEnv/2020 ansys/2021R1

unset SESSION_MANAGER

fluent -mpi=intel | cfx5

------------------------------------------------------------------------------------

module load StdEnv/2016 ansys/2020R2 (or older versions)

fluent | cfx5

WORKBENCH

module load StdEnv/2020 ansys/2022R1, or

module load StdEnv/2020 ansys/2021R2

runwb2

To run in parallel unTick the distributed box in the solve panel and specify a value for Cores uqual to the "salloc cpus value that you specified minus 1".

NOTE: Most pulldown menus and icons do not respond as one might expect when doing a mouse over or single clicking them. The following bullets provide a workaround to give equivalent control when working within the workbench gui on cluster compute nodes:

o To access the pulldown menus for the items in left side project tree, left click once to select the item then double right click your mouse button.

o To access the pulldown menus for the Upper icons with tiny down arrows, left click once to select the item then double left click your mouse button. The tiny down arrow for the Solve icon which provides access to the Solve Process settings is an exception ! To access this instead please click the tiny angled arrow found in the lower right corner of the Solve panel.

o While the pull downs found in the left side menu that typically default to Program Controlled work as expected when single clicked for various Ansys Systems, the width of the left side window is typically too narrow to see the full text for all of the pulled down items. This can rectified by dragging the vertical bar of the left window to make it wider. Doing so will also make the associated drop down arrows visible. The left side window can be restored to its default layout appearance by following the procedure in the previous bullet to specify Home -> Layout -> Reset Layout.

o If you are having trouble scrolling down to click File -> Close Mechanical or have minimized it and want to restore it to view then respectfully single right click -> close or single left click the Workbench button found along the bottom of the tigervnc gui frame to reselect it.

o If Mechanical will not start completely close Ansys as described in the previous bullet then exit and restart your salloc session. The cause of this inconvenience maybe a uncleanly exited previous ansys session run within the same salloc session.

------------------------------------------------------------------------------------

module load StdEnv/2016 ansys/2019R3 (not available on narval)

export PATH=$EBROOTNIXPKGS/bin:$PATH

runwb2

o Single right click to access left side project tree pulldown menus

o Single left click for the upper pulldown menus with tiny down arrows

o Tick distributed box, specify number Cores = salloc value minus 1

ANSYS EDT

module load CcEnv StdEnv/2020 ansysedt/2021R2

rm -rf ~/.mw (optionally force First-time configuration)

ansysedt

ENSIGHT

module load StdEnv/2020 ansys/2022R1; A=221; B=5.12.6, or

module load StdEnv/2020 ansys/2021R2; A=212; B=5.12.6, or

module load StdEnv/2020 ansys/2021R1; A=211; B=5.12.6, or

module load StdEnv/2016 ansys/2020R2; A=202; B=5.12.6, or

module load StdEnv/2016 ansys/2020R1; A=201; B=5.10.1, or

module load StdEnv/2016 ansys/2019R3; A=195; B=5.10.1, or

export LD_LIBRARY_PATH=$EBROOTANSYS/v$A/CEI/apex$A/machines/linux_2.6_64/qt-$B/lib

ensight -X

SSH Considerations Some ANSYS gui programs can be run remotely on a cluster compute node by X forwarding over ssh to your local desktop. Unlike VNC, this approach is untested and unsupported since it relies on a properly setup X display server for your particular operating system OR the selection, installation and configuration of a suitable X client emulator package such as MobaXterm. Most users will find interactive response times unacceptably slow for basic menu tasks let alone performing more complex tasks such as those involving graphics rendering. Startup times for gui programs can also be very slow depending on your internet connection. For example, in one test it took 40min to fully start ansysedt over ssh while starting it with vncviewer required on 34 seconds. Despite these stark facts the approach may be of interest if the only goal is to open a simulation and run some calculations in the gui. Therefore the basic steps are given here as a starting point: 1) ssh -Y username@graham.computecanada.ca 2) salloc --x11 --time=1:00:00 --cpus-per-task=1 --mem=16000 --account=def-mygroup 3) once on a compute node try running xclock to check the analog clock appears on your desktop, if it does then 4) load the needed modules and try running the program.

VDI Nodes

ANSYS programs can be run for upto 24hours on graham VDI Nodes using a maximum of 8cores and 128GB memory. The VDI System provides gpu OpenGL acceleration therefore it is ideal for performing tasks that benefit from high performance graphics. One might use VDI to create or modify simulation input files, post process data or visualize simulation results. To get started, login to gra-vdi.computecanada.ca with TigerVNC then open a new terminal window and start one of the following supported program versions as shown below. The vertical bar | notation is used to separate the various ANSYS commands.

FLUIDS

module load CcEnv StdEnv/2020 ansys/2022R1, or

module load CcEnv StdEnv/2020 ansys/2021R2, or

module load CcEnv StdEnv/2020 ansys/2021R1

unset SESSION_MANAGER

fluent | cfx5 | icemcfd

NOTE: If icemcfd crashes on startup click Settings -> Display -> X11

------------------------------------------------------------------------------------

module load CcEnv StdEnv/2016 ansys/2020R2, or

module load CcEnv StdEnv/2016 ansys/2020R1, or

module load CcEnv StdEnv/2016 ansys/2019R3

export HOOPS_PICTURE=opengl

fluent | cfx5 | icemcfd

WORKBENCH

module load CcEnv StdEnv/2020 ansys/2022R1

module load CcEnv StdEnv/2020 ansys/2021R2

runwb2

------------------------------------------------------------------------------------

module load SnEnv ansys/2021R1, or

module load SnEnv ansys/2020R2

runwb2

------------------------------------------------------------------------------------

module load CcEnv StdEnv/2016 ansys/2020R1, or

module load CcEnv StdEnv/2016 ansys/2019R3

export PATH=$EBROOTNIXPKGS/bin:$PATH

runwb2

NOTE: The pull downs in the left side menu that typically default to Program Controlled do NOT work when various Ansys Systems are loaded such as Steady-State Thermal, Explicit Dynamics and so forth. Trying to use the pull downs repeatedly on gra-vdi may crash Ansys in such a way that "pkill -9 ansys" should be run afterward to completely clear any lingering license server connections before restarting workbench. As a workaround, quit out of gra-vdi and then open your project under workbench on a compute node within an salloc session. Make the desired pulldown changes then click Home -> Save Project -> Close Mechanical, File -> Exit then also exit your salloc session. Now return to gra-vdi where the changed settings should appear when you reopen the project under workbench. This message will be updated if a fix for this issue is found.

ANSYS EDT

module load CcEnv StdEnv/2020 ansysedt/2021R2

rm -rf ~/.mw (optional: force first-time configuration)

ansysedt

ENSIGHT

module load SnEnv ansys/2022R1, or

module load SnEnv ansys/2021R2, or (see NOTE below)

module load SnEnv ansys/2021R1, or (see NOTE below)

module load SnEnv ansys/2020R2, or (see NOTE below)

module load SnEnv ansys/2020R1, or

module load SnEnv ansys/2019R3, or

module load SnEnv ansys/2019R2

ensight

NOTE: Please be aware versions 2021R2, 2021R1, 2020R2 have a compatibility issue on gra-vdi that effects how some tic boxes and menu tabs in the left hand panels update when clicked. As a workaround, try dragging all panels that are overlaid (and thus have selector tabs) outside the main window so they are separate (and thus no longer have selector tabs). If a panel becomes blurred click the tiny maximize button in its upper right hand corner (to the left of the tiny x button) to refresh it. If you still experience this problem then 1) use a different version not impacted by this issue, or 2) use the same version on a compute node. If none of these solutions work then open a problem ticket and let us know.

Site Specific Usage

Sharcnet License

The SHARCNET Ansys license is free for use by any Compute Canada user on any Compute Canada system. Similar to the commercial version the software has no solver or geometry limits however it may only be used for the purpose of Publishable Academic Research. The license was upgraded from CFD to MCS (Multiphysics Campus Solution) in May of 2020 and includes the following ANSYS products: HF, EM, Electronics HPC, Mechanical and CFD as described here. Neither LS-DYNA or Lumerical are included. In July of 2021 an additional 1024 anshpc licenses were added as a result researchers now start upto 4 jobs Compute Canada wide using a total of 124 anshpc (approximately double that of 2020) plus 4 anshpc per job. Therefore a single 128 core job OR two 64 core jobs can be submitted to run on four OR two full 32 core Graham Broadwell nodes respectively. A further limit increase to 172 anshpc is presently being considered to support launching 176 core jobs onto four full 44 core Graham Cascade Lake nodes. The SHARCNET Ansys License is made available on a first come first serve basis. Therefore if a larger than usual number of ANSYS jobs are submitted on a given day some jobs could fail on startup should insufficient licenses be available. These events however are expected to be rare given the recent increase in anshpc licenses. If your research requires more licenses than are available from SHARCNET then a dedicated researcher purchased license will be required. Researchers can purchase an Ansys license directly from Simutech where an extra 20% country wide uplift fee must be added if the cluster where the license will be used is not co-located at your institution. A dedicated Ansys license can be hosted on a local institutional license server OR transferred part or in full on the SHARCNET Ansys License server. In the former case the researcher would simply need to reconfigure their ~/.licenses/ansys.lic file on graham (or beluga or cedar) as described at the top of this wiki page. In the later case the researcher should instead open a ticket with SHARCNET to inquire about starting the license transfer process. Depending on the SHARCNET Ansys license utilization the per user feature limits may need to be changed. Advanced notice will be posted here a minimum of 2 weeks in advance. Large jobs that do not achieve an effective cpu utilization of at least 30% will be flagged by the system and you will likely be contacted by a Compute Canada analyst.

License Server File

To use the Sharcnet ANSYS license configure your ansys.lic file as follows:

[gra-login1:~/.licenses] cat ansys.lic
setenv("ANSYSLMD_LICENSE_FILE", "1055@license3.sharcnet.ca")
setenv("ANSYSLI_SERVERS", "2325@license3.sharcnet.ca")

Query License Server

To show the number of license in use by your username and the total in use by all users run:

ssh graham.computecanada.ca
module load ansys
lmutil lmstat -c $ANSYSLMD_LICENSE_FILE -a | grep "Users of\|$USER"

If you discover any licenses unexpectedly in use by your username (usually due to ansys not exiting cleanly on gra-vdi) then connect to the node where its running, open a terminal window and run the following command to terminate the rogue processes pkill -9 -e -u $USER -f "ansys" after which your licenses should be freed. Note that gra-vdi consists of two nodes (gra-vdi3 and gra-vdi4) which researchers are randomly placed on when connecting to gra-vdi.computecanada.ca with tigervnc. Therefore its necessary to specify the full hostname (gra-vdi3.sharcnet.ca or grav-vdi4.sharcnet.ca) when connecting with tigervnc to ensure you login to the correct node before running pkill.

Local VDI Modules

When using gra-vdi researchers have the choice of loading ANSYS modules from the global Compute Canada environment (after loading CcEnv) or loading ANSYS modules installed locally on the machine itself (after loading SnEnv). The local modules maybe of interest as they include some Ansys programs and versions not yet supported by the Compute Canada environment for graphics use on gra-vdi or the clusters. When starting programs from local Ansys modules, users can select the CMC license server or accept the default Sharcnet License server. Presently the settings from ~/.licenses/ansys.lic are not used by the local Ansys modules except when starting runwb2 where they will override the default Sharcnet License server settings. Suitable usage of Ansys programs on gra-vdi includes: running a single test job interactively with upto 8cores and/or 128G ram, create or modify simulation input files, post process or visualize data.

ansys Modules

1. Connect to gra-vdi.computecanada.ca with TigerVNC
2. Open a new terminal window and load a module:

   module load SnEnv ansys/2021R2, or

   module load SnEnv ansys/2021R1, or

   module load SnEnv ansys/2020R2, or

   module load SnEnv ansys/2020R1, or

   module load SnEnv ansys/2019R3

3. Start an ANSYS program by issuing one of the following:

   runwb2|fluent|cfx5|icemcfd|apdl

4. Press y then enter to accept the conditions
5. Press enter to accept the n option and use the SHARCNET license server by default (in the case of runwb2 ~/.licenses/ansysedt.lic will be used if present otherwise ANSYSLI_SERVERS and ANSYSLMD_LICENSE_FILE will be used if set in your environment for example to some other remote license server). If you change n to y and hit enter the CMC license server will be used.

where cfx5 from step 3. above provides the option to start the following components:

   1) CFX-Launcher  (cfx5 -> cfx5launch)
   2) CFX-Pre       (cfx5pre)
   3) CFD-Post      (cfdpost -> cfx5post)
   4) CFX-Solver    (cfx5solve)

ansysedt Modules

1. Connect to gra-vdi.computecanada.ca with TigerVNC
2. Open a new terminal window and load a module:

   module load SnEnv ansysedt/2021R2, or

   module load SnEnv ansysedt/2021R1

3. Start the ANSYS Electromagnetics Desktop program by typing the following command: ansysedt
4. Press y then enter to accept the conditions.
5. Press enter to accept the n option and use the SHARCNET license server by default (note that ~/.licenses/ansysedt.lic will be used if present otherwise ANSYSLI_SERVERS and ANSYSLMD_LICENSE_FILE will be used if set in your environment for example to some other remote license server). If you change n to y and hit enter then the CMC license server will be used.

License feature preferences previously setup with anslic_admin are no longer supported following the recent SHARCNET license server update (Sept9/2021). If a license problem occurs try removing the ~/.ansys directory in your home account to clear the settings. If problems persist please contact our Technical support and provide the contents your ~/.licenses/ansys.lic file.

Additive Manufacturing

To get started configure your ~/.licenses/ansys.lic file to point to a license server that has a valid ANSYS Mechanical License. This must be done on all systems where you plan to run the software.

Enable Additive

To enable ANSYS Additive Manufacturing in your project do the following 3 steps:

Start Workbench

• start workbench as described in the Graphical Use - WORKBENCH section found above.

Install Extension

• click Extensions -> Install Extension
• specify the following /path/to/AdditiveWizard.wbex then click Open: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ansys/2019R3/v195/aisol/WBAddins/MechanicalExtensions/AdditiveWizard.wbex

Load Extension

• click Extensions -> Manage Extensions and tick Additive Wizard
• click the ACT Start Page tab X to return to your Project tab

Run Additive

Gra-vdi

A user can run a single ANSYS Additive Manufacturing job on gra-vdi with upto 16cores as follows:

• Start Workbench On Gra-vdi as described above in Enable Additive
• click File -> Open and select test.wbpj then click Open
• click View -> reset workspace if you get a grey screen
• start Mechanical, Clear Generated Data, tick Distributed, specify Cores
• click File -> Save Project -> Solve

Check utilization:

• open another terminal and run: top -u $USER
• kill rogue processes from previous runs if required: pkill -9 -e -u $USER -f "ansys"

Cluster

Project preparation:

Before submitting a newly uploaded Additive project to a cluster queue (with sbatch scriptname) certain preparations must be done. To begin, open your simulation with Workbench gui (as described in the Enable Additive section above) in the same directory that your job will be submitted from and then save it again. Be sure to use the same ansys module version that will be used for the job. Next create a slurm script (as explained in the Cluster Batch Job Submission - WORKBENCH section above). To perform parametric studies change Update() to UpdateAllDesignPoints() in the slurm script. Determine the optimal number of cores and memory by submitting several short test jobs. To avoid needing to manually clear the solution and recreate all the design points in Workbench between each test run, either 1) change Save(Overwrite=True) to Save(Overwrite=False) or 2) save a copy of the original YOURPROJECT.wbpj file and corresponding YOURPROJECT_files directory. Optionally create and then manually run a replay file on the cluster in the respective test case directory between each run, noting that single replay file can be used in different directories by opening it in a text editor and changing the internal FilePath setting.

module load ansys/2019R3
rm -f test_files/.lock
runwb2 -R myreplay.wbjn

Resource utilization:

Once your additive job has been running for a few minutes a snapshot of its resource utilization on the compute node(s) can be obtained with the following the srun command. Sample output corresponding to an eight core submission script is shown next. It can be seen that two nodes were selected by the scheduler:

[gra-login1:~] srun --jobid=myjobid top -bn1 -u $USER | grep R | grep -v top
  PID USER   PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+  COMMAND
22843 demo   20   0 2272124 256048  72796 R  88.0  0.2  1:06.24  ansys.e
22849 demo   20   0 2272118 256024  72822 R  99.0  0.2  1:06.37  ansys.e
22838 demo   20   0 2272362 255086  76644 R  96.0  0.2  1:06.37  ansys.e
  PID USER   PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+  COMMAND
 4310 demo   20   0 2740212 271096 101892 R 101.0  0.2  1:06.26  ansys.e
 4311 demo   20   0 2740416 284552  98084 R  98.0  0.2  1:06.55  ansys.e
 4304 demo   20   0 2729516 268824 100388 R 100.0  0.2  1:06.12  ansys.e
 4305 demo   20   0 2729436 263204 100932 R 100.0  0.2  1:06.88  ansys.e
 4306 demo   20   0 2734720 431532  95180 R 100.0  0.3  1:06.57  ansys.e

Scaling tests:

After a job completes its "Job Wall-clock time" can be obtained from seff myjobid. Using this value scaling tests can be performed by submitting short test jobs with an increasing number of cores. If the Wall-clock time decreases by ~50% when the number of cores are doubled then additional cores maybe considered.

Online Documentation

The full ANSYS documentation for versions back to 19.2 can be accessed by following these steps:

1. connect to gra-vdi.computecanada.ca with tigervnc as described in VDI Nodes
2. if Firefox browser or ANSYS Workbench are open then close them now
3. start Firefox by clicking: Applications -> Internet -> Firefox
4. open a new terminal window by clicking: Applications -> System Tools -> Mate Terminal
5. start Workbench by typing the following in your terminal: module load CcEnv StdEnv ansys; runwb2
6. in the upper Workbench menu bar click: Help -> ANSYS Workbench Help
7. the Ansys documentation page should immediately appear in Firefox
8. at this point Workbench is no longer needed so close it by clicking the Unsaved Project - Workbench tab located along the bottom frame (doing this will bring workbench into focus) and then click File -> Exit
9. in the top middle of the Ansys documentation page click the word HOME located just left of API DOCS
10. now scroll down and you should see a list of ANSYS product icons and/or Alphabetical Ranges
11. select a product to view the documentation for, such as Fluent. the documentation for the latest release version will be displayed by default. change the documentation version by clicking the Release Year R pull down located above and just to the right of the the Ansys documentation page search bar.
12. to search for documentation corresponding to a different ANSYS product click HOME again

Message Passing Interface