Ansys

Ansys is a software suite for engineering simulation and 3-D design. It includes packages such as Ansys Fluent and Ansys CFX.

Licensing

We are a hosting provider for Ansys. This means that we have the software installed on our clusters, but we do not provide a generic license accessible to everyone. However, many institutions, faculties, and departments already have licenses that can be used on our clusters. Once the legal aspects are worked out for licensing, there will be remaining technical aspects. The license server on your end will need to be reachable by our compute nodes. This will require our technical team to get in touch with the technical people managing your license software. In some cases, this has already been done. You should then be able to load the Ansys module, and it should find its license automatically. If this is not the case, please contact our technical support so that they can arrange this for you.

Configuring your license file

Our module for Ansys is designed to look for license information in a few places. One of those places is your /home folder. You can specify your license server by creating a file named $HOME/.licenses/ansys.lic consisting of two lines as shown. Customize the file to replacing FLEXPORT, INTEPORT and LICSERVER with appropriate values for your server.

FILE: ansys.lic

setenv("ANSYSLMD_LICENSE_FILE", "FLEXPORT@LICSERVER")

setenv("ANSYSLI_SERVERS", "INTEPORT@LICSERVER")

The following table provides established values for the CMC and SHARCNET license servers. To use a different server, locate the corresponding values as explained in Local license servers.

TABLE: Preconfigured license servers

License	System/Cluster	LICSERVER	FLEXPORT	INTEPORT	VENDPORT	NOTICES
CMC	beluga	10.20.73.21	6624	2325	n/a	None
CMC	cedar	172.16.0.101	6624	2325	n/a	None
CMC	graham	199.241.167.222	6624	2325	n/a	None
CMC	narval	10.100.64.10	6624	2325	n/a	None
SHARCNET	beluga/cedar/graham/gra-vdi/narval	license3.sharcnet.ca	1055	2325	n/a	None
SHARCNET	niagara	localhost	1055	2325	1793	None

Researchers who purchase a CMC license subscription must send their Alliance account username to <cmcsupport@cmc.ca> otherwise license checkouts will fail. The number of cores that can be used with a CMC license is described in the Other Tricks and Tips sections of the Ansys Electronics Desktop and Ansys Mechanical/Fluids quick start guides.

Local license servers

Before a local Ansys license server can be reached from an Alliance cluster, firewall changes will need to be done on both the server side and the Alliance side. For many local institutional servers this work has already been done. In such cases you simply need to contact your local Ansys license server administrator and request 1) the fully qualified hostname (LICSERVER) of the server; 2) the Ansys flex port commonly 1055 (FLEXPORT); and 3) the Ansys licensing interconnect port commonly 2325 (INTEPORT). With this information you can then immediately configure your ansys.lic file as described above and theoretically begin submitting jobs.

If however your local license server has never been setup for use on the Alliance, you will additionally need to request 4) the static vendor port number (VENDPORT) number from your local Ansys server administrator. Once you have gathered all four pieces of information send it to technical support being sure to mention which Alliance cluster(s) you want to run Ansys on. We will then arrange for the Alliance firewall to be opened so that license requests on the cluster(s) can reach your server. You will then also receive a range of IP addresses to pass to your server administrator so the local firewall can likewise be opened to allow inbound license connections to reach your server on the 3 ports (FLEXPORT, INTEPORT, VENDPORT) from the requested Alliance system(s).

Checking license usage

Ansys comes with an lmutil tool that can be used to check your license usage. Before using it verify your ansys.lic is configured. Then run the following two commands on a cluster that you are set up to use:

[name@server ~]$ module load ansys/2023R2
[name@server ~]$ $EBROOTANSYS/v232/licensingclient/linx64/lmutil lmstat -c $ANSYSLMD_LICENSE_FILE -S ansyslmd

If you load a different version of the Ansys module, you will need to modify the path to the lmutil command.

Version compatibility

Platform Support

The Ansys Platform Support page states "the current release has been tested to read and open databases from the five previous releases". This implies that simulations developed using older versions of Ansys should generally work with newer module versions (forward compatible five releases). The reverse however cannot be assumed to be true. The Platform Support link also provides version based software and hardware compatibility information to determine optimal (supported) platform infrastructure that Ansys can be run on. The features supported under Windows VS Linux systems can be displayed by clicking the Platform Support by Application / Product link. Similar information for all of the above maybe found by clicking the Previous Releases link located at the lower left corner of the Platform Support page.

What's New

Ansys posts Product Release and Updates for the latest releases. Similar information for previous releases can generally be pulled up for various application topics by visiting the Ansys blog page and using the FILTERS search bar. For example searching on What’s New Fluent 2024 gpu pulls up a document with title What’s New for Ansys Fluent in 2024 R1? containing a wealth of the latest gpu support information. Specifying a version number in the Press Release search bar is also a good way to find new release information. At the time of this writing Ansys 2024R2 is the current release and will be installed when interest is expressed or there is evident need to support newer hardware or solver capabilities. To request a new version be installed submit a problem ticket.

Service Packs

Ansys regularly releases service packs to fix and enhance various issues with its major releases. Therefore starting with Ansys 2024 a separate ansys module will appear on the clusters with a decimal and two digits appearing after the release number whenever a service pack is been installed over the initial release. For example, the initial 2024 release without any service pack applied maybe loaded by doing module load ansys/2024R1 while a module with Service Pack 3 applied maybe loaded by doing module load ansys/2024R1.03 instead. If a service pack is already available by the time a new release is to be installed, then most likely only a module for that service pack number will be installed unless otherwise a requeste to install the initial release also is received.

Most users will likely want to load the latest module version equipped with latest installed service pack which can be achieved by simply doing module load ansys. While its not expected service packs will impact numerical results, the changes they make are extensive and so if computations have already been done with the initial release or an earlier service pack than some groups may prefer to continue using it. Having separate modules for each service pack makes this possible. Starting with Ansys 2024R1 a detailed description of what each service pack does can be found by searching this link for Service Pack Details. Future versions will presumably be similarly searchable by manually modifying the version number contained in the link.

Cluster batch job submission

The Ansys software suite comes with multiple implementations of MPI to support parallel computation. Unfortunately, none of them support our Slurm scheduler. For this reason, we need special instructions for each Ansys package on how to start a parallel job. In the sections below, we give examples of submission scripts for some of the packages. While the slurm scripts should work with on all clusters, Niagara users may need to make some additional changes covered here.

Ansys Fluent

Typically, you would use the following procedure to run Fluent on one of our clusters:

1. Prepare your Fluent job using Fluent from the Ansys Workbench on your desktop machine up to the point where you would run the calculation.
2. Export the "case" file with File > Export > Case… or find the folder where Fluent saves your project's files. The case file will often have a name like FFF-1.cas.gz.
3. If you already have data from a previous calculation, which you want to continue, export a "data" file as well (File > Export > Data…) or find it in the same project folder (FFF-1.dat.gz).
4. Transfer the case file (and if needed the data file) to a directory on the /project or /scratch filesystem on the cluster. When exporting, you can save the file(s) under a more instructive name than FFF-1.* or rename them when they are uploaded.
5. Now you need to create a "journal" file. It's purpose is to load the case file (and optionally the data file), run the solver and finally write the results. See examples below and remember to adjust the filenames and desired number of iterations.
6. If jobs frequently fail to start due to license shortages and manual resubmission of failed jobs is not convenient, consider modifying your script to requeue your job (up to 4 times) as shown under the by node + requeue tab further below. Be aware that doing this will also requeue simulations that fail due to non-license related issues (such as divergence), resulting in lost compute time. Therefore it is strongly recommended to monitor and inspect each Slurm output file to confirm each requeue attempt is license related. When it is determined that a job is requeued due to a simulation issue, immediately manually kill the job progression with scancel jobid and correct the problem.
7. After running the job, you can download the data file and import it back into Fluent with File > Import > Data….

Slurm scripts

General purpose

Most Fluent jobs should use the following by node script to minimize solution latency and maximize performance over as few nodes as possible. Very large jobs, however, might wait less in the queue if they use a by core script. However, the startup time of a job using many nodes can be significantly longer, thus offsetting some of the benefits. In addition, be aware that running large jobs over an unspecified number of potentially very many nodes will make them far more vulnerable to crashing if any of the compute nodes fail during the simulation. The scripts will ensure Fluent uses shared memory for communication when run on a single node or distributed memory (utilizing MPI and the appropriate HPC interconnect) when run over multiple nodes. The two narval tabs maybe be useful to provide a more robust alternative if fluent hangs during the initial auto mesh partitioning phase when using the standard intel based scripts with the parallel solver. The other option would be to manually perform the mesh partitioning in the fluent gui then trying to run the job again on the cluster with the intel scripts. Doing so will allow you to inspect the partition statistics and specify the partitioning method to obtain an optimal result. The number of mesh partitions should be an integral multiple of the there number of cores. For optimal efficiency ensure at least 10000 cells per core otherwise specifying too many cores will eventually result in the poor performance as the scaling drops off.

Multinode (by node)Multinode (by core)Multinode (by node, narval)Multinode (by core, narval)Multinode (by node, niagara)

File : script-flu-bynode-intel.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes
#SBATCH --ntasks-per-node=32  # Specify number of cores per node (graham 32 or 44, cedar 48, beluga 40, narval 64, or less)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change

rm -f cleanup* core*

#module load StdEnv/2020
#module load ansys/2019R3     # or newer versions (narval only)
#module load ansys/2021R2     # or newer versions (beluga, cedar, graham)

module load StdEnv/2023
module load ansys/2023R2      # or newer versions (beluga, cedar, graham, narval)

if [[ "${CC_CLUSTER}" == narval ]]; then
 if [ "$EBVERSIONGENTOO" == 2020 ]; then
   module load intel/2021 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
   export HCOLL_RCACHE=^ucs
 elif [ "$EBVERSIONGENTOO" == 2023 ]; then
   module load intel/2023 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT
 fi
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou
fi

File : script-flu-bycore-intel.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=1            # Uncomment to specify number of compute nodes (optional)
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change

rm -f cleanup* core*

#module load StdEnv/2020
#module load ansys/2019R3     # or newer versions (narval only)
#module load ansys/2021R2     # or newer versions (beluga, cedar, graham)

module load StdEnv/2023
module load ansys/2023R2      # or newer module versions (beluga, cedar, graham, narval)

if [[ "${CC_CLUSTER}" == narval ]]; then
 if [ "$EBVERSIONGENTOO" == 2020 ]; then
   module load intel/2021 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
   export HCOLL_RCACHE=^ucs
 elif [ "$EBVERSIONGENTOO" == 2023 ]; then
   module load intel/2023 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT
 fi
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
fi

File : script-flu-bynode-openmpi.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes
#SBATCH --ntasks-per-node=64  # Specify number of cores per node (narval 64 or less)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change

rm -f cleanup* core*

module load StdEnv/2023       
module load ansys/2023R2      # or newer versions (narval)

export OPENMPI_ROOT=$EBROOTOPENMPI
slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/mf-$SLURM_JOB_ID
for i in `cat /tmp/mf-$SLURM_JOB_ID | uniq`; do echo "${i}:$(cat /tmp/mf-$SLURM_JOB_ID | grep $i | wc -l)" >> /tmp/machinefile-$SLURM_JOB_ID; done
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=openmpi -ssh -i sample.jou
fi

File : script-flu-bycore-openmpi.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=1            # Uncomment to specify number of compute nodes (optional)
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change

rm -f cleanup* core*

module load StdEnv/2023       
module load ansys/2023R2      # or newer versions (narval)

export OPENMPI_ROOT=$EBROOTOPENMPI
slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/mf-$SLURM_JOB_ID
for i in `cat /tmp/mf-$SLURM_JOB_ID | uniq`; do echo "${i}:$(cat /tmp/mf-$SLURM_JOB_ID | grep $i | wc -l)" >> /tmp/machinefile-$SLURM_JOB_ID; done
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=openmpi -ssh -i sample.jou
fi

File : script-flu-bynode-intel-nia.sh

#!/bin/bash

#SBATCH --account=def-group      # Specify account name
#SBATCH --time=00-03:00          # Specify time limit dd-hh:mm
#SBATCH --nodes=1                # Specify number of compute nodes
#SBATCH --ntasks-per-node=80     # Specify number cores per node (niagara 80 or less)
#SBATCH --mem=0                  # Do not change (allocate all memory per compute node)
#SBATCH --cpus-per-task=1        # Do not change (required parameter)

rm -f cleanup* core*

module load CCEnv StdEnv/2023    # Do not change
module load ansys/2023R2         # or newer versions (niagara)

# Customize this section to specify your remote license server
# These settings are used instead of your ~/.licenses/ansys.lic
LICSERVER=license3.sharcnet.ca   # Specify license server hostname
FLEXPORT=1055                    # Specify server flex port
INTEPORT=2325                    # Specify server interconnect port
VENDPORT=1793                    # Specify server vendor port

ssh nia-gw -fNL $FLEXPORT:$LICSERVER:$FLEXPORT      # Do not change
ssh nia-gw -fNL $INTEPORT:$LICSERVER:$INTEPORT      # Do not change
ssh nia-gw -fNL $VENDPORT:$LICSERVER:$VENDPORT      # Do not change
export ANSYSLMD_LICENSE_FILE=$FLEXPORT@localhost    # Do not change
export ANSYSLI_SERVERS=$INTEPORT@localhost          # Do not change

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

if [ ! -L "$HOME/.ansys" ]; then
  echo "ERROR: A link to a writable .ansys directory does not exist."
  echo 'Remove ~/.ansys if one exists and then run: ln -s $SCRATCH/.ansys ~/.ansys'
  echo "Then try submitting your job again. Aborting the current job now!"
elif [ ! -L "$HOME/.fluentconf" ]; then
  echo "ERROR: A link to a writable .fluentconf directory does not exist."
  echo 'Remove ~/.fluentconf if one exists and run: ln -s $SCRATCH/.fluentconf ~/.fluentconf'
  echo "Then try submitting your job again. Aborting the current job now!"
elif [ ! -L "$HOME/.flrecent" ]; then
  echo "ERROR: A link to a writable .flrecent file does not exist."
  echo 'Remove ~/.flrecent if one exists and then run: ln -s $SCRATCH/.flrecent ~/.flrecent'
  echo "Then try submitting your job again. Aborting the current job now!"
else
  mkdir -pv $SCRATCH/.ansys
  mkdir -pv $SCRATCH/.fluentconf
  touch $SCRATCH/.flrecent
  # Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename
  if [ "$SLURM_NNODES" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou
  fi
fi

License requeue

The slurm scripts in this section are suitable for Fluent jobs that thoroughly tested but typically require multiple requeue attempts to checkout licenses. They are not recommended for Fluent jobs that may 1) run for a long time before crashing 2) run to completion but contain unresolved journal file warnings, since in both cases the simulations will be repeated from the beginning until the maximum number of requeue attempts specified by the array value is reached. For these types of jobs the general purpose scripts above should be used instead.

Multinode (by node + requeue)Multinode (by core + requeue)

File : script-flu-bynode+requeue.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes
#SBATCH --ntasks-per-node=32  # Specify number of cores per node (graham 32 or 44, cedar 48, beluga 40, or less)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of requeue attempts (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2020
#module load ansys/2019R3     # or newer versions (narval only)
#module load ansys/2021R2     # or newer versions (beluga, cedar, graham)

module load StdEnv/2023
module load ansys/2023R2      # or newer versions (beluga, cedar, graham, narval)

if [[ "${CC_CLUSTER}" == narval ]]; then
 if [ "$EBVERSIONGENTOO" == 2020 ]; then
   module load intel/2021 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
   export HCOLL_RCACHE=^ucs
 elif [ "$EBVERSIONGENTOO" == 2023 ]; then
   module load intel/2023 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT
 fi
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else 
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
fi
if [ $? -eq 0 ]; then
    echo "Job completed successfully! Exiting now."
    scancel $SLURM_ARRAY_JOB_ID
else
    echo "Job attempt $SLURM_ARRAY_TASK_ID of $SLURM_ARRAY_TASK_COUNT failed due to license or simulation issue!"
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
       echo "Resubmitting job now …"
    else
       echo "All job attempts failed exiting now."
    fi
fi

File : script-flu-bycore+requeue.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=1            # Uncomment to specify number of compute nodes (optional) 
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of requeue attempts (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2020
#module load ansys/2019R3     # or newer versions (narval only)
#module load ansys/2021R2     # or newer versions (beluga, cedar, graham)

module load StdEnv/2023
module load ansys/2023R2      # or newer versions (beluga, cedar, graham, narval)

if [[ "${CC_CLUSTER}" == narval ]]; then
 if [ "$EBVERSIONGENTOO" == 2020 ]; then
   module load intel/2021 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
   export HCOLL_RCACHE=^ucs
 elif [ "$EBVERSIONGENTOO" == 2023 ]; then
   module load intel/2023 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT
 fi
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else 
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
fi
if [ $? -eq 0 ]; then
    echo "Job completed successfully! Exiting now."
    scancel $SLURM_ARRAY_JOB_ID
else
    echo "Job attempt $SLURM_ARRAY_TASK_ID of $SLURM_ARRAY_TASK_COUNT failed due to license or simulation issue!"
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
       echo "Resubmitting job now …"
    else
       echo "All job attempts failed exiting now."
    fi
fi

Solution restart

The following two scripts are provided to automate restarting very large jobs that require more than the typical seven-day maximum runtime window available on most clusters. Jobs are restarted from the most recent saved time step files. A fundamental requirement is the first time step can be completed within the requested job array time limit (specified at the top of your Slurm script) when starting a simulation from an initialized solution field. It is assumed that a standard fixed time step size is being used. To begin, a working set of sample.cas, sample.dat and sample.jou files must be present. Next edit your sample.jou file to contain /solve/dual-time-iterate 1 and /file/auto-save/data-frequency 1. Then create a restart journal file by doing cp sample.jou sample-restart.jou and edit the sample-restart.jou file to contain /file/read-cas-data sample-restart instead of /file/read-cas-data sample and comment out the initialization line with a semicolon for instance ;/solve/initialize/initialize-flow. If your 2nd and subsequent time steps are known to run twice as fast (than the initial time step), edit sample-restart.jou to specify /solve/dual-time-iterate 2. By doing this, the solution will only be restarted after two 2 time steps are completed following the initial time step. An output file for each time step will still be saved in the output subdirectory. The value 2 is arbitrary but should be chosen such that the time for 2 steps fits within the job array time limit. Doing this will minimize the number of solution restarts which are computationally expensive. If your first time step performed by sample.jou starts from a converged (previous) solution, choose 1 instead of 2 since likely all time steps will require a similar amount of wall time to complete. Assuming 2 is chosen, the total time of simulation to be completed will be 1*Dt+2*Nrestart*Dt where Nrestart is the number of solution restarts specified in the script. The total number of time steps (and hence the number of output files generated) will therefore be 1+2*Nrestart. The value for the time resource request should be chosen so the initial time step and subsequent time steps will complete comfortably within the Slurm time window specifiable up to a maximum of "#SBATCH --time=07-00:00" days.

Multinode (by node + restart)Multinode (by core + restart)

File : script-flu-bynode+restart.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=07-00:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes
#SBATCH --ntasks-per-node=32  # Specify number of cores per node (graham 32 or 44, cedar 48, beluga 40, narval 64, or less)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of solution restarts (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2020
#module load ansys/2019R3     # or newer versions (narval only)
#module load ansys/2021R2     # or newer versions (beluga, cedar, graham)

module load StdEnv/2023
module load ansys/2023R2      # or newer versions (beluga, cedar, graham, narval)

if [[ "${CC_CLUSTER}" == narval ]]; then
 if [ "$EBVERSIONGENTOO" == 2020 ]; then
   module load intel/2021 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
   export HCOLL_RCACHE=^ucs
 elif [ "$EBVERSIONGENTOO" == 2023 ]; then
   module load intel/2023 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT
 fi
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample-restart.jou
  fi
else 
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample-restart.jou 
  fi
fi
if [ $? -eq 0 ]; then
    echo
    echo "SLURM_ARRAY_TASK_ID  = $SLURM_ARRAY_TASK_ID"
    echo "SLURM_ARRAY_TASK_COUNT = $SLURM_ARRAY_TASK_COUNT"
    echo
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
      echo "Restarting job with the most recent output dat file …"
      ln -sfv output/$(ls -ltr output | grep .cas | tail -n1 | awk '{print $9}') sample-restart.cas.gz
      ln -sfv output/$(ls -ltr output | grep .dat | tail -n1 | awk '{print $9}') sample-restart.dat.gz
      ls -lh cavity* output/*
    else
      echo "Job completed successfully! Exiting now."
      scancel $SLURM_ARRAY_JOB_ID
     fi
else
     echo "Simulation failed. Exiting …"
fi

File : script-flu-bycore+restart.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=1            # Uncomment to specify number of compute nodes (optional)
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of restart aka time steps (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2020
#module load ansys/2019R3     # or newer versions (narval only)
#module load ansys/2021R2     # or newer versions (beluga, cedar, graham)

module load StdEnv/2023
module load ansys/2023R2      # or newer versions (beluga, cedar, graham, narval)

if [[ "${CC_CLUSTER}" == narval ]]; then
 if [ "$EBVERSIONGENTOO" == 2020 ]; then
   module load intel/2021 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
   export HCOLL_RCACHE=^ucs
 elif [ "$EBVERSIONGENTOO" == 2023 ]; then
   module load intel/2023 intelmpi
   export INTELMPI_ROOT=$I_MPI_ROOT
 fi
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample-restart.jou
  fi
else 
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample-restart.jou 
  fi
fi
if [ $? -eq 0 ]; then
    echo
    echo "SLURM_ARRAY_TASK_ID  = $SLURM_ARRAY_TASK_ID"
    echo "SLURM_ARRAY_TASK_COUNT = $SLURM_ARRAY_TASK_COUNT"
    echo
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
      echo "Restarting job with the most recent output dat file …"
      ln -sfv output/$(ls -ltr output | grep .cas | tail -n1 | awk '{print $9}') sample-restart.cas.gz
      ln -sfv output/$(ls -ltr output | grep .dat | tail -n1 | awk '{print $9}') sample-restart.dat.gz
      ls -lh cavity* output/*
    else
      echo "Job completed successfully! Exiting now."
      scancel $SLURM_ARRAY_JOB_ID
     fi
else
     echo "Simulation failed. Exiting …"
fi

Journal files

Fluent journal files can include basically any command from Fluent's Text-User-Interface (TUI); commands can be used to change simulation parameters like temperature, pressure and flow speed. With this you can run a series of simulations under different conditions with a single case file, by only changing the parameters in the journal file. Refer to the Fluent User's Guide for more information and a list of all commands that can be used. The following journal files are set up with /file/cff-files no to use the legacy .cas/.dat file format (the default in module versions 2019R3 or older). Set this instead to /file/cff-files yes to use the more efficient .cas.h5/.dat.h5 file format (the default in module versions 2020R1 or newer).

Journal file (steady, case)Journal file (steady, case + data)Journal file (transient)

File : sample1.jou

; SAMPLE FLUENT JOURNAL FILE - STEADY SIMULATION
; ----------------------------------------------
; lines beginning with a semicolon are comments

; Overwrite files by default
/file/confirm-overwrite no

; Preferentially read/write files in legacy format
/file/cff-files no

; Read input case and data files
/file/read-case-data FFF-in

; Run the solver for this many iterations
/solve/iterate 1000

; Overwrite output files by default
/file/confirm-overwrite n

; Write final output data file
/file/write-case-data FFF-out

; Write simulation report to file (optional)
/report/summary y "My_Simulation_Report.txt"

; Cleanly shutdown fluent
/exit

File : sample2.jou

; SAMPLE FLUENT JOURNAL FILE - STEADY SIMULATION
; ----------------------------------------------
; lines beginning with a semicolon are comments

; Overwrite files by default
/file/confirm-overwrite no

; Preferentially read/write files in legacy format
/file/cff-files no

; Read input files
/file/read-case-data FFF-in

; Write a data file every 100 iterations
/file/auto-save/data-frequency 100

; Retain data files from 5 most recent iterations
/file/auto-save/retain-most-recent-files y

; Write data files to output sub-directory (appends iteration)
/file/auto-save/root-name output/FFF-out

; Run the solver for this many iterations
/solve/iterate 1000

; Write final output case and data files
/file/write-case-data FFF-out

; Write simulation report to file (optional)
/report/summary y "My_Simulation_Report.txt"

; Cleanly shutdown fluent
/exit

File : sample3.jou

; SAMPLE FLUENT JOURNAL FILE - TRANSIENT SIMULATION
; -------------------------------------------------
; lines beginning with a semicolon are comments

; Overwrite files by default
/file/confirm-overwrite no

; Preferentially read/write files in legacy format
/file/cff-files no

; Read the input case file
/file/read-case FFF-transient-inp

; For continuation (restart) read in both case and data input files
;/file/read-case-data FFF-transient-inp

; Write a data (and maybe case) file every 100 time steps
/file/auto-save/data-frequency 100
/file/auto-save/case-frequency if-case-is-modified

; Retain only the most recent 5 data (and maybe case) files
/file/auto-save/retain-most-recent-files y

; Write to output sub-directory (appends flowtime and timestep)
/file/auto-save/root-name output/FFF-transient-out-%10.6f

; ##### Settings for Transient simulation :  #####

; Set the physical time step size
/solve/set/time-step 0.0001

; Set the number of iterations for which convergence monitors are reported
/solve/set/reporting-interval 1

; ##### End of settings for Transient simulation #####

; Initialize using the hybrid initialization method
/solve/initialize/hyb-initialization

; Set max number of iters per time step and number of time steps
;/solve/set/max-iterations-per-time-step 75
;/solve/dual-time-iterate 1000 ,
/solve/dual-time-iterate 1000 75

; Write final case and data output files
/file/write-case-data FFF-transient-out

; Write simulation report to file (optional)
/report/summary y Report_Transient_Simulation.txt

; Cleanly shutdown fluent
/exit

Ansys CFX

Slurm scripts

MultinodeSingle node

File : script-cfx-dist.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=2             # Specify multiple (1 or more) compute nodes
#SBATCH --ntasks-per-node=32  # Specify cores per node (graham 32 or 44, cedar 32 or 48, beluga 40, narval 64)
#SBATCH --mem=0               # Allocate all memory per compute node
#SBATCH --cpus-per-task=1     # Do not change

module load StdEnv/2020       # Applies to: beluga, cedar, graham, narval
module load ansys/2021R1      # Or newer module versions

NNODES=$(slurm_hl2hl.py --format ANSYS-CFX)

# append additional command line options as required
if [ "$CC_CLUSTER" = cedar ]; then
  cfx5solve -def YOURFILE.def -start-method "Open MPI Distributed Parallel" -par-dist $NNODES
else
  cfx5solve -def YOURFILE.def -start-method "Intel MPI Distributed Parallel" -par-dist $NNODES
fi

File : script-cfx-local.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify single compute node (do not change)
#SBATCH --ntasks-per-node=4   # Specify total cores (narval up to 64)
#SBATCH --mem=16G             # Specify 0 to use all node memory
#SBATCH --cpus-per-task=1     # Do not change

module load StdEnv/2020       # Applies to: beluga, cedar, graham, narval
module load ansys/2021R1      # Or newer module versions

# append additional command line options as required
if [ "$CC_CLUSTER" = cedar ]; then
  cfx5solve -def YOURFILE.def -start-method "Open MPI Local Parallel" -part $SLURM_CPUS_ON_NODE
else
  cfx5solve -def YOURFILE.def -start-method "Intel MPI Local Parallel" -part $SLURM_CPUS_ON_NODE
fi

Note: You may get the following error in your output file which does not seem to affect the computation: /etc/tmi.conf: No such file or directory.

Workbench

Before submitting a project file to the queue on a cluster (for the first time) follow these steps to initialize it.

1. Connect to the cluster with TigerVNC.
2. Switch to the directory where the project file is located (YOURPROJECT.wbpj) and start Workbench with the same Ansys module you used to create your project.
3. In Workbench, open the project with File -> Open.
4. In the main window, right-click on Setup and select Clear All Generated Data.
5. In the top menu bar pulldown, select File -> Exit to exit Workbench.
6. In the Ansys Workbench popup, when asked The current project has been modified. Do you want to save it?, click on the No button.
7. Quit Workbench and submit your job using one of the Slurm scripts shown below.

To avoid writing the solution when a running job successfully completes \remove ;Save(Overwrite=True) from the last line of your script. Doing this will make it easier to run multiple test jobs (for scaling purposes when changing ntasks), since the initialized solution will not be overwritten each time. Alternatively, keep a copy of the initialized YOURPROJECT.wbpj file and YOURPROJECT_files subdirectory and restore them after the solution is written.

Slurm scripts

A project file can be submitted to the queue by customizing one of the following scripts and then running the sbatch script-wbpj.sh command:

Single node (StdEnv/2020)

File : script-wbpj-2020.sh

#!/bin/bash

#SBATCH --account=def-account
#SBATCH --time=00-03:00                # Time (DD-HH:MM)
#SBATCH --mem=16G                      # Total Memory (set to 0 for all node memory)
#SBATCH --ntasks=4                     # Number of cores
#SBATCH --nodes=1                      # Do not change (multi-node not supported)
##SBATCH --exclusive                   # Uncomment for scaling testing
##SBATCH --constraint=broadwell        # Applicable to graham or cedar

module load StdEnv/2020 ansys/2021R2   # OR newer Ansys modules

if [ "$SLURM_NNODES" == 1 ]; then
  MEMPAR=0                               # Set to 0 for SMP (shared memory parallel)
else
  MEMPAR=1                               # Set to 1 for DMP (distributed memory parallel)
fi

rm -fv *_files/.lock
MWFILE=~/.mw/Application\ Data/Ansys/`basename $(find $EBROOTANSYS/v* -maxdepth 0 -type d)`/SolveHandlers.xml
sed -re "s/(.AnsysSolution>+)[a-zA-Z0-9]*(<\/Distribute.)/\1$MEMPAR\2/" -i "$MWFILE"
sed -re "s/(.Processors>+)[a-zA-Z0-9]*(<\/MaxNumber.)/\1$SLURM_NTASKS\2/" -i "$MWFILE"
sed -i "s!UserConfigured=\"0\"!UserConfigured=\"1\"!g" "$MWFILE"

export KMP_AFFINITY=disabled
export I_MPI_HYDRA_BOOTSTRAP=ssh

runwb2 -B -E "Update();Save(Overwrite=True)" -F YOURPROJECT.wbpj

Mechanical

The input file can be generated from within your interactive Workbench Mechanical session by clicking Solution -> Tools -> Write Input Files then specify File name: YOURAPDLFILE.inp and Save as type: APDL Input Files (*.inp). APDL jobs can then be submitted to the queue by running the sbatch script-name.sh command.

Slurm scripts

The Ansys modules used in each of the following scripts have been tested on Graham and should work without issue (uncomment one). Once the scripts have been tested on other clusters, they will be updated if required.

Single node (stdenv/2020)Multinode script (stdenv/2020)

File : script-smp-2020.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory for all cores
#SBATCH --ntasks=8             # Specify number of cores (1 or more)
#SBATCH --nodes=1              # Specify one node (do not change)

unset SLURM_GTIDS

module load StdEnv/2020

#module load ansys/2021R2
#module load ansys/2022R1
module load ansys/2022R2

mapdl -smp -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -I YOURAPDLFILE.inp

rm -rf results-*
mkdir results-$SLURM_JOB_ID
cp -a --no-preserve=ownership $SLURM_TMPDIR/* results-$SLURM_JOB_ID

File : script-dis-2020.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem-per-cpu=2G       # Specify memory per core
#SBATCH --ntasks=8             # Specify number of cores (2 or more)
##SBATCH --nodes=2             # Specify number of nodes (optional)
##SBATCH --ntasks-per-node=4   # Specify cores per node (optional)

unset SLURM_GTIDS

module load StdEnv/2020

module load ansys/2022R2

mapdl -dis -mpi openmpi -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -I YOURAPDLFILE.inp

rm -rf results-*
mkdir results-$SLURM_JOB_ID
cp -a --no-preserve=ownership $SLURM_TMPDIR/* results-$SLURM_JOB_ID

Ansys allocates 1024 MB total memory and 1024 MB database memory by default for APDL jobs. These values can be manually specified (or changed) by adding arguments -m 1024 and/or -db 1024 to the mapdl command line in the above scripts. When using a remote institutional license server with multiple Ansys licenses, it may be necessary to add -p aa_r or -ppf anshpc, depending on which Ansys module you are using. As always, perform detailed scaling tests before running production jobs to ensure that the optimal number of cores and minimum amount memory is specified in your scripts. The single node (SMP Shared Memory Parallel) scripts will typically perform better than the multinode (DIS Distributed Memory Parallel) scripts and therefore should be used whenever possible. To help avoid compatibility issues the Ansys module loaded in your script should ideally match the version used to to generate the input file:

 [gra-login2:~/ansys/mechanical/demo] cat YOURAPDLFILE.inp | grep version
! ANSYS input file written by Workbench version 2019 R3

Ansys EDT

Ansys EDT can be run interactively in batch (non-gui) mode by first starting an salloc session with options salloc --time=3:00:00 --tasks=8 --mem=16G --account=def-account and then copy-pasting the full ansysedt command found in the last line of script-local-cmd.sh, being sure to manually specify $YOUR_AEDT_FILE.

Slurm scripts

Ansys Electronic Desktop jobs may be submitted to a cluster queue with the sbatch script-name.sh command using either of the following single node scripts. As of January 2023, the scripts had only been tested on Graham and therefore may be updated in the future as required to support other clusters. Before using them, specify the simulation time, memory, number of cores and replace YOUR_AEDT_FILE with your input file name. A full listing of command line options can be obtained by starting Ansys EDT in graphical mode with commands ansysedt -help or ansysedt -Batchoptionhelp to obtain scrollable graphical popups.

Single node (command line)Single node (options file)

File : script-local-cmd.sh

#!/bin/bash

#SBATCH --account=account      # Specify your account (def or rrg)
#SBATCH --time=00-01:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory (set to 0 to use all compute node memory)
#SBATCH --ntasks=8             # Specify cores (beluga 40, cedar 32 or 48, graham 32 or 44, narval 64)
#SBATCH --nodes=1              # Request one node (Do Not Change)

module load StdEnv/2020
module load ansysedt/2021R2

# Uncomment next line to run a test example:
cp -f $EBROOTANSYSEDT/AnsysEM21.2/Linux64/Examples/HFSS/Antennas/TransientGeoRadar.aedt .

# Specify input file such as:
YOUR_AEDT_FILE="TransientGeoRadar.aedt"

# Remove previous output:
rm -rf $YOUR_AEDT_FILE.* ${YOUR_AEDT_FILE}results

# ---- do not change anything below this line ---- #

echo -e "\nANSYSLI_SERVERS= $ANSYSLI_SERVERS"
echo "ANSYSLMD_LICENSE_FILE= $ANSYSLMD_LICENSE_FILE"
echo -e "SLURM_TMPDIR= $SLURM_TMPDIR on $SLURMD_NODENAME\n"

export KMP_AFFINITY=disabled
ansysedt -monitor -UseElectronicsPPE -ng -distributed -machinelist list=localhost:1:$SLURM_NTASKS \
-batchoptions "TempDirectory=$SLURM_TMPDIR HPCLicenseType=pool HFSS/EnableGPU=0" -batchsolve "$YOUR_AEDT_FILE"

File : script-local-opt.sh

#!/bin/bash

#SBATCH --account=account      # Specify your account (def or rrg)
#SBATCH --time=00-01:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory (set to 0 to allocate all compute node memory)
#SBATCH --ntasks=8             # Specify cores (beluga 40, cedar 32 or 48, graham 32 or 44, narval 64)
#SBATCH --nodes=1              # Request one node (Do Not Change)

module load StdEnv/2020
module load ansysedt/2021R2

# Uncomment next line to run a test example:
cp -f $EBROOTANSYSEDT/AnsysEM21.2/Linux64/Examples/HFSS/Antennas/TransientGeoRadar.aedt .

# Specify input filename such as:
YOUR_AEDT_FILE="TransientGeoRadar.aedt"

# Remove previous output:
rm -rf $YOUR_AEDT_FILE.* ${YOUR_AEDT_FILE}results

# Specify options filename:
OPTIONS_TXT="Options.txt"

# Write sample options file
rm -f $OPTIONS_TXT
cat > $OPTIONS_TXT <<EOF
\$begin 'Config'
'TempDirectory'='$SLURM_TMPDIR'
'HPCLicenseType'='pool'
'HFSS/EnableGPU'=0
\$end 'Config'
EOF

# ---- do not change anything below this line ---- #

echo -e "\nANSYSLI_SERVERS= $ANSYSLI_SERVERS"
echo "ANSYSLMD_LICENSE_FILE= $ANSYSLMD_LICENSE_FILE"
echo -e "SLURM_TMPDIR= $SLURM_TMPDIR on $SLURMD_NODENAME\n"

export KMP_AFFINITY=disabled
ansysedt -monitor -UseElectronicsPPE -ng -distributed -machinelist list=localhost:1:$SLURM_NTASKS \
-batchoptions $OPTIONS_TXT -batchsolve "$YOUR_AEDT_FILE"

Graphical use

Ansys programs may be run interactively in GUI mode on cluster compute nodes or Graham VDI Nodes.

Compute nodes

Ansys can be run interactively on a single compute node for up to 24 hours. This approach is ideal for testing large simulations since all cores and memory can be requested with salloc as described in TigerVNC. Once connected with vncviewer, any of the following program versions can be started after loading the required modules as shown below.

Fluids

module load StdEnv/2020

module load ansys/2021R1 (or newer versions)

fluent -mpi=intel, or,

QTWEBENGINE_DISABLE_SANDBOX=1 cfx5

Mapdl

module load StdEnv/2020

module load ansys/2021R2 (or newer versions)

mapdl -g, or via launcher,

launcher --> click RUN button

Workbench

module load StdEnv/2020

module load ansys/2021R2 (or newer versions)

xfwm4 --replace & (only needed if using Ansys Mechanical)

export QTWEBENGINE_DISABLE_SANDBOX=1 (only needed if using CFD-Post)

runwb2

NOTES :When running an Analysis Program such as Mechanical or Fluent in parallel on a single node, untick Distributed and specify a value of cores equal to your salloc session setting minus 1. The pulldown menus in the Ansys Mechanical workbench do not respond properly. As a workaround run xfwm4 --replace on the command line before starting workbench as shown. To make xfwm4 your default edit $HOME/.vnc/xstartup and change mate-session to xfce4-session. Lastly, fluent from ansys/2022R2 does not currently work on compute nodes please use a different version.

Ansys EDT

Start an interactive session using the following form of the salloc command (to specify cores and available memory):

salloc --time=3:00:00 --nodes=1 --cores=8 --mem=16G --account=def-group

xfwm4 --replace & (then hit enter twice)

module load StdEnv/2020 ansysedt/2021R2, or

module load StdEnv/2020 ansysedt/2023R2, or

module load StdEnv/2023 ansysedt/2023R2, or newer

ansysedt

o Click Tools -> Options -> HPC and Analysis Options -> Edit then :

1) untick Use Automatic Settings box (required one time only)

2) under Machines tab do not change Cores (auto-detected from slurm)

o To run interactive analysis click: Project -> Analyze All

Ensight

module load StdEnv/2020 ansys/2022R2; A=222; B=5.12.6, or

module load StdEnv/2020 ansys/2022R1; A=221; B=5.12.6, or

module load StdEnv/2020 ansys/2021R2; A=212; B=5.12.6, or

module load StdEnv/2020 ansys/2021R1; A=211; B=5.12.6, or

export LD_LIBRARY_PATH=$EBROOTANSYS/v$A/CEI/apex$A/machines/linux_2.6_64/qt-$B/lib

ensight -X

Note: ansys/2022R2 Ensight is lightly tested on compute nodes. Please let us know if you find any problems using it.

Rocky

module load rocky/2023R2 ansys/2023R2 (or newer versions)

Rocky (reads ~/licenses/ansys.lic if present, otherwise defaults to SHARCNET server), or

Rocky-int (interactively select CMC or SHARCNET server, also reads ~/licenses/ansys.lic)

RockySolver (run rocky from the command line, currently untested, specify "-h" for help)

RockySchedular (resource manager to submit multiple jobs on present node)

o Rocky is (currently) only available on gra-vdi and graham cluster (no workbench support on linux)

o Release pdfs can be found under /opt/software/rocky/2023R2/docs (read them with mupdf)

o The SHARCNET license now includes Rocky (free for all researchers to use)

VDI nodes

Ansys programs can be run for up to 24 hours on graham VDI Nodes using a maximum of 8 cores and 128GB memory. The VDI System provides GPU OpenGL acceleration therefore it is ideal for performing tasks that benefit from high performance graphics. One might use VDI to create or modify simulation input files, post-process data or visualize simulation results. To get started, log in to gra-vdi.computecanada.ca with TigerVNC, open a new terminal window and start one of the following supported program versions as shown below. The vertical bar | notation is used to separate the various commands. The maximum job size for any parallel job run on gra-vdi should be limited to 16cores to avoid overloading the servers and impacting other users. To run two simultaneous gui jobs (16core max each) connect once with vnc to gra-vdi3.sharcnet.ca then connect again to gra-vdi4.sharecnet.ca likewise with vnc. Next start an interactive gui session for the ansys program you are using in the desktop on each machine. Note that simultaneous simulations should in general be run in different directories to avoid file conflict issues. Unlike compute nodes vnc connections (which impose slurm limits through salloc) there is no time limit constraint on gra-vdi when running simulations.

Fluids

module load CcEnv StdEnv/2020

module load ansys/2021R1 (or newer versions)

unset SESSION_MANAGER

fluent | cfx5 | icemcfd

o Where unsetting SESSION_MANAGER prevents the following Qt message from appearing when starting fluent:

[Qt: Session management error: None of the authentication protocols specified are supported]

o In the event the following message appears in a popup window when starting icemcfd ...

[Error segmentation violation - exiting after doing an emergency save]

... do not click the popup OK button otherwise icemcfd will crash. Instead do the following (one time only):

click the Settings Tab -> Display -> tick X11 -> Apply -> OK -> File -> Exit

The error popup should no longer appear when icemcfd is restarted.

Mapdl

module load CcEnv StdEnv/2020

module load ansys/2021R1 (or newer versions)

mapdl -g, or via launcher,

unset SESSION_MANAGER; launcher --> click RUN button

Workbench

module load SnEnv

module load ansys/2020R2 (or newer versions)

export HOOPS_PICTURE=opengl

runwb2

o The export line avoids the following tui Warning from appearing when fluent starts:

[Software rasterizer found, hardware acceleration will be disabled.]

Alternatively the HOOPS_PICTURE environment variable can be set inside workbench by doing:

Fluent Launcher --> Environment Tab --> HOOPS_PICTURE=opengl (without the export)

NOTE1: When running Mechanical in Workbench on gra-vdi be sure to tic Distributed in the upper ribbon Solver panel and specify a maximum value of 24 cores. When running Fluent on gra-vdi instead untic Distributed and specify a maximum value of 12 cores. Do not attempt to use more than 128GB memory otherwise Ansys will hit the hard limit and be killed. If you need more cores or memory then please use a cluster compute node to run your graphical session on (as described in the previous Compute nodes section above). When doing old pre-processing or post-processing work with Ansys on gra-vdi and not running calculation, please only use 4 cores otherwise hpc licenses will be checked out unnecessarily.

NOTE2: On very rare occasions the Ansys workbench gui will freeze or become unresponsive in some way. If this happens open a new terminal window and run pkill -9 -e -u $USER -f "ansys|fluent|mwrpcss|mwfwrapper|ENGINE|mono" to fully kill off Ansys. Likewise if Ansys crashes or vncviewer disconnect before Ansys could be shutdown cleanly then try running the pkill command if Ansys does not run normally afterwards. In general, if Ansys is not behaving properly and you suspect one of the aforementioned causes try pkill before opening a problem ticket.

Ansys EDT

Open a terminal window and load the module:

module load SnEnv ansysedt/2023R2, or

module load SnEnv ansysedt/2021R2

Type ansysedt in the terminal and wait for the gui to start

The following only needs to be done once:

click Tools -> Options -> HPC and Analysis Options -> Options

change HPC License pulldown to Pool (allows > 4 cores to be used)

click OK

---------- EXAMPLES ----------

To copy the 2023R2 Antennas examples directory into your account:

login to a cluster such as graham

module load ansysedt/2023R2

mkdir -p ~/Ansoft/$EBVERSIONANSYSEDT; cd ~/Ansoft/$EBVERSIONANSYSEDT; rm -rf Antennas

cp -a $EBROOTANSYSEDT/v232/Linux64/Examples/HFSS/Antennas ~/Ansoft/$EBVERSIONANSYSEDT

To run an example:

open a simulation .aedt file then click HFSS -> Validation Check

(if errors are reported by the validation check, close then reopen the simulation and repeat as required)

to run simulation click Project -> Analyze All

to quit without saving the converged solution click File -> Close -> No

If the program crashes and won't restart try running the following commands:

pkill -9 -u $USER -f "ansys*|mono|mwrpcss|apip-standalone-service"

rm -rf ~/.mw (ansysedt will re-run first-time configuration on startup)

Ensight

module load SnEnv ansys/2019R2 (or newer)

ensight

Rocky

module load clumod rocky/2023R2 CcEnv StdEnv/2020 ansys/2023R2 (or newer versions)

Rocky (reads ~/licenses/ansys.lic if present, otherwise defaults to SHARCNET server), or

Rocky-int (interactively select CMC or SHARCNET server, also reads ~/licenses/ansys.lic)

RockySolver (run rocky from the command line, currently untested, specify "-h" for help)

RockySchedular (resource manager to submit multiple jobs on present node)

o Rocky is (currently) only available on gra-vdi and graham cluster (no workbench support on linux)

o Release pdfs can be found under /opt/software/rocky/2023R2/docs (read them with mupdf)

o A maximum of 2 Rocky gui jobs can be run by connecting directly to gra-vdi3 and gra-vdi4.sharcnet.ca

o The SHARCNET license now includes Rocky (free for all researchers to use)

SSH issues

Some Ansys GUI programs can be run remotely on a cluster compute node by X forwarding over SSH to your local desktop. Unlike VNC, this approach is untested and unsupported since it relies on a properly setup X display server for your particular operating system OR the selection, installation and configuration of a suitable X client emulator package such as MobaXterm. Most users will find interactive response times unacceptably slow for basic menu tasks let alone performing more complex tasks such as those involving graphics rendering. Startup times for GUI programs can also be very slow depending on your Internet connection. For example, in one test it took 40 minutes to fully start ansysedt over SSH while starting it with vncviewer required only 34 seconds. Despite the potential slowness when connecting over SSH to run GUI programs, doing so may still be of interest if your only goal is to open a simulation and perform some basic menu operations or run some calculations. The basic steps are given here as a starting point: 1) ssh -Y username@graham.computecanada.ca; 2) salloc --x11 --time=1:00:00 --mem=16G --cpus-per-task=4 [--gpus-per-node=1] --account=def-mygroup; 3) once connected onto a compute node try running xclock. If the clock appears on your desktop, proceed to load the desired Ansys module and try running the program.

Site-specific usage

SHARCNET license

The SHARCNET Ansys license is free for academic use by any Alliance researcher on any Alliance system. The installed software does not have any solver or geometry limits. The SHARCNET license may only be used for the purpose of Publishable Academic Research. Producing results for private commercial purposes is strictly prohibited. The SHARCNET license was upgraded from CFD to MCS (Multiphysics Campus Solution) in May of 2020. It includes the following products: HF, EM, Electronics HPC, Mechanical and CFD as described here. In 2023 Rocky for Linux (no Workbench support) was also added. Neither LS-DYNA or Lumerical are included in the SHARCNET license. Note that since all the Alliance clusters are Linux based, SpaceClaim cannot be used on our systems. In July of 2021 an additional 1024 anshpc licenses were added to the previous 512 pool. Before running large parallel jobs, scaling tests should be run for any given simulation. Parallel jobs that do not achieve at least 50% CPU utilization may be flagged by the system for a follow-up by our support team.

As of December 2022, each researcher can run 4 jobs using a total of 252 anshpc (plus 4 anshpc per job). Thus any of the following uniform job size combinations are possible: one 256 core job, two 130 core jobs, three 88 core jobs, or four 67 core jobs according to ( (252 + 4*num_jobs) / num_jobs ). Since the best parallel performance is usually achieved by using all cores on packed compute nodes (aka full nodes), one can determine the number of full nodes by dividing the total anshpc cores with the compute node size. For example, consider Graham which has many 32-core (Broadwell) and some 44-core (Cascade) compute nodes, the maximum number of nodes that could be requested when running various size jobs on 32-core nodes would be: 256/32=8, 130/32=~4, 88/32=~2 or 67/32=~2 to run 1, 2, 3 or 4 simultaneous jobs respectively. To express this in equation form, for a given compute node size on any cluster, the number of compute nodes can be calculated by ( 252 + (4*num_jobs) ) / (num_jobs*cores_per_node) ) then round down and finally determine the total cores to request by multiplying the even number of nodes by the number of cores_per_node.

The SHARCNET Ansys license is made available on a first come first serve basis. Should an unusually large number of Ansys jobs be submitted on a given day some jobs could fail on startup should insufficient licenses be available. These events however have become very rare given the recent increase in anshpc licenses. If your research requires more licenses than SHARCNET can provide, a dedicated researcher license may be purchased (and hosted) on an Ansys license server at your local institution.

If more cores are needed (beyond that which the SHARCNET license can provide per user) a research group can purchase a license directly from SimuTech to host on their own institutional license server. Note however that an extra 20% country-wide uplift fee must be payed if the cluster(s) to be used are not co-located at the institution. Waterloo researchers who only use Graham will therefore be exempt since Graham is physically located at their campus.

Researchers can also purchase their own ansys license subscription from CMC and use their remote license servers. Doing so will have several benefits 1) a local institutional license server is not needed 2) a physical license does not need to be obtained upon each renewal 3) the license can be used almost anywhere including at home, institutions, or any alliance cluster across Canada and 4) download and installation instructions for the windows version of ansys are provided so researchers can run spaceclaim on their own computer (not possible on the Alliance since all systems are linux based). There is however one potentially serious limitation to be aware of, according to the CMC Ansys Quick Start Guides one Ansys user may run a simulation on a maximum number of 64 cores.

License server file

To use the SHARCNET Ansys license on any Alliance cluster, simply configure your ansys.lic file as follows:

[username@cluster:~] cat ~/.licenses/ansys.lic
setenv("ANSYSLMD_LICENSE_FILE", "1055@license3.sharcnet.ca")
setenv("ANSYSLI_SERVERS", "2325@license3.sharcnet.ca")

Query license server

To show the number of licenses in use by your username and the total in use by all users, run:

ssh graham.computecanada.ca
module load ansys
lmutil lmstat -c $ANSYSLMD_LICENSE_FILE -a | grep "Users of\|$USER"

If you discover any licenses unexpectedly in use by your username (usually due to ansys not exiting cleanly on gra-vdi) then connect to the node where its running, open a terminal window and run the following command to terminate the rogue processes pkill -9 -e -u $USER -f "ansys" after which your licenses should be freed. Note that gra-vdi consists of two nodes (gra-vdi3 and gra-vdi4) which researchers are randomly placed on when connecting to gra-vdi.computecanada.ca with TigerVNC. Therefore it's necessary to specify the full hostname (gra-vdi3.sharcnet.ca or grav-vdi4.sharcnet.ca) when connecting with tigervnc to ensure you log in to the correct node before running pkill.

Local VDI modules

When using gra-vdi, researchers have the choice of loading Ansys modules from our global environment (after loading CcEnv) or loading Ansys modules installed locally on the machine itself (after loading SnEnv). The local modules may be of interest as they include some Ansys programs and versions not yet supported by our standard environment. When starting programs from local Ansys modules, you can select the CMC license server or accept the default SHARCNET license server. Presently, the settings from ~/.licenses/ansys.lic are not used by the local Ansys modules except when starting runwb2 where they will override the default SHARCNET license server settings. Suitable usage of Ansys programs on gra-vdi includes: running a single test job interactively with up to 8 cores and/or 128G RAM, create or modify simulation input files, post process or visualize data.

Ansys modules

1. Connect to gra-vdi.computecanada.ca with TigerVNC.
2. Open a new terminal window and load a module:

   module load SnEnv ansys/2021R2, or

   module load SnEnv ansys/2021R1, or

   module load SnEnv ansys/2020R2, or

   module load SnEnv ansys/2020R1, or

   module load SnEnv ansys/2019R3

3. Start an Ansys program by issuing one of the following:

   runwb2|fluent|cfx5|icemcfd|apdl

4. Press y and Enter to accept the conditions
5. Press Enter to accept the n option and use the SHARCNET license server by default (in the case of runwb2 ~/.licenses/ansysedt.lic will be used if present, otherwise ANSYSLI_SERVERS and ANSYSLMD_LICENSE_FILE will be used if set in your environment for example to some other remote license server). If you change n to y and hit y. the CMC license server will be used.

where cfx5 from step 3. above provides the option to start the following components:

   1) CFX-Launcher  (cfx5 -> cfx5launch)
   2) CFX-Pre       (cfx5pre)
   3) CFD-Post      (cfdpost -> cfx5post)
   4) CFX-Solver    (cfx5solve)

ansysedt modules

1. Connect to gra-vdi.computecanada.ca with TigerVNC.
2. Open a new terminal window and load a module:

   module load SnEnv ansysedt/2021R2, or

   module load SnEnv ansysedt/2021R1

3. Start the Ansys Electromagnetics Desktop program by typing the following command: ansysedt
4. Press y and Enter to accept the conditions.
5. Press Enter to accept the n option and use the SHARCNET license server by default (note that ~/.licenses/ansysedt.lic will be used if present, otherwise ANSYSLI_SERVERS and ANSYSLMD_LICENSE_FILE will be used if set in your environment for example to some other remote license server). If you change n to y and hit enter, the CMC license server will be used.

License feature preferences previously setup with anslic_admin are no longer supported following the recent SHARCNET license server update (2021-09-09). If a license problem occurs, try removing the ~/.ansys directory in your /home account to clear the settings. If problems persist please contact our technical support and provide the contents of your ~/.licenses/ansys.lic file.

Additive Manufacturing

To get started configure your ~/.licenses/ansys.lic file to point to a license server that has a valid Ansys Mechanical License. This must be done on all systems where you plan to run the software.

Enable Additive

This section describes how to make the Ansys Additive Manufacturing ACT extension available for use in your project. The steps must be performed on each cluster for each ansys module version where the extension will be used. Any extensions needed by your project will also need to be installed on the cluster as described below. If you get warnings about missing un-needed extensions (such as ANSYSMotion) then uninstall them from your project.

Download Extension

• download AdditiveWizard.wbex from https://catalog.ansys.com/
• upload AdditiveWizard.wbex to the cluster where it will be used

Start Workbench

• follow the Workbench section in Graphical use above.
• File -> Open your project file (ending in .wbpj) into Workbench gui

Open Extension Manager

• click ACT Start Page and the ACT Home page tab will open
• click Manage Extensions and the Extension Manager will open

Install Extension

• click the box with the large + sign under the search bar
• navigate to select and install your AdditiveWizard.wbex file

Load Extension

• click to highlight the AdditiveWizard box (loads the AdditiveWizard extension for current session only)
• click lower right corner arrow in the AdditiveWizard box and select Load extension (loads the extension for current AND future sessions)

Unload Extension

• click to un-highlight the AdditiveWizard box (unloads extension for the current session only)
• click lower right corner arrow in the AdditiveWizard box and select Do not load as default (extension will not load for future sessions)

Run Additive

Gra-vdi

A user can run a single Ansys Additive Manufacturing job on gra-vdi with up to 16 cores as follows:

• Start Workbench on Gra-vdi as described above in Enable Additive.
• click File -> Open and select test.wbpj then click Open
• click View -> reset workspace if you get a grey screen
• start Mechanical, Clear Generated Data, tick Distributed, specify Cores
• click File -> Save Project -> Solve

Check utilization:

• open another terminal and run: top -u $USER **OR** ps u -u $USER | grep ansys
• kill rogue processes from previous runs: pkill -9 -e -u $USER -f "ansys|mwrpcss|mwfwrapper|ENGINE"

Please note that rogue processes can persistently tie up licenses between gra-vdi login sessions or cause other unusual errors when trying to start gui programs on gra-vdi. Although rare, rogue processes can occur if an ansys gui session (fluent, workbench, etc) is not cleanly terminated by the user before vncviewer is terminated either manually or unexpectedly - for instance due to a transient network outage or hung filesystem. If the latter is to blame then the processes may not by killable until normal disk access is restored.

Cluster

Project preparation:

Before submitting a newly uploaded Additive project to a cluster queue (with sbatch scriptname) certain preparations must be done. To begin, open your simulation with Workbench gui (as described in the Enable Additive section above) in the same directory that your job will be submitted from and then save it again. Be sure to use the same ansys module version that will be used for the job. Next create a Slurm script (as explained in the Cluster Batch Job Submission - WORKBENCH section above). To perform parametric studies change Update() to UpdateAllDesignPoints() in the Slurm script. Determine the optimal number of cores and memory by submitting several short test jobs. To avoid needing to manually clear the solution and recreate all the design points in Workbench between each test run, either 1) change Save(Overwrite=True) to Save(Overwrite=False) or 2) save a copy of the original YOURPROJECT.wbpj file and corresponding YOURPROJECT_files directory. Optionally create and then manually run a replay file on the cluster in the respective test case directory between each run, noting that a single replay file can be used in different directories by opening it in a text editor and changing the internal FilePath setting.

module load ansys/2019R3
rm -f test_files/.lock
runwb2 -R myreplay.wbjn

Resource utilization:

Once your additive job has been running for a few minutes a snapshot of its resource utilization on the compute node(s) can be obtained with the following srun command. Sample output corresponding to an eight core submission script is shown next. It can be seen that two nodes were selected by the scheduler:

[gra-login1:~] srun --jobid=myjobid top -bn1 -u $USER | grep R | grep -v top
  PID USER   PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+  COMMAND
22843 demo   20   0 2272124 256048  72796 R  88.0  0.2  1:06.24  ansys.e
22849 demo   20   0 2272118 256024  72822 R  99.0  0.2  1:06.37  ansys.e
22838 demo   20   0 2272362 255086  76644 R  96.0  0.2  1:06.37  ansys.e
  PID USER   PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+  COMMAND
 4310 demo   20   0 2740212 271096 101892 R 101.0  0.2  1:06.26  ansys.e
 4311 demo   20   0 2740416 284552  98084 R  98.0  0.2  1:06.55  ansys.e
 4304 demo   20   0 2729516 268824 100388 R 100.0  0.2  1:06.12  ansys.e
 4305 demo   20   0 2729436 263204 100932 R 100.0  0.2  1:06.88  ansys.e
 4306 demo   20   0 2734720 431532  95180 R 100.0  0.3  1:06.57  ansys.e

Scaling tests:

After a job completes its "Job Wall-clock time" can be obtained from seff myjobid. Using this value scaling tests can be performed by submitting short test jobs with an increasing number of cores. If the Wall-clock time decreases by ~50% when the number of cores is doubled then additional cores may be considered.

Online documentation

The full Ansys documentation for versions back to 19.2 can be accessed by following these steps:

1. Connect to gra-vdi.computecanada.ca. with tigervnc as described in TigerVNC.
2. If the Firefox browser or the Ansys Workbench is open, close it now.
3. Start Firefox by clicking Applications -> Internet -> Firefox.
4. Open a new terminal window by clicking Applications -> System Tools -> Mate Terminal.
5. Start Workbench by typing the following in your terminal: module load CcEnv StdEnv/2023 ansys; runwb2
6. Go to the upper Workbench menu bar and click Help -> ANSYS Workbench Help. The Workbench Users' Guide should appear loaded in Firefox.
7. At this point Workbench is no longer needed so close it by clicking the >Unsaved Project - Workbench tab located along the bottom frame (doing this will bring Workbench into focus) and then click File -> Exit.
8. In the top middle of the Ansys documentation page, click the word HOME located just left of API DOCS.
9. Now scroll down and you should see a list of Ansys product icons and/or alphabetical ranges.
10. Select a product to view its documentation. The documentation for the latest release version will be displayed by default. Change the version by clicking the Release Year pull down located above and just to the right of the Ansys documentation page search bar.
11. To search for documentation corresponding to a different Ansys product, click HOME again.

Message Passing Interface