Alliance Doc

Ansys

Revision as of 19:43, 18 December 2023 by Roberpj (talk | contribs)
Other languages:

Ansys is a software suite for engineering simulation and 3-D design. It includes packages such as Ansys Fluent and Ansys CFX.

Licensing

We are a hosting provider for Ansys. This means that we have the software installed on our clusters, but we do not provide a generic license accessible to everyone. However, many institutions, faculties, and departments already have licenses that can be used on our clusters. Once the legal aspects are worked out for licensing, there will be remaining technical aspects. The license server on your end will need to be reachable by our compute nodes. This will require our technical team to get in touch with the technical people managing your license software. In some cases, this has already been done. You should then be able to load the Ansys module, and it should find its license automatically. If this is not the case, please contact our technical support so that they can arrange this for you.

Configuring your license file

Our module for Ansys is designed to look for license information in a few places. One of those places is your /home folder. You can specify your license server by creating a file named $HOME/.licenses/ansys.lic consisting of two lines as shown. Customize the file to replacing FLEXPORT, INTEPORT and LICSERVER with appropriate values for your server.

FILE: ansys.lic
setenv("ANSYSLMD_LICENSE_FILE", "FLEXPORT@LICSERVER")
setenv("ANSYSLI_SERVERS", "INTEPORT@LICSERVER")

The following table provides established values for the CMC and SHARCNET license servers. To use a different server, locate the corresponding values as explained in Local license servers.

TABLE: Preconfigured license servers
License System/Cluster LICSERVER FLEXPORT INTEPORT VENDPORT NOTICES
CMC beluga 10.20.73.21 6624 2325 n/a None
CMC cedar 172.16.0.101 6624 2325 n/a None
CMC graham 199.241.167.222 6624 2325 n/a None
CMC narval 10.100.64.10 6624 2325 n/a None
SHARCNET beluga/cedar/graham/gra-vdi/narval license3.sharcnet.ca 1055 2325 n/a None
SHARCNET niagara localhost 1055 2325 1793 None

Researchers who purchase a CMC license subscription must send their Alliance account username to <cmcsupport@cmc.ca> otherwise license checkouts will fail. The number of cores that can be used with a CMC license is described in the Other Tricks and Tips sections of the Ansys Electronics Desktop and Ansys Mechanical/Fluids quick start guides.

Local license servers

Before a local Ansys license server can be reached from an Alliance system, firewall changes will need to be done on both the server side and the Alliance side. For many institutional servers this work has already been done. In such cases you simply need to contact your local Ansys license server administrator and request 1) the fully qualified hostname (LICSERVER) of the server; 2) the Ansys flex port commonly 1055 (FLEXPORT); and 3) the Ansys licensing interconnect port commonly 2325 (INTEPORT). With this information you can then immediately configure your ansys.lic file as described above and theoretically begin using the software. If however the license server has never been set up for use on the Alliance systems that you want to use it on, then you will also need to request the 4) static vendor port number (VENDPORT) number from your administrator.
Once you have all the information together, send it to technical support and mention which systems you want to use to run Ansys jobs. We will then arrange for the Alliance firewall to be opened so outbound license connections from the Ansys software can reach your license server. You will also be sent a range of IP addresses so that your server administrator can likewise open the local firewall to allow inbound license connections to your server on the required ports (FLEXPORT, INTEPORT, VENDPORT) from the requested Alliance systems.

Checking license usage

Ansys comes with an lmutil tool that can be used to check your license usage. Before using it verify your ansys.lic is configured. Then run the following two commands on a cluster that you are set up to use: 

[name@server ~]$ module load ansys
[name@server ~]$ $EBROOTANSYS/shared_files/licensing/linx64/lmutil lmstat -c $ANSYSLMD_LICENSE_FILE -S ansyslmd

Version compatibility

As explained in Ansys Platform Support, the current release (2021R2) was tested to read and open databases from the five previous releases. In addition, some products can read and open databases from releases before Ansys 18.1.

Cluster batch job submission

The Ansys software suite comes with multiple implementations of MPI to support parallel computation. Unfortunately, none of them support our Slurm scheduler. For this reason, we need special instructions for each Ansys package on how to start a parallel job. In the sections below, we give examples of submission scripts for some of the packages. While the slurm scripts should work with on all clusters, Niagara users may need to make some additional changes covered here.

Ansys Fluent

Typically, you would use the following procedure to run Fluent on one of our clusters:

  1. Prepare your Fluent job using Fluent from the Ansys Workbench on your desktop machine up to the point where you would run the calculation.
  2. Export the "case" file with File > Export > Case… or find the folder where Fluent saves your project's files. The case file will often have a name like FFF-1.cas.gz.
  3. If you already have data from a previous calculation, which you want to continue, export a "data" file as well (File > Export > Data…) or find it in the same project folder (FFF-1.dat.gz).
  4. Transfer the case file (and if needed the data file) to a directory on the /project or /scratch filesystem on the cluster. When exporting, you can save the file(s) under a more instructive name than FFF-1.* or rename them when they are uploaded.
  5. Now you need to create a "journal" file. It's purpose is to load the case file (and optionally the data file), run the solver and finally write the results. See examples below and remember to adjust the filenames and desired number of iterations.
  6. If jobs frequently fail to start due to license shortages and manual resubmission of failed jobs is not convenient, consider modifying your script to requeue your job (up to 4 times) as shown under the by node + requeue tab further below. Be aware that doing this will also requeue simulations that fail due to non-license related issues (such as divergence), resulting in lost compute time. Therefore it is strongly recommended to monitor and inspect each Slurm output file to confirm each requeue attempt is license related. When it is determined that a job is requeued due to a simulation issue, immediately manually kill the job progression with scancel jobid and correct the problem.
  7. After running the job, you can download the data file and import it back into Fluent with File > Import > Data….

Slurm scripts

General purpose

Most Fluent jobs should use the following by node script to minimize solution latency and maximize performance over as few nodes as possible. Very large jobs, however, might wait less in the queue if they use a by core script. However, the startup time of a job using many nodes can be significantly longer, thus offsetting some of the benefits. In addition, be aware that running large jobs over an unspecified number of potentially very many nodes will make them far more vulnerable to crashing if any of the compute nodes fail during the simulation. The scripts will ensure Fluent uses shared memory for communication when run on a single node or distributed memory (utilizing MPI and the appropriate HPC interconnect) when run over multiple nodes.

File : script-flu-bynode.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes (1 or more)
#SBATCH --ntasks-per-node=32  # Specify number of cores per node (graham 32 or 44, cedar 48, beluga 40, narval 64)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change

rm -f cleanup* core*

#module load StdEnv/2016      # Applies to cedar, beluga, graham
#module load ansys/2020R2     # or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # or newer module versions

module load StdEnv/2020       # Applies to cedar, beluga, graham
module load ansys/2021R2      # or newer module versions

if [[ "${CC_CLUSTER}" == narval ]]; then
 module load intel/2021 intelmpi
 export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou
fi


File : script-flu-bycore.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=2            # Optional (uncomment to specify 1 or more compute nodes)
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change

rm -f cleanup* core*

#module load StdEnv/2016      # Applies to cedar, beluga, graham
#module load ansys/2020R2     # or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # or newer module versions

module load StdEnv/2020       # Applies to cedar, beluga, graham
module load ansys/2021R2      # or newer module versions

if [[ "${CC_CLUSTER}" == narval ]]; then
 module load intel/2021 intelmpi
 export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
fi


License requeue

These two scripts can be used to automatically requeue Fluent jobs that exit abnormally during startup and require several requeue attempts to check out the required licenses from a remote server. The cause could be due to the license server randomly experiencing heavy load, transient network instability or a shortage of licenses. It is not recommended to use these scripts with Fluent jobs that start normally and run a long time before crashing (for instance due to a node failure, divergence, or running out Slurm runtime before completing normally) since they will be restarted from the beginning instead of using the most recent saved solution data file thus resulting in a significant amount of lost compute time and resources.

File : script-flu-bynode+requeue.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes (1 or more)
#SBATCH --ntasks-per-node=32  # Specify number of cores per node (graham 32 or 44, cedar 48, beluga 40, narval 64)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of requeue attempts (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2016      # Applies to cedar, beluga, graham
#module load ansys/2020R2     # or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # or newer module versions

module load StdEnv/2020       # Applies to cedar, beluga, graham
module load ansys/2021R2      # or newer module versions

if [[ "${CC_CLUSTER}" == narval ]]; then
 module load intel/2021 intelmpi
 export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else 
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
fi
if [ $? -eq 0 ]; then
    echo "Job completed successfully! Exiting now."
    scancel $SLURM_ARRAY_JOB_ID
else
    echo "Job attempt $SLURM_ARRAY_TASK_ID of $SLURM_ARRAY_TASK_COUNT failed due to license or simulation issue!"
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
       echo "Resubmitting job now …"
    else
       echo "All job attempts failed exiting now."
    fi
fi


File : script-flu-bycore+requeue.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=2            # Optional (uncomment to specify 1 or more compute nodes)
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of requeue attempts (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2016      # Applies to cedar, beluga, graham
#module load ansys/2020R2     # or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # or newer module versions

module load StdEnv/2020       # Applies to cedar, beluga, graham
module load ansys/2021R2      # or newer module versions

if [[ "${CC_CLUSTER}" == narval ]]; then
 module load intel/2021 intelmpi
 export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
 fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
else 
 fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
fi
if [ $? -eq 0 ]; then
    echo "Job completed successfully! Exiting now."
    scancel $SLURM_ARRAY_JOB_ID
else
    echo "Job attempt $SLURM_ARRAY_TASK_ID of $SLURM_ARRAY_TASK_COUNT failed due to license or simulation issue!"
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
       echo "Resubmitting job now …"
    else
       echo "All job attempts failed exiting now."
    fi
fi


Solution restart

The following two scripts are provided to automate restarting very large jobs that require more than the typical seven-day maximum runtime window available on most clusters. Jobs are restarted from the most recent saved time step files. A fundamental requirement is the first time step can be completed within the requested job array time limit (specified at the top of your Slurm script) when starting a simulation from an initialized solution field. It is assumed that a standard fixed time step size is being used. To begin, a working set of sample.cas, sample.dat and sample.jou files must be present. Next edit your sample.jou file to contain /solve/dual-time-iterate 1 and /file/auto-save/data-frequency 1. Then create a restart journal file by doing cp sample.jou sample-restart.jou and edit the sample-restart.jou file to contain /file/read-cas-data sample-restart instead of /file/read-cas-data sample and comment out the initialization line with a semicolon for instance ;/solve/initialize/initialize-flow. If your 2nd and subsequent time steps are known to run twice as fast (than the initial time step), edit sample-restart.jou to specify /solve/dual-time-iterate 2. By doing this, the solution will only be restarted after two 2 time steps are completed following the initial time step. An output file for each time step will still be saved in the output subdirectory. The value 2 is arbitrary but should be chosen such that the time for 2 steps fits within the job array time limit. Doing this will minimize the number of solution restarts which are computationally expensive. If your first time step performed by sample.jou starts from a converged (previous) solution, choose 1 instead of 2 since likely all time steps will require a similar amount of wall time to complete. Assuming 2 is chosen, the total time of simulation to be completed will be 1*Dt+2*Nrestart*Dt where Nrestart is the number of solution restarts specified in the script. The total number of time steps (and hence the number of output files generated) will therefore be 1+2*Nrestart. The value for the time resource request should be chosen so the initial time step and subsequent time steps will complete comfortably within the Slurm time window specifiable up to a maximum of "#SBATCH --time=07-00:00" days.

File : script-flu-bynode+restart.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=07-00:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify number of compute nodes (1 or more)
#SBATCH --ntasks-per-node=32  # Specify number of cores per node (graham 32 or 44, cedar 48, beluga 40, narval 64)
#SBATCH --mem=0               # Do not change (allocates all memory per compute node)
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of solution restarts (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2016      # Applies to cedar, beluga, graham
#module load ansys/2020R2     # or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # or newer module versions

module load StdEnv/2020       # Applies to cedar, beluga, graham
module load ansys/2021R2      # or newer module versions

if [[ "${CC_CLUSTER}" == narval ]]; then
 module load intel/2021 intelmpi
 export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NNODES * SLURM_NTASKS_PER_NODE * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample-restart.jou
  fi
else 
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample-restart.jou 
  fi
fi
if [ $? -eq 0 ]; then
    echo
    echo "SLURM_ARRAY_TASK_ID  = $SLURM_ARRAY_TASK_ID"
    echo "SLURM_ARRAY_TASK_COUNT = $SLURM_ARRAY_TASK_COUNT"
    echo
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
      echo "Restarting job with the most recent output dat file …"
      ln -sfv output/$(ls -ltr output | grep .cas | tail -n1 | awk '{print $9}') sample-restart.cas.gz
      ln -sfv output/$(ls -ltr output | grep .dat | tail -n1 | awk '{print $9}') sample-restart.dat.gz
      ls -lh cavity* output/*
    else
      echo "Job completed successfully! Exiting now."
      scancel $SLURM_ARRAY_JOB_ID
     fi
else
     echo "Simulation failed. Exiting …"
fi


File : script-flu-bycore+restart.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
##SBATCH --nodes=2            # Optional (uncomment to specify 1 or more compute nodes)
#SBATCH --ntasks=16           # Specify total number of cores
#SBATCH --mem-per-cpu=4G      # Specify memory per core
#SBATCH --cpus-per-task=1     # Do not change
#SBATCH --array=1-5%1         # Specify number of restart aka time steps (2 or more, 5 is shown)

rm -f cleanup* core*

#module load StdEnv/2016      # Applies to cedar, beluga, graham
#module load ansys/2020R2     # or older module versions

#module load StdEnv/2020      # Applies to narval only
#module load ansys/2019R3     # or newer module versions

module load StdEnv/2020       # Applies to cedar, beluga, graham
module load ansys/2021R2      # or newer module versions

if [[ "${CC_CLUSTER}" == narval ]]; then
 module load intel/2021 intelmpi
 export INTELMPI_ROOT=$I_MPI_ROOT/mpi/latest
 unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
 unset I_MPI_ROOT
fi

slurm_hl2hl.py --format ANSYS-FLUENT > /tmp/machinefile-$SLURM_JOB_ID
NCORES=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Specify 2d, 2ddp, 3d or 3ddp and replace sample with your journal filename …
if [ "$SLURM_NNODES" == 1 ]; then
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample.jou
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -i sample-restart.jou
  fi
else 
  if [ "$SLURM_ARRAY_TASK_ID" == 1 ]; then
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample.jou 
  else
    fluent -g 2ddp -t $NCORES -affinity=0 -cnf=/tmp/machinefile-$SLURM_JOB_ID -mpi=intel -ssh -i sample-restart.jou 
  fi
fi
if [ $? -eq 0 ]; then
    echo
    echo "SLURM_ARRAY_TASK_ID  = $SLURM_ARRAY_TASK_ID"
    echo "SLURM_ARRAY_TASK_COUNT = $SLURM_ARRAY_TASK_COUNT"
    echo
    if [ $SLURM_ARRAY_TASK_ID -lt $SLURM_ARRAY_TASK_COUNT ]; then
      echo "Restarting job with the most recent output dat file …"
      ln -sfv output/$(ls -ltr output | grep .cas | tail -n1 | awk '{print $9}') sample-restart.cas.gz
      ln -sfv output/$(ls -ltr output | grep .dat | tail -n1 | awk '{print $9}') sample-restart.dat.gz
      ls -lh cavity* output/*
    else
      echo "Job completed successfully! Exiting now."
      scancel $SLURM_ARRAY_JOB_ID
     fi
else
     echo "Simulation failed. Exiting …"
fi


Journal files

Fluent journal files can include basically any command from Fluent's Text-User-Interface (TUI); commands can be used to change simulation parameters like temperature, pressure and flow speed. With this you can run a series of simulations under different conditions with a single case file, by only changing the parameters in the journal file. Refer to the Fluent User's Guide for more information and a list of all commands that can be used. The following journal files are set up with /file/cff-files no to use the legacy .cas/.dat file format (the default in module versions 2019R3 or older). Set this instead to /file/cff-files yes to use the more efficient .cas.h5/.dat.h5 file format (the default in module versions 2020R1 or newer).

File : sample1.jou

; SAMPLE FLUENT JOURNAL FILE - STEADY SIMULATION
; ----------------------------------------------
; lines beginning with a semicolon are comments

; Read/Write ANSYS common fluids format (CFF) files? [yes by default]
/file/cff-files no

; Read input case and data files:
/file/read-case-data FFF-in

; Run the solver for this many iterations:
/solve/iterate 1000

; Overwrite output files by default:
/file/confirm-overwrite n

; Write final output data file:
/file/write-data FFF-out

; Write simulation report to file (optional):
/report/summary y "My_Simulation_Report.txt"

; Exit fluent (required to requeue):
exit


File : sample2.jou

; SAMPLE FLUENT JOURNAL FILE - STEADY SIMULATION
; ----------------------------------------------
; lines beginning with a semicolon are comments

; Read/Write ANSYS common fluids format (CFF) files? [yes by default]
/file/cff-files no

; Read input files (.cas/.dat or .cas.h5/.dat.h5):
/file/read-case-data FFF-in

; Write a data file every 100 iterations:
/file/auto-save/data-frequency 100

; Retain data files from 5 most recent iterations:
/file/auto-save/retain-most-recent-files y

; Write data files to output sub-directory (appends iteration)
/file/auto-save/root-name output/FFF-out

; Run the solver for this many iterations:
/solve/iterate 1000

; Write final output case and data files:
/file/write-case-data FFF-out

; Write simulation report to file (optional):
/report/summary y "My_Simulation_Report.txt"

; Exit fluent (required to requeue):
exit


File : sample3.jou

; SAMPLE FLUENT JOURNAL FILE - TRANSIENT SIMULATION
; -------------------------------------------------
; lines beginning with a semicolon are comments

; Read/Write ANSYS common fluids format (CFF) files? [yes by default]
/file/cff-files no

; Read only the input case file:
/file/read-case FFF-transient-inp

; For continuation (restart) read in both case and data input files:
;/file/read-case-data FFF-transient-inp

; Write a data (and maybe case) file every 100 time steps:
/file/auto-save/data-frequency 100
/file/auto-save/case-frequency if-case-is-modified

; Retain only the most recent 5 data (and maybe case) files:
/file/auto-save/retain-most-recent-files y

; Write to output sub-directory (appends flowtime and timestep)
/file/auto-save/root-name output/FFF-transient-out-%10.6f

; ##### Settings for Transient simulation :  #####

; Set the physical time step size
/solve/set/time-step 0.0001

; Set the max number of iterations per time step:
/solve/set/max-iterations-per-time-step 20

; Set the number of iterations for which convergence monitors are reported:
/solve/set/reporting-interval 1

; ##### End of settings for Transient simulation #####

; Initialize using the hybrid initialization method:
/solve/initialize/hyb-initialization

; Perform unsteady iterations for a specified number of time steps:
/solve/dual-time-iterate 1000 ,

; Write final case and data output files:
/file/write-case-data FFF-transient-out

; Write simulation report to file (optional):
/report/summary y Report_Transient_Simulation.txt

; Exit fluent (required to requeue):
exit


Ansys CFX

Slurm scripts

File : script-cfx-dist.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=2             # Specify multiple (1 or more) compute nodes
#SBATCH --ntasks-per-node=32  # Specify cores per node (graham 32 or 44, cedar 32 or 48, beluga 40, narval 64)
#SBATCH --mem=0               # Allocate all memory per compute node
#SBATCH --cpus-per-task=1     # Do not change

module load StdEnv/2020       # Applies to: beluga, cedar, graham, narval
module load ansys/2021R1      # Or newer module versions

NNODES=$(slurm_hl2hl.py --format ANSYS-CFX)

# append additional command line options as required
if [ "$CC_CLUSTER" = cedar ]; then
  cfx5solve -def YOURFILE.def -start-method "Open MPI Distributed Parallel" -par-dist $NNODES
else
  cfx5solve -def YOURFILE.def -start-method "Intel MPI Distributed Parallel" -par-dist $NNODES
fi
File : script-cfx-local.sh

#!/bin/bash

#SBATCH --account=def-group   # Specify account name
#SBATCH --time=00-03:00       # Specify time limit dd-hh:mm
#SBATCH --nodes=1             # Specify single compute node (do not change)
#SBATCH --ntasks-per-node=4   # Specify total cores (narval up to 64)
#SBATCH --mem=16G             # Specify 0 to use all node memory
#SBATCH --cpus-per-task=1     # Do not change

module load StdEnv/2020       # Applies to: beluga, cedar, graham, narval
module load ansys/2021R1      # Or newer module versions

# append additional command line options as required
if [ "$CC_CLUSTER" = cedar ]; then
  cfx5solve -def YOURFILE.def -start-method "Open MPI Local Parallel" -part $SLURM_CPUS_ON_NODE
else
  cfx5solve -def YOURFILE.def -start-method "Intel MPI Local Parallel" -part $SLURM_CPUS_ON_NODE
fi

Note: You may get the following error in your output file which does not seem to affect the computation: /etc/tmi.conf: No such file or directory.

Workbench

Before submitting a project file to the queue on a cluster (for the first time) follow these steps to initialize it.

  1. Connect to the cluster with TigerVNC.
  2. Switch to the directory where the project file is located (YOURPROJECT.wbpj) and start Workbench with the same Ansys module you used to create your project.
  3. In Workbench, open the project with File -> Open.
  4. In the main window, right-click on Setup and select Clear All Generated Data.
  5. In the top menu bar pulldown, select File -> Exit to exit Workbench.
  6. In the Ansys Workbench popup, when asked The current project has been modified. Do you want to save it?, click on the No button.
  7. Quit Workbench and submit your job using one of the Slurm scripts shown below.

To avoid writing the solution when a running job successfully completes \remove ;Save(Overwrite=True) from the last line of your script. Doing this will make it easier to run multiple test jobs (for scaling purposes when changing ntasks), since the initialized solution will not be overwritten each time. Alternatively, keep a copy of the initialized YOURPROJECT.wbpj file and YOURPROJECT_files subdirectory and restore them after the solution is written. For APDL-based simulations submitted under the legacy StdEnv/2016 environment, nodes=1 may be either removed from the script or changed to be greater than 1 to permit computations across multiple nodes.

Slurm scripts

A project file can be submitted to the queue by customizing one of the following scripts and then running the sbatch script-wbpj.sh command:

File : script-wbpj-2020.sh

#!/bin/bash

#SBATCH --account=def-account
#SBATCH --time=00-03:00                # Time (DD-HH:MM)
#SBATCH --mem=16G                      # Total Memory (set to 0 for all node memory)
#SBATCH --ntasks=4                     # Number of cores
#SBATCH --nodes=1                      # Do not change (multi-node not supported)
##SBATCH --exclusive                   # Uncomment for scaling testing
##SBATCH --constraint=broadwell        # Applicable to graham or cedar

module load StdEnv/2020 ansys/2021R2   # OR newer Ansys modules

if [ "$SLURM_NNODES" == 1 ]; then
  MEMPAR=0                               # Set to 0 for SMP (shared memory parallel)
else
  MEMPAR=1                               # Set to 1 for DMP (distributed memory parallel)
fi

rm -fv *_files/.lock
MWFILE=~/.mw/Application\ Data/Ansys/`basename $(find $EBROOTANSYS/v* -maxdepth 0 -type d)`/SolveHandlers.xml
sed -re "s/(.AnsysSolution>+)[a-zA-Z0-9]*(<\/Distribute.)/\1$MEMPAR\2/" -i "$MWFILE"
sed -re "s/(.Processors>+)[a-zA-Z0-9]*(<\/MaxNumber.)/\1$SLURM_NTASKS\2/" -i "$MWFILE"
sed -i "s!UserConfigured=\"0\"!UserConfigured=\"1\"!g" "$MWFILE"

export KMP_AFFINITY=disabled
export I_MPI_HYDRA_BOOTSTRAP=ssh

runwb2 -B -E "Update();Save(Overwrite=True)" -F YOURPROJECT.wbpj


File : script-wbpj-2016.sh

#!/bin/bash

#SBATCH --account=def-account
#SBATCH --time=00-03:00                # Time (DD-HH:MM)
#SBATCH --mem=16G                      # Total Memory (set to 0 for all node memory)
#SBATCH --ntasks=4                     # Number of cores
#SBATCH --nodes=1                      # Do not change

module load StdEnv/2016 ansys/2019R3   # Do not change

MEMPAR=1                               # Do not change

rm -fv *_files/.lock
MWFILE=~/.mw/Application\ Data/Ansys/`basename $(find $EBROOTANSYS/v* -maxdepth 0 -type d)`/SolveHandlers.xml
sed -re "s/(.AnsysSolution>+)[a-zA-Z0-9]*(<\/Distribute.)/\1$MEMPAR\2/" -i "$MWFILE"
sed -re "s/(.Processors>+)[a-zA-Z0-9]*(<\/MaxNumber.)/\1$((SLURM_NTASKS-1))\2/" -i "$MWFILE"

export KMP_AFFINITY=balanced
export I_MPI_HYDRA_BOOTSTRAP=ssh
export PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin:$PATH

runwb2 -B -E "Update();Save(Overwrite=True)" -F YOURPROJECT.wbpj


Mechanical

The input file can be generated from within your interactive Workbench Mechanical session by clicking Solution -> Tools -> Write Input Files then specify File name: YOURAPDLFILE.inp and Save as type: APDL Input Files (*.inp). APDL jobs can then be submitted to the queue by running the sbatch script-name.sh command.

Slurm scripts

The Ansys modules used in each of the following scripts have been tested on Graham and should work without issue (uncomment one). Once the scripts have been tested on other clusters, they will be updated if required.

File : script-smp-2016.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory for all cores
#SBATCH --ntasks=8             # Specify number of cores (1 or more)
#SBATCH --nodes=1              # Specify one node (do not change)

unset SLURM_GTIDS

module load StdEnv/2016

#module load ansys/19.1
#module load ansys/19.2
#module load ansys/2019R2
#module load ansys/2019R3
#module load ansys/2020R1
module load ansys/2020R2

mapdl -smp -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -I YOURAPDLFILE.inp

rm -rf results-*
mkdir results-$SLURM_JOB_ID
cp -a --no-preserve=ownership $SLURM_TMPDIR/* results-$SLURM_JOB_ID


File : script-smp-2020.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory for all cores
#SBATCH --ntasks=8             # Specify number of cores (1 or more)
#SBATCH --nodes=1              # Specify one node (do not change)

unset SLURM_GTIDS

module load StdEnv/2020

#module load ansys/2021R2
#module load ansys/2022R1
module load ansys/2022R2

mapdl -smp -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -I YOURAPDLFILE.inp

rm -rf results-*
mkdir results-$SLURM_JOB_ID
cp -a --no-preserve=ownership $SLURM_TMPDIR/* results-$SLURM_JOB_ID


File : script-dis-2016.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem-per-cpu=2G       # Specify memory per core
#SBATCH --ntasks=8             # Specify number of cores (2 or more)
##SBATCH --nodes=2             # Specify number of nodes (optional)
##SBATCH --ntasks-per-node=4   # Specify cores per node (optional)

unset SLURM_GTIDS

module load StdEnv/2016

#module load ansys/2019R3
module load ansys/2020R1

export I_MPI_HYDRA_BOOTSTRAP=ssh; export KMP_AFFINITY=compact
mapdl -dis -mpi intelmpi -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -I YOURAPDLFILE.inp

rm -rf results-*
mkdir results-$SLURM_JOB_ID
cp -a --no-preserve=ownership $SLURM_TMPDIR/* results-$SLURM_JOB_ID


File : script-dis-2020.sh

#!/bin/bash
#SBATCH --account=def-account  # Specify your account
#SBATCH --time=00-03:00        # Specify time (DD-HH:MM)
#SBATCH --mem-per-cpu=2G       # Specify memory per core
#SBATCH --ntasks=8             # Specify number of cores (2 or more)
##SBATCH --nodes=2             # Specify number of nodes (optional)
##SBATCH --ntasks-per-node=4   # Specify cores per node (optional)

unset SLURM_GTIDS

module load StdEnv/2020

module load ansys/2022R2

mapdl -dis -mpi openmpi -b nolist -np $SLURM_NTASKS -dir $SLURM_TMPDIR -I YOURAPDLFILE.inp

rm -rf results-*
mkdir results-$SLURM_JOB_ID
cp -a --no-preserve=ownership $SLURM_TMPDIR/* results-$SLURM_JOB_ID


Ansys allocates 1024 MB total memory and 1024 MB database memory by default for APDL jobs. These values can be manually specified (or changed) by adding arguments -m 1024 and/or -db 1024 to the mapdl command line in the above scripts. When using a remote institutional license server with multiple Ansys licenses, it may be necessary to add -p aa_r or -ppf anshpc, depending on which Ansys module you are using. As always, perform detailed scaling tests before running production jobs to ensure that the optimal number of cores and minimum amount memory is specified in your scripts. The single node (SMP Shared Memory Parallel) scripts will typically perform better than the multinode (DIS Distributed Memory Parallel) scripts and therefore should be used whenever possible. To help avoid compatibility issues the Ansys module loaded in your script should ideally match the version used to to generate the input file: 

 [gra-login2:~/ansys/mechanical/demo] cat YOURAPDLFILE.inp | grep version
! ANSYS input file written by Workbench version 2019 R3

Ansys EDT

Ansys EDT can be run interactively in batch (non-gui) mode by first starting an salloc session with options salloc --time=3:00:00 --tasks=8 --mem=16G --account=def-account and then copy-pasting the full ansysedt command found in the last line of script-local-cmd.sh, being sure to manually specify $YOUR_AEDT_FILE.

Slurm scripts

Ansys Electronic Desktop jobs may be submitted to a cluster queue with the sbatch script-name.sh command using either of the following single node scripts. As of January 2023, the scripts had only been tested on Graham and therefore may be updated in the future as required to support other clusters. Before using them, specify the simulation time, memory, number of cores and replace YOUR_AEDT_FILE with your input file name. A full listing of command line options can be obtained by starting Ansys EDT in graphical mode with commands ansysedt -help or ansysedt -Batchoptionhelp to obtain scrollable graphical popups.

File : script-local-cmd.sh

#!/bin/bash

#SBATCH --account=account      # Specify your account (def or rrg)
#SBATCH --time=00-01:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory (set to 0 to use all compute node memory)
#SBATCH --ntasks=8             # Specify cores (beluga 40, cedar 32 or 48, graham 32 or 44, narval 64)
#SBATCH --nodes=1              # Request one node (Do Not Change)

module load StdEnv/2020
module load ansysedt/2021R2

# Uncomment next line to run a test example:
cp -f $EBROOTANSYSEDT/AnsysEM21.2/Linux64/Examples/HFSS/Antennas/TransientGeoRadar.aedt .

# Specify input file such as:
YOUR_AEDT_FILE="TransientGeoRadar.aedt"

# Remove previous output:
rm -rf $YOUR_AEDT_FILE.* ${YOUR_AEDT_FILE}results

# ---- do not change anything below this line ---- #

echo -e "\nANSYSLI_SERVERS= $ANSYSLI_SERVERS"
echo "ANSYSLMD_LICENSE_FILE= $ANSYSLMD_LICENSE_FILE"
echo -e "SLURM_TMPDIR= $SLURM_TMPDIR on $SLURMD_NODENAME\n"

export KMP_AFFINITY=disabled
ansysedt -monitor -UseElectronicsPPE -ng -distributed -machinelist list=localhost:1:$SLURM_NTASKS \
-batchoptions "TempDirectory=$SLURM_TMPDIR HPCLicenseType=pool HFSS/EnableGPU=0" -batchsolve $YOUR_AEDT_FILE


File : script-local-opt.sh

#!/bin/bash

#SBATCH --account=account      # Specify your account (def or rrg)
#SBATCH --time=00-01:00        # Specify time (DD-HH:MM)
#SBATCH --mem=16G              # Specify memory (set to 0 to allocate all compute node memory)
#SBATCH --ntasks=8             # Specify cores (beluga 40, cedar 32 or 48, graham 32 or 44, narval 64)
#SBATCH --nodes=1              # Request one node (Do Not Change)

module load StdEnv/2020
module load ansysedt/2021R2

# Uncomment next line to run a test example:
cp -f $EBROOTANSYSEDT/AnsysEM21.2/Linux64/Examples/HFSS/Antennas/TransientGeoRadar.aedt .

# Specify input filename such as:
YOUR_AEDT_FILE="TransientGeoRadar.aedt"

# Remove previous output:
rm -rf $YOUR_AEDT_FILE.* ${YOUR_AEDT_FILE}results

# Specify options filename:
OPTIONS_TXT="Options.txt"

# Write sample options file
rm -f $OPTIONS_TXT
cat > $OPTIONS_TXT <<EOF
\$begin 'Config'
'TempDirectory'='$SLURM_TMPDIR'
'HPCLicenseType'='pool'
'HFSS/EnableGPU'=0
\$end 'Config'
EOF

# ---- do not change anything below this line ---- #

echo -e "\nANSYSLI_SERVERS= $ANSYSLI_SERVERS"
echo "ANSYSLMD_LICENSE_FILE= $ANSYSLMD_LICENSE_FILE"
echo -e "SLURM_TMPDIR= $SLURM_TMPDIR on $SLURMD_NODENAME\n"

export KMP_AFFINITY=disabled
ansysedt -monitor -UseElectronicsPPE -ng -distributed -machinelist list=localhost:1:$SLURM_NTASKS \
-batchoptions $OPTIONS_TXT -batchsolve $YOUR_AEDT_FILE


Graphical use

Ansys programs may be run interactively in GUI mode on cluster compute nodes or Graham VDI Nodes.

Compute nodes

Ansys can be run interactively on a single compute node for up to 24 hours. This approach is ideal for testing large simulations since all cores and memory can be requested with salloc as described in TigerVNC. Once connected with vncviewer, any of the following program versions can be started after loading the required modules as shown below.

Fluids

module load StdEnv/2020
module load ansys/2021R1 (or newer versions)
fluent -mpi=intel, or,
QTWEBENGINE_DISABLE_SANDBOX=1 cfx5
------------------------------------------------------------------------------------
module load StdEnv/2016
module load ansys/2020R2 (or older versions)
fluent -mpi=intel, or
QTWEBENGINE_DISABLE_SANDBOX=1 cfx5
o Where QTWEBENGINE_DISABLE_SANDBOX=1 prevents cfx5 -> CFD-Post from crashing on startup with message:
[Unexpected end of transmission from GUI. Engine state was dumped ...]

Mapdl

module load StdEnv/2020
module load ansys/2021R2 (or newer versions)
mapdl -g, or via launcher,
launcher --> click RUN button

Workbench

module load StdEnv/2020
module load ansys/2021R2 (or newer versions)
xfwm4 --replace & (only needed if using Ansys Mechanical)
export QTWEBENGINE_DISABLE_SANDBOX=1 (only needed if using CFD-Post)
runwb2
------------------------------------------------------------------------------------
module load StdEnv/2016 (not available on Narval)
module load ansys/2019R3 (or older versions)
export PATH=$EBROOTNIXPKGS/bin:$PATH
xfwm4 --replace & (only needed if using Ansys Mechanical)
export QTWEBENGINE_DISABLE_SANDBOX=1 (only needed if using CFD-Post)
runwb2
o Where QTWEBENGINE_DISABLE_SANDBOX=1 prevents CFX -> Results from crashing on startup with message:
[Error! Application has terminated before command could be sent]

NOTES :When running an Analysis Program such as Mechanical or Fluent in parallel on a single node, untick Distributed and specify a value of cores equal to your salloc session setting minus 1. The pulldown menus in the Ansys Mechanical workbench do not respond properly. As a workaround run xfwm4 --replace on the command line before starting workbench as shown. To make xfwm4 your default edit $HOME/.vnc/xstartup and change mate-session to xfce4-session. Lastly, fluent from ansys/2022R2 does not currently work on compute nodes please use a different version.

Ansys EDT

Start an interactive session using the following form of the salloc command (to specify cores and available memory):
salloc --time=3:00:00 --nodes=1 --cores=8 --mem=16G --account=def-group
xfwm4 --replace & (then hit enter twice)
module load StdEnv/2020 ansysedt/2021R2, or
module load StdEnv/2020 ansysedt/2023R2
ansysedt
o Click Tools -> Options -> HPC and Analysis Options -> Edit then :
1) untick Use Automatic Settings box (required one time only)
2) under Machines tab do not change Cores (auto-detected from slurm)
o To run interactive analysis click: Project -> Analyze All
o Run first-time configuration again : rm -rf ~/.mw (optional)

Ensight

module load StdEnv/2020 ansys/2022R2; A=222; B=5.12.6, or
module load StdEnv/2020 ansys/2022R1; A=221; B=5.12.6, or
module load StdEnv/2020 ansys/2021R2; A=212; B=5.12.6, or
module load StdEnv/2020 ansys/2021R1; A=211; B=5.12.6, or
module load StdEnv/2016 ansys/2020R2; A=202; B=5.12.6, or
module load StdEnv/2016 ansys/2020R1; A=201; B=5.10.1, or
module load StdEnv/2016 ansys/2019R3; A=195; B=5.10.1, or
export LD_LIBRARY_PATH=$EBROOTANSYS/v$A/CEI/apex$A/machines/linux_2.6_64/qt-$B/lib
ensight -X

Note: ansys/2022R2 Ensight is lightly tested on compute nodes. Please let us know if you find any problems using it.

Rocky

module load rocky/2023R2 ansys/2023R2 (or newer versions)
Rocky (reads ~/licenses/ansys.lic if present, otherwise defaults to SHARCNET server), or
Rocky-int (interactively select CMC or SHARCNET server, also reads ~/licenses/ansys.lic)
RockySolver (run rocky from the command line, currently untested, specify "-h" for help)
RockySchedular (resource manager to submit multiple jobs on present node)
o Rocky is (currently) only available on gra-vdi and graham cluster (no workbench support on linux)
o Release pdfs can be found under /opt/software/rocky/2023R2/docs (read them with mupdf)
o The SHARCNET license now includes Rocky (free for all researchers to use)

VDI nodes

Ansys programs can be run for up to 24 hours on graham VDI Nodes using a maximum of 8 cores and 128GB memory. The VDI System provides GPU OpenGL acceleration therefore it is ideal for performing tasks that benefit from high performance graphics. One might use VDI to create or modify simulation input files, post-process data or visualize simulation results. To get started, log in to gra-vdi.computecanada.ca with TigerVNC, open a new terminal window and start one of the following supported program versions as shown below. The vertical bar | notation is used to separate the various commands.

Fluids

module load CcEnv StdEnv/2020 ansys/2021R1 (or newer)
unset SESSION_MANAGER
fluent | cfx5 | icemcfd
o Where unsetting SESSION_MANAGER prevents the following Qt message from appearing when starting fluent:
[Qt: Session management error: None of the authentication protocols specified are supported]
o In the event the following message appears in a popup window when starting icemcfd ...
[Error segmentation violation - exiting after doing an emergency save]
... do not click the popup OK button otherwise icemcfd will crash. Instead do the following (one time only):
click the Settings Tab -> Display -> tick X11 -> Apply -> OK -> File -> Exit
The error popup should no longer appear when icemcfd is restarted.
------------------------------------------------------------------------------------
module load CcEnv StdEnv/2016 ansys/2020R2, or
module load CcEnv StdEnv/2016 ansys/2020R1, or
module load CcEnv StdEnv/2016 ansys/2019R3
export HOOPS_PICTURE=opengl
fluent | cfx5 | icemcfd

Mapdl

module load StdEnv/2020
module load ansys/2021R2 (or newer versions)
mapdl -g, or via launcher,
unset SESSION_MANAGER; launcher --> click RUN button

Workbench

module load CcEnv StdEnv/2020 ansys/2021R2 (or newer)
export HOOPS_PICTURE=opengl
runwb2
o The export line prevents the following abbreviated tui Warning from appearing when fluent starts:
[Software rasterizer found, hardware acceleration will be disabled.]
Alternatively HOOPS_PICTURE can be set after workbench has started by doing the following:
Fluent Launcher --> Environment Tab --> HOOPS_PICTURE=opengl (without the export)
------------------------------------------------------------------------------------
module load SnEnv ansys/2021R1, or
module load SnEnv ansys/2020R2
runwb2
------------------------------------------------------------------------------------
module load CcEnv StdEnv/2016 ansys/2020R1, or
module load CcEnv StdEnv/2016 ansys/2019R3
export PATH=$EBROOTNIXPKGS/bin:$PATH
runwb2

NOTES: To run an Analysis Program in parallel within Workbench on gra-vdi such as Mechanical or Fluent, untick Distributed and specify a value of cores no greater than 12. To run with more cores than 12 please use a compute node instead. If an Ansys program refuses to start following a previous run there maybe hung processes if the program was not shutdown cleanly or possibly the program was minimized instead. To clear out any old processes run pkill -9 -e -u $USER -f "ansys|fluent|mwrpcss|mwfwrapper|ENGINE|mono" and then try starting the program again.

Ansys EDT

module load SnEnv ansysedt/2021R2, or
module load SnEnv ansysedt/2023R2
ansysedt
Click Tools -> Options -> HPC and Analysis Options -> Edit then :
1) untick Use Automatic Settings box (required one time only)
2) under Machines tab change Cores to 16 or less
To run interactive analysis click: Project -> Analyze All
Optionally force "Running first-time configuration" again : rm -rf ~/.mw

Ensight

module load SnEnv ansys/2019R2 (or newer)
ensight

Rocky

module load clumod rocky/2023R2 CcEnv StdEnv/2020 ansys/2023R2 (or newer versions)
Rocky (reads ~/licenses/ansys.lic if present, otherwise defaults to SHARCNET server), or
Rocky-int (interactively select CMC or SHARCNET server, also reads ~/licenses/ansys.lic)
RockySolver (run rocky from the command line, currently untested, specify "-h" for help)
RockySchedular (resource manager to submit multiple jobs on present node)
o Rocky is (currently) only available on gra-vdi and graham cluster (no workbench support on linux)
o Release pdfs can be found under /opt/software/rocky/2023R2/docs (read them with mupdf)
o The SHARCNET license now includes Rocky (free for all researchers to use)

SSH issues

Some Ansys GUI programs can be run remotely on a cluster compute node by X forwarding over SSH to your local desktop. Unlike VNC, this approach is untested and unsupported since it relies on a properly setup X display server for your particular operating system OR the selection, installation and configuration of a suitable X client emulator package such as MobaXterm. Most users will find interactive response times unacceptably slow for basic menu tasks let alone performing more complex tasks such as those involving graphics rendering. Startup times for GUI programs can also be very slow depending on your Internet connection. For example, in one test it took 40 minutes to fully start ansysedt over SSH while starting it with vncviewer required only 34 seconds. Despite the potential slowness when connecting over SSH to run GUI programs, doing so may still be of interest if your only goal is to open a simulation and perform some basic menu operations or run some calculations. The basic steps are given here as a starting point: 1) ssh -Y username@graham.computecanada.ca; 2) salloc --x11 --time=1:00:00 --mem=16G --cpus-per-task=4 [--gpus-per-node=1] --account=def-mygroup; 3) once connected onto a compute node try running xclock. If the clock appears on your desktop, proceed to load the desired Ansys module and try running the program.

Site-specific usage

SHARCNET license

The SHARCNET Ansys license is free for academic use by any Alliance researcher on any Alliance system. The installed software does not have any solver or geometry limits. The SHARCNET license may only be used for the purpose of Publishable Academic Research. Producing results for private commercial purposes is strictly prohibited. The SHARCNET license was upgraded from CFD to MCS (Multiphysics Campus Solution) in May of 2020. It includes the following products: HF, EM, Electronics HPC, Mechanical and CFD as described here. In 2023 Rocky for Linux (no Workbench support) was also added. Neither LS-DYNA or Lumerical are included in the SHARCNET license. Note that since all the Alliance clusters are Linux based, SpaceClaim cannot be used on our systems. In July of 2021 an additional 1024 anshpc licenses were added to the previous 512 pool. Before running large parallel jobs, scaling tests should be run for any given simulation. Parallel jobs that do not achieve at least 50% CPU utilization may be flagged by the system for a follow-up by our support team.

As of December 2022, each researcher can run 4 jobs using a total of 252 anshpc (plus 4 anshpc per job). Thus any of the following uniform job size combinations are possible: one 256 core job, two 130 core jobs, three 88 core jobs, or four 67 core jobs according to ( (252 + 4*num_jobs) / num_jobs ). Since the best parallel performance is usually achieved by using all cores on packed compute nodes (aka full nodes), one can determine the number of full nodes by dividing the total anshpc cores with the compute node size. For example, consider Graham which has many 32-core (Broadwell) and some 44-core (Cascade) compute nodes, the maximum number of nodes that could be requested when running various size jobs on 32-core nodes would be: 256/32=8, 130/32=~4, 88/32=~2 or 67/32=~2 to run 1, 2, 3 or 4 simultaneous jobs respectively. To express this in equation form, for a given compute node size on any cluster, the number of compute nodes can be calculated by ( 252 + (4*num_jobs) ) / (num_jobs*cores_per_node) ) then round down and finally determine the total cores to request by multiplying the even number of nodes by the number of cores_per_node.

The SHARCNET Ansys license is made available on a first come first serve basis. Should an unusually large number of Ansys jobs be submitted on a given day some jobs could fail on startup should insufficient licenses be available. These events however have become very rare given the recent increase in anshpc licenses. If your research requires more licenses than SHARCNET can provide, a dedicated researcher license may be purchased (and hosted) on an Ansys license server at your local institution.

If more cores are needed (beyond that which the SHARCNET license can provide per user) a research group can purchase a license directly from SimuTech to host on their own institutional license server. Note however that an extra 20% country-wide uplift fee must be payed if the cluster(s) to be used are not co-located at the institution. Waterloo researchers who only use Graham will therefore be exempt since Graham is physically located at their campus.

Researchers can also purchase their own ansys license subscription from CMC and use their remote license servers. Doing so will have several benefits 1) a local institutional license server is not needed 2) a physical license does not need to be obtained upon each renewal 3) the license can be used almost anywhere including at home, institutions, or any alliance cluster across Canada and 4) download and installation instructions for the windows version of ansys are provided so researchers can run spaceclaim on their own computer (not possible on the Alliance since all systems are linux based). There is however one potentially serious limitation to be aware of, according to the CMC Ansys Quick Start Guides one Ansys user may run a simulation on a maximum number of 64 cores.

License server file

To use the SHARCNET Ansys license on any Alliance cluster, simply configure your ansys.lic file as follows: 

[username@cluster:~] cat ~/.licenses/ansys.lic
setenv("ANSYSLMD_LICENSE_FILE", "1055@license3.sharcnet.ca")
setenv("ANSYSLI_SERVERS", "2325@license3.sharcnet.ca")

Query license server

To show the number of licenses in use by your username and the total in use by all users, run: 

ssh graham.computecanada.ca
module load ansys
lmutil lmstat -c $ANSYSLMD_LICENSE_FILE -a | grep "Users of\|$USER"

If you discover any licenses unexpectedly in use by your username (usually due to ansys not exiting cleanly on gra-vdi) then connect to the node where its running, open a terminal window and run the following command to terminate the rogue processes pkill -9 -e -u $USER -f "ansys" after which your licenses should be freed. Note that gra-vdi consists of two nodes (gra-vdi3 and gra-vdi4) which researchers are randomly placed on when connecting to gra-vdi.computecanada.ca with TigerVNC. Therefore it's necessary to specify the full hostname (gra-vdi3.sharcnet.ca or grav-vdi4.sharcnet.ca) when connecting with tigervnc to ensure you log in to the correct node before running pkill.

Local VDI modules

When using gra-vdi, researchers have the choice of loading Ansys modules from our global environment (after loading CcEnv) or loading Ansys modules installed locally on the machine itself (after loading SnEnv). The local modules may be of interest as they include some Ansys programs and versions not yet supported by our standard environment. When starting programs from local Ansys modules, you can select the CMC license server or accept the default SHARCNET license server. Presently, the settings from ~/.licenses/ansys.lic are not used by the local Ansys modules except when starting runwb2 where they will override the default SHARCNET license server settings. Suitable usage of Ansys programs on gra-vdi includes: running a single test job interactively with up to 8 cores and/or 128G RAM, create or modify simulation input files, post process or visualize data.

Ansys modules

  1. Connect to gra-vdi.computecanada.ca with TigerVNC.
  2. Open a new terminal window and load a module:
    module load SnEnv ansys/2021R2, or
    module load SnEnv ansys/2021R1, or
    module load SnEnv ansys/2020R2, or
    module load SnEnv ansys/2020R1, or
    module load SnEnv ansys/2019R3
  3. Start an Ansys program by issuing one of the following:
    runwb2|fluent|cfx5|icemcfd|apdl
  4. Press y and Enter to accept the conditions
  5. Press Enter to accept the n option and use the SHARCNET license server by default (in the case of runwb2 ~/.licenses/ansysedt.lic will be used if present, otherwise ANSYSLI_SERVERS and ANSYSLMD_LICENSE_FILE will be used if set in your environment for example to some other remote license server). If you change n to y and hit y. the CMC license server will be used.

where cfx5 from step 3. above provides the option to start the following components: 

   1) CFX-Launcher  (cfx5 -> cfx5launch)
   2) CFX-Pre       (cfx5pre)
   3) CFD-Post      (cfdpost -> cfx5post)
   4) CFX-Solver    (cfx5solve)

ansysedt modules

  1. Connect to gra-vdi.computecanada.ca with TigerVNC.
  2. Open a new terminal window and load a module:
    module load SnEnv ansysedt/2021R2, or
    module load SnEnv ansysedt/2021R1
  3. Start the Ansys Electromagnetics Desktop program by typing the following command: ansysedt
  4. Press y and Enter to accept the conditions.
  5. Press Enter to accept the n option and use the SHARCNET license server by default (note that ~/.licenses/ansysedt.lic will be used if present, otherwise ANSYSLI_SERVERS and ANSYSLMD_LICENSE_FILE will be used if set in your environment for example to some other remote license server). If you change n to y and hit enter, the CMC license server will be used.

License feature preferences previously setup with anslic_admin are no longer supported following the recent SHARCNET license server update (2021-09-09). If a license problem occurs, try removing the ~/.ansys directory in your /home account to clear the settings. If problems persist please contact our technical support and provide the contents of your ~/.licenses/ansys.lic file.

Additive Manufacturing

To get started configure your ~/.licenses/ansys.lic file to point to a license server that has a valid Ansys Mechanical License. This must be done on all systems where you plan to run the software.

Enable Additive

This section describes how to make the Ansys Additive Manufacturing ACT extension available for use in your project. The steps must be performed on each cluster for each ansys module version where the extension will be used. Any extensions needed by your project will also need to be installed on the cluster as described below. If you get warnings about missing un-needed extensions (such as ANSYSMotion) then uninstall them from your project.

Download Extension

Start Workbench

  • follow the Workbench section in Graphical use above.
  • File -> Open your project file (ending in .wbpj) into Workbench gui

Open Extension Manager

  • click ACT Start Page and the ACT Home page tab will open
  • click Manage Extensions and the Extension Manager will open

Install Extension

  • click the box with the large + sign under the search bar
  • navigate to select and install your AdditiveWizard.wbex file

Load Extension

  • click to highlight the AdditiveWizard box (loads the AdditiveWizard extension for current session only)
  • click lower right corner arrow in the AdditiveWizard box and select Load extension (loads the extension for current AND future sessions)

Unload Extension

  • click to un-highlight the AdditiveWizard box (unloads extension for the current session only)
  • click lower right corner arrow in the AdditiveWizard box and select Do not load as default (extension will not load for future sessions)

Run Additive

Gra-vdi

A user can run a single Ansys Additive Manufacturing job on gra-vdi with up to 16 cores as follows:

  • Start Workbench on Gra-vdi as described above in Enable Additive.
  • click File -> Open and select test.wbpj then click Open
  • click View -> reset workspace if you get a grey screen
  • start Mechanical, Clear Generated Data, tick Distributed, specify Cores
  • click File -> Save Project -> Solve

Check utilization:

  • open another terminal and run: top -u $USER **OR** ps u -u $USER | grep ansys
  • kill rogue processes from previous runs: pkill -9 -e -u $USER -f "ansys|mwrpcss|mwfwrapper|ENGINE"

Please note that rogue processes can persistently tie up licenses between gra-vdi login sessions or cause other unusual errors when trying to start gui programs on gra-vdi. Although rare, rogue processes can occur if an ansys gui session (fluent, workbench, etc) is not cleanly terminated by the user before vncviewer is terminated either manually or unexpectedly - for instance due to a transient network outage or hung filesystem. If the latter is to blame then the processes may not by killable until normal disk access is restored.

Cluster

Project preparation:

Before submitting a newly uploaded Additive project to a cluster queue (with sbatch scriptname) certain preparations must be done. To begin, open your simulation with Workbench gui (as described in the Enable Additive section above) in the same directory that your job will be submitted from and then save it again. Be sure to use the same ansys module version that will be used for the job. Next create a Slurm script (as explained in the Cluster Batch Job Submission - WORKBENCH section above). To perform parametric studies change Update() to UpdateAllDesignPoints() in the Slurm script. Determine the optimal number of cores and memory by submitting several short test jobs. To avoid needing to manually clear the solution and recreate all the design points in Workbench between each test run, either 1) change Save(Overwrite=True) to Save(Overwrite=False) or 2) save a copy of the original YOURPROJECT.wbpj file and corresponding YOURPROJECT_files directory. Optionally create and then manually run a replay file on the cluster in the respective test case directory between each run, noting that a single replay file can be used in different directories by opening it in a text editor and changing the internal FilePath setting. 

module load ansys/2019R3
rm -f test_files/.lock
runwb2 -R myreplay.wbjn

Resource utilization:

Once your additive job has been running for a few minutes a snapshot of its resource utilization on the compute node(s) can be obtained with the following srun command. Sample output corresponding to an eight core submission script is shown next. It can be seen that two nodes were selected by the scheduler: 

[gra-login1:~] srun --jobid=myjobid top -bn1 -u $USER | grep R | grep -v top
  PID USER   PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+  COMMAND
22843 demo   20   0 2272124 256048  72796 R  88.0  0.2  1:06.24  ansys.e
22849 demo   20   0 2272118 256024  72822 R  99.0  0.2  1:06.37  ansys.e
22838 demo   20   0 2272362 255086  76644 R  96.0  0.2  1:06.37  ansys.e
  PID USER   PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+  COMMAND
 4310 demo   20   0 2740212 271096 101892 R 101.0  0.2  1:06.26  ansys.e
 4311 demo   20   0 2740416 284552  98084 R  98.0  0.2  1:06.55  ansys.e
 4304 demo   20   0 2729516 268824 100388 R 100.0  0.2  1:06.12  ansys.e
 4305 demo   20   0 2729436 263204 100932 R 100.0  0.2  1:06.88  ansys.e
 4306 demo   20   0 2734720 431532  95180 R 100.0  0.3  1:06.57  ansys.e

Scaling tests:

After a job completes its "Job Wall-clock time" can be obtained from seff myjobid. Using this value scaling tests can be performed by submitting short test jobs with an increasing number of cores. If the Wall-clock time decreases by ~50% when the number of cores is doubled then additional cores may be considered.

Online documentation

The full Ansys documentation for versions back to 19.2 can be accessed by following these steps:

  1. Connect to gra-vdi.computecanada.ca. with tigervnc as described in TigerVNC.
  2. If the Firefox browser or the Ansys Workbench is open, close it now.
  3. Start Firefox by clicking Applications -> Internet -> Firefox.
  4. Open a new terminal window by clicking Applications -> System Tools -> Mate Terminal.
  5. Start Workbench by typing the following in your terminal: module load CcEnv StdEnv ansys; runwb2.
  6. In the upper Workbench menu bar click Help -> ANSYS Workbench Help; the Ansys documentation page should immediately appear in Firefox.
  7. At this point Workbench is no longer needed so close it by clicking the >Unsaved Project - Workbench tab located along the bottom frame (doing this will bring Workbench into focus) and then click File -> Exit.
  8. In the top middle of the Ansys documentation page, click the word HOME located just left of API DOCS.
  9. Now scroll down and you should see a list of Ansys product icons and/or alphabetical ranges.
  10. Select a product to view its documentation. The documentation for the latest release version will be displayed by default. Change the version by clicking the Release Year pull down located above and just to the right of the Ansys documentation page search bar.
  11. To search for documentation corresponding to a different Ansys product, click HOME again.

Message Passing Interface

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Ansys&oldid=148701"