LS-DYNA

Introduction

LS-DYNA is available on all our systems. It is used for many applications to solve problems in multiphysics, solid mechanics, heat transfer and fluid dynamics. Analyses are performed as separate phenomena or coupled physics simulations such as thermal stress or fluid structure interaction. LSTC was recently purchased by ANSYS so the LS-DYNA software may eventually be exclusively provided as part of the ANSYS module. For now we recommend using the LS-DYNA software traditionally provided by LSTC as documented in this wiki page.

Licensing

The Alliance is a hosting provider for LS-DYNA. This means that we have LS-DYNA software installed on our clusters. The Alliance does NOT however provide a generic license accessible to everyone or provide license hosting services. Instead many institutions, faculties, and departments already have licenses that can be used on our clusters. Before using such licenses some cluster specific network changes may need to be done (to ensure its reachable from compute nodes). In cases where a license has already been used on a particular cluster these changes may already be done. Users unable to locate or arrange for a license on campus may contact CMC Microsystems. Licenses purchased from CMC do not require the overhead of hosting a local license server since they are hosted on remote server system that CMC manages with the added benefit of being usable anywhere. If you have your own server and need a quote for a locally managed license consider contacting Simutech or contact ANSYS directly. SHARCNET does not provide any free LS-DYNA licenses or license hosting services at this time.

Initial Setup and Testing

If your (existing or new) license server has never been used on the cluster where you plan to run jobs, firewall changes will first need to be done on both the cluster side and server side. This will typically require involvement from both our technical team and the technical people managing your license software. To arrange this, send an email containing the service port and IP address of your floating license server to Technical support. To check if your license file is working run the following commands:

module load ls-dyna
ls-dyna_s or ls-dyna_d

where its not necessary to specify any input file or arguments to run this test. The output header should contain a (non-empty) value for Licensed to: with the exception of CMC license servers. Press ^C to quit the program and return to the command line.

Configuring your license

In 2019 Ansys purchased the Livermore Software Technology Corporation (LSTC), the developer of LS-DYNA. LS-DYNA licenses issued by Ansys since that time use ANSYS License servers. Licenses issued by LSTC may still use an LSTC license server. Also, some of our users obtain an LS-DYNA license through CMC Microsystems. This section explains how to configure your account or job script for each of these cases.

LSTC License

If you have a license issued to run on a LSTC License Server, there are two options to specify it:

Option 1) Specify your license server by creating a small file named ls-dyna.lic with the following contents:

File : ls-dyna.lic

#LICENSE_TYPE: network
#LICENSE_SERVER:<port>@<server>

where <port> is an integer number and <server> is the hostname of your LSTC License server. Put this file in directory $HOME/.licenses/ on each cluster where you plan to submit jobs. The values in the file are picked up by lsdyna when it runs. This occurs because the alliance module system sets LSTC_FILE=/home/$USER.licenses/ls-dyna.lic whenever you load ls-dyna or ls-dyna-mpi modules. This approach is recommended for users with a license hosted on a LSTC License Server since (compared to the next option) the identical settings will automatically be used by all jobs you submit on the cluster (without the need to specify them in each individual slurm script or setting them in your environment).

Option 2) Specify your license server by setting the following two environment variables in your slurm scripts:

export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=<port>@<server>

where <port> is an integer number and <server> is the hostname or IP address of your LSTC License server. These variables will take priority over any values specified in your ~/.licenses/ls-dyna.lic file which must exist (even if its empty) for any ls-dyna or ls-dyna-mpi module to successfully load; To ensure it exists run touch ~/.licenses/ls-dyna.lic once on the command line on each cluster you will submit jobs on. For further details see the official documentation.

ANSYS License

If your LS-DYNA license is hosted on an ANSYS License Server then set the following two environment variables in your slurm scripts:

export LSTC_LICENSE=ansys
export ANSYSLMD_LICENSE_FILE=<port>@<server>

where <port> is an integer number and <server> is the hostname or IP address of your ANSYS License server. These variables cannot be defined in your ~/.licenses/ls-dyna.lic file. The file however must exist (even if its empty) for any ls-dyna module to load. To ensure this run touch ~/.licenses/ls-dyna.lic once from the command line (or each time in your slurm scripts). Note that only module versions >= 12.2.1 will work with ANSYS License Servers.

SHARCNET

The SHARCNET ansys license supports running SMP and MPP lsdyna jobs. It can be used freely by anyone (on a core and job limited basis tba) on graham, narval or cedar cluster by adding the following lines into your slurm script:

export LSTC_LICENSE=ansys
export ANSYSLMD_LICENSE_FILE=1055@license3.sharcnet.ca

CMC License

If your LS-DYNA license was purchased from CMC then set the following two environment variables depending on which cluster you are using:

export LSTC_LICENSE=ansys
Beluga:  export ANSYSLMD_LICENSE_FILE=6624@10.20.73.21
Cedar:   export ANSYSLMD_LICENSE_FILE=6624@172.16.121.25
Graham:  export ANSYSLMD_LICENSE_FILE=6624@199.241.167.222
Narval:  export ANSYSLMD_LICENSE_FILE=6624@10.100.64.10
Niagara: export ANSYSLMD_LICENSE_FILE=6624@172.16.205.199

where the various IP address correspond to the respective CADpass servers. No firewall changes are required to use a CMC license on any cluster since these have already been done. Since the remote CMC server that hosts ls-dyna licenses is ANSYS based, these variables cannot be defined in your ~/.licenses/ls-dyna.lic file. The file however must exist (even if its empty) for any ls-dyna module to load. To ensure this is the case run touch ~/.licenses/ls-dyna.lic once from the command line (or each time in your slurm scripts). Note that only module versions >= 13.1.1 will work with ANSYS License Servers.

Cluster batch job submission

LS-DYNA provides binaries for running jobs on a single compute node (SMP - Shared Memory Parallel using OpenMP) or across a multiple compute nodes (MPP - Message Passing Parallel using MPI). This section provides slurm scripts for each job type.

Single node jobs

Modules for running jobs on a single compute node can be listed with: module spider ls-dyna. Jobs may be submitted to the queue with: sbatch script-smp.sh. The following sample slurm script shows how to to run LS-DYNA with 8 cores on a single cluster compute node. The AUTO setting allows explicit simulations to allocate more memory than the 100M word size default at runtime:

File : script-smp.sh

#!/bin/bash
#SBATCH --account=def-account   # Specify
#SBATCH --time=0-03:00          # D-HH:MM
#SBATCH --cpus-per-task=8       # Specify number of cores
#SBATCH --mem=16G               # Specify total memory
#SBATCH --nodes=1               # Do not change

#export RSNT_ARCH=avx2          # Uncomment on beluga for versions < 14.1.0

module load StdEnv/2020         # Versions < 14.1.0
module load ls-dyna/13.1.1

#module load StdEnv/2023        # Versions > 13.1.1 (coming soon)
#module load intel/2023.2.1
#module load ls-dyna/12.2.1

#export LSTC_LICENSE=ansys      # Specify an ANSYS License Server
#export ANSYSLMD_LICENSE_FILE=<port>@<server>

export LSTC_MEMORY=AUTO        

ls-dyna_s ncpu=$SLURM_CPUS_ON_NODE i=airbag.deploy.k memory=100M

where 
ls-dyna_s = single precision smp solver
ls-dyna_d = double precision smp solver

Multiple node jobs

The module versions available for running jobs on multiple nodes can be listed with module spider ls-dyna-mpi. To submit jobs to the queue use: sbatch script-mpp.sh. Sample scripts for submitting jobs to a specified total number of whole nodes *OR* a specified total number of cores follow:

Specify node count

Jobs can be submitted to a specified number of whole compute nodes with the following script:

File : script-mpp-bynode.sh

#!/bin/bash
#SBATCH --account=def-account    # Specify
#SBATCH --time=0-03:00           # D-HH:MM
#SBATCH --ntasks-per-node=32     # Specify number cores per node (graham 32 or 44, cedar 48, beluga 40)
#SBATCH --nodes=2                # Specify number compute nodes (1 or more)
#SBATCH --mem=0                  # Use all memory per compute node (do not change)

#export RSNT_ARCH=avx2           # Uncomment on beluga for versions < 14.1.0

module load StdEnv/2020          # Versions < 14.1.0
module load ls-dyna-mpi/13.1.1 

#module load StdEnv/2023         # Versions > 13.1.1 (coming soon)
#module load intel/2023.2.1
#module load ls-dyna-mpi/12.2.1

#export LSTC_LICENSE=ansys       # Specify an ANSYS License Server
#export ANSYSLMD_LICENSE_FILE=<port>@<server>

export LSTC_MEMORY=AUTO 

srun ls-dyna_d i=airbag.deploy.k memory=8G memory2=200M

where 
ls-dyna_s = single precision mpp solver
ls-dyna_d = double precision mpp solver

Specify core count

Jobs can be submitted to an arbitrary number of compute nodes by specifying the number of cores. This approach allows the slurm scheduler to determine the optimal number of compute nodes to help minimize job wait time in the queue. Memory limits are applied per core therefore a sufficiently large value of mem-per-cpu must be specified so the master processor can successfully distribute and manage the computations. Note that requesting a total amount of memory instead of using the mem-per-cpu option may not be as efficient as the other job submission methods described so far.

File : script-mpp-bycore.sh

#!/bin/bash
#SBATCH --account=def-account     # Specify
#SBATCH --time=0-03:00            # D-HH:MM
#SBATCH --ntasks=64               # Specify total number of cores
#SBATCH --mem-per-cpu=2G          # Specify memory per core

#export RSNT_ARCH=avx2           # Uncomment on beluga for versions < 14.1.0

module load StdEnv/2020           # Versions < 14.1.0
module load ls-dyna-mpi/13.1.1

#module load StdEnv/2023         # Versions > 13.1.1 (coming soon)
#module load intel/2023.2.1
#module load ls-dyna-mpi/12.2.1

#export LSTC_LICENSE=ansys       # Specify an ANSYS License Server
#export ANSYSLMD_LICENSE_FILE=<port>@<server>

export LSTC_MEMORY=AUTO

srun ls-dyna_d i=airbag.deploy.k

where 
ls-dyna_s = single precision mpp solver
ls-dyna_d = double precision mpp solver

Performance Testing

Depending on the simulation LS-DYNA may (or may not) be able to efficiently use very many cores in parallel. Therefore before running a full simulation, standard scaling tests should be run to determine the optimal number of cores that can be used before simulation slowdown occurs, where the seff jobnumber command can be used to determine the Job Wall-clock time, CPU Efficiency and Memory Efficiency of successfully completed test jobs. In addition, recent testing with airbag jobs submitted to the queue on different clusters found significantly better performance on cedar and narval than graham. The testing was done with 6 cores on a single node using the ls-dyna/12.2.1 module and 6 cores evenly distributed across two nodes using the ls-dyna-mpi/12.2.1 module. Although limited the results point out that significant performance variance can occur on different systems for a given simulation setup. Therefore before running full LD-DYNA simulations its recommended to both A) conduct standard scaling tests on a given cluster and B) run identical test cases on each cluster before settling on an optimal job size, module version and cluster configuration.

Visualization with LS-PrePost

LSTC provides LS-PrePost for pre- and post-processing of LS-DYNA models. This program is made available by a separate module. It does not require a license and can be used on any cluster node or the Graham VDI nodes:

Cluster nodes

Connect to a compute node or to a login node with TigerVNC and open a terminal:

module load StdEnv/2020

module load ls-prepost/4.8
lsprepost

module load ls-prepost/4.9
lsprepost OR lspp49

VDI nodes

Connect to gra-vdi with TigerVNC and open a new terminal:

module load CcEnv StdEnv/2020

module load ls-prepost/4.8
lsprepost

module load ls-prepost/4.9
lsprepost OR lspp49

Message Passing Interface