Abaqus: Difference between revisions

__FORCETOC__

<translate>

[https://www.3ds.com/products-services/simulia/products/abaqus/ Abaqus FEA] is a software suite for finite element analysis and computer-aided engineering.

Abaqus software modules are available on our clusters; however, you must provide your own license. To configure your account on a cluster, log in and create a file named <code>$HOME/.licenses/abaqus.lic</code> containing the following two lines which support versions 202X and 6.14.1 respectively. Next, replace <code>port@server</code> with the flexlm port number and server IP address (or fully qualified hostname) of your Abaqus license server.

{{File

|name=abaqus.lic

}}

If your license has not been set up for use on an Alliance cluster, some additional configuration changes by the Alliance system administrator and your local system administrator will need to be done. Such changes are necessary to ensure the flexlm and vendor TCP ports of your Abaqus server are reachable from all cluster compute nodes when jobs are run via the queue. So we may help you get this done, write to [[Technical support|technical support]]. Please be sure to include the following three items:

* flexlm port number

You will then be sent a list of cluster IP addresses so that your administrator can open the local server firewall to allow connections from the cluster on both ports. Please note that a special license agreement must generally be negotiated and signed by SIMULIA and your institution before a local  license may be used remotely on Alliance hardware.

Below are prototype Slurm scripts for submitting thread and mpi-based parallel simulations to single or multiple compute nodes.  Most users will find it sufficient to use one of the <b>project directory scripts</b> provided in the <i>Single node computing</i> sections. The optional <code>memory=</code> argument found in the last line of the scripts is intended for larger memory or problematic jobs where 3072MB offset value may require tuning.  A listing of all Abaqus command line arguments can be obtained by loading an Abaqus module and running: <code>abaqus -help | less</code>.

Single node jobs that run less than one day should find the <i>project directory script</i> located in the first tab sufficient. However, single node jobs that run for more than a day should use one of the restart scripts.  Jobs that create large restart files will benefit by writing to local disk through the use of the SLURM_TMPDIR environment variable utilized in the <b>temporary directory scripts</b> provided in the two rightmost tabs of the single node standard and explicit analysis sections.  The restart scripts shown here will continue jobs that have been terminated early for some reason.  Such job failures can occur if a job reaches its maximum requested runtime before completing and is killed by the queue or if the compute node the job was running on crashed due to an unexpected hardware failure.  Other restart types are possible by further tailoring of the input file (not shown here) to continue a job with additional steps or change the analysis (see the documentation for version specific details).

Jobs that require large memory or larger compute resources (beyond that which a single compute node can provide) should use the mpi scripts in the <b>multiple node sections</b> below to distribute computing over arbitrary node ranges determined automatically by the scheduler.  Short scaling test jobs should be run to determine wall-clock times (and memory requirements) as a function of the number of cores (2, 4, 8, etc.) to determine the optimal number before running any long jobs.  

Abaqus solvers support thread-based and mpi-based parallelization.  Scripts for each type are provided below for running Standard Analysis type jobs on Single or Multiple nodes respectively.  Scripts to perform multiple node job restarts are not currently provided.

<tabs>

<tab name="project directory script">

##SBATCH --gres=gpu:a100:1    # Uncomment to specify gpu

module load StdEnv/2020        # Latest installed version

module load abaqus/2021        # Latest installed version

#module load StdEnv/2016      # Uncomment to use

#module load abaqus/2020      # Uncomment to use

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "ABAQUSLM_LICENSE_FILE=$ABAQUSLM_LICENSE_FILE"

rm -f testsp1* testsp2*

abaqus job=testsp1 input=mystd-sim.inp \

}}

To write restart data every N=12 time increments specify in the input file:

  *RESTART, WRITE, OVERLAY, FREQUENCY=12

  order_parallel=OFF

</tab>

<tab name="project directory restart script">

##SBATCH --gres=gpu:a100:1    # Uncomment to specify gpu

module load StdEnv/2020        # Latest installed version

module load abaqus/2021        # Latest installed version

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "ABAQUSLM_LICENSE_FILE=$ABAQUSLM_LICENSE_FILE"

rm -f testsp2* testsp1.lck

abaqus job=testsp2 oldjob=testsp1 input=mystd-sim-restart.inp \

}}

The restart input file should contain:

  *HEADING

  *RESTART, READ

</tab>

<tab name="temporary directory script">

##SBATCH --gres=gpu:a100:1    # Uncomment to specify gpu

module load StdEnv/2020        # Latest installed version

module load abaqus/2021        # Latest installed version

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "SLURM_TMPDIR = " $SLURM_TMPDIR

rm -f testst1* testst2*

mkdir $SLURM_TMPDIR/scratch

}}

To write restart data every N=12 time increments specify in the input file:

  *RESTART, WRITE, OVERLAY, FREQUENCY=12

  egrep -i "step|start" testst*.com testst*.msg testst*.sta

</tab>

<tab name="temporary directory restart script">

##SBATCH --gres=gpu:a100:1    # Uncomment to specify gpu

module load StdEnv/2020        # Latest installed version

module load abaqus/2021        # Latest installed version

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "SLURM_TMPDIR = " $SLURM_TMPDIR

rm -f testst2* testst1.lck

cp testst1* $SLURM_TMPDIR

}}

The restart input file should contain:

  *HEADING

</tabs>

Users with large memory or compute needs (and correspondingly large licenses) can use the following script to perform mpi-based computing over an arbitrary range of nodes ideally left to the scheduler to  automatically determine.  A companion template script to perform restart of multinode jobs is not provided due to additional limitations when they can be used.

{{File

   |name="scriptsp1-mpi.txt"

#SBATCH --cpus-per-task=1      # Do not change !

module load StdEnv/2020        # Latest installed version

module load abaqus/2021        # Latest installed version

unset SLURM_GTIDS

#export MPI_IC_ORDER='tcp'

echo "ABAQUSLM_LICENSE_FILE=$ABAQUSLM_LICENSE_FILE"

rm -f testsp1-mpi*

unset hostlist

nodes="$(slurm_hl2hl.py --format MPIHOSTLIST {{!}} xargs)"

echo "$mphostlist" > abaqus_v6.env

abaqus job=testsp1-mpi input=mystd-sim.inp \

   scratch=$SLURM_TMPDIR cpus=$SLURM_NTASKS interactive mp_mode=mpi \

}}

Abaqus solvers support thread-based and mpi-based parallelization.  Scripts for each type are provided below for running explicit analysis type jobs on single or multiple nodes respectively.  Template scripts to perform multinode job restarts are not currently provided pending further testing.

<tabs>

<tab name="project directory script">

#SBATCH --nodes=1              # do not change

module load StdEnv/2020

module load abaqus/2021

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "ABAQUSLM_LICENSE_FILE=$ABAQUSLM_LICENSE_FILE"

rm -f testep1* testep2*

abaqus job=testep1 input=myexp-sim.inp \

}}

To write restart data for a total of 12 time increments specify in the input file:

  *RESTART, WRITE, OVERLAY, NUMBER INTERVAL=12, TIME MARKS=NO

  egrep -i "step|restart" testep*.com testep*.msg testep*.sta

</tab>

<tab name="project directory restart script">

#SBATCH --nodes=1              # do not change

module load StdEnv/2020

module load abaqus/2021

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "ABAQUSLM_LICENSE_FILE=$ABAQUSLM_LICENSE_FILE"

rm -f testep2* testep1.lck

for f in testep1*; do [[ -f ${f} ]] && cp -a "$f" "testep2${f#testep1}"; done

}}

No input file modifications are required to restart the analysis.

</tab>

<tab name="temporary directory script">

#SBATCH --nodes=1              # do not change

module load StdEnv/2020

module load abaqus/2021

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "SLURM_TMPDIR = " $SLURM_TMPDIR

rm -f testet1* testet2*

cd $SLURM_TMPDIR

}}

To write restart data for a total of 12 time increments specify in the input file:

  *RESTART, WRITE, OVERLAY, NUMBER INTERVAL=12, TIME MARKS=NO

  egrep -i "step|restart" testet*.com testet*.msg testet*.sta

</tab>

<tab name="temporary directory restart script">

#SBATCH --nodes=1              # do not change

module load StdEnv/2020

module load abaqus/2021

unset SLURM_GTIDS

export MPI_IC_ORDER='tcp'

echo "SLURM_TMPDIR = " $SLURM_TMPDIR

rm -f testet2* testet1.lck

for f in testet1*; do cp -a "$f" $SLURM_TMPDIR/"testet2${f#testet1}"; done

}}

No input file modifications are required to restart the analysis.

</tab>

</tabs>