cc_staff
1,894
edits
Abaqus: Difference between revisions
Jump to navigation
Jump to search
m
no edit summary
mNo edit summary |
mNo edit summary |
||
| Line 277: | Line 277: | ||
#mp_host_split=1 # number of mp processes per host >= 1 (uncomment to specify) | #mp_host_split=1 # number of mp processes per host >= 1 (uncomment to specify) | ||
}} | }} | ||
== Explicit analysis == <!--T:20833--> | == Explicit analysis == <!--T:20833--> | ||
| Line 509: | Line 493: | ||
}} | }} | ||
== | == Memory estimates == <!--T:20840--> | ||
=== Single process === <!--T:20841--> | |||
An estimate for the total slurm node memory (--mem=) required for a simulation to run fully in ram (without being virtualized to scratch disk) can be obtained by examining the Abaqus output <code>test.dat</code> file. For example, a simulation that requires a fairly large amount of memory might show: | An estimate for the total slurm node memory (--mem=) required for a simulation to run fully in ram (without being virtualized to scratch disk) can be obtained by examining the Abaqus output <code>test.dat</code> file. For example, a simulation that requires a fairly large amount of memory might show: | ||
</translate> | </translate> | ||
| Line 525: | Line 512: | ||
<translate> | <translate> | ||
<!--T:20841--> | <!--T:20841--> | ||
Alternatively the total memory estimate for a single node threaded process, could be obtained by running the simulation interactively on a compute node and then monitor the memory consumption using the ps or top commands. The follows described how to do the later::<br> | |||
1) ssh into a cluster, obtain an allocation on a compute node (such as gra100), | 1) ssh into a cluster, obtain an allocation on a compute node (such as gra100), and start your simulation running: | ||
<!--T:20914--> | <!--T:20914--> | ||
{{Commands | {{Commands | ||
|salloc --time=0:30:00 --cpus-per-task=8 --mem=64G --account=def-piname | |salloc --time=0:30:00 --cpus-per-task=8 --mem=64G --account=def-piname | ||
|module load | |module load StdEnv/2020 | ||
|module load abaqus/2021 | |||
|unset SLURM_GTIDS | |unset SLURM_GTIDS | ||
|abaqus job=test input=Sample.inp scratch=$ | |abaqus job=test input=Sample.inp scratch=$SLURM_TMPDIR cpus=8 mp_mode=threads interactive | ||
}} | }} | ||
<!--T:20916--> | <!--T:20916--> | ||
2) ssh into the cluster again, ssh into the compute node | 2) ssh into the cluster again, then ssh into the compute node reserved by salloc and run top ie. | ||
<!--T:20915--> | <!--T:20915--> | ||
| Line 548: | Line 536: | ||
<!--T:20844--> | <!--T:20844--> | ||
To completely satisfy the recommended "MEMORY TO OPERATIONS REQUIRED MINIMIZE I/O" (MRMIO) value, at least the same amount of non-swapped physical memory (RES) must be available to Abaqus. Since the RES will in general be less than the virtual memory (VIRT) by some relatively constant amount for a given simulation, it is necessary to slightly over-allocate the requested Slurm node memory <code>-mem=</code>. In the above sample Slurm script, this over-allocation has been hardcoded to a conservative value of 3072MB based on initial testing of the standard Abaqus solver. To avoid long queue wait times associated with large values of MRMIO, it may be worth investigating the simulation performance impact associated with reducing the RES memory that is made available to Abaqus significantly below the MRMIO. This can be done by lowering the <code>-mem=</code> value which in turn will set an artificially low value of <code>memory=</code> in the Abaqus command (found in the last line of the slurm script). In doing this one should be careful the RES does not dip below the MINIMUM MEMORY REQUIRED (MMR) otherwise Abaqus will exit due to Out of Memory (OOM). As an example, if your MRMIO is 96GB try running a series of short test jobs with <code>#SBATCH --mem=8G, 16G, 32G, 64G</code> until an acceptable minimal performance impact is found, noting that smaller values will result in increasingly larger scratch space used by temporary files. | To completely satisfy the recommended "MEMORY TO OPERATIONS REQUIRED MINIMIZE I/O" (MRMIO) value, at least the same amount of non-swapped physical memory (RES) must be available to Abaqus. Since the RES will in general be less than the virtual memory (VIRT) by some relatively constant amount for a given simulation, it is necessary to slightly over-allocate the requested Slurm node memory <code>-mem=</code>. In the above sample Slurm script, this over-allocation has been hardcoded to a conservative value of 3072MB based on initial testing of the standard Abaqus solver. To avoid long queue wait times associated with large values of MRMIO, it may be worth investigating the simulation performance impact associated with reducing the RES memory that is made available to Abaqus significantly below the MRMIO. This can be done by lowering the <code>-mem=</code> value which in turn will set an artificially low value of <code>memory=</code> in the Abaqus command (found in the last line of the slurm script). In doing this one should be careful the RES does not dip below the MINIMUM MEMORY REQUIRED (MMR) otherwise Abaqus will exit due to Out of Memory (OOM). As an example, if your MRMIO is 96GB try running a series of short test jobs with <code>#SBATCH --mem=8G, 16G, 32G, 64G</code> until an acceptable minimal performance impact is found, noting that smaller values will result in increasingly larger scratch space used by temporary files. | ||
=== Multi process === <!--T:20833--> | |||
To determine the required slurm memory for multi-node slurm scripts, memory estimates (per compute process) required to minimize I/O are given in the output dat file of completed jobs. If mp_host_split is not specified (or is set to 1) then the total number of compute processes will equal the number of nodes. The mem-per-cpu value can then be roughly determined by multiplying the largest memory estimate bye the number of nodes and then dividing by the number or ntasks. If however a value for mp_host_split is specified (greater than 1) than the mem-per-cpu value can be roughly determined from the largest memory estimate times the number of nodes times the value of mp_host_split divided by the number of tasks. Note that mp_host_split must be less than or equal to the number of cores per node assigned by slurm at runtime otherwise abaqus will terminate. This scenario can be controlled by uncommenting to specify a value for tasks-per-node. The following definitive statement is given in every output dat file and mentioned here for reference: | |||
<!--T:20885--> | |||
THE UPPER LIMIT OF MEMORY THAT CAN BE ALLOCATED BY ABAQUS WILL IN GENERAL DEPEND ON THE VALUE OF | |||
THE "MEMORY" PARAMETER AND THE AMOUNT OF PHYSICAL MEMORY AVAILABLE ON THE MACHINE. PLEASE SEE | |||
THE "ABAQUS ANALYSIS USER'S MANUAL" FOR MORE DETAILS. THE ACTUAL USAGE OF MEMORY AND OF DISK | |||
SPACE FOR SCRATCH DATA WILL DEPEND ON THIS UPPER LIMIT AS WELL AS THE MEMORY REQUIRED TO MINIMIZE | |||
I/O. IF THE MEMORY UPPER LIMIT IS GREATER THAN THE MEMORY REQUIRED TO MINIMIZE I/O, THEN THE ACTUAL | |||
MEMORY USAGE WILL BE CLOSE TO THE ESTIMATED "MEMORY TO MINIMIZE I/O" VALUE, AND THE SCRATCH DISK | |||
USAGE WILL BE CLOSE-TO-ZERO; OTHERWISE, THE ACTUAL MEMORY USED WILL BE CLOSE TO THE PREVIOUSLY | |||
MENTIONED MEMORY LIMIT, AND THE SCRATCH DISK USAGE WILL BE ROUGHLY PROPORTIONAL TO THE DIFFERENCE | |||
BETWEEN THE ESTIMATED "MEMORY TO MINIMIZE I/O" AND THE MEMORY UPPER LIMIT. HOWEVER ACCURATE | |||
ESTIMATE OF THE SCRATCH DISK SPACE IS NOT POSSIBLE. | |||
= Graphical use = <!--T:20845--> | = Graphical use = <!--T:20845--> | ||