Best practices for job submission: Difference between revisions

* Ultimately, the goal should be to '''ensure that the CPU efficiency of your jobs is very close to 100%''', as measured by the field <code>CPU Efficiency</code> in the output from the <code>seff</code> command.

* Be sure that this GPU you had to wait so much longer to obtain is '''being used as efficiently as possible''' and that it is really contributing to improved performance in your jobs.

** A considerable amount of software does have a GPU option, for example such widely used packages as [[NAMD]] and [[GROMACS]], but only a small part of these programs' functionality has been modified to make use of GPUs. For this reason, it is wiser to '''first test a small sample calculation both with and without a GPU''' to see what kind of speed-up you obtain from the use of this GPU.

** Because of the high cost of GPU nodes, a job using '''a single GPU''' should run significantly faster than if it was using a full CPU node.

** If your job '''only finishes 5% or 10% more quickly with a GPU, it's probably not worth''' the effort of waiting to get a node with a GPU as it will be idle during much of your job's execution.

* '''Other tools for monitoring the efficiency''' of your GPU-based jobs include <code>[https://developer.nvidia.com/nvidia-system-management-interface nvidia-smi]</code>, <code>nvtop</code> and, if you're using software based on [[TensorFlow]], the [[TensorFlow#TensorBoard|TensorBoard]] utility.