Running jobs: Difference between revisions

  #SBATCH --mail-type=ALL

Output from a non-interactive Slurm job is normally ''buffered'', which means that there is usually a delay between when data is written by the job and when you can see the output on a login node.  Depending on the application you are running and the load on the filesystem, this delay can range from less than a second to many minutes to the end of the job.

There are methods to reduce or eliminate the buffering, but we do not recommend using them because buffering is vital to preserving the overall performance of the filesystem.  If you need to monitor the output from a job in "real time", we recommend you run an [[#Interactive_jobs|interactive job]] as described above.

Sometimes a submitted job writes no output to the log file for an extended period of time, looking like it is hanging. A common reason for this is the aggressive [[#Output_buffering|buffering]] performed by the Slurm scheduler, which will aggregate many output lines before flushing them to the log file. Often the output file will only be written after the job completes. If you wish to monitor the progress of your submitted job as it runs, consider running an [[#Interactive_jobs|interactive job]]

Another option is to redirect output to a separate file. For example, in your submit file you may run your, e.g. python, job as <code>python myscript.py > mylog.txt</code>. If you wish for the output to also be present in the Slurm log, you may also use the <code>tee</code> command, as follows: <code>python myscript.py | tee mylog.txt</code>.