Prolonging terminal sessions

For working on the clusters, most users will need to use SSH to connect to the cluster for job submission and monitoring, editing files and so forth. Keeping this SSH connection alive for a long period of time, ranging from hours to days, may be necessary for some users and this page proposes a variety of techniques for keeping such a terminal session alive.

SSH configuration

One simple solution is to modify the configuration of your SSH client to prolong the connection. On MacOS and Linux the client configuration is found in $HOME/.ssh/config while in Windows it is located in C:\Users\<username>\.ssh\config. Note that the file may not exist initially, so you will need to create it; you should add the lines

Host *
    ServerAliveInterval 240

This addition will ensure the transmission of a sign-of-life signal over the SSH connection to the cluster every four minutes (240 seconds), which should help to keep your SSH connection alive even if it is idle for several hours.

Terminal multiplexers

The programs screen and tmux are examples of a terminal multiplexer - a program which allows you to detach your terminal session entirely, where it will keep on running on its own until you choose to reattach to it. With such a program, you can logout from the cluster, turn off the workstation or hibernate the laptop you use to connect to the cluster and when you're ready to start working again the next day, reattach to your session and start from right where you left off. In the context of our clusters, one thing you will need to keep in mind is that we have several login nodes for each cluster and your screen or tmux session is specific to a login node. If you wish to reattach to a session, you will first need to ensure you're connected to the right login node which of course means remembering which login node you had been using before. Equally so, we make no guarantee that a particular login node won't be rebooted, which will kill any detached terminal sessions on that node.

TMUX

tmux is a terminal multiplexer that allows multiple virtual sessions in a single terminal session. You can also disconnect from an SSH server without interrupting its process(es).

Cheat sheet

For a complete reference, see this page.

Command	Description
tmux	Start a server
Ctrl+B D	Disconnect from server
tmux a	Reconnect to server
Ctrl+B C	Create a new window
Ctrl+B N	Go to next window
Ctrl+B [	Enable "copy" mode, allowing to scroll with the mouse and Page-Up Page-Down
Esc	Disable "copy" mode

Launch tmux inside a job submitted through tmux

If you submit a job with tmux and try to start tmux within the same job, you will get the lost server error message. This happens because the $TMUX environment variable pointing to the tmux server is propagated to the job. The value of the variable is not valid and you can reset it with:

[name@server ~]$ unset TMUX

However, nested use of tmux is not recommended. To send commands to a nested tmux, one has to hit Ctrl+B twice; for example, to create a new window, one has to use Ctrl+B Ctrl+B C. Consider using screen inside your job (if you are using tmux on a login node).