SSH security improvements

SSH security improvements flowchart (click to enlarge)

SSH is the software protocol that you use to connect to Compute Canada clusters. It protects the security of your communication by verifying the server’s identity and yours against known identification data, and by encrypting the connection. Because security risks evolve over time, Compute Canada will soon end support for certain SSH options which are no longer deemed secure. You will have to make some changes on your part in order to continue using our clusters. The changes are outlined in the flowchart to the right, and explained in greater detail below.

SSH changes (September-October 2019)

Email explaining these changes (with "IMPORTANT" in the subject line) was sent to all users on July 29, and with more detail on September 16.

What is changing?

The following SSH security improvements have been implemented on September 24, 2019 on Graham, and one week later on Béluga and Cedar:

1. Disable certain encryption algorithms.
2. Disable certain public key types.
3. Regenerate the cluster's host keys.

If you do not understand the significance of "encryption algorithms", "public keys", or "host keys", do not be alarmed. Simply follow the steps outlined below. If testing indicates you need to update or change your SSH client, you may find page useful.

Because users do not connect to Arbutus via SSH but through a web interface, the upcoming changes do not concern them.

There were earlier, less comprehensive updates made to both Niagara (on May 31, 2019; see here) and Graham (early August) which would have triggered some of the same messages and errors.

Updating your client's known host list

The first time you login to a Compute Canada cluster after the changes, you will probably see a warning message like the following:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:109
ED25519 host key for graham.computecanada.ca has changed and you have requested strict checking.
Host key verification failed.
Killed by signal 1.

This warning is displayed because the host keys on the cluster (in this case Graham) were changed, and your SSH client software remembers the old host keys. (It does this to prevent "man-in-the-middle" attacks.) This will happen for your SSH client on each device from which you connect, so you may see it multiple times.

You may also get a warning regarding "DNS spoofing", which is related to the same change.

MobaXterm, PuTTY, or WinSCP

If you are using MobaXterm, PuTTY, or WinSCP as your ssh (or scp) client under Windows, the warning will appear in a pop-up window and will allow you to accept the new host key by clicking "Yes". Only click yes if the fingerprint matches one listed in SSH host key fingerprints at the bottom of this page. If the fingerprint does not match any on the list, do not accept the connection, and contact Technical support with the details.

macOS, Linux, GitBash or Cygwin

If you are using the command line ssh command on macOS, Linux, GitBash or Cygwin, you should tell your system to "forget" the old host key by running one of the following commands:

• Graham

for h in 2620:123:7002:4::{2..5} 199.241.166.{2..5} {gra-login{1..3},graham,gra-dtn,gra-dtn1,gra-platform,gra-platform1}.{sharcnet,computecanada}.ca; do ssh-keygen -R $h; done

• Cedar

for h in 206.12.124.{2,6} cedar{1,5}.cedar.computecanada.ca cedar.computecanada.ca; do ssh-keygen -R $h; done

• Beluga

for h in beluga{,{1..4}}.{computecanada,calculquebec}.ca 132.219.136.{1..4}; do ssh-keygen -R $h; done

• Mp2

for h in ip{15..20}-mp2.{computecanada,calculquebec}.ca 204.19.23.2{15..20}; do ssh-keygen -R $h; done

Afterwards, the next time you ssh to the cluster you'll be asked to confirm the new host keys, e.g.:

$ ssh graham.computecanada.ca
The authenticity of host 'graham.computecanada.ca (142.150.188.70)' can't be established.
ED25519 key fingerprint is SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
ED25519 key fingerprint is MD5:bc:93:0c:64:f7:e7:cf:d9:db:81:40:be:4d:cd:12:5c.
Are you sure you want to continue connecting (yes/no)? 

Only type yes if the fingerprint matches one listed in the SSH host key fingerprints at the bottom of this page. If the fingerprint does not match any on the list below, do not accept the connection, and contact Technical support with the details.

Troubleshooting

Can I test my SSH client before the change?

Yes. We have set up a server that you can use to test how your SSH client software works with the new protocols. Connect to it at ssh-test.computecanada.ca using your regular Compute Canada credentials. If you can connect successfully then your client will be able to connect to our clusters after the SSH changes are made in the summer of 2019; however you will still need to update the SSH host keys that your client has stored locally.

See the list of SSH host key fingerprints.

My SSH key no longer works

If you're being asked for a password, but were using SSH keys previously on the same system, it's likely because 1024-bit DSA & RSA keys have been disabled.

You need to generate a new stronger key. The process for doing this depends on the operating system you use, either Windows or Linux/macOS. Those instructions also describe how to add your client's new public key to the remote host, so that you can authenticate with the key rather than needing to provide a password.

I can't connect!

If you see any of the following error messages:

Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
Unable to negotiate with 142.150.188.70 port 22: no matching mac found.

you need to upgrade your SSH client to one of the compatible clients shown below.

Which clients are compatible with the new configuration?

The list below is not exhaustive, but we have tested the configuration with the following clients. Earlier versions of these clients may or may not work. We recommend that you update your operating system and SSH client to the latest version compatible with your hardware.

Linux clients

• OpenSSH_7.4p1, OpenSSL 1.0.2k-fips (CentOS 7.5, 7.6)
• OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f (Ubuntu 14)

OS X clients

You can determine the exact version of your SSH client on OS X using the command ssh -V.

• OpenSSH 7.4p1, OpenSSL 1.0.2k (Homebrew)
• OpenSSH 7.9p1, LibreSSL 2.7.3 (OS X 10.14.5)

Windows clients

• MobaXterm Home Edition v11.1
• PuTTY 0.72
• Windows Services for Linux (WSL) v1
  • Ubuntu 18.04 (OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n)
  • openSUSE Leap 15.1 (OpenSSH_7.9p1, OpenSSL 1.1.0i-fips)

iOS clients

• Termius, 4.3.12

SSH host key fingerprints

To retrieve the host fingerprints remotely, one can use the following commands:

ssh-keyscan <hostname> | ssh-keygen -E md5 -l -f -
ssh-keyscan <hostname> | ssh-keygen -E sha256 -l -f -

Listed below are the SSH fingerprints for our clusters. If the fingerprint you get does not match any on the list below, do not accept the connection, and contact Technical support with the details.

Béluga

ED25519

SHA256:lwmU2AS/oQ0Z2M1a31yRAxlKPcMlQuBPFP+ji/HorHQ

MD5:2d:d7:cc:d0:85:f9:33:c1:44:80:38:e7:68:ce:38:ce

RSA

SHA256:7ccDqnMTR1W181U/bSR/Xg7dR4MSiilgzDlgvXStv0o

MD5:7f:11:29:bf:61:45:ae:7a:07:fc:01:1f:eb:8c:cc:a4

Cedar

ED25519

• SHA256:a4n68wLDqJhxtePn04T698+7anVavd0gdpiECLBylAU
• MD5:f8:6a:45:2e:b0:3a:4b:16:0e:64:da:fd:68:74:6a:24

RSA

• SHA256:91eMtc/c2vBrAKM0ID7boyFySo3vg2NEcQcC69VvCg8
• MD5:01:27:45:a0:fd:34:27:9e:77:66:b0:97:55:10:0e:9b

Graham

ED25519

• SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk
• MD5:bc:93:0c:64:f7:e7:cf:d9:db:81:40:be:4d:cd:12:5c

RSA

• SHA256:tB0gbgW4PV+xjNysyll6JtDi4aACmSaX4QBm6CGd3RM
• MD5:21:51:ca:99:15:a8:f4:92:3b:8e:37:e5:2f:12:55:d3

Narval

ED25519

• SHA256:pTKCWpDC142truNtohGm10+lB8gVyrp3Daz4iR5tT1M
• MD5:79:d5:b2:8b:c6:2c:b6:3b:79:d2:75:0e:3b:31:46:17

RSA

• SHA256:tC0oPkkY2TeLxqYHgfIVNq376+RfBFFUZaswnUeeOnw
• MD5:bc:63:b5:f9:e6:48:a3:b7:0d:4a:23:26:a6:31:19:ef

Niagara

ED25519

• SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc
• MD5:b4:ae:76:a5:2b:37:8d:57:06:0e:9a:de:62:00:26:be

RSA

• SHA256:k6YEhYsI73M+NJIpZ8yF+wqWeuXS9avNs2s5QS/0VhU
• MD5:98:e7:7a:07:89:ef:3f:d8:68:3d:47:9c:6e:a6:71:5e

ssh-test.computecanada.ca

ED25519 (256b)

SHA256:Tpu6li6aynYkhmB83Q9Sh7x8qdhT8Mbw4QcDxTaHgxY

MD5:33:8f:f8:57:fa:46:f9:7f:aa:73:e2:0b:b1:ce:66:38

RSA (4096b)

SHA256:DMSia4nUKIyUhO5axZ/As4I8uqlaX0jPcJvcK93D2H0

MD5:a7:08:00:7c:eb:81:f2:f7:2f:5a:92:b0:85:e3:e8:5d

Mp2

ED25519 (256b)

SHA256:hVAo6KoqKOEbtOaBh6H6GYHAvsStPsDEcg4LXBQUP50

MD5:44:71:28:23:9b:a1:9a:93:aa:4b:9f:af:8d:9b:07:01

RSA (4096b)

SHA256:XhbK4jWsnoNNjoBudO6zthlgTqyKkFDtxiuNY9md/aQ

MD5:88:ef:b0:37:26:75:a2:93:91:f6:15:1c:b6:a7:a9:37

Siku

ED25519 (256b)

SHA256:F9GcueU8cbB0PXnCG1hc4URmYYy/8JbnZTGo4xKflWU

MD5:44:2b:1d:40:31:60:1a:83:ae:1d:1a:20:eb:12:79:93

RSA (2048b)

SHA256:cpx0+k52NUJOf8ucEGP3QnycnVkUxYeqJQMp9KOIFrQ

MD5:eb:44:dc:42:70:32:f7:61:c5:db:3a:5c:39:04:0e:91