SSH security improvements: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
m (add reference at top, to basic ssh page.)
No edit summary
 
(26 intermediate revisions by 10 users not shown)
Line 7: Line 7:


<!--T:37-->
<!--T:37-->
[[SSH]] is the software protocol that you use to connect to Compute Canada clusters.  It protects the security of your data by verifying the server’s identity and yours against known identification data, and by encrypting the connection.  Because security risks evolve over time, Compute Canada will soon end support for certain SSH options which are no longer deemed secure.  You will have to make some changes on your part in order to continue using our clusters.  The changes are outlined in the flowchart to the right, and explained in greater detail below.
[[SSH]] is the software protocol that you use to connect to Compute Canada clusters.  It protects the security of your communication by verifying the server’s identity and yours against known identification data, and by encrypting the connection.  Because security risks evolve over time, Compute Canada will soon end support for certain SSH options which are no longer deemed secure.  You will have to make some changes on your part in order to continue using our clusters.  The changes are outlined in the flowchart to the right, and explained in greater detail below.


=SSH changes (Summer 2019)= <!--T:1-->
=SSH changes (September-October 2019)= <!--T:1-->
 
<!--T:44-->
Email explaining these changes (with "IMPORTANT" in the subject line) was sent to all users on July 29, and with more detail on September 16.


<!--T:40-->
<!--T:40-->
<!--T:2
<!--T:2
With constant increase in computing power over time, some encryption algorithms  
With constant increase in computing power over time, some encryption algorithms  
and protocols which were reasonably secure ten or fifteen years ago now pose an unacceptable risk of the connection being compromised by a third party. For this reason, Compute Canada is modifying its policies and practices regarding [[SSH]], the principal tool used to provide secure access to its clusters. Some users may have to update their SSH client software, some may have to generate a new public/private key-pair, and everyone will have to update the local copy of the "host key" which is used to identify each Compute Canada cluster.  -->  
and protocols which were reasonably secure ten or fifteen years ago now pose an unacceptable risk of the connection being compromised by a third party. For this reason, Compute Canada is modifying its policies and practices regarding [[SSH]], the principal tool used to provide secure access to its clusters. Some users may have to update their SSH client software, some may have to generate a new public/private key-pair, and everyone will have to update their local copy of the "host key" which is used to identify each Compute Canada cluster.  -->  


== What is changing? == <!--T:3-->
== What is changing? == <!--T:3-->


<!--T:4-->
<!--T:4-->
During the summer of 2019, we will make the following SSH security improvements on Compute Canada clusters:
The following SSH security improvements have been implemented on September 24, 2019 on Graham, and one week later on Béluga and Cedar:


<!--T:5-->
<!--T:5-->
Line 31: Line 34:
<!--T:41-->
<!--T:41-->
Because users do not connect to Arbutus via SSH but through a web interface, the upcoming changes do not concern them.
Because users do not connect to Arbutus via SSH but through a web interface, the upcoming changes do not concern them.
<!--T:42-->
There were earlier, less comprehensive updates made to both Niagara (on May 31, 2019; see [https://docs.scinet.utoronto.ca/index.php/SSH_Changes_in_May_2019 here]) and Graham (early August) which would have triggered some of the same messages and errors.


== Updating your client's known host list == <!--T:7-->
== Updating your client's known host list == <!--T:7-->


<!--T:8-->
<!--T:8-->
The first time you login to a Compute Canada cluster after the changes, you will probably see the following warning message:
The first time you login to a Compute Canada cluster after the changes, you will probably see a warning message like the following:


<!--T:9-->
</translate>
<pre>
<pre>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Line 55: Line 61:
</pre>
</pre>


<translate>
<!--T:10-->
<!--T:10-->
This warning is displayed because the host keys on the cluster (in this case [[Graham]]) were changed, and your SSH client software remembers the old host keys.  (It does this to prevent [https://en.wikipedia.org/wiki/Man-in-the-middle_attack "man-in-the-middle" attacks].)  
This warning is displayed because the host keys on the cluster (in this case [[Graham]]) were changed, and your SSH client software remembers the old host keys.  (It does this to prevent [https://en.wikipedia.org/wiki/Man-in-the-middle_attack "man-in-the-middle" attacks].) This will happen for your SSH client on each device from which you connect, so you may see it multiple times.


<!--T:11-->
<!--T:11-->
Line 64: Line 71:


<!--T:13-->
<!--T:13-->
If you are using MobaXterm, PuTTY, or WinSCP as your ssh (or scp) client under Windows, the warning will appear in a pop-up window and will allow you to accept the new host key by clicking "Yes".  '''Only click yes if the fingerprint matches one listed in [[SSH_changes#SSH_host_key_fingerprints|SSH host key fingerprints]]''' at the bottom of this page. If the fingerprint does not match any on the list, do not accept the connection, and contact [[Technical support]] with the details.
If you are using [[Connecting with MobaXTerm|MobaXterm]], [[Connecting with PuTTY|PuTTY]], or [https://winscp.net/eng/download.php WinSCP] as your ssh (or scp) client under Windows, the warning will appear in a pop-up window and will allow you to accept the new host key by clicking "Yes".  '''Only click yes if the fingerprint matches one listed in [[SSH_changes#SSH_host_key_fingerprints|SSH host key fingerprints]]''' at the bottom of this page. If the fingerprint does not match any on the list, do not accept the connection, and contact [[Technical support]] with the details.


=== macOS, Linux, GitBash or Cygwin === <!--T:14-->
=== macOS, Linux, GitBash or Cygwin === <!--T:14-->
Line 70: Line 77:
<!--T:15-->
<!--T:15-->
If you are using the command line <tt>ssh</tt> command on macOS, Linux, GitBash or Cygwin, you should tell your system to "forget" the old host key by running one of the following commands:
If you are using the command line <tt>ssh</tt> command on macOS, Linux, GitBash or Cygwin, you should tell your system to "forget" the old host key by running one of the following commands:
</translate>


  <!--T:16-->
* Graham
ssh-keygen -R graham.computecanada.ca
  for h in 2620:123:7002:4::{2..5} 199.241.166.{2..5} {gra-login{1..3},graham,gra-dtn,gra-dtn1,gra-platform,gra-platform1}.{sharcnet,computecanada}.ca; do ssh-keygen -R $h; done
ssh-keygen -R cedar.computecanada.ca
* Cedar
ssh-keygen -R beluga.computecanada.ca
for h in 206.12.124.{2,6} cedar{1,5}.cedar.computecanada.ca cedar.computecanada.ca; do ssh-keygen -R $h; done
* Beluga
for h in beluga{,{1..4}}.{computecanada,calculquebec}.ca 132.219.136.{1..4}; do ssh-keygen -R $h; done
* Mp2
for h in ip{15..20}-mp2.{computecanada,calculquebec}.ca 204.19.23.2{15..20}; do ssh-keygen -R $h; done


<translate>
<!--T:17-->
<!--T:17-->
Afterwards, the next time you ssh to the cluster you'll be asked to confirm the new host keys, e.g.:
Afterwards, the next time you ssh to the cluster you'll be asked to confirm the new host keys, e.g.:
</translate>


  <!--T:18-->
  $ ssh graham.computecanada.ca
$ ssh graham.computecanada.ca
  The authenticity of host 'graham.computecanada.ca (142.150.188.70)' can't be established.
  The authenticity of host 'graham.computecanada.ca (142.150.188.70)' can't be established.
  ED25519 key fingerprint is SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
  ED25519 key fingerprint is SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
Line 86: Line 99:
  Are you sure you want to continue connecting (yes/no)?  
  Are you sure you want to continue connecting (yes/no)?  


<translate>
<!--T:19-->
<!--T:19-->
'''Only type yes if the fingerprint matches one listed in the [[SSH_changes#SSH_host_key_fingerprints|SSH host key fingerprints]]''' at the bottom of this page. If the fingerprint does not match any on the list below, do not accept the connection, and contact [[Technical support]] with the details.
'''Only type yes if the fingerprint matches one listed in the [[SSH_changes#SSH_host_key_fingerprints|SSH host key fingerprints]]''' at the bottom of this page. If the fingerprint does not match any on the list below, do not accept the connection, and contact [[Technical support]] with the details.


== Troubleshooting == <!--T:20-->
== Troubleshooting == <!--T:20-->
Line 114: Line 128:


<!--T:25-->
<!--T:25-->
If you see one of the following error messages:
If you see any of the following error messages:
 
</translate>
<!--T:26-->
Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
  Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
  Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
  Unable to negotiate with 142.150.188.70 port 22: no matching mac found.
  Unable to negotiate with 142.150.188.70 port 22: no matching mac found.
 
<translate>
<!--T:27-->
<!--T:27-->
you need to upgrade your SSH client to one of the compatible clients shown below.
you need to upgrade your SSH client to one of the compatible clients shown below.


=== Which clients are compatible with the new configuration? === <!--T:28-->
=== Which clients are compatible with the new configuration? === <!--T:28-->
The list below is not exhaustive, but we have tested the configuration with the following clients. Earlier versions of these clients may or may not work. We recommend that you update your operating system and SSH client to the latest version compatible with your hardware.
'''The list below is not exhaustive''', but we have tested the configuration with the following clients. Earlier versions of these clients may or may not work. We recommend that you update your operating system and SSH client to the latest version compatible with your hardware.


==== Linux clients ==== <!--T:29-->
==== Linux clients ==== <!--T:29-->
Line 137: Line 150:


==== Windows clients ==== <!--T:31-->
==== Windows clients ==== <!--T:31-->
* MobaXterm Personal Edition v11.1
* [[Connecting with MobaXTerm|MobaXterm Home Edition]] v11.1
* [[Connecting with PuTTY|PuTTY]] 0.72
* Windows Services for Linux (WSL) v1
** Ubuntu 18.04 (OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n)
** openSUSE Leap 15.1 (OpenSSH_7.9p1, OpenSSL 1.1.0i-fips)


==== iOS clients ==== <!--T:32-->
==== iOS clients ==== <!--T:32-->
* Termius, 4.3.12
* Termius, 4.3.12


= SSH host key fingerprints = <!--T:33-->
To retrieve the host fingerprints remotely, one can use the following commands:


= SSH host key fingerprints = <!--T:33-->
</translate>
ssh-keyscan <hostname> | ssh-keygen -E md5 -l -f -
ssh-keyscan <hostname> | ssh-keygen -E sha256 -l -f -
<translate>
 
<!--T:43-->
Listed below are the SSH fingerprints for our clusters. If the fingerprint you get does not match any on the list below, do not accept the connection, and contact [[Technical support]] with the details.
{{:SSH_host_keys}}
{{:SSH_host_keys}}


</translate>
</translate>

Latest revision as of 14:49, 28 April 2020

Other languages:


SSH security improvements flowchart (click to enlarge)

SSH is the software protocol that you use to connect to Compute Canada clusters. It protects the security of your communication by verifying the server’s identity and yours against known identification data, and by encrypting the connection. Because security risks evolve over time, Compute Canada will soon end support for certain SSH options which are no longer deemed secure. You will have to make some changes on your part in order to continue using our clusters. The changes are outlined in the flowchart to the right, and explained in greater detail below.

SSH changes (September-October 2019)

Email explaining these changes (with "IMPORTANT" in the subject line) was sent to all users on July 29, and with more detail on September 16.


What is changing?

The following SSH security improvements have been implemented on September 24, 2019 on Graham, and one week later on Béluga and Cedar:

  1. Disable certain encryption algorithms.
  2. Disable certain public key types.
  3. Regenerate the cluster's host keys.

If you do not understand the significance of "encryption algorithms", "public keys", or "host keys", do not be alarmed. Simply follow the steps outlined below. If testing indicates you need to update or change your SSH client, you may find page useful.

Because users do not connect to Arbutus via SSH but through a web interface, the upcoming changes do not concern them.

There were earlier, less comprehensive updates made to both Niagara (on May 31, 2019; see here) and Graham (early August) which would have triggered some of the same messages and errors.

Updating your client's known host list

The first time you login to a Compute Canada cluster after the changes, you will probably see a warning message like the following: 

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:109
ED25519 host key for graham.computecanada.ca has changed and you have requested strict checking.
Host key verification failed.
Killed by signal 1.

This warning is displayed because the host keys on the cluster (in this case Graham) were changed, and your SSH client software remembers the old host keys. (It does this to prevent "man-in-the-middle" attacks.) This will happen for your SSH client on each device from which you connect, so you may see it multiple times.

You may also get a warning regarding "DNS spoofing", which is related to the same change.

MobaXterm, PuTTY, or WinSCP

If you are using MobaXterm, PuTTY, or WinSCP as your ssh (or scp) client under Windows, the warning will appear in a pop-up window and will allow you to accept the new host key by clicking "Yes". Only click yes if the fingerprint matches one listed in SSH host key fingerprints at the bottom of this page. If the fingerprint does not match any on the list, do not accept the connection, and contact Technical support with the details.

macOS, Linux, GitBash or Cygwin

If you are using the command line ssh command on macOS, Linux, GitBash or Cygwin, you should tell your system to "forget" the old host key by running one of the following commands:

  • Graham
for h in 2620:123:7002:4::{2..5} 199.241.166.{2..5} {gra-login{1..3},graham,gra-dtn,gra-dtn1,gra-platform,gra-platform1}.{sharcnet,computecanada}.ca; do ssh-keygen -R $h; done
  • Cedar
for h in 206.12.124.{2,6} cedar{1,5}.cedar.computecanada.ca cedar.computecanada.ca; do ssh-keygen -R $h; done
  • Beluga
for h in beluga{,{1..4}}.{computecanada,calculquebec}.ca 132.219.136.{1..4}; do ssh-keygen -R $h; done
  • Mp2
for h in ip{15..20}-mp2.{computecanada,calculquebec}.ca 204.19.23.2{15..20}; do ssh-keygen -R $h; done

Afterwards, the next time you ssh to the cluster you'll be asked to confirm the new host keys, e.g.: 

$ ssh graham.computecanada.ca
The authenticity of host 'graham.computecanada.ca (142.150.188.70)' can't be established.
ED25519 key fingerprint is SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
ED25519 key fingerprint is MD5:bc:93:0c:64:f7:e7:cf:d9:db:81:40:be:4d:cd:12:5c.
Are you sure you want to continue connecting (yes/no)?

Only type yes if the fingerprint matches one listed in the SSH host key fingerprints at the bottom of this page. If the fingerprint does not match any on the list below, do not accept the connection, and contact Technical support with the details.

Troubleshooting

Can I test my SSH client before the change?

Yes. We have set up a server that you can use to test how your SSH client software works with the new protocols. Connect to it at ssh-test.computecanada.ca using your regular Compute Canada credentials. If you can connect successfully then your client will be able to connect to our clusters after the SSH changes are made in the summer of 2019; however you will still need to update the SSH host keys that your client has stored locally.

See the list of SSH host key fingerprints.

My SSH key no longer works

If you're being asked for a password, but were using SSH keys previously on the same system, it's likely because 1024-bit DSA & RSA keys have been disabled.

You need to generate a new stronger key. The process for doing this depends on the operating system you use, either Windows or Linux/macOS. Those instructions also describe how to add your client's new public key to the remote host, so that you can authenticate with the key rather than needing to provide a password.

I can't connect!

If you see any of the following error messages: 

Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
Unable to negotiate with 142.150.188.70 port 22: no matching mac found.

you need to upgrade your SSH client to one of the compatible clients shown below.

Which clients are compatible with the new configuration?

The list below is not exhaustive, but we have tested the configuration with the following clients. Earlier versions of these clients may or may not work. We recommend that you update your operating system and SSH client to the latest version compatible with your hardware.

Linux clients

  • OpenSSH_7.4p1, OpenSSL 1.0.2k-fips (CentOS 7.5, 7.6)
  • OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f (Ubuntu 14)

OS X clients

You can determine the exact version of your SSH client on OS X using the command ssh -V.

  • OpenSSH 7.4p1, OpenSSL 1.0.2k (Homebrew)
  • OpenSSH 7.9p1, LibreSSL 2.7.3 (OS X 10.14.5)

Windows clients

  • MobaXterm Home Edition v11.1
  • PuTTY 0.72
  • Windows Services for Linux (WSL) v1
    • Ubuntu 18.04 (OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n)
    • openSUSE Leap 15.1 (OpenSSH_7.9p1, OpenSSL 1.1.0i-fips)

iOS clients

  • Termius, 4.3.12

SSH host key fingerprints

To retrieve the host fingerprints remotely, one can use the following commands: 

ssh-keyscan <hostname> | ssh-keygen -E md5 -l -f -
ssh-keyscan <hostname> | ssh-keygen -E sha256 -l -f -

Listed below are the SSH fingerprints for our clusters. If the fingerprint you get does not match any on the list below, do not accept the connection, and contact Technical support with the details.

Béluga

ED25519
SHA256:lwmU2AS/oQ0Z2M1a31yRAxlKPcMlQuBPFP+ji/HorHQ
MD5:2d:d7:cc:d0:85:f9:33:c1:44:80:38:e7:68:ce:38:ce
RSA
SHA256:7ccDqnMTR1W181U/bSR/Xg7dR4MSiilgzDlgvXStv0o
MD5:7f:11:29:bf:61:45:ae:7a:07:fc:01:1f:eb:8c:cc:a4

Cedar

ED25519
  • SHA256:a4n68wLDqJhxtePn04T698+7anVavd0gdpiECLBylAU
  • MD5:f8:6a:45:2e:b0:3a:4b:16:0e:64:da:fd:68:74:6a:24
RSA
  • SHA256:91eMtc/c2vBrAKM0ID7boyFySo3vg2NEcQcC69VvCg8
  • MD5:01:27:45:a0:fd:34:27:9e:77:66:b0:97:55:10:0e:9b

Graham

ED25519
  • SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk
  • MD5:bc:93:0c:64:f7:e7:cf:d9:db:81:40:be:4d:cd:12:5c
RSA
  • SHA256:tB0gbgW4PV+xjNysyll6JtDi4aACmSaX4QBm6CGd3RM
  • MD5:21:51:ca:99:15:a8:f4:92:3b:8e:37:e5:2f:12:55:d3

Narval

ED25519
  • SHA256:pTKCWpDC142truNtohGm10+lB8gVyrp3Daz4iR5tT1M
  • MD5:79:d5:b2:8b:c6:2c:b6:3b:79:d2:75:0e:3b:31:46:17
RSA
  • SHA256:tC0oPkkY2TeLxqYHgfIVNq376+RfBFFUZaswnUeeOnw
  • MD5:bc:63:b5:f9:e6:48:a3:b7:0d:4a:23:26:a6:31:19:ef

Niagara

ED25519
  • SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc
  • MD5:b4:ae:76:a5:2b:37:8d:57:06:0e:9a:de:62:00:26:be
RSA
  • SHA256:k6YEhYsI73M+NJIpZ8yF+wqWeuXS9avNs2s5QS/0VhU
  • MD5:98:e7:7a:07:89:ef:3f:d8:68:3d:47:9c:6e:a6:71:5e

ssh-test.computecanada.ca

ED25519 (256b)
SHA256:Tpu6li6aynYkhmB83Q9Sh7x8qdhT8Mbw4QcDxTaHgxY
MD5:33:8f:f8:57:fa:46:f9:7f:aa:73:e2:0b:b1:ce:66:38
RSA (4096b)
SHA256:DMSia4nUKIyUhO5axZ/As4I8uqlaX0jPcJvcK93D2H0
MD5:a7:08:00:7c:eb:81:f2:f7:2f:5a:92:b0:85:e3:e8:5d

Mp2

ED25519 (256b)
SHA256:hVAo6KoqKOEbtOaBh6H6GYHAvsStPsDEcg4LXBQUP50
MD5:44:71:28:23:9b:a1:9a:93:aa:4b:9f:af:8d:9b:07:01
RSA (4096b)
SHA256:XhbK4jWsnoNNjoBudO6zthlgTqyKkFDtxiuNY9md/aQ
MD5:88:ef:b0:37:26:75:a2:93:91:f6:15:1c:b6:a7:a9:37

Siku

ED25519 (256b)
SHA256:F9GcueU8cbB0PXnCG1hc4URmYYy/8JbnZTGo4xKflWU
MD5:44:2b:1d:40:31:60:1a:83:ae:1d:1a:20:eb:12:79:93
RSA (2048b)
SHA256:cpx0+k52NUJOf8ucEGP3QnycnVkUxYeqJQMp9KOIFrQ
MD5:eb:44:dc:42:70:32:f7:61:c5:db:3a:5c:39:04:0e:91
Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=SSH_security_improvements&oldid=83271"