Meltdown and Spectre bugs

Meltdown and Spectre are bugs related to speculative execution in a variety of CPU architectures developed during the past ten to fifteen years and which affect in particular processors from Intel and AMD, including those in use on Compute Canada clusters. A detailed discussion of the two bugs can be found on this page and Compute Canada personnel are currently patching all of the systems vulnerable to attacks based on these bugs. What sort of performance degradation users will observe as a consequence of the patches is dependent on the software you are using and how it interacts with the operating system but in general the more filesystem activity and other input/output operations that a program performs during its execution, the more likely it is to suffer from a slowdown. Some benchmarks of the performance loss for AI and machine learning codes are publicly available and we recommend that users consider running some simple tests of their own to see if there is any substantial loss of performance with their own code(s).

What are the impacts ?

Availability impacts

Updates to patch the vulnerabilities require updating the operating system and rebooting the nodes. For compute nodes, this is typically done in a rolling fashion, resulting in nodes being unavailable for a short period of time. This may impair the scheduling of large jobs, but typically goes unnoticed by users. Some nodes, such as login nodes and cloud hosts will however see a short interruption of service.

Performance impacts

Many groups around the world, including within Compute Canada, are running various benchmarks. Numbers cited can sometimes be scary (up to 30 or even 50% performance hit), while other are very minimal. Tasks which do a lot of input/output (reading and writing files) seem to be most impacted. Such tasks include databases, file transfer (rsync). Typical high performance tasks and GPU tasks are not affected nearly as much. Different processor generations are also affected at different levels. If you suspect your jobs may incur a large performance penalty, we recommend that you compare its performance before and after. You can also contact our Technical support to get help, but keep in mind that code modification may be required to lessen the performance impact on any given code.

Below, you will find various links pointing to performance benchmarks. Keep in mind that those were not necessarily run on hardware and operating system similar to what Compute Canada clusters are running.

What is Compute Canada doing about it ?

Teams managing the Compute Canada clusters are acting digilently to update their servers as needed and as patches are being released by various vendors. Many servers have already been patched, but some may require more updates as vendors release new patches.

References

1. Other general information about Spectre and Meltdown is available on the US-CERT web site.
   • Includes comprehensive links to vendor patch sites.