Meltdown and Spectre bugs
This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.
Meltdown and Spectre are bugs related to speculative execution in a variety of CPU architectures developed during the past ten to fifteen years and which affect in particular processors from Intel and AMD, including those in use on Compute Canada clusters. A detailed discussion of the two bugs can be found on this page. Compute Canada personnel are currently patching systems deemed sensitive to this vulnerability.
What are the impacts ?
Availability impacts
Updates to patch the vulnerabilities require updating the operating system and rebooting the nodes. For compute nodes, this is typically done in a rolling fashion, resulting in nodes being unavailable for a short period of time. This may impair the scheduling of large jobs, but typically goes unnoticed by users. Some nodes, such as login nodes and cloud hosts, will however see a short interruption of service.
Performance impacts
Many groups around the world, including within Compute Canada, are running benchmarks to evaluate the effects of the operating system patches on performance. Certain figures that have been cited are alarming (up to a 30% or even 50% performance hit), while others are very minimal.
Tasks which involve a lot of input/output (reading and writing files) seem to be most heavily affected. Examples include databases, or file transfers (e.g. rsync). Most high performance computing jobs should be minimally affected since the vast majority of the run time is spent computing rather than doing input and output. Different processor generations are also affected to different degrees, with the most notable performance degradation reported for older processors. If you suspect your jobs may incur a large performance penalty, we recommend that you compare its performance before and after. (Naturally, this will only be useful if you have performance measurements which were taken before the patches were applied.) You can also contact our Technical support to get help, but keep in mind that code modification may be required to lessen the performance impact on any given code.
Below, you will find various links pointing to performance benchmarks. Keep in mind that those were not necessarily run on hardware and operating systems similar to what Compute Canada clusters are running.
What is Compute Canada doing about it ?
Teams managing the Compute Canada clusters are acting diligently to update their servers as needed and as patches are released by various vendors. Many servers have already been patched, but some may require more updates as vendors release new patches.
What should I do about it ?
Security-wise, please rest assured that Compute Canada team members are taking every action possible to ensure that systems we run are secure. If you are operating your own virtual machine in our cloud, you are however responsible for updating its operating system to include the latest security patches (see next subsection).
Performance-wise, if you believe that your application may be severely impacted by the security patches, please contact our Technical support team. We encourage you to bring forward comparative performance numbers of your application (job run time after and before for example). Keep in mind however that mitigating the performance impact of the security patches is likely to require some modification to the code you are running, and may not always be possible.
I have a virtual machine running on the Compute Canada Cloud
Updating your Linux virtual machine's operating system to the latest version frequently over the coming days will ensure you include the latest security patches to address these bugs, see updating your VM for specific instructions on how to update your VM.
References
- Other general information about Spectre and Meltdown is available on the US-CERT web site.
- Includes comprehensive links to vendor patch sites.
- Initial Benchmarks Of The Performance Impact Resulting From Linux's x86 Security Changes
- Further Analyzing The Intel CPU "x86 PTI Issue" On More Systems
- The Meltdown bug and the KPTI patch: How does it impact ML performance?
- Ellexus whitepaper explaining HPC performance issues