Picking up another blog post that I had in my drafts waiting to be finished, and I never did. In this Change ESXi host to use ESXi Side-Channel-Aware Scheduler v2 (SCAv2), I will explain how to change your VMware infrastructure to use SCAV2 instead of SCAv1.
A quick explanation about ESXi Side-Channel-Aware Scheduler v2 (SCAv2) and L1TF.
‘L1 Terminal Fault – VMM’ (L1TF – VMM) Speculative-Execution vulnerability in Intel processors was reported some time ago, particularly in August 2018 with the CVE-2018-3646, and is a security vulnerability that attackIntel CPUs and the option to protect our VMware infrastructure was to use ESXi Side-Channel-Aware Scheduler v2 (SCAv1)
“Intel has disclosed details on a new class of CPU speculative-execution vulnerabilities known collectively as “L1 Terminal Fault” that can occur on past and current Intel processors (from at least 2009 – 2018) [See Table 1 for supported vSphere processors that are affected].
Like Meltdown, Rogue System Register Read, and “Lazy FP state restore”, the “L1 Terminal Fault” vulnerability can occur when affected Intel microprocessors speculate beyond an unpermitted data access. By continuing the speculation in these cases, the affected Intel microprocessors expose a new side-channel for attack. (Note, however, that architectural correctness is still provided as the speculative operations will be later nullified at instruction retirement.)”
One of the ways to protect the CPU was to disable Hyperthreading on ESXi. To protect our ESXi CPUs and VMs from this security vulnerability, VMware launched a PowerCli script HTAware Mitigation Tool where we could test our VMware environment to check if they were protected or not against this vulnerability and at the same time do the needed changes to protect the environment. Many VMware administrators change their environment using ESXi Side-Channel-Aware Scheduler v2 (SCAv1) so that environment protected.
How does it work?
But this option had some side backs. First, we need to disable Hyperthreading, and we also lose performance around 25/30% in our environment. But with vSphere 6.7 U2 VMware launched ESXi Side-Channel-Aware Scheduler v2 (SCAv2) that improve the performance and also enable Hyperthreading.
Important Note: This is a big subject and it is impossible to explain in a few words. There a lot of documentation regarding testing both options and what is the impact on our VMs using SCAv1 or SCAv2 option. If you do not know the subject, I propose you read a bit more before you make any changes to your environment.
Two examples of using SCAv1 or SCAv2:
In the next image, I can check the VM performance between the two options.
In this next image, we can check what is enabled and what is disabled in your ESXi hosts in the two options.
As we can see in the above images, SCAv2 as better performance, and also Hyperthreading is enabled, and we can have the double of vCPUS to allocate in our VMs.
- VMware article about ‘L1 Terminal Fault – VMM’ (L1TF – VMM)
- VMware Security Advisories – VMSA-2018-0020
- VMware Performance Impact Statement for ‘L1 Terminal Fault – VMM’ (L1TF – VMM)
- New Scheduler Option for vSphere 6.7 U2
- Which vSphere CPU Scheduler to Choose
- Performance of vSphere 6.7 Scheduling Options (PDF)
How to switch from SCAv1 to SCAv2?
Note: This blog post is only for those who have already enabled their system with SCAv1 and want to change to SCAv2. Is not install from scratch the SCAv2. You can following the steps to do this but is slightly different.
After you had downloaded the zip (script), you need to extract and import the module to your PowerCli.
Note: Before start using the PowerShell/PowerCli, you should update to the latest version.
Import the module and check witch new command you can use for HTAwareMitigation.
Get-Command -Module HTAwareMitigation
You now have three new commands to use in HTAwareMitigation.
- Get-HTAwareMitigationAnalysis – Analyses your environment and report witch ones can be changed or not.
- Get-HTAwareMitigationConfig – Check or change the configuration.
- Get-HTAwareMitigationSuppression – Check or enable the warning in the ESXi host about L1TF.
After the HTAwareMitigation module is imported, I will run an analysis on my vCenter (we can do this per Cluster).
First, I have connected to the vCenter.
Note: I previously added the entries for the variable $vCenter, $user and $pwd
Connect-VIServer $vCenter -User $user -Password $pwd
Get-HTAwareMitigationAnalysis -Server $vCenter
I will make the changes in two Clusters (10 ESXi hosts each). One has HPE servers and the other only Dell Server.
As we see in the next images, Hyperthreading is disabled in all ESXi hosts.
After the HTAwareMitigationAnalysis finishes, it will create some files.
Note: For this blog post, I will only use the report Output.html, other files are more used if we are applying the HTAwareMitigation for the first time and will only affect the Mitigation in the servers that are fit to be Mitigated. So the rest of the file is out of the scope of this blog post.
Opening the Output.html, we get all the information about our Clusters regarding ESXi hosts and VMs. If everything is ok and we can apply to all we will see all gree, yellow means yes but may lose performance, and red means the changes will have a big impact on the VMs performance.
As we can see in my report, everything is green. This is because the Meditation was already set using SCAv1.
Next, I will check all ESXi Scheduler configuration.
Get-HTAwareMitigationConfig -ClusterName Infrastructure-Cluster
As we can see in the next image and 2 ESXi hosts as an example, my CurrentScheduler is SCAv1.
If I try to enable the Mitigation for this Cluster, this is what I get.
Set-HTAwareMitigationConfig -ClusterName Infrastructure-Cluster -Enable
As we can see in the image above, we get the information that risks with SCAv1. So what we need to do is now do the same but selecting the SCAv2.
Set-HTAwareMitigationConfig -SCAv2 -ClusterName Infrastructure-Cluster
I have changed for both Clusters that I need to change. And we have the Scheduler changed to SVAv2. A reboot is required, so the changes are effective.
Before I reboot the ESXi hosts, I run Get-HTAwareMitigationConfig -ClusterName Infrastructure-Cluster again.
As we can see in the image, we are still using SCAv1, but SCAv2 is now configured and just need a reboot.
I reboot all ESXi hosts that were changed, and now let recheck the Scheduler.
As we can see in the next image, SCAv2 is now the active Scheduler.
Let us check the Server again in the HyperThreading. As we can see now is enable, and now we have more available vCPUs in the VMs and also a better performance.
Before I finish this blog post, just a final step to disable the Mitigation Warnings, in one of the Cluster, I had the SuppressHyperthreadWarning disabled. Meaning I never receive a warning about these L1TF vulnerabilities.
This is the message:
To enable again that ESXi hosts give me warnings about a possible threat, I need to disable the suppression warning.
Set-HTAwareMitigationSuppression -ClusterName Infrastructure-Cluster -Disable
As we can see, SuppressHyperthreadWarning has now being changed to 0, which means it is active.
With the last step, we finish this Change ESXi host to use ESXi Side-Channel-Aware Scheduler v2 (SCAv2).
This Intel vulnerabilities still exist. Even Intel has stated that their CPUs are protected against these threats after CPU Cascade Lake, which is not 100% true. Since we have seen new threats and that those new CPUs are not 100% bulletproof.
About Cascade Lake, this is the statement from Intel:
“These changes will begin with our next-generation Intel® Xeon® Scalable processors (code-named Cascade Lake) as well as 8th Generation Intel® Core™ processors expected to ship in the second half of 2018”
But still, Cascade Lake can be affected by Zombieload v2 in the Intel TSX. But as long we have at least vSphere 6.7 u2 and Bios firmware updated, we are ok.
There is always some CPU vulnerabilities, and companies are always trying to mitigate those vulnerabilities by launching patches or firmware updates.
Some of the new CPU vulnerabilities:
- vSphere & Intel JCC, TAA, and MCEPSC/IFU: What You Need to Know
- VMware Security Advisories – VMSA-2019-0008.2
- VMware Security Advisories – VMSA-2019-0020
For a secure infrastructure, always have your environment up to date with the latest security patches and servers firmware with the latest updates.
I hope this Change ESXi host to use ESXi Side-Channel-Aware Scheduler v2 (SCAv2) helps you to understand not only the Mitigation and L1TF vulnerability but also what is the difference between the Scheduler SCAv1 and SCAv2 and what is the best for your environment. Even this is a subject that is already known for 2 years now, using the SCAv2 is important to understand. Also, since you have your environment using Scheduler SCAv1, how to change the ESXi host to use Scheduler SCAv2.
Check also my Runecast blog post about how they can help you check vulnerabilities in your infrastructure.
Note: Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.