{"id":4255,"date":"2018-08-15T19:21:36","date_gmt":"2018-08-15T15:21:36","guid":{"rendered":"https:\/\/rakhesh.com\/?p=4255"},"modified":"2018-08-15T19:21:36","modified_gmt":"2018-08-15T15:21:36","slug":"what-is-esx-problem-hyperthreading-unmitigated","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/virtualization\/what-is-esx-problem-hyperthreading-unmitigated\/","title":{"rendered":"What is esx.problem.hyperthreading.unmitigated?"},"content":{"rendered":"<p>Upgraded one of our ESXi hosts with the <a href=\"https:\/\/www.vmware.com\/security\/advisories\/VMSA-2018-0020.html\">latest patches released today<\/a> that are aimed at fixing the L1 Terminal Fault issues. After that the host started giving this warning: <code>esx.problem.hyperthreading.unmitigated<\/code>. No idea what it&#8217;s supposed to mean!<\/p>\n<p>Went to Configure &gt; Settings &gt; Advanced System Settings and searched for anything with &#8220;hyperthread&#8221; in it. Found <code>VMkernel.Boot.hyperthreadingMitigation<\/code>, which was set to &#8220;false&#8221; but sounded suspiciously similar to the warning I had. Changed it to &#8220;true&#8221;, rebooted the host, and Googled on this setting to come across <a href=\"https:\/\/kb.vmware.com\/s\/article\/55806\">this KB article<\/a>. It&#8217;s a good read but here&#8217;s some excerpts if you are interested in only the highlights:<\/p>\n<blockquote>\n<p>Like Meltdown, Rogue System Register Read, and &#8220;Lazy FP state restore&#8221;, the \u201cL1 Terminal Fault\u201d vulnerability can occur when affected Intel microprocessors speculate beyond an unpermitted data access. By continuing the speculation in these cases, the affected Intel microprocessors expose a new side-channel for attack. (Note, however, that architectural correctness is still provided as the speculative operations will be later nullified at instruction retirement.)<\/p>\n<p>CVE-2018-3646 is one of these Intel microprocessor vulnerabilities and impacts hypervisors. It <em>may allow a malicious VM running on a given CPU core to effectively infer contents of the hypervisor&#8217;s or another VM&#8217;s privileged information residing at the same time in the same core&#8217;s L1 Data cache. Because current Intel processors share the physically-addressed L1 Data Cache across both logical processors of a Hyperthreading (HT) enabled core, indiscriminate simultaneous scheduling of software threads on both logical processors creates the potential for further information leakage.<\/em> CVE-2018-3646 has two currently known attack vectors which will be referred to here as &#8220;Sequential-Context&#8221; and &#8220;Concurrent-Context.\u201d Both attack vectors must be addressed to mitigate CVE-2018-3646..<\/p>\n<p><strong>Attack Vector Summary<\/strong><\/p>\n<ul>\n<li>Sequential-context attack vector: a malicious VM can potentially infer recently accessed L1 data of a previous context (hypervisor thread or other VM thread) on either logical processor of a processor core.<\/li>\n<li>Concurrent-context attack vector: a malicious VM can potentially infer recently accessed L1 data of a concurrently executing context (hypervisor thread or other VM thread) on the other logical processor of the hyperthreading-enabled processor core.<\/li>\n<\/ul>\n<p><strong>Mitigation Summary<\/strong><\/p>\n<ul>\n<li>Mitigation of the Sequential-Context attack vector is achieved by vSphere updates and patches. This mitigation is enabled by default and does not impose a significant performance impact. Please see resolution section for details.<\/li>\n<li>Mitigation of the Concurrent-context attack vector <em>requires enablement of a new feature known as the ESXi Side-Channel-Aware Scheduler<\/em>. The initial version of this feature will only schedule the hypervisor and VMs on one logical processor of an Intel Hyperthreading-enabled core. This feature may impose a non-trivial performance impact and is not enabled by default. <\/li>\n<\/ul>\n<\/blockquote>\n<p>So that&#8217;s what the warning was about. To enable the ESXi Side Channel Aware scheduler we need to set the key above to &#8220;true&#8221;. More excerpts:<\/p>\n<blockquote>\n<p>The Concurrent-context attack vector is mitigated through enablement of the ESXi Side-Channel-Aware Scheduler which is included in the updates and patches listed in VMSA-2018-0020. This scheduler is not enabled by default. Enablement of this scheduler may impose a non-trivial performance impact on applications running in a vSphere environment. The goal of the Planning Phase is to understand if your current environment has sufficient CPU capacity to enable the scheduler without operational impact.<\/p>\n<p>The following list summarizes potential problem areas after enabling the ESXi Side-Channel-Aware Scheduler:<\/p>\n<ul>\n<li>VMs configured with vCPUs greater than the physical cores available on the ESXi host<\/li>\n<li>VMs configured with custom affinity or NUMA settings<\/li>\n<li>VMs with latency-sensitive configuration<\/li>\n<li>ESXi hosts with Average CPU Usage greater than 70%<\/li>\n<li>Hosts with custom CPU resource management options enabled<\/li>\n<li>HA Clusters where a rolling upgrade will increase Average CPU Usage above 100%<\/li>\n<\/ul>\n<p>Note: It may be necessary to acquire additional hardware, or rebalance existing workloads, before enablement of the ESXi Side-Channel-Aware Scheduler. <em>Organizations can choose not to enable the ESXi Side-Channel-Aware Scheduler after performing a risk assessment and accepting the risk posed by the Concurrent-context attack vector. This is NOT RECOMMENDED and VMware cannot make this decision on behalf of an organization.<\/em><\/p>\n<\/blockquote>\n<p>So to fix the second issue we need to enable the new scheduler. That can have a performance hit, so best to enable it manually so you are aware and can keep an eye on the load and performance hits. Also, if you are not in a shared environment and don&#8217;t care, you don&#8217;t need to enable it either. Makes sense.<\/p>\n<p>That warning message could have been a bit more verbose though! :)<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Upgraded one of our ESXi hosts with the latest patches released today that are aimed at fixing the L1 Terminal Fault issues. After that the host started giving this warning: esx.problem.hyperthreading.unmitigated. No idea what it&#8217;s supposed to mean! Went to Configure &gt; Settings &gt; Advanced System Settings and searched for anything with &#8220;hyperthread&#8221; in it. &hellip; <a href=\"https:\/\/rakhesh.com\/virtualization\/what-is-esx-problem-hyperthreading-unmitigated\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">What is esx.problem.hyperthreading.unmitigated?<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[183],"tags":[389,847,846],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/4255"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=4255"}],"version-history":[{"count":0,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/4255\/revisions"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=4255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=4255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=4255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}