Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

What is esx.problem.hyperthreading.unmitigated?

Upgraded one of our ESXi hosts with the latest patches released today that are aimed at fixing the L1 Terminal Fault issues. After that the host started giving this warning: esx.problem.hyperthreading.unmitigated. No idea what it’s supposed to mean!

Went to Configure > Settings > Advanced System Settings and searched for anything with “hyperthread” in it. Found VMkernel.Boot.hyperthreadingMitigation, which was set to “false” but sounded suspiciously similar to the warning I had. Changed it to “true”, rebooted the host, and Googled on this setting to come across this KB article. It’s a good read but here’s some excerpts if you are interested in only the highlights:

Like Meltdown, Rogue System Register Read, and “Lazy FP state restore”, the “L1 Terminal Fault” vulnerability can occur when affected Intel microprocessors speculate beyond an unpermitted data access. By continuing the speculation in these cases, the affected Intel microprocessors expose a new side-channel for attack. (Note, however, that architectural correctness is still provided as the speculative operations will be later nullified at instruction retirement.)

CVE-2018-3646 is one of these Intel microprocessor vulnerabilities and impacts hypervisors. It may allow a malicious VM running on a given CPU core to effectively infer contents of the hypervisor’s or another VM’s privileged information residing at the same time in the same core’s L1 Data cache. Because current Intel processors share the physically-addressed L1 Data Cache across both logical processors of a Hyperthreading (HT) enabled core, indiscriminate simultaneous scheduling of software threads on both logical processors creates the potential for further information leakage. CVE-2018-3646 has two currently known attack vectors which will be referred to here as “Sequential-Context” and “Concurrent-Context.” Both attack vectors must be addressed to mitigate CVE-2018-3646..

Attack Vector Summary

  • Sequential-context attack vector: a malicious VM can potentially infer recently accessed L1 data of a previous context (hypervisor thread or other VM thread) on either logical processor of a processor core.
  • Concurrent-context attack vector: a malicious VM can potentially infer recently accessed L1 data of a concurrently executing context (hypervisor thread or other VM thread) on the other logical processor of the hyperthreading-enabled processor core.

Mitigation Summary

  • Mitigation of the Sequential-Context attack vector is achieved by vSphere updates and patches. This mitigation is enabled by default and does not impose a significant performance impact. Please see resolution section for details.
  • Mitigation of the Concurrent-context attack vector requires enablement of a new feature known as the ESXi Side-Channel-Aware Scheduler. The initial version of this feature will only schedule the hypervisor and VMs on one logical processor of an Intel Hyperthreading-enabled core. This feature may impose a non-trivial performance impact and is not enabled by default.

So that’s what the warning was about. To enable the ESXi Side Channel Aware scheduler we need to set the key above to “true”. More excerpts:

The Concurrent-context attack vector is mitigated through enablement of the ESXi Side-Channel-Aware Scheduler which is included in the updates and patches listed in VMSA-2018-0020. This scheduler is not enabled by default. Enablement of this scheduler may impose a non-trivial performance impact on applications running in a vSphere environment. The goal of the Planning Phase is to understand if your current environment has sufficient CPU capacity to enable the scheduler without operational impact.

The following list summarizes potential problem areas after enabling the ESXi Side-Channel-Aware Scheduler:

  • VMs configured with vCPUs greater than the physical cores available on the ESXi host
  • VMs configured with custom affinity or NUMA settings
  • VMs with latency-sensitive configuration
  • ESXi hosts with Average CPU Usage greater than 70%
  • Hosts with custom CPU resource management options enabled
  • HA Clusters where a rolling upgrade will increase Average CPU Usage above 100%

Note: It may be necessary to acquire additional hardware, or rebalance existing workloads, before enablement of the ESXi Side-Channel-Aware Scheduler. Organizations can choose not to enable the ESXi Side-Channel-Aware Scheduler after performing a risk assessment and accepting the risk posed by the Concurrent-context attack vector. This is NOT RECOMMENDED and VMware cannot make this decision on behalf of an organization.

So to fix the second issue we need to enable the new scheduler. That can have a performance hit, so best to enable it manually so you are aware and can keep an eye on the load and performance hits. Also, if you are not in a shared environment and don’t care, you don’t need to enable it either. Makes sense.

That warning message could have been a bit more verbose though! :)

Interesting podcast episodes

Quick shoutout to some interesting podcast episodes I listened to lately. Sorry they are Overcast links than links to the podcast site. I am being lazy here.

  • The Tradeoffs of Information Hiding in the Control Plane – this one’s from the Packet Pushers network and while the title sounds very techie it is actually a discussion about a book written by the podcast host and the person he is talking to. The book seems interesting, I must buy it sometime to read (or at least add to my library).
  • Episode 221 of The Committed podcast – again an interview, with the author of a productivity book. It’s less of an interview (as both podcasts are) and more of a discussion. Both host and author share a lot of their workflow and apps they use. The apps are mostly Mac or iOS based but it’s a good listen.
  • Episode 222 of The Committed podcast – listening to this currently. I liked the discussion. It’s about books and reading and I resonated with a lot of the discussion. Especially a bit where one of the hosts mentions that he has cut down on his audiobook and podcast listening recently as they were taking up all his time, and started listening to more music. Same here. In my case audiobooks were taking up all my ear time so I have cut them down over the month to listen to more podcasts and also a lot more music than I usually do. Hope that pattern sticks! It’s difficult because my huge Audible library of unheard books make me feel guilty and so I tend to subconsciously prioritize audiobooks unless I actively counter this tendency. :)

New MacBook Air

So I finally dipped my feet into the Mac ecosystem and bought myself a MacBook Air. Yes, I know it’s 3 years old but what the heck – it was the cheapest Mac I could buy! Went for the 8GB/ 256GB i5 version as that’s the one I found on a deal with our local online shopping provider. Might have gone with a different spec if I decided to go with the version available officially with Apple but a) that had a UK English keyboard and b) the same model there was about 33% more expensive so if I were to get a better spec’d one I’d be spending a lot lot more (bringing the costs up to the MacBook range). 

One thing about MacBook purchases though – it isn’t easy. I mean, with an iPhone. you only have to choose along the color & size, and then pick the capacity you want. But with Macs I have to worry about size, CPU (i5 or i7), RAM, and storage; and each choices ups the price by so much! And more than the price the choices just exhaust. It’s the paradox of choice concept (I’ve read the book) and the feeling is similar to Windows laptops where there’s so many choices and you just get bogged down trying to pick what you want and eventually let go of the idea itself. Which is what I had done here (let go of the idea) until my wife suggested this MacBook Air model that was on a deal and I thought what the heck and just purchased it. My focus here is to get something that will get me a toehold in the Mac ecosystem and probably settling on price as a criteria than anything else was what was needed. 

Oh, and the MacBook Air is the only one with a decent set of ports. Yikes! All the other MacBooks have just USB-C ports so there’s the additional cost of dongles and the hassle of having to carry them around. If it wasn’t for the dongles and the fact that the MacBook has a 2nd generation butterfly keyboard which is known for problems (which is fixed in the MacBook Pro’s 3rd generation keyboard) I might have gone for the MacBook. It has more colors too. 

Anyways, back to the MacBook Air. I’ve had it for less than a day now so these are just initial thoughts. 

  • I love the keyboard and size. There’s a lot of room for the hand, and the keys feel good to type on. It’s a very “lapable” laptop. 
  • I thought I’d be put off by the 1440×900 screen as I am so used to full HD nowadays and when I had recently tried using a 1440×900 external monitor I didn’t like it at all, but no I don’t mind this screen. Yes I notice the difference but I don’t mind it. 
  • I like the feel of the OS. I had various people tell me it is complicated and unintuitive etc. but I don’t see that. I love the two finger way of scrolling up and down pages and going back and forward, and the three finger way of moving across apps. That feels very intuitive and much better than having a touch screen. There’s a lot more gestures but I am yet to get the hang of that. I tried to memorize those initially but then figured I’ll pick them up as I go. I think I know the main ones that I am interested in at least. 
  • It’s a jarring experience going to the App Store and seeing all the prices! Boy. It’s like the pre-iPhone days when software used to be expensive. Pretty much everything is US$10 and above, and if something is free it is bound to have a in-add purchase. Even the same app which for iPhone & iPad is (say) US$5 would be US$50 or above here! I imagine it is because the code base is different and so there’s more effort? I don’t know. That’s something I am having trouble getting my head around. The Windows OS store apps are much cheaper (but yeah there aren’t many). Anyways, the App Store is like a trip back in time to expensive software. I don’t think I’ll be buying much apps. Or I hope I won’t be buying much apps – it is not a sustainable option. 
  • The laptop came with MacOS High Sierra 10.13.1 and I couldn’t update to the latest 10.13.6 via the App Store. I downloaded it and tried to upgrade manually, but that failed saying the volume doesn’t meet some pre-requisites. I downloaded 10.13.2 and 10.13.3 and was able to upgrade to them manually, but 10.13.4 fails with the same error. That’s when I came across the macOS Recovery options, especially the Internet Recovery option which you get to by pressing Option-Command (⌘)-R (instead of just Command (⌘)-R for regular recovery). Internet Recovery actually connects to the Internet (it prompts you for Wi-Fi details etc) and can download the latest version and do a fresh install. When I tried this it complained my disk was still being encrypted and so it cannot upgrade. Am guessing that is why the update previously failed so I’ll wait for the encryption to finish and try again. That is so cool though, being able to connect to the Internet and do a recovery! Windows recovery options are nothing compared to this. Even the Recovery screen has a good GUI etc. (of course, that’s easy for Apple to do as it controls the hardware; versus Microsoft which can’t cater for every single display where Windows might be installed on). 
    • Update: After encryption completed I was able to install 10.13.4 successfully. I tried to just to 10.13.6 directly but that failed. I realized that these updates are deltas so I’ll just have to install 10.13.5 and then 10.13.6. Tried that and now my system is finally up to date. Yay! Pity MacOS doesn’t do cumulative updates. 
  • What else? The Finder is good, the uniform way in which each app shows a menubar where you can go and find its options etc. is good. I love the UI as expected for its consistency and sleekness. I also loved how I could just click on the Apple icon and go to “About this Mac” to quickly find its OS version, free storage etc. I don’t know why I liked that, but I found it incredibly thoughtful of Apple to present this information via this option. 
  • There’s still (obviously) a lot to pick up. Keyboard shortcuts and gestures etc. 
  • Oh, forgot. Installing apps from outside the App Store is cute in the way you download the DMG file and then (in most cases) just drag and drop the application to the Applications folder. I remember reading somewhere that in the Mac each application is sandboxed to its own hierarchy or something so it’s not like Windows or Linux where everything just writes to a common place and there’s dependencies and DLL hell etc. 
  • I love how the MacOS restores all my previously open apps after a reboot/ shutdown. It’s just the other day I was wishing Windows could do something similar (my laptop crashed and I had to restore all my Windows) and it was pleasant to see the MacOS do exactly this whenever I’d reboot. Such a user friendly and useful thing to do!

More later!

… forcefulness (personality) of the magician’s character

A paragraph from “Jonathan Strange and Mr. Norrell”, which I am still reading.

“But in the end,” added Dr John, “it is by the imposition of his will upon his patient that the doctor effects his cure. It is the forcefulness of the doctor’s own character which determines his success or failure. It was observed by many people that our father could subdue lunatics merely by fixing them with his eye.”

“Really?” said Strange, becoming interested in spite of himself. “I had never thought of it before, but something of the sort is certainly true of magic. There are all sorts of occasions when the success of a piece of magic depends upon the forcefulness of the magician’s character.”

So true!

[Aside] OS/2 Museum

Oh, this is lovely. This OS/2 Museum blog. Such a trip down memory lane! :)

I came across the blog via a post from it (“How fast is a PS/2 keyboard“). OS/2 is a OS I wanted to try when I was a kid but never got a chance. Just seeing the floppy disk image in the blog header makes me smile with nostalgia!

DNS SRV records used by AD

Just thought I’d put these here for my own easy reference. I keep forgetting these records and when there’s an issue I end up Googling and trying to find them! These are DNS records you can query to see if clients are able to lookup the PDC, GC, KDC, and DC of the domain you specify via DNS. If this is broken nothing else will work. :)

PDC _ldap._tcp.pdc._msdcs.<DnsDomainName>
GC _ldap._tcp.gc._msdcs.<DnsDomainName>
KDC _kerberos._tcp.dc._msdcs.<DnsDomainName>
DC _ldap._tcp.dc._msdcs.<DnsDomainName>

You would look this up using nslookup -type=SRV <Record>.

As a refresher, SRV records are of the form _Service._Proto.Name TTL Class SRV Priority Weight Port Target. The _Service._Proto.Name is what we are looking up above, just that our name space is _msdcs.<DnsDomainName>.

Vocal Harmonizing

A few days ago I was listening to “Agar Tum Saath Ho” from the excellent movie “Tamashaa” and noticed for the first time (yeah after nearly 2-3 years of regularly listening to that song coz it is one of my favorites!) that Arijith has someone else singing along with him in the background. I had previously seen A.R. Rahman employ this in other favorites of mine like “Piya Haji Ali” (from the otherwise unremarkable “Fiza”) and also “Noon-Un-Ala-Noor” (from the artsy-but-worth-a-watch “Meenaxi”). But in both these cases I knew you the background singer was – it was obvious from the artists section of the song. But with “Agar Tum Saath Ho” I never noticed this other singer until a few days ago when I kind of slept in my bus ride home listening to this song on loop, and I think my mind just relaxed and stopped thinking other stuff … it just soaked in the song, was in the moment so to say, and I heard the other singer as obvious as anything else.

Turns out this other singer was Arijith himself, but in a different pitch (thanks Quora) and this technique is called vocal harmonizing. Nice, I didn’t know of this.

While typing this post I was post I was listening to “Aanandhame” from the movie “Aravindante Athithikal” (which I previously mentioned, I love its songs) and noticed that it too employs something similar. While Anne Amie is the primary voice, you can also hear Vineeth Sreenivasan lightly in the background singing along with her. Adds a lot of the feel of the song.

Speaking of “Aravindante Athithikal”, a lovely first half a very draggy second half. Wish the movie had just stuck on with the theme of first half or concluded there if it had nothing more to say. The second half would even have been fine if it didn’t drag so much towards the end about finding the mother!

New AirPods

So I finally purchased a pair of Apple AirPods. There was a deal going on and I got a good additional 20% as there was an offer on my credit card.

  • I can’t control the volume with it (except using Siri).
  • I have to choose between whether I want to be able to pause the music via double tap or go to previous or next tracks. I can customize the double tap on either side AirPod so I only have two choices really.
  • Good thing though is that I can pause by removing either of the AirPods.
  • The fit is good too. I expected it to fall out as Apple EarPods have never fit me; but no, this one stays. Good job!
  • Audio quality is ok as expected. No large sound stage. No bass (I don’t mind that). Good for podcasts and audiobooks which is my use case.
  • The lack of much controls customization irks me though. No other vendor would have been able to get away with that in my opinion.
  • Update after using it for a day: I love the fact that I can use it just one AirPod at a time. That’s super handy. That alone plus the small size and that it’s light and that it fits in my ear and I barely notice it makes it a very useful gadget.

Lovin’ iPhone portrait mode

I started using a work provided iPhone 8 recently, side by side to my personal iPhone 7 Plus. I opted for a golden iPhone 8 and I love that look on the glass back. In terms of prettiness I so much prefer the iPhone 8 to the 7 Plus. It’s a much less finger print magnet too. I think it’s my fingers – they sweat – so the back of the iPhone 7 Plus gets all sweaty after a while of use. But no such issues with the iPhone 8.

I don’t think I’ll ever buy a personal non Plus size phone though. I don’t use the portrait mode much but I miss the dual lens on the Plus when I take pics with the iPhone 8. And the size of the Plus is convenient for typing and watching movies. I notice that I tend to use the iPhone 8 more as a phone or checking work emails or browsing something which quickly, but long term I prefer the iPhone 7 Plus for the size.

Here’s a nice (personal opinion!) pic I took with the iPhone 7 Plus in portrait mode now. That’s what prompted this post.

In other news I have purchased a TORRO case for the iPhone 7 Plus. They look so good! It was an impulse purchase and I hope to get it tomorrow.