Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Recent Posts

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

Hyper-V differencing disks with an answer file

If you follow the differencing VHD approach from my previous posts (this & this) you’ll notice the boot process starts off by getting the devices ready, does a reboot or two, and then you are taken to a prompt to set the Administrator password.

Do that and you are set. There’s no other prompting in terms of selecting the image, partitioning etc (because we have bypassed all these stages of the install process).

Would be good if I could somehow specify the admin password and the server name automatically – say via an answer file. That’ll take care of the two basic stuff I do always any way. My admin password is common for all the machines, and the server name is same as the VM name, so these can be figured out automatically to use with an answer file.

The proper way to create an answer file is too much work and not really needed here. So I Googled for answer files, found one, removed most of it as it was unnecessary, and the result is something like this:

If you replace the text marked with –REPLACE– with the computer name, and save this to the c:\ of a newly created VM, the password and computer name will be automatically set for you!

So here’s what I do in addition to the steps before.

Create the differencing VHD as usual

Save the XML file above as “Unattend.xml”. Doesn’t matter where you save it as long as you, I’ll assume it’s in my current directory. If it is saved anyplace else replace the path accordingly in the second cmdlet below.

Mount the VHD, copy the answer file over replacing the machine name with what you want, dismount the VHD. Finito!

That’s it really.

A different way to manipulate the XML file

I used the -replace operator above to make changes to the XML file. But I can do things differently too as PowerShell understands XML natively.

Three cmdlets instead of one, but this might feel “neater”.

Notes of UEFI, GPT, UEFI boot process, disk partitions, and Hyper-V differencing disks with a Generation 2 VM

In my previous post I had talked about creating differencing VHDs for use with Hyper-V. While making that post I realized that what I what I was doing doesn’t work with Generation 2 VMs. Investigating a bit into that bought me to my old friend UEFI. I say “old friend” because UEFI is something I have been reading off and on the past few months – mainly due to my interest in encryption. For instance, my laptop with Self Encrypting SSDs can only be managed by BitLocker if I install Windows 8 in UEFI mode. By default it had installed in BIOS mode (and was continuing to when I re-installed) so a few months ago I had read about UEFI and figured how to install Windows 8 on that laptop in UEFI mode.

Then at work we started getting UEFI computers and so I spent some time going through the firmware on those computers just to get a hang of UEFI.

And then last month I bought a Notion Ink Cain tablet, and to get encryption working on it I had to enable Secure Boot (which is a part of UEFI) so once again I found myself reading about UEFI. That was a fun exercise (and something I am yet to post about) so I have been meaning to write about UEFI for a while just that I never got around to it. Since I stumbled upon UEFI again today, might as well do so now.

So what is UEFI? Simply put UEFI is a firmware specification that’s meant to replace BIOS. Most modern laptops and desktops come with UEFI but it looks and behaves like BIOS so you might not notice the difference until you delve in. In this post I’ll focus on the boot process of BIOS and UEFI as that’s what I am interested in.

BIOS boot process

With BIOS you have an MBR (Master Boot Record). In BIOS you specify the boot order of disks, and each of these disks is searched for the MBR by BIOS. The MBR is the first sector of a disk and it contains information on the partitions in the disk as well as a special program (called a “boot loader”) which can load OSes from these partitions. Since the MBR is at a standard location the BIOS can pass control to the boot loader located there. The BIOS doesn’t need to know anything about the OSes or their file systems – things are dumb & simple.

BIOS has limitations in terms of the size of disks it can work with, the limited space available to the boot loader (because of which you have to use quirks like “chain loaders” and such), and so on. BIOS is good, but its time has come … its replacement is UEFI.

What is UEFI?

BIOS stands for “Basic Input/ Output System”. UEFI stands for “Unified Extensible Firmware Interface”. UEFI began as EFI, and was developed by Intel but is now managed by the UEFI Forum. Both BIOS and UEFI aren’t a specific piece of software. Rather, they are specifications that define the interface between the firmware and OS. The UEFI specification is more managed. There are many versions of the specification, with each version adding more capabilities. For instance, version 2.2 added the Secure Boot protocol stuff. Version 2.1 added cryptography stuff. As of this writing UEFI is at version 2.4.

In contrast, BIOS doesn’t have a specification as such. Various BIOS implementations have their own feature set and there’s no standard.

For backward compatibility UEFI can behave like BIOS. The UEFI specification defines a Compatibility Support Module (CSM) which can emulate BIOS. Bear in mind, it is still UEFI firmware, just that it behaves like BIOS firmware without any of the additional UEFI features or advantages. You can’t have both UEFI and BIOS on a computer – only one of them is present, after all they are both firmware!

UEFI classes

The UEFI forum defines four classes for computers:

  1. Class 0 – The computer has no UEFI, only BIOS.
  2. Class 1 – The computer has UEFI with CSM only. So it has UEFI but behaves in a BIOS compatible mode.
  3. Class 2 – The computer has UEFI and CSM. So it can behave as BIOS compatible mode if need be.
  4. Class 3 – The computer has UEFI only, no CSM.

It’s important to be aware of what class your computer is. Hyper-V Generation 2 VMs, for instance, behave as Class 3 computers. They have no CSM. (Moreover Hyper-V Generation 2 does not have a 32-bit implementation of UEFI so only 64-bit guest OSes are supported).

UEFI and GPT

UEFI has a different boot process to BIOS. For starters, it doesn’t use the MBR. UEFI uses a newer partitioning scheme called GPT (GUID Partition Table) that doesn’t have many of MBRs limitations.

If your disk partitioning is MBR and you system has UEFI firmware, it will boot but in CSM mode. So be sure to choose GPT partitioning if you want to use UEFI without CSM.

Also, even though your machine has UEFI, when trying to install Windows it might boot the Windows installer in CSM mode. When you press F9 or whatever key to select the boot media, usually there’s an option which lets you boot in UEFI mode or BIOS/ CSM mode. Sometimes the option isn’t explicit and if the boot media has both UEFI and BIOS boot files, the wrong one may be chosen and UEFI will behave in CSM mode. It is possible to detect which mode Windows PE (which runs during Windows install) is running in. It is also possible to force the install media to boot in UEFI or CSM mode by deleting the boot files of the mode you don’t want.

My laptop, for instance, is UEFI. But each time I’d install Windows 8 onto it it would pick up the BIOS boot loader files and boot in CSM mode. Since I wanted to use UEFI for some of its features, I used Rufus to create a bootable USB of the media (be sure to select “GPT partitioning for UEFI computers”) and when I booted from it Windows installed in UEFI mode. The trick isn’t the GPT partitioning. The trick is that by telling Rufus we want to boot on an UEFI computer, it omits the BIOS specific boot loader files from the USB. It is not necessariy to use Rufus – the process can be done manually too.

UEFI and GPT work with both 32-bit and 64-bit Windows. The catch is that to booting from GPT is only supported for 64-bit Windows running in UEFI. So while you can have 32-bit Windows running in UEFI, it will need an MBR partition to boot from. What this means is that such a system will be running as UEFI Class 2 as that’s the only one which supports UEFI and MBR partitions (essentially the system has UEFI but behaves as BIOS compatible mode).

UEFI classes and MBR/GPT partitioning

With Windows you can use MBR or GPT partitions on your computer depending on its class. From this Microsoft page:

  • UEFO Class 0 – Uses MBR partitions.
  • UEFI Class 1 – Uses GPT partitions.
  • UEFI Class 2 – Uses GPT partitions. This class of UEFI support includes CSM so if MBR partitions are present UEFI will run in compatibility mode.
  • UEFI Class 3 – Uses GPT partitions.

I am not clear why Class 1 only uses GPT partitions. Considering Class 1 is UEFI with CSM only and CSM supports MBR, I would have thought Class 1 supports only MBR partitions.

UEFI boot process

The UEFI boot process is more complicated than BIOS. That doesn’t mean it’s difficult to understand or unnecessarily complicated. What I meant is that it isn’t as simple as having an MBR with a boot loader, as in the case of BIOS. You can’t expect to pass along a VHD file created with BIOS in mind to a machine having only UEFI and expect it to work (as was my case). You need to tweak things so the boot process works with UEFI.

An excellent blog post on the UEFI boot process is this. If you have the time and inclination, go read it! You will be glad you did. What follows are my notes from that post and some others.

  • The UEFI specifications define a type of executable (think .exe files) that all UEFI firmware must support. Each OS that wishes the UEFI firmware to be able to boot it will provide a boot loader of this type. That’s it. OS provides such a boot loader, UEFI loads it.
  • In BIOS the boot loader was present in the MBR. Where is it present in UEFI? In order to be not limited by space like BIOS was, UEFI defines a special partition where boot loaders can be stored. The partition doesn’t have to be of a specific size or at a specific location. The spec requires that all UEFI firmware must be able to read a variant of the FAT file system that’s defined in the spec. (UEFI firmware can read other file system types too if they so wish, but support for this variant of FAT is a must). So UEFI boot loaders are stored in a special partition that’s of file system type FAT (the variant defined by UEFI). And to denote this partition as the special partition it has a different type (i.e. it doesn’t say FAT32 or NTFS or EXT2FS etc, it says ESP (EFI System Partition)). Simple! (Oh, and there can be multiple ESP partitions too if you so wish!)

The above design makes UEFI much more reliable than BIOS. Whereas with the latter you could only store a very limited boot loader at a specific space on the disk – and that boot loader usually chain loaded the OSes – with UEFI you can store boot loaders (in the EFI executable format) of each OS in the ESP partition that’s of file system type FAT (the variant defined by UEFI). Already you have a lot more flexibility compared to BIOS.

To tie all these together UEFI has a boot manager. The boot manager is what looks at all the boot loader entries and creates a menu for booting them. The menu isn’t a static one – the firmware can create a menu on the fly based on boot loaders present across multiple disks attached to the computer. And this boot manager can be managed by tools in the installed OS too. (Sure you could do similar things with Linux boot loaders such as GRUB, but the neat thing here is that the functionality is provided by the firmware – independent of the OS – which is really where it should be! It’s because BIOS was so limited that we had fancy boot loaders like GRUB that worked around it).

If you go down to the section entitled “The UEFI boot manager” in the post I linked to earlier you’ll see an example of a boot manager output. No point me paraphrasing what the author has said, so best to go and check there. I’ll mention one interesting point though:

  • Remember I said there are ESP partitions and they contain the OS boot loaders? So, for instance, you could have an UEFI boot manager entry like HD(1,800,61800,6d98f360-cb3e-4727-8fed-5ce0c040365d)File(\EFI\fedora\grubx64.efi) which points to the partition called HD(1,800,61800,6d98f360-cb3e-4727-8fed-5ce0c040365d) (the naming convention follows a EFI_DEVICE_PATH_PROTOCOL specification) and specifically the \EFI\fedora\grubx64.efi file as the boot loader.
  • What you can also have, however, is a generic entry such as HD(2,0,00). Note there’s no boot loader specified here, and probably no specific ESP partition either. What happens in such cases is that the boot manager will go through each ESP partition on that disk, check for a file \EFI\BOOT\BOOT{machine type short-name}.EFI, and try loading that. This way the UEFI spec allows for one to boot from a hard disk without specifying the OS or path to the boot loader, as long as the disk contains a “default” boot loader as per the naming convention above. This is what happens, for instance, when you boot a Windows 8 DVD, for instance. If you put in such a DVD in your computer and check, you’ll see the root folder has a folder called EFI that contains a sub folder called BOOT which contains a file called bootx64.efi.

Another example and screenshot of the UEFI boot manager can be found at this link.

Tying this in with my WIM to VHD case

If you have read this far, it’s obvious what’s wrong with my VHD file. When the Gen 2 VM boots up – and it uses UEFI as it’s a Gen 2 VM – it will look for a ESP partition with the UEFI boot loader but won’t find any (as my VHD has only one partition and that too of type NTFS). So what I need to do is create an ESP partition and copy the boot loaders to it as required. Also, I am using MBR style partitioning and a Gen 2 VM firmware is Class 3, so I must switch to GPT.

In fact, while I am at it why don’t I partition everything properly. When I install Windows manually (server or desktop) it creates many partitions so this looks like a good opportunity to read up on the Windows partitioning scheme and create any other required partitions on my base disk.

Understanding (GPT/UEFI) disk partitions for Windows

There are three Microsoft pages I referred to:

Read those for more details than what I post below.

The following partitions are required:

  • System partition: This is the EFI System Partition (ESP). Minimum size of the partition is 100 MB, FAT32 formatted. For Windows, the ESP contains the NTLDR, HAL, and other files and drivers required to boot the system. The partition GUID for ESP is DEFINE_GUID (PARTITION_SYSTEM_GUID, 0xC12A7328L, 0xF81F, 0x11D2, 0xBA, 0x4B, 0x00, 0xA0, 0xC9, 0x3E, 0xC9, 0x3B) (on an MBR partition the ID is 0xEF; but remember, Windows doesn’t support booting into UEFI mode from MBR partitions). The type of this partition is c12a7328-f81f-11d2-ba4b-00a0c93ec93b. Windows does not support having two ESPs on a single disk.
  • Microsoft Reserved Partition (MSR): Whereas with BIOS/ MBR one could have hidden sectors, UEFI does away with all that. So Microsoft recommends a reserved partition be set aside instead of such hidden sectors. The size of this partition is 128 MB (for drives larger than 16GB; else the size is 32 MB). It does not have any data – think of the MSR as free space set aside for future use by Windows – it is used when any disk operations require extra space and/ or partitions and they can’t use the existing space and/ or partitions. The partition GUID for MSR is DEFINE_GUID (PARTITION_MSFT_RESERVED_GUID, 0xE3C9E316L, 0x0B5C, 0x4DB8, 0x81, 0x7D, 0xF9, 0x2D, 0xF0, 0x02, 0x15, 0xAE). The type of this partition is e3c9e316-0b5c-4db8-817d-f92df00215ae.

The order of these partitions is: ESP, followed by any OEM partitions, followed by MSR, followed by the OS & data partitions. (See this link for a nice picture).

Apart from the two above, Microsoft recommends two other partitions (note: these recommended, not required):

  • Windows Recovery Environment (Windows RE) tools partition: This must be at least 300 MB, preferably 500 MB or larger, and contains the Windows RE tools image (winre.wim) which is about 300 MB in size. It is preferred that these tools are on a separate partition in case the main partition is BitLocker encrypted, and even otherwise to ensure the files in this partition are preserved in case the main partition is wiped out. The type of this partition is de94bba4-06d1-4d40-a16a-bfd50179d6ac.
  • Recovery image partition: This must be at least 2 GB, preferably 3 GB, and contains the Windows recovery image (install.wim) which is about 2 GB in size. This partition must be placed after all other partitions so its space can be reclaimed later if need be. The type of this partition is de94bba4-06d1-4d40-a16a-bfd50179d6ac.

Finally, the disk has basic data partitions which are the usual partitions containing OS and data. These partitions have a GUID DEFINE_GUID (PARTITION_BASIC_DATA_GUID, 0xEBD0A0A2L, 0xB9E5, 0x4433, 0x87, 0xC0, 0x68, 0xB6, 0xB7, 0x26, 0x99, 0xC7). The minimum size requirement for the partition containing the OS the 20 GB for 64-bit and 16 GB for 32-bit. The OS partition must be formatted as NTFS. The type of these partitions are ebd0a0a2-b9e5-4433-87c0-68b6b72699c7.

The order of all these partitions is: Windows RE tools, followed by ESP, followed by any OEM partitions, followed by MSR, followed by the data partitions, and finally the Recovery image partition.

It is worth pointing out that when you are installing Windows via an answer file it is possible to create all the above partitions via an answer file. But in my scenario, I am applying a WIM image to a VHD partition manually and creating all the partitions myself so I need a way to do this manually.

Let’s make some partitions!

Now back to my VHDs. To recap, previously I had shown how I apply an OS image from a WIM file to a (base) VHD and then make differencing VHDs off that base VHD for my Hyper-V VMs. The VHD thus created works well for Generation 1 VMs but fails for Generation 2 VMs. As we have learnt from the current post that’s because (a) I was using MBR partitions instead of GPT and (b) I hadn’t created any ESP partitions for the UEFI firmware to pick a boot loader from. Hyper-V Generation 2 VMs have a Class 3 UEFI firmware, so they don’t do any of the CSM/ BIOS compatibility stuff.

As before, create a new VHD and initialize it. Two changes from before are that I am now using a size of 25 GB instead of 20GB and that I initialize the disk as GPT.

Confirm that the disk is visible and note its number:

By default the newly created disk has a 128 MB MSR partition. Since the ESP has to be before this partition let’s remove that.

Then create new partitions:

Just double-checking:

Wunderbar!

Next I apply the image I want as before:

That takes care of the data partition. Let’s look at the other ones now.

WinRE tools partition

This is the first partition on the disk. I will (1) format it as FAT32, (2) mount it to a temporary drive letter, (3) copy the WinRE.WIM file from E:\Windows\System32\Recovery (change E: to whatever letter is assigned to the OS partition), (4) register the Windows RE image with the OS, and (5) dismount it.

Thanks to this TechNet article on how to register the Windows RE image. The WinRE.WIM image can be customized too with drivers and other tools if required but I won’t be doing any of that here.

Thanks to one of my readers (Exotic Hadron) for pointing out that the winre.wim file is only present in %SYSTEMROOT%\System32\Recovery if Windows was installed by expanding install.wim (like in the above case). On a typical system where Windows is installed via the setup program the file won’t be present here.

Just double-checking that Windows RE is registered correctly:

EFI System Partition

This is the second partition on the disk. As before I will format this as FAT32 and mount to a temporary drive letter. (Note: I use a different cmdlet to assign drive letter here, but you can use the previous cmdlet too).

Format the partition as FAT32. The Format-Volume cmdlet doesn’t work here (am guessing it can’t work with “System” partitions) so I use the usual format command instead:

Get the boot loaders over to this drive, confirm they are copied, and remove the drive letter:

Phew!

Recovery Image partition

Last thing, let’s sort out the recovery image partition.

I am going to skip this (even though I made the partition as an example) because there’s no point wasting the space in my use case. All one has to do to sort out the recovery image partition is to mount it like with the WinRE tools partition and copy over the install.wim file to it. Then use the ReAgentc.exe command to register that image with that installation of Windows. (See steps 5-7 of this link).

That’s it!

Now dismount the VHD, make a differencing VHD as before, create a VM with this VHD and fire away!

And voila! It is booting!

boot-success

 

Update: I came across this interesting post by Mark Russinovich on disk signature collisions and how that affects the BCD. Thought I should link it here as it makes for a good read in the context of what I talk about above. 

Notes on Hyper-V differencing disks

I am using Hyper-V again to setup a lab on one of my laptops. Most of the VMs are going to be Server 2012 (GUI or Core) and rather than “waste” space for each VM I decided to create a base VHD and use.

Generation 1 VM only!

The steps below only work for a Generation 1 VM. If you want to use a Generation 2 VM read this post to get an idea of what I am doing, but don’t follow all the steps. Instead, check out my next post where I go into UEFI and GPT and the boot process there, and follow the steps there.

Using the VHD created below in a Generation 1 VM gives the following error when booting: Boot failed. EFI SCSI Device. No Operating System was Loaded. Press a key to retry the boot sequence ...

Here’s how I went about doing it. (None of this is new material, the below is just notes to my future self). Also, all these steps are on my Windows 8.1 with Update laptop, so it might not work on older versions of Windows (I know for sure that these won’t work on Windows 7).

First, mount the Server 2012 ISO:

The ISO is mounted at F: drive. The sources\install.wim file in this drive contains the WIM file with all the Server 2012 images. A WIM file (short for “Windows IMaging” file) is a file-based image. Unlike a block-based image (which is what most of us are familiar with from tools such as Ghost) a file-based image contains the files and file-system as it. In addition to that the files are compressed, and duplicate file names point to one actual file, so you could have a WIM file that contains all the editions of Server 2012 but the size of the WIM file isn’t equal to the size of one of these multiplied by the number of editions. Since Vista WIM files are what Microsoft’s been using to install the OS. The install program sets up your hard disk and then dumps a WIM file onto it. After a reboot, the machine boots into this dumped image and continues configuring and customizing.

The beauty of this is that one can dump a specified image from a WIM file onto a virtual hard disk too – which is what I am going to do here. I select an image from my WIM file, create a VHD file, and apply this image to the VHD file. Then when I boot up from this VHD file in a newly created VM the setup process continues as it is. The difference here is that I am going to apply to the image to a VHD file, and then use that VHD file as a base image and create new “differencing” VHD files off that. This way each newly created VM uses the differencing VHD file, and the size of that is more or less equal to the changes I make in the VM. That’s way better than having multiple VHD files all of which contain the same OS and similar files and together take up a lot of space!

Back to the WIM file, find the image that we are interested in:

I am interested in the first image here (Serer 2012 R2 Standard Core). I want to apply this image to a VHD. So I create a 20GB VHDX file:

Mount the VHDX file:

Note that I use Get-Disk above. That’s because the VHDX doesn’t have a file system yet, and hence no volume. I will have to prepare the VHDX before I can apply the previous WIM image onto it. So let’s do that.

Create a partition and format it as NTFS. Assign a drive label while we are at it.

NOTE: It is possible to combine all the cmdlets above from New-VHD to Format-Volume into one long pipe. That way easier to copy paste than the multiple cmdlets above. Here’s the combo version:

Now I apply the WIM image to this drive letter (which is actually the VHDX file I created earlier):

This cmdlet will take a while to complete (and it doesn’t offer much by way of a progress bar)

NOTE: You can use the DISM command too instead of the cmdlet above. That has a more informative progress bar:

Install a boot loader on the VHD so it boots.

Don’t forget the boot loader!

I had missed this step when I first wrote this post. Apologies if anyone followed the steps and ended up with non-bootable VHD. I realized this omission only after I tried booting the VHD and got the following error:

bcderror

Finally dismount the VHDX:

Dismount the ISO file too if you are done with it:

Finally, I make a new differencing VHD:

That’s it. Now I can create a VM and assign the WINDC01.vhdx disk to it. As far as the VM (or even us if we mount the VHDX file directly) are concerned the WINDC01.vhdx file is identical to the 2012R2.vhdx file it is based up. Just that what the file actually contains is only the differences from the base file. Any references to files in the base VHD file are looked up there transparently; any references to new/ changed files are looked up in the differencing VHD file.

Generation 1 VM only!

As mentioned at the beginning of this post, the VHDX file created using the above steps only works with a Generation 1 VM. With a Generation 2 VM you get an error like this:

hyper-v-error

This is because Generation 2 VMs are using UEFI and they have a different boot process. Check my follow-up post on what to do with a Generation 2 VM. .