Came across this PDF from VMware while reading on memory management. It’s dated, but a good read. Below are some notes I took while reading it. Wanted to link to the PDF and also put these somewhere; hence this post.
Some terminology:
- Host physical memory <–[mapped to]– Guest physical memory (continuous virtual address space presented by Hypervisor to Guest OS) <–[mapped to]– Guest virtual memory (continuous virtual address space presented by Guest OS to its applications).
- Guest virtual -> Guest physical mapping is in Guest OS page tables
- Guest physical -> Host physical mapping is in pmap data structure
- There’s also a shadow page table that the Hypervisor maintains for Guest virtual -> Guest physical
- A VM does Guest virtual -> Guest physical mapping via hardware Translation Lookup Buffers (TLBs). The hypervisor intercepts calls to these; and uses these to keep its shadow page tables up to date.
- Guest physical memory -> Guest swap device (disk) == Guest level paging.
- Guest physical memory -> Host swap device (disk) == Hypervisor swapping.
Some interesting bits on the process:
- Applications use OS provided interfaces to allocate & de-allocate memory.
- OSes have different implementations on how memory is classified as free or allocated. For example: two lists.
- A VM has no pre-allocated physical memory.
- Hypervisor maintains its own data structures for free and allocated memory for a VM.
- Allocating memory for a VM is easy. When the VM Guest OS makes a request to a certain location, it will generate a page fault. The hypervisor can capture that and allocate memory.
- De-allocation is tricky because there’s no way for the hypervisor to know the memory is not in use. These lists are internal to the OS. So there’s no straight-forward way to take back memory from a VM.
- The host physical memory assigned to a VM doesn’t keep growing indefinitely though as the guest OS will free and allocate within the range assigned to it, so it will stick within what it has. And side by side the hypervisor tries to take back memory anyways.
- Only when the VM tries to access memory that is not actually mapped to host physical memory does a page fault happen. The hypervisor will intercept that and allocate memory.
- For de-allocation, the hypervisor adds the VM assigned memory to a free list. Actual data in the physical memory may not be modified. Only when that physical memory is subsequently allocated to some other VM does it get zeroed out.
- Ballooning is one way of reclaiming memory from the VM. This is a driver loaded in the Guest OS.
- Hypervisor tells ballooning driver how much memory it needs back.
- Driver will pin those memory pages using Guest OS APIs (so the Guest OS thinks those pages are in use and should not assign to anyone else).
- Driver will inform Hypervisor it has done this. And Hypervisor will remove the physical backing of those pages from physical memory and assign it to other VMs.
- Basically the balloon driver inflates the VM’s memory usage, giving it the impression a lot of memory is in use. Hence the term “balloon”.
- Another way is Hypervisor swapping. In this the Hypervisor swaps to physical disk some of the physical memory it has assigned to the VM. So what the VM thinks is physical memory is actually on disk. This is basically swapping – just that it’s done by Hypervisor, instead of Guest OS.
- This is not at all preferred coz it’s obviously going to affect VM performance.
- Moreover, the Guest OS too could swap the same memory pages to its disk if it is under memory pressure. Hence double paging.
- Ballooning is slow. Hypervisor swapping is fast. Ballooning is preferred though; Hypervisor swapping is only used when under lots of pressure.
- Host (Hypervisor) has 4 memory states (view this via
esxtop, pressm).- High == All Good
- Soft == Start ballooning. (Starts before the soft state is actually reached).
- Hard == Hypervisor swapping too.
- Low == Hypervisor swapping + block VMs that use more memory than their target allocations.