Notes on Volume Shadow Copy in Windows (or How to backup open PST files via Robocopy)

Since XP and Server 2003 Windows has had this cool feature called Volume Shadow Copy (also known as Shadow Copy or Volume Snapshot Service (VSS)). It’s a cool feature in that it lets you take read-only snapshots of your file-system so you can then trawl through it or take backups and such. When Windows create system restore points or does backups this is what it uses. Without VSS Windows wouldn’t be able to backup files that are in use by the system/ applications as these files are locked; but with VSS it can take a “snapshot” of everything as they are at that point in time and the backup program or system restore can access the files in this snapshot. 

A good overview of the Volume Shadow Copy service can be found in these two (1 & 2) TechNet articles. What follows is a tl;dr version but I suggest you read the original articles. 

  • Volume Shadow Copy consists of 4 components:
    1. the VSS requester, which is the software that actually requests for a shadow copy to be made – for e.g. Windows Backup;
    2. the VSS provider, which is a component that actually creates and maintains shadow copies – for e.g. Windows includes a VSS provider that allows for Copy-on-Write (COW) snapshots, SAN solutions usually have their own VSS providers;
      1. there are three types of providers – hardware providers, software providers, and the system provider which is a part of Windows (see the section on Shadow Copy Providers in this link)
    3. the VSS writer, which is a component of Windows or installed software whose role is to ensure that any data belonging to Windows (e.g. Registry) or such software (e.g. Active Directory, SQL or Exchange databases) is in a consistent state for the shadow copy to be made; 
      1. Windows includes many writers (see the section on In-Box VSS writers in this link)
      2. Databases such as Active Directory and Exchange use transaction logs. Which is to say changes are not written to the database directly, rather they are written to memory first and a note made in a special file (the “transaction log”). During periods of non-peak activity changes in the transaction log are committed to the actual database. This way even if the database were to crash during a transaction, when it comes back up the transaction log can be “replayed” and any uncommitted transactions can be added to the database. Here are some links which explain this concept for Active Directory (1 & 2) and Exchange (1).
      3. What the VSS writer component of Active Directory or Exchange does is that when a snapshot is taken of the database it will be in a consistent state wherein any pending transactions are written to it or are waiting to be written, never in a state where the transactions are being committed to the database.  
    4. the VSS service, which is a coordinator for all the components above. 
  • Here’s how it all falls into place:
    1. The requester tells the service that it wants a shadow copy and so it needs a list of the writers on the system with their metadata. 
    2. The service asks all the writers to get back with the information. Each writer creates an XML description of its components and data stores. Each writer also defines a restore method. 
    3. The service passes the above information to the requester. The requester selects the components it wants to shadow copy.
    4. The service now notifies the writers of the selected components to prepare their data (complete any open transactions, flush caches, roll transaction logs, etc). 
    5. Each writer does so and notifies the service. 
    6. The service tells each writer to temporarily freeze all application I/O write requests (read I/O requests are still allowed).
      1. This freeze is not allowed to take longer than 60 seconds. 
      2. The service also flushes the file system cache.
    7. The service then notifies the provider. The provider makes a shadow copy – this is not allowed to last for longer than 10 seconds, during which all I/O requests to that file system are frozen.
    8. Once copy is done, the service releases the pending I/O requests. 
    9. The service also notifies the writers that they are free to resume writing data as before. 
    10. The service informs the requester on the location of the shadow copy. 
  • There are three methods a provider can use to shadow copy:
    1. Complete copy – makes a read-only clone of the original volume – this is only done by hardware providers
    2. Redirect on Write – leaves the original volume untouched, but creates a new volume and any changes are now redirected to this new volume
    3. Copy on Write (COW) – does not make a copy of the original volume, but any changes made after the shadow copy are copied to a “differences area” and when the shadow copy needs to be accessed it can be logically constructed based on the original volume and the changes in the “differences area” – this can be done by both software and hardware providers.
  • Paging and other temporary files are automatically excluded from snapshots. The FilesNotToSnapshot registry key can be used to exclude additional files from a shadow copy (it is meant to be used by applications, not end users).
  • For the system provider the shadow copy storage area (also called the “differences area” above) must be an NTFS volume. The volume to be shadow copied need not be an NTFS volume. 
    1. The “differences area” need not necessarily be on the same volume as the one being shadow copied. If the volume already has a “differences area”, that is used. Otherwise a “differences area” can be manually associated with a volume. Else a volume is automatically selected based on available free space (with preference being given to a space on the volume that’s being shadow copied). 
    2. Although the volume being shadow copied can be non-NTFS, if you are creating persistent shadow copies then it must be NTFS. (Persistent shadow copies exist even after the shadow copy operation is done. Non-persistent shadow copies exist only for the duration of the operation – such as a backup – and are deleted afterwards). 
  • Maximum number of shadow copied volumes is 64. Maximum number of shadow copies created by the system provider for a volume is 512. 
  • There are three tools (VSS requesters) you can use to manually make and manage shadow copies. 
    1. DiskShadow which is present on Windows Server 2008/ Windows Vista and upwards (only on the server versions I think, as I couldn’t find it on my Windows 7 or Windows 8 install)
    2. VssAdmin which is present on Windows Server 2003/ Windows XP and upwards (the client versions are different from the server versions I think, the version on my Windows 7 and Windows 8 machines didn’t have an option to create shadows but only list and delete shadows)
    3. VShadow which is a part of an SDK package (see this thread for more info) and is better than VssAdmin. Good thing about VShadow is that someone has kindly extracted it from the SDK and made available for download. So if you are on a client version of Windows that’s probably what you will go with. 
  • The FAQ section on this link has more bits and pieces of use (no point in me regurgitating it here! :)). 

Practical Example

I have an USB stick with PST files and such. I can’t take a backup of this USB while Outlook is running as the PST files are locked by it. So what I need to do is use VSS to manually make a shadow copy, then expose said shadow copy as a drive/ path, and use my regular backup tool (robocopy in this case) to backup the files in this shadow copy. Easy peasy? It is!

First, I downloaded the VShadow tools.

To create a shadow here’s the command to use:

 This, however, creates a non-persistent shadow copy. I want a persistent shadow copy though as I want to access it even after VShadow exits. So I add a switch to create persistent copies. 

 Here’s a snippet of the output:

Notice how the VSS service determines which component are to be excluded from the shadow and accordingly excludes those Writers? Then it contacts them (not shown) and proceeds to create a shadow. The end result of this operation will be output similar to this:

A shadow copy is a snapshot of a volume at one point of time. Each shadow copy has a GUID. A shadow copy set is a collection of shadow copies of various volumes all taken at the point of time (so it’s a collection of shadow copies of multiple volumes taken at the same point of time). Each shadow copy set has a persistent GUID. These details are shown in the output above, along with the original volume name and others. The shadow copy gets a device name of its which you can mount via mklink (be sure to add a trailing slash to the device name, i.e. \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy16 becomes \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy16\):

or

Former creates a directory symbolic link, latter creates a junction. Doesn’t make a difference in this case. The directory must not exist beforehand

You can mount persistent shadow copies via their GUID too as below:

This mounts the shadow copy of GUID {dbc75a3a-0a6c-4540-8ba2-c2d3665b3cc1} at the specified drive letter. You don’t specifically need a drive letter. Any empty folder can be used too as a mount point. Even better, you can expose a shadow copy as a shared folder. The command below will expose the previous shadow copy as a shared folder of the specified name. 

It’s also possible to expose only a sub-folder of the shadow copy as the shared folder. 

Wouldn’t it be cool though if there was a way for the shadow creating command to pass on the GUID and other details so the exposing/ mounting command can easily do that? That would make it very easy in batch files and such, and indeed VShadow has such an option. The command below, which is a modification of the original command, will create a new CMD file which when run populates a set of environment variables that contain the shadow GUID and such. 

 The contents of the vss-setvar.cmd file will be similar to this (this output is from a different shadow copy hence the GUIDs vary from the output above):

As you can see all it does is set some environment variables containing the snapshot ID and snapshot set ID. What this means is that you could have a batch file with the following commands:

Nice, isn’t it? On the second line when you call vss-setvar.cmd it will set the environment variables for you and then you can use the other commands to expose the shadow copy and finally delete it once its backed up. 

It’s also possible to run a command as part of the shadow copy process. As in, this command will run when the shadow copy is created but before Vshadow exists. This is useful when you are dealing with non-persistent shadow copies as the copy will still be present when the command runs. You can run any command or a CMD file, but no parameters can be passed. Here’s how (use the -exec switch):

 For instance, here’s what the vss-robocopy.cmd could contain:

Now the robocopy backup happens as soon as the shadow copy finishes but before Vshadow exists. This means I don’t have to use persistent shadow copies like before because when the backup happens the shadow copy is temporarily exposed any way. I find this technique better – feels more neat to me. 

While reading for this post I came across a page with more Vshadow examples. Also, here’s someone using PowerShell to mount all shadow copies – good stuff. maybe it will be of use to me later!

Lastly, if you are trying to make a persistent shadow copy and get the following error:

 It’s quite likely that the volume you are snapshotting isn’t NTFS formatted. Remember, persistent copies require the volume to be NTFS