I’m running Hyper-V on a Windows Server 2022 host. I recently discovered that my largest virtual machine was in “differencing” mode, i.e. only writing changes to checkpoint AVHDX files, also known as snapshots. Each day when Synology Active Backup ran, one snapshot stopped and the next began.
That left me with 15 snapshots of two VHDX files, for a total of 30 snapshots comprising 493GB:
Unfortunately, the Hyper-V Manager did not show any checkpoints, so I was unable to trigger a merge from there:
However I was able to confirm using the Inspect Disk function that each file had a parent file. It became clear that I would need to merge each file into its parent up to an including the original VHDX file:
I will spare you the long saga and repeated failures getting these files merged and try to present a summary for “next time.” (hopefully never!).
First, Two Backups
Before you do anything, back up your perfectly running VM using Active Backup and/or another program. Then shut down the VM and copy the entire VM folder (including Virtual Hard Disks and Virtual Machine subfolder) to another location. I copied across the network to another server; it took about 2.5 hours to copy 860GB.
If you have to start over (as I did a couple times), you’ll have to copy the files back, then use icacls to re-add the virtual machine’s ID to the files. I blogged this here and Microsoft has an article here. Note that icacls does allow wildcards, so I was able to use these two commands (substitute your own server path and VM ID):
icacls "D:\Hyper-V\SRV01\Virtual Machines\*.*" /grant "NT VIRTUAL MACHINE\7E77503A-A26B-4BB5-9846-396F49A30141":(F) icacls "D:\Hyper-V\SRV01\Virtual Hard Disks\*.*" /grant "NT VIRTUAL MACHINE\7E77503A-A26B-4BB5-9846-396F49A30141":(F)
Automatic Merge
Several online articles suggest that snapshots should merge automatically when you shut down the virtual machine. That didn’t happen. In fact, I’m pretty sure I rebooted and it didn’t happen.
What did work in the end was completely shutting down the VM, then starting it from the Hyper-V Manager. When I started the machine, it immediately showed “Merge in progress”:
To my surprise, I didn’t even have to wait until the merge completed before I was able to log in to the machine. Well maybe drive C had to finish, but It kept merging drive D after I logged in.
Manual Merge
If you have to merge manually, you’ll find lots of suggestions out there.
- You can in fact merge one-by-one from the Hyper-V Manager using Edit Disk, but I didn’t trust myself to keep the backup chains straight.
- The Merge-VMDisk command in the AZSBTools package looked promising and I started down that path. Unfortunately, it does not identify the entire chain to be merged; it only merges one snapshot at a time. I ran it repeated until I realized I might do better with the Microsoft option (next). But Merge-VMDisk was somehow updating the main virtual machine with the identifier of the latest disk, so after completing the merge with the the other script, I was left with a running VM that was missing a week’s worth of data—it thought it had been improperly shut down a week earlier.
- It turns out that Microsoft has perhaps the most usable instructions on dealing with this situation. See Merging checkpoints when you can’t view them in Hyper-V Manager. I did a little tweaking of their scripts to run them from .ps1 files. The Get-VHDChain function spits out a nice list of files (just make sure the count of AVHDX files matches what you see in Windows Explorer):
After that, the second script can create an ordered list of Merge-VHD commands that you can run to merge the disks.
If I had this to do over and the automatic merge wasn’t working, I’d start with the Microsoft scripts.
Possible Cause
The way this is supposed to work is that the Synology application-aware Active Backup triggers a snapshot, makes its application-consistent backup, then releases the snapshot to be merged back into the main VHDXs. It’s unclear what caused this perpetual snapshot mode in the first place. My hunch is after I first set up this machine and migrated all my data, the delta of the new files was so large that one backup had not completed merging the AVHDXs before the next backup (possibly the Veeam backup inside the VM) began. This thought comes from the Unitrends backup program, which has a setting to guard against this behavior.
If You’re in a Hurry
If this is a production server that people are waiting on and merging isn’t going well, it might be faster to just to a bare metal restore of the Synology Active Backup (the first backup taken above) to a new location. I actually did this while I was trying other merge steps, restoring to a secondary server, and it seems to have worked fine. I kept that restored machine off the network and was ultimately able to complete the merge, but this would have been my fallback if merging didn’t work.
A Final Tip
There are no doubt exceptions, but under normal circumstances, a production Hyper-V server shouldn’t have any AVHDX files except briefly during a backup. If you have a Remote Monitoring and Management (RMM) service, consider adding a daily check to your Hyper-V servers to to let you know if any AVHDX files are still around at, say, 6:00am.