I recently migrated to Server 2016 with the Essentials role. The server runs as a Hyper-V virtual machine. Windows Backup writes to a drive hosted on a VHDX file. Almost from the beginning, I had problems with Windows Backup failing while trying to back up System State.
Here is the Event Log error message with a perhaps-relevant bit of the EventData:
Log Name: Microsoft-Windows-Backup
Source: Microsoft-Windows-Backup
Date: 8/18/2017 1:06:24 PM
Event ID: 5
Level: Error
User: SYSTEM
Description:
The backup operation that started at ‘2017-08-18T20:00:32.276165000Z’ has failed with following error code ‘0x80780034’ (Windows Backup failed to create the shadow copy on the storage location.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
<EventData>
<Data Name=”SystemStateBackup”><SystemState IsPresent=”1″ HResult=”-2139619299″ DetailedHResult=”-2147212522″ /></Data>
</EventData>
</Event>
And here is the failure message in Windows Backup:
The full message is “The operation ended before completion. Detailed error: Another shadow copy creation is already in progress. Wait a few moments and try again.”
Tried and Failed
Fail 1
The first thing I did was convert the Hyper-V backup volume from dynamically expanding to fixed. I thought maybe Windows was having trouble creating the shadow copy in time. Didn’t help.
Fail 2
This article suggests setting the maximum shadow copy storage on the target volume to Unlimited. I tried that. (Normally I use shadow storage limits to keep free space trimmed as I described at the bottom of this post.)
Fail 3
This Experts Exchange post suggested that one must fix corrupt service definitions. Seemed like a long shot but I ran a modified version of the PowerShell script and did in fact find an invalid path:
Another way to find service entries with bad paths, as described here, is to use System Information (msinfo32.exe). Go to Software Environment > Services and sort the grid by Path. Look for malformed paths. However, note that the service in question is set to Manual start, so I was beginning to doubt that this was actually causing my issue:
That service corresponds to this registry entry. The Data path should begin with C:\windows\, which I added.
I ran the script again to confirm that the service entries were clean. Didn’t help my System State backup issue.
The Solution: CrashPlan Exclusions
A while later, I was researching another error, this one in the Application event log:
Log Name: Application
Source: VSS
Date: 8/18/2017 7:01:26 AM
Event ID: 12294
Level: Error
Description:
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine returned E_INVALIDARG. Routine details GetSnapshot({00000000-0000-0000-0000-000000000000},000002C68381AC30).Operation:
Get Shadow Copy PropertiesContext:
Execution Context: Coordinator
These seemed to happen at random times, but always in batches of four almost identical errors. I realized that the time of this error corresponded to the time that CrashPlan tried to run a backup (which it does every 15 minutes). I found these messages in the C:\ProgramData\CrashPlan\log\backup_files.log.0 file:
I 08/17/17 07:16PM 41 [MCB01 Backup] Completed backup to CrashPlan PRO Online in 18 minutes: 15 files (42.40MB) backed up, 56.10KB encrypted and sent @ 1.1Kbps (Effective rate: 7.5Mbps) [21,3968,299680,3952,9,178,46]
I 08/17/17 07:16PM 41 - Unable to backup 7 files (next attempt within 15 minutes)
W 08/17/17 07:16PM 41 - C:\Users\All Users\Microsoft\Search\Data\Applications\Windows\edb.jtx {\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18240\Users\All Users\Microsoft\Search\Data\Applications\Windows\edb.jtx}
W 08/17/17 07:16PM 41 - C:\Users\All Users\Microsoft\Search\Data\Applications\Windows\edbtmp.jtx {\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18236\Users\All Users\Microsoft\Search\Data\Applications\Windows\edbtmp.jtx}
W 08/17/17 07:16PM 41 - C:\Users\All Users\Microsoft\Search\Data\Applications\Windows\Windows.jfm {\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18244\Users\All Users\Microsoft\Search\Data\Applications\Windows\Windows.jfm}
W 08/17/17 07:16PM 41 - C:\Users\All Users\Microsoft\Windows Defender\IMpService77BDAF73-B396-481F-9042-AD358843EC24.lock
W 08/17/17 07:16PM 41 - C:\Users\All Users\Microsoft\Windows Defender\Scans\History\CacheManager\B6899568-3B78-4C04-ADD9-3B9ECC38318B-1.bin
W 08/17/17 07:16PM 41 - C:\Users\All Users\Microsoft\Windows Defender\Scans\MpDiag.bin
W 08/17/17 07:16PM 41 - C:\Users\MyUser\AppData\Local\Packages\Microsoft.Windows.Cortana_cw5n1h2txyewy\Settings\settings.dat.LOG1
I didn’t understand why CrashPlan was backing up Windows Search and Windows Defender data. I wondered whether its repeated failures to get VSS snapshots were what was blocking the native Windows Backup from completing the snapshot needed for backing up System State.
CrashPlan support suggested deselecting these two folders so CrashPlan would not back them up:
C:\Users\All Users\Microsoft\
C:\Users\MyUser\AppData\
I instead chose to deselect only these two folders:
C:\Users\All Users\Microsoft\Search
C:\Users\All Users\Microsoft\WindowsDefender
The failure on Cortana in my user profile should be infrequent as I am not usually logged on to the server.
Excluding those folders from CrashPlan had two effects:
- CrashPlan backups now complete to 100%, with no VSS 12294 errors in the Application event log.
- Windows Server Backup is now able to back up System State every time.
This has been working for a full week now, so I’m hopeful that the issues with both backup programs are fixed.