Environment Details

The server is a Dell PowerEdge 1900 with 4GB of RAM and two 250GB hard drives configured as RAID 1 (mirrored). As mentioned, it runs Windows Server 2008 R2 with the Hyper-V role, and it hosts a single Server 2008 R2 guest. The guest runs as a domain controller, file server, and email server. Windows Server Backup runs daily on the host, taking an image of the C: drive (containing the host operating system, 16GB) and the D: drive (containing the guest virtual machine, about 48GB used on 80GB of fixed VHD space). Backups are stored on a Seagate Freeagent Go USB drive. Windows Server Backup also runs daily on the guest, saving its backups to a VHD on the same Freeagent USB drive.

The desktop is an Optiplex 960 with 4GB of RAM and one 300GB hard drive. Most important is the processor, an Intel Core 2 E8400 that does support Intel Virtualization Technology and Execute Disable, so it should run Hyper-V.

Recovery Options

There are at least three options for recovering this guest server (which is of course the critical machine):

1. Restore the host and guest from the host’s backup. This makes use of (and tests) Windows Server Backup’s ability to restore to dissimilar hardware, as reported here.

2. Install Server 2008 R2 and the Hyper-V role in a small partition and apply all operating system updates. Restore the guest VHD in a larger partition. Re-create the Hyper-V guest machine using the VHD. This avoids the dissimilar hardware restore at least for the host.

3. Restore the guest backup directly. This would be a virtual-to-physical restore, so also dissimilar hardware.

Option 1 should be the fastest and the most flexible, since it would also accommodate the scenario where multiple guests needed to be restored.

Recovery Experience

The recovery—and this post—became quite long. You may want to skip to the “Conclusions” section, then come back to the step-by-step logs for details as necessary

Time Log – Day 1

Here my log of the recovery process with notes for next time.

3:20pm  Insert Server 2008 R2 DVD into Optiplex 960 and attach USB drive containing backup data. During reboot, check BIOS to make sure Virtualization Technology and Execute-Disable are turned on. Boot from DVD.

3:22pm DVD boot is fast! Glad I didn’t waste time making a bootable thumb drive. Choose Repair your computer, then Restore your computer…. Can’t see external USB drive as backup source. Start googling.

3:35pm Oh yeah, the whole backup drive is Bitlocker-encrypted. Apparently Repair doesn’t detect this as there is no prompt to unlock. Press Shift-F10 to get a command prompt. Commands:
manage-bde –status to show drive (it’s mounted as G:)
manage-bde –unlock –h for parameter help
manage-bde –unlock G: –RecoveryPassword 111111-222222-…-8888888 to unlock using recovery key

3:56pm Drive unlocked. Close command prompt. Press Refresh in Repair wizard. Backup drive found. Choose most recent backup, accept defaults (do check Format and repartition disks, do not check Only restore system drives). Do not install drivers or change Advanced options. Start restore. “This might take from a few minutes to a few hours.”

4:06pm Drive C: restore complete.

4:28pm Drive D: restore about half done.

4:48pm Drive D: restore complete.

4:51pm Drive S: restore complete. “The restored drives are not encrypted.” Remove DVD, unplug USB hard drive, Restart Now.

4:53pm Stuck on boot. “Intel® Matrix Storage Manager” text on screen. I suspect boot drive number is off. Reboot from DVD.

4:58pm Choose Repair your computer, then Use recovery tools…. and Command Prompt. Start fiddling with diskpart:

diskpart
list disk
select disk 0
list volume – letters are all messed up
select volume 1
remove letter=C System Reserved should not have a letter
select volume 2
remove letter=F This is the boot volume
assign letter=C
active identify as the boot volume
select volume 3 this is already assigned drive D, so skip
select volume 4
remove letter=E
assign letter=S
list volumes confirm correct lettering
exit

5:16pm Can’t find a boot.ini file. A little googling confirms that has been replaced in Server 2008 and above with the Boot Manager Configuration file. Let’s just reboot and see what happens.

5:20pm This time we got far enough to see “BOOTMGR is missing. Press Ctrl+Alt+Del to restart”. Reboot from DVD.

5:24pm Operating system is not listed for repair, so follow Method 2 of accepted solution in this thread. From a command prompt, type
bootrec /rebuildbcd
and let it add C:\Windows to the boot list. Remove DVD and restart.

5:27pm Same message:  “BOOTMGR is missing.” Boot from DVD.

This time, operating system is listed, so follow Method 1 at the previous link. Select the operating system. Unfortunately there is no Startup Repair option (apparently that is only available on Windows Vista/7 media). Try a couple other bootrec options:
bootrec /fixmbr
bootrec /fixboot
and reboot.

5:40pm Same message. Giving up for today. Will contact Microsoft Partner Support for suggestions. In a real disaster recovery situation, I would be on the phone with support and/or moving to Recovery Option 2 above. In fact, since the guest’s VHD is already restored, I could probably just install a clean Server 2008 R2 with Hyper-V on top of the botched C drive and use that to load the guest. But I’d like to understand why Recovery Option 1 hasn’t worked and get it working if possible.

Time Log – Day 2

1:10pm MS Support suggests copying the bootmgr file from \Windows\Boot\PCAT\ on the DVD to the root of C:. That location does not exist on the DVD, but it does exist on the restored OS, so I copied that to the root of C:. Voila! Windows boots and immediately reboots, telling me it failed to start. Well at least it’s finding the BOOTMGR.

1:23pm Windows won’t start in Safe Mode either. Don’t even see a blue screen flicker by; just gets partway through the boot and restarts. (It finishes loading CLASSPNP.SYS but fails on whatever comes after that.) Start googling for how to disable Recovery Options > Automatically Restart when you can’t get into the OS.

1:55pm No joy finding out how to change boot behavior. Just for grins, try copying BOOTMGR from the root of the DVD drive to the root of C. Same problem:  won’t start in Safe Mode, just restarts without blue screen.

2:10pm Grasping at straws:  trying copying bootmgr.efi from the root of the DVD to the root of C. Same problem persists.

2:25pm Interesting thread here. Though not directly related, I see I didn’t try one of the recommend commands earlier. Try this while booted from the DVD:
bootrec /rebuildbcd
bootrec /fixmbr
bootrec /fixboot
G: switch to DVD drive
\boot\bootsect.exe /nt60 C: – this is new
After reboot, same issue.

2:40pm Follow an idea near the bottom of this thread and change Optiplex 960 BIOS to put SATA Operation in Legacy mode (was in RAID On mode). Still gets through CLASSPNP.SYS, but does not immediately reboot; it just hangs, won’t even accept Ctrl-Alt-Del. That’s different, so the SATA Operation mode may be the issue! Try the other two SATA modes.  RAID Autodetect / AHCI behaves like RAID On:  a reboot loop. RAID Autodetect / ATA … worked! It hung for a while after CLASSPNP.SYS, but then it gave me a Safe Mode logon screen.

2:53pm After logging on, I see popup telling me “You must restart to apply these changes.” First I went in to System properties > Advanced system settings > Startup and Recovery and unchecked “Automatically restart”. That should stop the reboot loop, should it recur.

2:57pm It rebooted quickly into full operating system and logged on.

Of course the NIC has changed. In Hyper-V manager, run Virtual Network Manager. Change the “Local Area Connection – Virtual Network” to use the External adapter in the Optiplex 960 (Intel 82567LM-3 Gigabit Network Connection).

3:15pm Errors in Optiplex’s Server event log confirm that this machine’s name conflicts with the live server. Temporarily shut down the live server.  Also disable port 25 on router to block incoming email (to keep email from going to the test machine when we start it.) Reboot the Optiplex.

3:18pm Start test guest server. “The virtual machine could not be started because the hypervisor is not running.” This corresponds to Hyper-V-Worker error 3112. The third suggestion may apply:  “If you have made changes to the Boot Configuration Data store, review these changes to ensure that the hypervisor is configured to launch automatically.” We have been mucking around with the boot manager. Hmm, I wonder if I had stayed with the version in the C:\Windows\Boot\PCAT folder instead of copying from the DVD… In fact, maybe I would not have had to rebuild the BCD if I’d just set the SATA mode first.

3:30pm Reboot to DVD. Delete C:\bootmgr, C:\bootmgr.efi, and C:\bootsqm.dat. Reboot. “BOOTMGR is missing.” Oh well. Copy bootmgr from C:\Windows\boot\PCAT to C:\. Windows boots fine. (I realized later that the bootmgr file was not the issue, it was the boot configuration, not stored in the bootmgr file.)

3:41pm I was thinking I would need to re-install the Hyper-V role to get the hypervisor running again, but then I came across this article that shows how to fix it with one command:
bcdedit /set hypervisorlaunchtype auto

3:52pm After rebooting, try starting the guest OS again. Same problem:  the hypervisor is not running. Now in the event log there is Hyper-V-Hypervisor error 41, “Hyper-V launch failed; Either VMX not present or not enabled in BIOS.” Huh? I know I enabled the virtualization stuff in the BIOS.

4:06pm This article suggests turning the BIOS options off and back on. And the first comment in this article says Trusted Execution should NOT be checked in the BIOS, which I agree is counter-intuitive. Followed both suggestions, with full power-off, then started Windows.

4:11pm No error 41 in the event log. Started guest. Failed to start because external drive is not attached to Optiplex. (The guest targets a VHD on that drive for backups.) Use Hyper-V manager to update the settings for the guest, removing that drive from the configuration.

4:13pm Start guest again. It tells me it had not shut down successfully (it wasn’t shut down at all; it was backed up from a VSS snapshot). Let it Start Windows Normally.

4:16pm Guest logon prompt. (Took a long time Applying Computer Settings.) After logon, “You must restart your computer to apply these changes.” Clicked Restart Now.

4:20pm Guest logon prompt. Logged on. Event log looks fairly normal (typical warnings from a startup). Since the guest server has a fixed IP, it automatically is online at the same address as the live server, ready to accept communications from client computers and the Internet. Tested a few components:

Web server works
Active directory works, including accepting logon from client, group policy, folder redirection
Shared drives are available and allow opening QuickBooks data on test server
Email server can retrieve POP mail and send mail

Theoretically at this point I should do a System State restore from the guest’s backup (see my article Careful with Image-Based Backup of Exchange and Active Directory). But I think we’ll call this good for now.

4:33pm Shut down test server, restart live server host and guest, and re-open port 25 on router to allow inbound email.

And a little bonus disaster recovery…

I was anticipating a quick restore of the Optiplex from the Windows Home Server backup. Here’s what happened.

4:53pm In Optiplex BIOS, set SATA Operation back to Raid On. Boot Optiplex from Windows Home Server PC Restore CD. Network driver was missing, but WHS provides an easy way to retrieve the required drivers (by logging on to the WHS console from another computer and copying a the “Windows Home Server Drivers for Restore” folder directly from the WHS backup). Unfortunately that’s the Windows 7 driver, and WHS restore needs the Vista driver.

5:43pm Vista NIC driver downloaded from Dell support site and supplied via USB stick. WHS restore started.

5:57pm Restore complete.

5:59pm Optiplex booted back to Windows 7.

Conclusions and Notes for Next Time

Obviously things did not go as smoothly as hoped. Two hours and 20 minutes on the first day plus three hours and 10 minutes on the second day to get a final logon prompt for the Hyper-V guest. Five and a half hours of solid futzing.

The good news is that it did work; in a real disaster, the client would be back in business.

The other good news is that this was a test, which re-affirms the importance of testing your disaster recovery plan. Next time, I’ll know to check a few key things during the restore process:

• In the Optiplex BIOS, set SATA Operation mode to to RAID Autodetect / ATA before starting the restore.
• In the Optiplex BIOS, set Virtualization, VT for Direct I/O Access, and Execute Disable to ON, but set Trusted Execution to OFF.
• Unlock Bitllocker-protected backup drives using manage-bde commands above.
• If just a blank screen appears after restore, use diskpart to assign correct drive letters and set the C: drive as Active.
• If BOOTMGR is missing, copy bootmgr from C:\Windows\boot\PCAT to C:\.
• Once the host starts, update the Hyper-V Virtual Network Manager to point to the new host NIC.
• If the guest references unavailable and unneeded hardware (e.g. an external backup drive), remove it from the Hyper-V configuration before starting the guest.
• If the guest won’t start because the hypervisor is not running, set it to auto-start using bcdedit command above.

Maybe I’ll try this again someday and see how fast it goes with all that in mind.

I decided to try the command line. Here I found that manage-bde –status listed two volumes out of about eight before aborting with the same error:

When I saw that error code 0×80070017 can mean that a file is missing or corrupt, it occurred to me that the old BitLocker auto-unlock keys are still on the restored system volume, but they cannot be accessed. So I tried manage-bde -autounlock -clearallkeys C:.

After this, manage-bde -status correctly lists all volumes, but manage-bde -autounlock -enable S: fails with the message:

ERROR: An error occurred (code 0×80310054):
The auto-unlock master key was not available from the operating system drive.

Maybe the master key will be re-created by a reboot? Sure enough, after rebooting and manually unlocking the drives again, I was finally able to enable auto-unlock on the encrypted data volumes.

System Volume Restore with BitLocker Data Volumes

Based on that experience, here is what I think should work next time for system volume restore:

1. Do bare metal restore of system volume C:. The volume is restored unencrypted.
2. Boot into OS and log in.
3. From an administrative command prompt, run manage-bde -autounlock -clearallkeys C:.
4. In the Control Panel, go to BitLocker Drive Encryption and enable Bitlocker on C:. Leave the encrypted data volumes in their locked state for now.
5. Reboot to activate BitLocker. This apparently creates the auto-unlock master key on the system volume.
6. Log in. In the Control Panel, go to BitLocker Drive Encryption and manually unlock encrypted data volumes.
7. For each encrypted data volume, click on Manage BitLocker and set Automatically unlock this drive on this computer.

Update 02/17/2011:  You may want to refer to the Manage-bde Parameter Reference on TechNet. Based on comments from Stephan (see below), a slightly modified procedure to try next time:

1. Do bare metal restore of system volume C:. The volume is restored unencrypted.
2. Boot into OS and log in.
3. In the Control Panel, go to BitLocker Drive Encryption and enable Bitlocker on C:. Leave the encrypted data volumes in their locked state for now.
4. Reboot to activate Bitlocker. This apparently creates the auto-unlock master key on the system volume. Log in.
5. From an administrative command prompt, run manage-bde -autounlock -clearallkeys C:.
6. In the Control Panel, go to BitLocker Drive Encryption and manually unlock encrypted data volumes.
7. From a command prompt, check for and delete old External Keys on data volumes (I haven’t tested this!):
   manage-bde -protectors -get S:
   manage-bde -protectors -delete S: -type ExternalKey
8. Back in the Control Panel, for each encrypted data volume, click on Manage BitLocker and set Automatically unlock this drive on this computer.

Recently a colleague noticed that a $495 program called Passware Kit Enterprise is claiming “Instant decryption of BitLocker To Go USB disks.” In fact they claim to be able to decrypt BitLocker and TrueCrypt disks, as well as PGP volumes. Really? How does that work? Are my efforts to encrypt sensitive data useless?

I should note that I am by no means a high-level security guru, but as an I.T. professional, I need to understand and advise customers on suggested best practices for mitigating security risks. Here’s what I found out in a few minutes of research on Passware and BitLocker.

Need a Live Memory Dump

This page at the Passware site describes the main prerequisite for decrypting a BitLocker or TrueCrypt volume:  the target computer must be running and you must be able to get a full memory dump. This makes sense, since the key to decrypt the drive must be stored in memory while the computer is running.

The page lists three tools for getting the memory image:

- Passware FireWire Memory Manager (included in $795 Passware Forensic edition). This requires a FireWire port on the target machine. Oh no, my server doesn’t support FireWire! If yours does, take it out. Note that this could be a vulnerability on laptops.

- Mantech Physical Memory Dump Utility. I downloaded this free tool from SourceForge. and ran it. Based on the Usage instructions, it can only dump the memory of the current computer, and only if you are logged on as an administrator. If you can log on as an administrator, the BitLocker volume is probably already unlocked; you don’t really need to crack the password.

- Win32dd, now part of MoonSols Windows Memory Toolkit. I downloaded the free Community Edition of this tool and checked the command-line parameters. It appears more sophisticated than Mantech:  it can run in client and server mode, allowing the client to send a dump across the network to the Win32dd server. But running the client would still require being logged on to the target machine.

Microsoft Response

This PC Magazine article includes a response from Microsoft about the Passware tool. Basically, it’s, “Yeah, we knew that.” To quote the quote, “We have always been up front in our discussions of Windows BitLocker and that it is intended to help protect data at rest (e.g. when the machine is powered off).”

For more arcane information about BitLocker that you ever wanted to know, see the Microsoft System Integrity Team Blog.

Conclusion

The rule of thumb with any full-disk encryption product seems to be for best protection, either prevent physical access to the machine (servers) or turn it off when not in use (laptops). Sleep is not enough; hibernation is probably okay. As long as the hacker does not have access to live system memory, they are stuck with a brute force attack on the volume, which could take a very long time.

Update March 2, 2011:  Passware now claims here that if the system was hibernated, the BitLocker keys “could be possibly recovered from the hiberfil.sys file.” Failing that, they resort to brute force. So turn your computer completely off, don’t just hibernate it.

Another popular topic is the need to store backup data off site so you can recover in case of disaster. In the small business arena, this is often accomplished by saving the data to external hard drives that are rotated off site.

But how secure are those backup drives once they leave your office? While a laptop may contain excerpts of data, that server backup drive contains all of your proprietary data, and likely private information about your clients as well. What happens if that drive is lost or stolen, either while en route or while stored off site?

BitLocker and External Drives

Windows Server 2008 introduced BitLocker as a built-in full-disk encryption (FDE) engine. Unfortunately, BitLocker makes it difficult to work with external drives in a server environment. Why? Because if you connect a BitLocker-encrypted USB drive to a computer, even if you have set it up to auto-unlock, it will not unlock until a user logs on. Apparently the unlock keys for removable drives are stored with the user profile, not the computer profile. Obviously in a server environment, this is unacceptable:  the server must have access to attached storage even if no one is logged on to the server console.

eSATA to the Rescue

Fortunately, there is a workaround:  use eSATA to connect external drives. Windows sees eSATA drives as internal drives, not removable drives, and you can enable Bitlocker with auto-unlock just as if they were internal data drives. In this example, the S: and T: drives are external eSATA drives, while the R: drive is an external USB drive which is only available in user-specific “Bitlocker To Go” mode:

When you turn on BitLocker, choose “Automatically unlock” so the drive’s key will be stored on the system volume:

Be sure to save the recovery key to a file and/or print it out. Keep it in a safe place so you can decrypt the volume elsewhere in a disaster recovery scenario. Test this by attaching the drive to another machine and making sure you can unlock it with the BitLocker recovery key.

Note that the system volume must be encrypted first, which means you must provide a Bitlocker “master” key to boot the server. See this post for a tip on how to use Bitlocker in a server environment when you need to be able to boot remotely.

If you are using the Enterprise or Datacenter editions of Windows Server 2008 R2, you may find that eSATA drives are offline by default. The simple fix is described here.

Safely Remove eSATA Drives

The next challenge is, how do we safely remove an eSATA drive so we can take it off site? Because Windows thinks eSATA drives are internal, not removable, they are not listed when you choose Safely Remove Hardware. Only the external USB drive is listed:

Here a small, free program called HotSwap! has proved invaluable. Just copy the 64-bit version of this program to a folder under C:\Program Files, run HotSwap!.exe, and a small icon with a red arrow will appear in your system tray. Right-click on the icon to see program options, or just left-click to remove a drive:

and within a few seconds the drive is ready for removal:

My understanding is that HotSwap! works by uninstalling the drive. You could do the same thing from Device Manager.

Not Perfect

HotSwap!, and probably the underlying device uninstallation approach, are not perfect. If you have an open file on the drive, for example a VHD attached to a running virtual machine, you may see this dialog:

If you close the file and try again, you may get the dreaded “system restart requested” dialog:

Once that happens, I’ve found that the only way to remove the drive safely is to restart the server.

Backup Staging

In general, my approach is to “stage” backups on the S: drive, which remains permanently attached to the server, then use a daily robocopy job to copy the S: drive to the T: drive, which rotates off site with another T: drive. This means that the S: drive is always available and always has the same shared folders. The T: drive is only accessed during the daily copy, so I shouldn’t run into the “system restart requested” issue when removing the T: drive. The one thing I still need to test is whether I can host a backup VHD on the S: drive and reliably copy it to the T: drive while the VHD is attached to the VM.

Microsoft’s Bitlocker can use a Trusted Platform Module (TPM) on the motherboard to provide a unified start-up experience, even unlocking system drives before a user logs on. But what if your machine does not have a TPM? How do you configure Bitlocker, and how do you boot the machine if you are not at the server location? I found that a Dell Remote Access Controller (DRAC) is all that is needed.

Allow BitLocker without a Compatible TPM

Scenario 5 of this Technet article has instructions for enabling Bitlocker without a TPM, but they are incorrect at a crucial point. In Windows Server 2008 R2, you will find the setting in Local Group Policy under Computer Configuration > Administrative Templates > Windows Components > BitLocker Drive Encryption > Operating System Drive > Require additional authentication at startup. Enable this policy, and you will be able to check the box Allow BitLocker without a compatible TPM. (Note:  for Server 2008 R2, set the policy for the “Windows 7 family”, not the one for “Windows Server 2008 and Windows Vista”.)

Once that Group Policy setting is made, you can go to Control Panel to turn on Bitlocker on the system drive. You’ll need to save the startup key to a USB flash drive. Be sure to save and print out the recovery key as well.

Boot without a USB Flash Drive

If you know you are going to need to reboot a remote server for system maintenance, you can temporarily disable Bitlocker as described near the bottom of this article. But what if you don’t do that? What happens if you boot with Bitlocker enabled but without the USB flash drive containing the startup key? You’ll see a screen like this:

So how do we use the DRAC to get into the system? My first attempt was to use the DRAC media redirection to plug the USB flash drive into my local machine. But for some reason the machine didn’t see the flash drive:

However I was able to access the console. When I pressed Enter, this screen appeared:

Once I typed in the 48-digit recovery key (not much harder than typing in a product key), Windows started:

Conclusion

Some consider Bitlocker without a TPM to be more secure because it requires an external USB flash drive to boot. Even if you have a TPM, you may want to configure Bitlocker to additionally require a PIN (two-factor authentication). Either way, an out-of-band remote access card like the Dell DRAC solves the problem of not being physically present when booting the machine.

Recently though I read about a password “bypass” program called Kon-Boot that dynamically replaces the Windows kernel during bootup and allows logging in with any password. I wondered if BitLocker was vulnerable to this kind of program. If a thief could simply log on to my BitLocker-protected system, the encryption would be useless.

So I decided to give it a try.

Caveat:  I have no idea if Kon-Boot can harm a computer and/or upload data. Even with a full backup, there is a risk that it might corrupt the BIOS or otherwise make the computer unusable. Use at your own risk.

BitLocker Kicks Kon-Boot

So I booted Windows 7 with the Kon-Boot disk in the CD drive. BitLocker promptly reported that “the system boot information has changed”:

When I pressed Enter to continue, BitLocker prompted me for my password (Label and ID blacked out):

I didn’t want to actually change the BIOS, so I didn’t provide the password. I got this screen:

After rebooting without the Kon-Boot CD, I got this heart-stopping message:

Had Kon-Boot in fact damaged my system? Fortunately after another reboot, Windows 7 came up fine.