A rep with Microsoft Partner Support spent quite a bit of time and effort trying to get to the root cause of this issue, but “no luck.” I decided to live with the annoying error messages in my daily report—then one day, after a reboot, I believe, they stopped. SBS 2008 was back to running only one backup as per schedule.

Back with a Vengeance

Then earlier this month, the issue suddenly returned, but worse: SBS was now trying to run up to five identical backup jobs at a time:

I found in this thread that at least half a dozen other users were experiencing the same issue. Fortunately, one of them had a good suggestion:  manually modify the task to prevent starting a new instance if the task is already running:

1. Open Task Scheduler.
2. Under Task Scheduler > Task Scheduler Library > Microsoft > Windows > Backup, double-click on the Microsoft-Windows-WindowsBackup task.
3. Click on the Settings tab.
4. Under If the task is already running, then the following rule applies change Run a new instance in parallel to Do not start a new instance.
5. Click on OK to save the changes.

This still doesn’t explain why the task tries to start multiple times in the first place. It’s certainly not the “SBS way” to have to manually edit the backup task in Task Scheduler. But at least it prevents multiple instances from attempting to start. A week later, I’m back to all green on my backup history:

Environment Details

The server is a Dell PowerEdge 1900 with 4GB of RAM and two 250GB hard drives configured as RAID 1 (mirrored). As mentioned, it runs Windows Server 2008 R2 with the Hyper-V role, and it hosts a single Server 2008 R2 guest. The guest runs as a domain controller, file server, and email server. Windows Server Backup runs daily on the host, taking an image of the C: drive (containing the host operating system, 16GB) and the D: drive (containing the guest virtual machine, about 48GB used on 80GB of fixed VHD space). Backups are stored on a Seagate Freeagent Go USB drive. Windows Server Backup also runs daily on the guest, saving its backups to a VHD on the same Freeagent USB drive.

The desktop is an Optiplex 960 with 4GB of RAM and one 300GB hard drive. Most important is the processor, an Intel Core 2 E8400 that does support Intel Virtualization Technology and Execute Disable, so it should run Hyper-V.

Recovery Options

There are at least three options for recovering this guest server (which is of course the critical machine):

1. Restore the host and guest from the host’s backup. This makes use of (and tests) Windows Server Backup’s ability to restore to dissimilar hardware, as reported here.

2. Install Server 2008 R2 and the Hyper-V role in a small partition and apply all operating system updates. Restore the guest VHD in a larger partition. Re-create the Hyper-V guest machine using the VHD. This avoids the dissimilar hardware restore at least for the host.

3. Restore the guest backup directly. This would be a virtual-to-physical restore, so also dissimilar hardware.

Option 1 should be the fastest and the most flexible, since it would also accommodate the scenario where multiple guests needed to be restored.

Recovery Experience

The recovery—and this post—became quite long. You may want to skip to the “Conclusions” section, then come back to the step-by-step logs for details as necessary

Time Log – Day 1

Here my log of the recovery process with notes for next time.

3:20pm  Insert Server 2008 R2 DVD into Optiplex 960 and attach USB drive containing backup data. During reboot, check BIOS to make sure Virtualization Technology and Execute-Disable are turned on. Boot from DVD.

3:22pm DVD boot is fast! Glad I didn’t waste time making a bootable thumb drive. Choose Repair your computer, then Restore your computer…. Can’t see external USB drive as backup source. Start googling.

3:35pm Oh yeah, the whole backup drive is Bitlocker-encrypted. Apparently Repair doesn’t detect this as there is no prompt to unlock. Press Shift-F10 to get a command prompt. Commands:
manage-bde –status to show drive (it’s mounted as G:)
manage-bde –unlock –h for parameter help
manage-bde –unlock G: –RecoveryPassword 111111-222222-…-8888888 to unlock using recovery key

3:56pm Drive unlocked. Close command prompt. Press Refresh in Repair wizard. Backup drive found. Choose most recent backup, accept defaults (do check Format and repartition disks, do not check Only restore system drives). Do not install drivers or change Advanced options. Start restore. “This might take from a few minutes to a few hours.”

4:06pm Drive C: restore complete.

4:28pm Drive D: restore about half done.

4:48pm Drive D: restore complete.

4:51pm Drive S: restore complete. “The restored drives are not encrypted.” Remove DVD, unplug USB hard drive, Restart Now.

4:53pm Stuck on boot. “Intel® Matrix Storage Manager” text on screen. I suspect boot drive number is off. Reboot from DVD.

4:58pm Choose Repair your computer, then Use recovery tools…. and Command Prompt. Start fiddling with diskpart:

diskpart
list disk
select disk 0
list volume – letters are all messed up
select volume 1
remove letter=C System Reserved should not have a letter
select volume 2
remove letter=F This is the boot volume
assign letter=C
active identify as the boot volume
select volume 3 this is already assigned drive D, so skip
select volume 4
remove letter=E
assign letter=S
list volumes confirm correct lettering
exit

5:16pm Can’t find a boot.ini file. A little googling confirms that has been replaced in Server 2008 and above with the Boot Manager Configuration file. Let’s just reboot and see what happens.

5:20pm This time we got far enough to see “BOOTMGR is missing. Press Ctrl+Alt+Del to restart”. Reboot from DVD.

5:24pm Operating system is not listed for repair, so follow Method 2 of accepted solution in this thread. From a command prompt, type
bootrec /rebuildbcd
and let it add C:\Windows to the boot list. Remove DVD and restart.

5:27pm Same message:  “BOOTMGR is missing.” Boot from DVD.

This time, operating system is listed, so follow Method 1 at the previous link. Select the operating system. Unfortunately there is no Startup Repair option (apparently that is only available on Windows Vista/7 media). Try a couple other bootrec options:
bootrec /fixmbr
bootrec /fixboot
and reboot.

5:40pm Same message. Giving up for today. Will contact Microsoft Partner Support for suggestions. In a real disaster recovery situation, I would be on the phone with support and/or moving to Recovery Option 2 above. In fact, since the guest’s VHD is already restored, I could probably just install a clean Server 2008 R2 with Hyper-V on top of the botched C drive and use that to load the guest. But I’d like to understand why Recovery Option 1 hasn’t worked and get it working if possible.

Time Log – Day 2

1:10pm MS Support suggests copying the bootmgr file from \Windows\Boot\PCAT\ on the DVD to the root of C:. That location does not exist on the DVD, but it does exist on the restored OS, so I copied that to the root of C:. Voila! Windows boots and immediately reboots, telling me it failed to start. Well at least it’s finding the BOOTMGR.

1:23pm Windows won’t start in Safe Mode either. Don’t even see a blue screen flicker by; just gets partway through the boot and restarts. (It finishes loading CLASSPNP.SYS but fails on whatever comes after that.) Start googling for how to disable Recovery Options > Automatically Restart when you can’t get into the OS.

1:55pm No joy finding out how to change boot behavior. Just for grins, try copying BOOTMGR from the root of the DVD drive to the root of C. Same problem:  won’t start in Safe Mode, just restarts without blue screen.

2:10pm Grasping at straws:  trying copying bootmgr.efi from the root of the DVD to the root of C. Same problem persists.

2:25pm Interesting thread here. Though not directly related, I see I didn’t try one of the recommend commands earlier. Try this while booted from the DVD:
bootrec /rebuildbcd
bootrec /fixmbr
bootrec /fixboot
G: switch to DVD drive
\boot\bootsect.exe /nt60 C: – this is new
After reboot, same issue.

2:40pm Follow an idea near the bottom of this thread and change Optiplex 960 BIOS to put SATA Operation in Legacy mode (was in RAID On mode). Still gets through CLASSPNP.SYS, but does not immediately reboot; it just hangs, won’t even accept Ctrl-Alt-Del. That’s different, so the SATA Operation mode may be the issue! Try the other two SATA modes.  RAID Autodetect / AHCI behaves like RAID On:  a reboot loop. RAID Autodetect / ATA … worked! It hung for a while after CLASSPNP.SYS, but then it gave me a Safe Mode logon screen.

2:53pm After logging on, I see popup telling me “You must restart to apply these changes.” First I went in to System properties > Advanced system settings > Startup and Recovery and unchecked “Automatically restart”. That should stop the reboot loop, should it recur.

2:57pm It rebooted quickly into full operating system and logged on.

Of course the NIC has changed. In Hyper-V manager, run Virtual Network Manager. Change the “Local Area Connection – Virtual Network” to use the External adapter in the Optiplex 960 (Intel 82567LM-3 Gigabit Network Connection).

3:15pm Errors in Optiplex’s Server event log confirm that this machine’s name conflicts with the live server. Temporarily shut down the live server.  Also disable port 25 on router to block incoming email (to keep email from going to the test machine when we start it.) Reboot the Optiplex.

3:18pm Start test guest server. “The virtual machine could not be started because the hypervisor is not running.” This corresponds to Hyper-V-Worker error 3112. The third suggestion may apply:  “If you have made changes to the Boot Configuration Data store, review these changes to ensure that the hypervisor is configured to launch automatically.” We have been mucking around with the boot manager. Hmm, I wonder if I had stayed with the version in the C:\Windows\Boot\PCAT folder instead of copying from the DVD… In fact, maybe I would not have had to rebuild the BCD if I’d just set the SATA mode first.

3:30pm Reboot to DVD. Delete C:\bootmgr, C:\bootmgr.efi, and C:\bootsqm.dat. Reboot. “BOOTMGR is missing.” Oh well. Copy bootmgr from C:\Windows\boot\PCAT to C:\. Windows boots fine. (I realized later that the bootmgr file was not the issue, it was the boot configuration, not stored in the bootmgr file.)

3:41pm I was thinking I would need to re-install the Hyper-V role to get the hypervisor running again, but then I came across this article that shows how to fix it with one command:
bcdedit /set hypervisorlaunchtype auto

3:52pm After rebooting, try starting the guest OS again. Same problem:  the hypervisor is not running. Now in the event log there is Hyper-V-Hypervisor error 41, “Hyper-V launch failed; Either VMX not present or not enabled in BIOS.” Huh? I know I enabled the virtualization stuff in the BIOS.

4:06pm This article suggests turning the BIOS options off and back on. And the first comment in this article says Trusted Execution should NOT be checked in the BIOS, which I agree is counter-intuitive. Followed both suggestions, with full power-off, then started Windows.

4:11pm No error 41 in the event log. Started guest. Failed to start because external drive is not attached to Optiplex. (The guest targets a VHD on that drive for backups.) Use Hyper-V manager to update the settings for the guest, removing that drive from the configuration.

4:13pm Start guest again. It tells me it had not shut down successfully (it wasn’t shut down at all; it was backed up from a VSS snapshot). Let it Start Windows Normally.

4:16pm Guest logon prompt. (Took a long time Applying Computer Settings.) After logon, “You must restart your computer to apply these changes.” Clicked Restart Now.

4:20pm Guest logon prompt. Logged on. Event log looks fairly normal (typical warnings from a startup). Since the guest server has a fixed IP, it automatically is online at the same address as the live server, ready to accept communications from client computers and the Internet. Tested a few components:

Web server works
Active directory works, including accepting logon from client, group policy, folder redirection
Shared drives are available and allow opening QuickBooks data on test server
Email server can retrieve POP mail and send mail

Theoretically at this point I should do a System State restore from the guest’s backup (see my article Careful with Image-Based Backup of Exchange and Active Directory). But I think we’ll call this good for now.

4:33pm Shut down test server, restart live server host and guest, and re-open port 25 on router to allow inbound email.

And a little bonus disaster recovery…

I was anticipating a quick restore of the Optiplex from the Windows Home Server backup. Here’s what happened.

4:53pm In Optiplex BIOS, set SATA Operation back to Raid On. Boot Optiplex from Windows Home Server PC Restore CD. Network driver was missing, but WHS provides an easy way to retrieve the required drivers (by logging on to the WHS console from another computer and copying a the “Windows Home Server Drivers for Restore” folder directly from the WHS backup). Unfortunately that’s the Windows 7 driver, and WHS restore needs the Vista driver.

5:43pm Vista NIC driver downloaded from Dell support site and supplied via USB stick. WHS restore started.

5:57pm Restore complete.

5:59pm Optiplex booted back to Windows 7.

Conclusions and Notes for Next Time

Obviously things did not go as smoothly as hoped. Two hours and 20 minutes on the first day plus three hours and 10 minutes on the second day to get a final logon prompt for the Hyper-V guest. Five and a half hours of solid futzing.

The good news is that it did work; in a real disaster, the client would be back in business.

The other good news is that this was a test, which re-affirms the importance of testing your disaster recovery plan. Next time, I’ll know to check a few key things during the restore process:

• In the Optiplex BIOS, set SATA Operation mode to to RAID Autodetect / ATA before starting the restore.
• In the Optiplex BIOS, set Virtualization, VT for Direct I/O Access, and Execute Disable to ON, but set Trusted Execution to OFF.
• Unlock Bitllocker-protected backup drives using manage-bde commands above.
• If just a blank screen appears after restore, use diskpart to assign correct drive letters and set the C: drive as Active.
• If BOOTMGR is missing, copy bootmgr from C:\Windows\boot\PCAT to C:\.
• Once the host starts, update the Hyper-V Virtual Network Manager to point to the new host NIC.
• If the guest references unavailable and unneeded hardware (e.g. an external backup drive), remove it from the Hyper-V configuration before starting the guest.
• If the guest won’t start because the hypervisor is not running, set it to auto-start using bcdedit command above.

Maybe I’ll try this again someday and see how fast it goes with all that in mind.

The Errors

Here’s the error in the Application event log on the Hyper-V host:

Log Name:      Application
Source:        Microsoft-Windows-Backup
Event ID:      521
Level:         Error
User:          SYSTEM
Description:  The backup operation that started at ‘‎2010‎-‎09‎-‎13T23:25:48.855000000Z’ has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code ‘2155348129′. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

That corresponds to this error in the Hyper-V-VMMS event log on the host:

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Event ID:      10137
Level:         Error
User:          SYSTEM
Description:  No snapshots to revert were found for virtual machine ‘<GuestOS>. (Virtual machine ID 7C88D4F5-71DF-4725-B4B2-E75181BE80EF)

The Wrong Answers

Googling that message takes you to MSKB 2004712, which tells you to enable automount, presumably on the host. However on my machine, automount is already enabled.

There’s a helpful troubleshooting walkthrough on TechNet relating to VSS issues and Windows Server Backup. Even though the example is a similar scenario, I don’t have any errors in the guest, so my issue must be different.

Trying to Work Out the Correct Configuration

Next I turned to the TechNet article Planning for Backup. In the section “Understanding online and offline backups,” I was mightily confused by this sentence: 

Volume Shadow Copy Service must be enabled on all volumes used by the virtual machine with a specific configuration.

This sentence is repeated in several places on the web, but the only clarification I found was at this German site. The author asks two important questions:

1. Does Volume Shadow Copy Service need to be enabled on the host or the guest OS?
2. What should the “specific configuration” look like?

Well, my Server 2003 R2 guest has two volumes, C: for its OS and D: for the data. Volume Shadow Copies are already enabled on D:. Alas, enabling Shadow Copies on C: as well as D: did not allow the host backup to complete, so I disabled them again on C:.

Okay, what about the host? My Hyper-V host also has a C: drive for the OS and a D: drive for data, which in this case are virtual hard drive configuration and disk files. Until now I had not enabled Shadow Copies on either drive. Enabling Shadow Copies on D: was not enough; I got the same error. Additionally enabling Shadow Copies on C: also did not resolve the issue.

For the sake of completeness, I re-enabled Shadow Copies on the guest’s volume C:. So now Shadow Copies are enabled on both C: and D: on both the host and guest. By now, it’s no surprise that the backup failed with the same error:  “No snapshots to revert were found for virtual machine ‘<GuestOS>’.”

Microsoft Support

Microsoft Partner Support clarified two things:

1. The Volume Shadow Copy Service must be enabled on the host, not the guest.

2. I need to check for duplicate disk signatures between the host and the guest.

Aha! I had in fact received warnings about disk signatures in the System event log:

Log Name:      System
Source:        partmgr
Event ID:      58
Level:         Warning
Description:  The disk signature of disk 4 is equal to the disk signature of disk 0.

However when I looked in diskmgmt.msc, there was no disk 4, so I figured it was some issue perhaps related to the virtual devices provided by Dell’s DRAC card. Apparently it was related to a disk attached for creating the backup…

Check and Fix the Disk Signature Conflict

After downloading Sector Inspector and running it on the host and guest machines, it was quickly apparent that since they both had Disk Signature 0×00000080, a disk signature collision could well be my problem.

The fix provided my Microsoft Support was quite simple:

1. Shut down the affected VMs. (I copied the guest VHD to another drive at this point for safety’s sake.)
2. Run the Disk Management MMC on the host.
3. Manually attach the all VHDs to the Hyper-V server via Disk Management MMC.
4. If any VHDs are attached in an “offline” state, a disk signature conflict was found. To fix the conflict, right click the disk and “online” the disk manually.
5. Once all VHD disks are attached in an “online” state, detach them one at a time.
6. Start the VMs.

When I ran Sector Inspector on the guest again, the Disk Signature had changed to 0×4a0d3609. This allowed the host Windows Server Backup to proceed.

Disk2Vhd Gotcha

In retrospect, I realized that the disk signature conflict was a “gotcha” from creating the guest VM using Sysinternals’ Disk2Vhd. Previously, the the Server 2003 R2 machine ran on bare metal. I used Sysinternals Disk2vhd to convert that to a VHD, then installed Server 2008 R2 on the same physical machine and created a VM guest with the Server 2003 R2. Disk2Vhd faithfully copied the disk signature of the source disk. When installed as a guest VM on the same physical disk, the collision occurred.

Can’t Add Second Drive

Using the Windows Server Backup GUI, I set up a backup schedule to do backups to the G: drive and ran backups for a few days. Then I swapped in a new drive. The backup failed with Event ID 561 followed by the more helpful Event ID 546:

Log Name:      Application
Source:        Microsoft-Windows-Backup
Event ID:      546
Task Category: None
Level:         Error
Description: The backup operation attempted at ‘‎2010‎-‎09‎-‎13T04:00:16.334849700Z’ has failed to start, error code ‘2155348061′ (Windows Backup cannot find any of the backup storage locations.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

Okay, so re-run the backup schedule wizard and choose Modify backup storage destinations. Interesting, the old drive G: is now listed as a Volume ID. I guess that makes sense since it is not mounted. Add the current drive G:

Click Next and then Finish the wizard. What? “The filename, directory name, or volume label syntax is incorrect.”

Knowledge Base Incomplete

This must be a common problems, because Microsoft Knowledge Base article 2009365 addresses it specifically:

Error message when you try to add an additional disk to a scheduled backup: “The filename, directory name, or volume label syntax is incorrect”

In my case, the drives are mounted in a Vantec EZ-Swap bay, so I can only mount one at a time. I’m therefore left with Option 3 of the KB article, namely running wbadmin to add the extra target drive. However when I follow their instructions to add the target using the Disk Identifier, I get a long message telling me that it will wipe out my old storage locations:

Apparently when you specify a Disk Identifier, it’s telling Windows Server Backup to consume and format the entire target disk. Since I use the disk for other backups, I need Windows Server Backup to treat it is as a Volume, not a Disk.

Specify Additional Volume Target

The solution is simple:  specify the drive letter instead of the Disk Identifier when adding the target. For example:

wbadmin enable backup –addTarget:G:

Once that Volume target was added, I was able to successfully run a backups to the second backup disk.

Another popular topic is the need to store backup data off site so you can recover in case of disaster. In the small business arena, this is often accomplished by saving the data to external hard drives that are rotated off site.

But how secure are those backup drives once they leave your office? While a laptop may contain excerpts of data, that server backup drive contains all of your proprietary data, and likely private information about your clients as well. What happens if that drive is lost or stolen, either while en route or while stored off site?

BitLocker and External Drives

Windows Server 2008 introduced BitLocker as a built-in full-disk encryption (FDE) engine. Unfortunately, BitLocker makes it difficult to work with external drives in a server environment. Why? Because if you connect a BitLocker-encrypted USB drive to a computer, even if you have set it up to auto-unlock, it will not unlock until a user logs on. Apparently the unlock keys for removable drives are stored with the user profile, not the computer profile. Obviously in a server environment, this is unacceptable:  the server must have access to attached storage even if no one is logged on to the server console.

eSATA to the Rescue

Fortunately, there is a workaround:  use eSATA to connect external drives. Windows sees eSATA drives as internal drives, not removable drives, and you can enable Bitlocker with auto-unlock just as if they were internal data drives. In this example, the S: and T: drives are external eSATA drives, while the R: drive is an external USB drive which is only available in user-specific “Bitlocker To Go” mode:

When you turn on BitLocker, choose “Automatically unlock” so the drive’s key will be stored on the system volume:

Be sure to save the recovery key to a file and/or print it out. Keep it in a safe place so you can decrypt the volume elsewhere in a disaster recovery scenario. Test this by attaching the drive to another machine and making sure you can unlock it with the BitLocker recovery key.

Note that the system volume must be encrypted first, which means you must provide a Bitlocker “master” key to boot the server. See this post for a tip on how to use Bitlocker in a server environment when you need to be able to boot remotely.

If you are using the Enterprise or Datacenter editions of Windows Server 2008 R2, you may find that eSATA drives are offline by default. The simple fix is described here.

Safely Remove eSATA Drives

The next challenge is, how do we safely remove an eSATA drive so we can take it off site? Because Windows thinks eSATA drives are internal, not removable, they are not listed when you choose Safely Remove Hardware. Only the external USB drive is listed:

Here a small, free program called HotSwap! has proved invaluable. Just copy the 64-bit version of this program to a folder under C:\Program Files, run HotSwap!.exe, and a small icon with a red arrow will appear in your system tray. Right-click on the icon to see program options, or just left-click to remove a drive:

and within a few seconds the drive is ready for removal:

My understanding is that HotSwap! works by uninstalling the drive. You could do the same thing from Device Manager.

Not Perfect

HotSwap!, and probably the underlying device uninstallation approach, are not perfect. If you have an open file on the drive, for example a VHD attached to a running virtual machine, you may see this dialog:

If you close the file and try again, you may get the dreaded “system restart requested” dialog:

Once that happens, I’ve found that the only way to remove the drive safely is to restart the server.

Backup Staging

In general, my approach is to “stage” backups on the S: drive, which remains permanently attached to the server, then use a daily robocopy job to copy the S: drive to the T: drive, which rotates off site with another T: drive. This means that the S: drive is always available and always has the same shared folders. The T: drive is only accessed during the daily copy, so I shouldn’t run into the “system restart requested” issue when removing the T: drive. The one thing I still need to test is whether I can host a backup VHD on the S: drive and reliably copy it to the T: drive while the VHD is attached to the VM.

The Script

/*
07/15/2010:  It's pretty difficult to move a full-text index to a new server,
             and the HelpDesk is so small that an index can be regenerated quickly.
             So in case the backup needs to be restored elsewhere,
             before doing the backup, drop and disable the full-text index.
             Re-enable the index after the backup completes.

             This approach assumes the database uses the Simple recovery model.
             It seems that otherwise, log backups are required to fully remove
             the full-text catalog.

             Note:  Adjust the path to your backup location (line 16) and the
                    name of the primary key index on the problems table (line 50).

            Copyright 2010 by Mark Berry, MCB Systems, www.mcbsys.com.
            Free for personal or commercial use.  Use at your own risk.
*/

DECLARE @BackupPath NVARCHAR(255)
DECLARE @dateYYYYMMDD NCHAR(8)

-- Set the next line to the actual backup path
SET @BackupPath = 'C:\Program Files (x86)\Microsoft SQL Server\MSSQL.3\MSSQL\Backup\'

-- Convert today's date to YYYYMMDD form.
-- Backup name will be @BackupPath + 'HelpDesk_db_YYYYMMDD.BAK
SET @dateYYYYMMDD = REPLACE(CONVERT(VARCHAR, GETDATE(),112),'/','')

/*
Step 1:  Remove Full-Text index and catalog, then disable full-text indexing.
*/
USE HelpDesk
DROP FULLTEXT INDEX ON problems
DROP FULLTEXT CATALOG HelpDesk_Problems_FullTextIndex
EXEC sp_fulltext_database 'disable'

/*
Step 2:  Back up the database
*/
SET @BackupPath = @BackupPath + N'HelpDesk_db_' + @dateYYYYMMDD + N'.BAK'
BACKUP DATABASE [HelpDesk]
TO  DISK = @BackupPath
WITH NOFORMAT, INIT,  NAME = N'HelpDesk-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10

/*
Step 3:  Enable full-text indexing, then recreate the catalog and index
         Note:  A full population is started automatically when the index is created..
*/
USE HelpDesk
EXEC sp_fulltext_database 'enable'
CREATE FULLTEXT CATALOG HelpDesk_Problems_FullTextIndex
    AS DEFAULT
CREATE FULLTEXT INDEX ON problems
    (title LANGUAGE 1033,
     [description] LANGUAGE 1033,
     solution LANGUAGE 1033)
    KEY INDEX PK__problems__060DEAE8
        ON HelpDesk_Problems_FullTextIndex
    WITH CHANGE_TRACKING AUTO

-- Next line can be used to list full-text catalogs
-- select * from sysfulltextcatalogs

Prepare IIS 7

Liberum Help Desk is a classic ASP application, so to run it under IIS 7, as explained in this Technet article, you’ll need to enable the following roles under IIS 7:

• ASP
• Request Filtering
• ISAPI Extensions

It turns out those are already enabled on SBS 2008, so I didn’t have to do anything here.

Set Up an Active Directory User

Add a new user in Active Directory that the web site can use for accessing the database, for example DOMAIN\HelpDeskUser. Create a complex password and set it to Never Expires. I left mine in the default Domain User group.

Create and Configure the Web Site

1. Copy the Help Desk files from your old web server to your new server.

2. Open IIS Manager and create a new web site pointing to the path where you just copied the Help Desk files. The Connect as button here only controls access to the physical path. I was able to leave it as the default (Application user – pass-through authentication).

3. Still in IIS Manager, click on Application Pools in the left pane, then right-click on the new application pool created for your web site and choose Advanced Settings.

3.1. Change the .NET Framework Version to No Managed Code.

3.2. Under Process Model > Identity, click on the small button and change the logon from Network Service to the Custom account that you created earlier (DOMAIN\HelpDeskUser).

4. Go back to the web site and double-click on ASP to open the ASP properties:

Set Enable Parent Paths to True:

5. Go back to the web site again and double-click on Authentication to open the Authentication properties. Right-click on Anonymous Authentication and change the Anonymous user identity to use the Application pool identity.

If you leave this set to IUSER, your database logins won’t work!

Update 7/21/2010:  Follow the steps in this post to allow your custom user to update the IIS persistent cache:  Template Persistent Cache Error with Classic ASP under IIS 7.

Move the Database

I spent a lot of time on with this step because it’s very difficult to move a database containing full-text index configuration if the FTTEXT path has changed. My HelpDesk database is relatively small, so the index generates in under 30 seconds. It should be much easier to just drop the full-text index on SQL 2000 before backing up the database for moving to SQL 2005. Although not the route I followed, I’ll document what (I think) is necessary to do that.

1. In SQL 2000 Enterprise Manager, right-click on the Full-Text Catalogs and Remove All Catalogs.

2. Back up the HelpDesk database to a file. Copy the file to your new server.

3. On your new server, if you haven’t already, install a new instance of Microsoft SQL Server 2005 Express Edition with Advanced Services Service Pack 2 (download). The Advanced Services edition is required for full-text search capabilities (which is a pretty awesome feature in a free product!). If, like me, you do not notice the optional Full-Text Search option during the initial installation, you’ll have to go back and do a Change installation to add that feature:

4. You’ll also need Microsoft SQL Server Management Studio Express (download).

5. Open SQL Management Studio. Under Security > Logins, create a new login that maps to the Active Directory user you created earlier (DOMAIN\HelpDeskUser).

6. Restore the HelpDesk database backup you created just above.

7 Under HelpDesk > Security > Users, create a new user that matches the Active Directory user you created earlier (again DOMAIN\HelpDeskUser). Assign the db_datareader and db_datawriter roles to the user.

8. In your Help Desk web site, edit settings.asp with a text editor. Because I installed a named instance of SQL 2005 Express, I had to change the Application(“SQLServer”) line. I was already using SQL Server with integrated security, so the Application(“DBType”) did not change.

9. At this point you should be able to log in to the Help Desk. Go to Administration > Configure Site and change your SMTP Server.

Note JMail 4.5 installed fine but I get an object error when trying to send a test mail from the Help Desk. I I enabled 32-bit applications as suggested in this thread, but that kills the site (Service Unavailable) and the OWA component of Exchange 2007 Service Pack 3 raises errors in the event log. I’m still working on this issue.

Update 7/23/2010:  Since 32-bit JMail is giving IIS with Exchange 2007 SP3 fits, I thought I’d check whether one of the other mail components supported by Liberum Help Desk is available in a 64-bit version. Success:  I installed the 64-bit version of ASPEmail from Persits Software, changed the Help Desk configuration to use ASPEmail, and voila!, I’m sending email.

Update 7/26/2010:  An extra setting is required in Exchange 2007 to allow ASPEmail to anonymsously send mail outside the network. See the bottom of this article.

Set Up Full-Text Indexing

If you plan to use full-text indexing, you’ll need to create the index. Some UI elements are missing from the Express edition, so it’s not quite as easy as it is under SQL 2000. Here are the basics.

1. Open SQL Management Studio. Right-click on the HelpDesk database and choose Properties, then Files. Check the Use full-text indexing box. If the box is grayed out, you may need to install the optional Full-Text Search feature—see above.

2. Open a new query and use a T-SQL command to create a full-text catalog. For example:

USE HelpDesk
CREATE FULLTEXT CATALOG HelpDesk_Problems_FullTextIndex
    AS DEFAULT

You can check for catalogs with this command:

SELECT * FROM sysfulltextcatalogs

3. Back in the main SQL Management Studio window, expand the HelpDesk database and the list of Tables. Right-click and choose Refresh (so the Management Studio will see the catalog you just created). Then right-click on the problems table and choose Design. Right-click in the design window and choose Fulltext Index. In the Full-text Index dialog, click on the Add button. Click the little button next to Columns and choose the title, description, and solution columns. Leave the other fields with their default values.

Click on Close, then exit the Design view, confirming that you do want to save changes to the problems table. At this point, you can go back into Design view, open the Full-text index dialog, and check whether population (crawl) is complete—mine took 10 seconds:

Back Up without the Full-Text Index

SQL 2005 Express is VSS-aware, so I’m counting on my daily server backups to include the HelpDesk database for disaster recovery. However I also want to be able to create a backup manually that does not include the full-text index, in case the backup needs to be restored elsewhere. To that end, I created the Back Up HelpDesk script which you are free to try and modify at your own risk. Note that this approach assumes the database uses the Simple recovery model. It seems that otherwise, log backups are required to fully remove the full-text catalog.

Credits

I found two posts by Ian Porter to be very helpful as I got started on full-text indexing under SQL 2005 Express:  re-installation and setup.

For details, many thanks to Marcin Policht for an amazing series of articles on full-text indexing in SQL 2005 Express. The series starts here.

Exchange Backup

For Exchange, you have to enable the Exchange VSS writer (disabled by default on SBS 2003). That and other tips in this post in the StorageCraft ShadowProtect forum.

Active Directory Backup

And this little surprise is buried deep within MSKB 888794, Considerations when hosting Active Directory domain controller in virtual hosting environments:  restore of Active Directory is only supported from a System State backup. “In particular, Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent.”

The explanation makes it sound like there is only a risk if you do replication, so if you only have one domain controller (no replication) you might get away with it. But it’s probably better to do a System State backup using a traditional backup tool regardless.

You also can’t add features remotely using Server Manager, so type these commands at the Hyper-V Server command prompt:

1. Start PowerShell:

powershell

2. Import the server manager module so you have access to those PowerShell commands:

import-module ServerManager

3. (Optional) Check available and installed features:

Get-WindowsFeature

4. (Optional) Get help on a PowerShell cmdlet:

Get-Help [cmdlet_name] -full
For example:  Get-Help Add-WindowsFeature -full

5. Install Windows Server Backup Features:

Add-WindowsFeature -Name Backup-Features -IncludeAllSubfeature

6. Quit PowerShell:

exit

7. Register the VSS writer for use with Windows Server Backup (see this TechNet article and MSKB 958662):

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\WindowsServerBackup\Application Support\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}"

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\WindowsServerBackup\Application Support\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}" /v "Application Identifier" /t REG_SZ /d Hyper-V

8. Check the registration of VSS writer for use with Windows Server Backup:

reg query "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\WindowsServerBackup\Application Support\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}" /s

Note You can also type regedit at the server’s command prompt to launch the familiar graphical registry editor and check the setting.

Using Windows Server Backup

Once installed, it seems you are left with the Wbadmin command line tool for running backups and restores unless you have a full installation of Windows Server 2008 with the Windows Server Backup MMC plug-in. That plug-in is not available in the Remote Administration Tools available for Windows 7.

According to this TechNet article, Wbadmin has been extended for Windows Server 2008 R2 (there are some cool new backup features in R2), however as of this writing, the Wbadmin command reference has not been updated. The built-in help has been updated, so for details you can, for example, type the following at the server command line:

Wbadmin -help
Wbadmin Enable Backup -help

Here’s the command to do a one-time backup of the system drive C: and the data drive D: to an external drive F:

Wbadmin start backup -backupTarget:F: -include:C:,D: -systemState -allCritical

The -systemState flag is actually redundant since backing up the system drive C: includes the system state. The -allcritical flag means “back up everything needed to restore the operating system to bare metal,” which in this case adds the 100MB System Reserved volume used by Bitlocker.