Often failed pings are related to the firewall on the target machine. For example, the Windows Firewall setup on my Server 2008 R2 only allows ICMPv4 pings on Private networks:

By temporarily disabling individual firewall profiles on the server, I found that I could ping the server if I turned off the Public firewall profile. Huh? Why does the server thing it is on a Public network?

The answer is Network Location Awareness. You’ve seen the prompt when you connect a laptop to a new wireless network:  is this a Home, Work, or Public network? Well I never saw that prompt on the server; apparently it just assumes that a new network (e.g. due to a new router) is Public, the most restrictive profile.

Note that this doesn’t affect domain-joined computers; they are automatically assigned to the Domain firewall profile. But it does affect my non-domain-joined Hyper-V server.

Change the Network Type

To tell the server it’s on a Private network, go to Network and Sharing Center, click on Public network:

Then choose Work network:

That’s it! The server now knows it’s on a Private network, and the Private firewall profile applies. As long as that profile accepts ICMP (ping) requests, you should be able to ping the server.

Symptoms

How do you know you’re approaching expiration? You’ll see this warning in your Hyper-V-VMMS event log:

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Event ID:      12510
Task Category: None
Level:         Warning
User:          SYSTEM
Description:  The certificate used for server authentication will expire within 30 days. Remote access to virtual machines will not be possible after the certificate expires. Please renew or recreate the certificate.

To check the certificate, you’ll need to create a new certificate viewer for the Hyper-V Virtual Machine Management service. (For instructions, in this TechNet article, see step 2 under “Deploying a certificate issued by a CA.”) You’ll see that the machine’s certificate will expire soon:

A copy of the certificate is also found in the vmms\Trusted Root Certification Authorities store.

Solution

Microsoft Support recommended following the instructions in 241735 to create a self-signed certificate that is good until 2050. For that you need:

• This PowerShell script: Hyper-V_Cert.zip. This is a slightly modified version of the script posted on TechNet here. The TechNet version improperly appends “WORKGROUP” to the certificate’s Subject Name if your server is not domain-joined, which prevents Hyper-V from using the certificate. See this article for more info on creating and updating Hyper-V certificates. As always, use downloaded scripts at your own risk!
• MakeCert.exe from the Windows SDK. Use the web installer (be sure you’re on a 64-bit machine) and select only the Tools. The total download is about 80MB:

Running the Script

Put the script and makecert.exe in the same folder on your Hyper-V server and run the cert.ps1 PowerShell script.

Note The script will save and and restore all running VMs!

I assume that saving and restoring VMs will disrupt clients that are relying on a constant server connection, so do this when no one is around.

If you see “WORKGROUP” when you start the script, you’re using the version from TechNet. When the script completes, you’ll need to delete the certificate from the certificate store and use the version above.

You should instead see something like this if your server is not in a domain:

When the script completes, you should see the new certificate in the vmms\Trusted Root Certification Authorities store:

No copy is created in the vmms\Personal store.

If you stop and start the Virtual Machine Management service, you should no longer see the event log warning about the pending certificate expiration.

Environment Details

The server is a Dell PowerEdge 1900 with 4GB of RAM and two 250GB hard drives configured as RAID 1 (mirrored). As mentioned, it runs Windows Server 2008 R2 with the Hyper-V role, and it hosts a single Server 2008 R2 guest. The guest runs as a domain controller, file server, and email server. Windows Server Backup runs daily on the host, taking an image of the C: drive (containing the host operating system, 16GB) and the D: drive (containing the guest virtual machine, about 48GB used on 80GB of fixed VHD space). Backups are stored on a Seagate Freeagent Go USB drive. Windows Server Backup also runs daily on the guest, saving its backups to a VHD on the same Freeagent USB drive.

The desktop is an Optiplex 960 with 4GB of RAM and one 300GB hard drive. Most important is the processor, an Intel Core 2 E8400 that does support Intel Virtualization Technology and Execute Disable, so it should run Hyper-V.

Recovery Options

There are at least three options for recovering this guest server (which is of course the critical machine):

1. Restore the host and guest from the host’s backup. This makes use of (and tests) Windows Server Backup’s ability to restore to dissimilar hardware, as reported here.

2. Install Server 2008 R2 and the Hyper-V role in a small partition and apply all operating system updates. Restore the guest VHD in a larger partition. Re-create the Hyper-V guest machine using the VHD. This avoids the dissimilar hardware restore at least for the host.

3. Restore the guest backup directly. This would be a virtual-to-physical restore, so also dissimilar hardware.

Option 1 should be the fastest and the most flexible, since it would also accommodate the scenario where multiple guests needed to be restored.

Recovery Experience

The recovery—and this post—became quite long. You may want to skip to the “Conclusions” section, then come back to the step-by-step logs for details as necessary

Time Log – Day 1

Here my log of the recovery process with notes for next time.

3:20pm  Insert Server 2008 R2 DVD into Optiplex 960 and attach USB drive containing backup data. During reboot, check BIOS to make sure Virtualization Technology and Execute-Disable are turned on. Boot from DVD.

3:22pm DVD boot is fast! Glad I didn’t waste time making a bootable thumb drive. Choose Repair your computer, then Restore your computer…. Can’t see external USB drive as backup source. Start googling.

3:35pm Oh yeah, the whole backup drive is Bitlocker-encrypted. Apparently Repair doesn’t detect this as there is no prompt to unlock. Press Shift-F10 to get a command prompt. Commands:
manage-bde –status to show drive (it’s mounted as G:)
manage-bde –unlock –h for parameter help
manage-bde –unlock G: –RecoveryPassword 111111-222222-…-8888888 to unlock using recovery key

3:56pm Drive unlocked. Close command prompt. Press Refresh in Repair wizard. Backup drive found. Choose most recent backup, accept defaults (do check Format and repartition disks, do not check Only restore system drives). Do not install drivers or change Advanced options. Start restore. “This might take from a few minutes to a few hours.”

4:06pm Drive C: restore complete.

4:28pm Drive D: restore about half done.

4:48pm Drive D: restore complete.

4:51pm Drive S: restore complete. “The restored drives are not encrypted.” Remove DVD, unplug USB hard drive, Restart Now.

4:53pm Stuck on boot. “Intel® Matrix Storage Manager” text on screen. I suspect boot drive number is off. Reboot from DVD.

4:58pm Choose Repair your computer, then Use recovery tools…. and Command Prompt. Start fiddling with diskpart:

diskpart
list disk
select disk 0
list volume – letters are all messed up
select volume 1
remove letter=C System Reserved should not have a letter
select volume 2
remove letter=F This is the boot volume
assign letter=C
active identify as the boot volume
select volume 3 this is already assigned drive D, so skip
select volume 4
remove letter=E
assign letter=S
list volumes confirm correct lettering
exit

5:16pm Can’t find a boot.ini file. A little googling confirms that has been replaced in Server 2008 and above with the Boot Manager Configuration file. Let’s just reboot and see what happens.

5:20pm This time we got far enough to see “BOOTMGR is missing. Press Ctrl+Alt+Del to restart”. Reboot from DVD.

5:24pm Operating system is not listed for repair, so follow Method 2 of accepted solution in this thread. From a command prompt, type
bootrec /rebuildbcd
and let it add C:\Windows to the boot list. Remove DVD and restart.

5:27pm Same message:  “BOOTMGR is missing.” Boot from DVD.

This time, operating system is listed, so follow Method 1 at the previous link. Select the operating system. Unfortunately there is no Startup Repair option (apparently that is only available on Windows Vista/7 media). Try a couple other bootrec options:
bootrec /fixmbr
bootrec /fixboot
and reboot.

5:40pm Same message. Giving up for today. Will contact Microsoft Partner Support for suggestions. In a real disaster recovery situation, I would be on the phone with support and/or moving to Recovery Option 2 above. In fact, since the guest’s VHD is already restored, I could probably just install a clean Server 2008 R2 with Hyper-V on top of the botched C drive and use that to load the guest. But I’d like to understand why Recovery Option 1 hasn’t worked and get it working if possible.

Time Log – Day 2

1:10pm MS Support suggests copying the bootmgr file from \Windows\Boot\PCAT\ on the DVD to the root of C:. That location does not exist on the DVD, but it does exist on the restored OS, so I copied that to the root of C:. Voila! Windows boots and immediately reboots, telling me it failed to start. Well at least it’s finding the BOOTMGR.

1:23pm Windows won’t start in Safe Mode either. Don’t even see a blue screen flicker by; just gets partway through the boot and restarts. (It finishes loading CLASSPNP.SYS but fails on whatever comes after that.) Start googling for how to disable Recovery Options > Automatically Restart when you can’t get into the OS.

1:55pm No joy finding out how to change boot behavior. Just for grins, try copying BOOTMGR from the root of the DVD drive to the root of C. Same problem:  won’t start in Safe Mode, just restarts without blue screen.

2:10pm Grasping at straws:  trying copying bootmgr.efi from the root of the DVD to the root of C. Same problem persists.

2:25pm Interesting thread here. Though not directly related, I see I didn’t try one of the recommend commands earlier. Try this while booted from the DVD:
bootrec /rebuildbcd
bootrec /fixmbr
bootrec /fixboot
G: switch to DVD drive
\boot\bootsect.exe /nt60 C: – this is new
After reboot, same issue.

2:40pm Follow an idea near the bottom of this thread and change Optiplex 960 BIOS to put SATA Operation in Legacy mode (was in RAID On mode). Still gets through CLASSPNP.SYS, but does not immediately reboot; it just hangs, won’t even accept Ctrl-Alt-Del. That’s different, so the SATA Operation mode may be the issue! Try the other two SATA modes.  RAID Autodetect / AHCI behaves like RAID On:  a reboot loop. RAID Autodetect / ATA … worked! It hung for a while after CLASSPNP.SYS, but then it gave me a Safe Mode logon screen.

2:53pm After logging on, I see popup telling me “You must restart to apply these changes.” First I went in to System properties > Advanced system settings > Startup and Recovery and unchecked “Automatically restart”. That should stop the reboot loop, should it recur.

2:57pm It rebooted quickly into full operating system and logged on.

Of course the NIC has changed. In Hyper-V manager, run Virtual Network Manager. Change the “Local Area Connection – Virtual Network” to use the External adapter in the Optiplex 960 (Intel 82567LM-3 Gigabit Network Connection).

3:15pm Errors in Optiplex’s Server event log confirm that this machine’s name conflicts with the live server. Temporarily shut down the live server.  Also disable port 25 on router to block incoming email (to keep email from going to the test machine when we start it.) Reboot the Optiplex.

3:18pm Start test guest server. “The virtual machine could not be started because the hypervisor is not running.” This corresponds to Hyper-V-Worker error 3112. The third suggestion may apply:  “If you have made changes to the Boot Configuration Data store, review these changes to ensure that the hypervisor is configured to launch automatically.” We have been mucking around with the boot manager. Hmm, I wonder if I had stayed with the version in the C:\Windows\Boot\PCAT folder instead of copying from the DVD… In fact, maybe I would not have had to rebuild the BCD if I’d just set the SATA mode first.

3:30pm Reboot to DVD. Delete C:\bootmgr, C:\bootmgr.efi, and C:\bootsqm.dat. Reboot. “BOOTMGR is missing.” Oh well. Copy bootmgr from C:\Windows\boot\PCAT to C:\. Windows boots fine. (I realized later that the bootmgr file was not the issue, it was the boot configuration, not stored in the bootmgr file.)

3:41pm I was thinking I would need to re-install the Hyper-V role to get the hypervisor running again, but then I came across this article that shows how to fix it with one command:
bcdedit /set hypervisorlaunchtype auto

3:52pm After rebooting, try starting the guest OS again. Same problem:  the hypervisor is not running. Now in the event log there is Hyper-V-Hypervisor error 41, “Hyper-V launch failed; Either VMX not present or not enabled in BIOS.” Huh? I know I enabled the virtualization stuff in the BIOS.

4:06pm This article suggests turning the BIOS options off and back on. And the first comment in this article says Trusted Execution should NOT be checked in the BIOS, which I agree is counter-intuitive. Followed both suggestions, with full power-off, then started Windows.

4:11pm No error 41 in the event log. Started guest. Failed to start because external drive is not attached to Optiplex. (The guest targets a VHD on that drive for backups.) Use Hyper-V manager to update the settings for the guest, removing that drive from the configuration.

4:13pm Start guest again. It tells me it had not shut down successfully (it wasn’t shut down at all; it was backed up from a VSS snapshot). Let it Start Windows Normally.

4:16pm Guest logon prompt. (Took a long time Applying Computer Settings.) After logon, “You must restart your computer to apply these changes.” Clicked Restart Now.

4:20pm Guest logon prompt. Logged on. Event log looks fairly normal (typical warnings from a startup). Since the guest server has a fixed IP, it automatically is online at the same address as the live server, ready to accept communications from client computers and the Internet. Tested a few components:

Web server works
Active directory works, including accepting logon from client, group policy, folder redirection
Shared drives are available and allow opening QuickBooks data on test server
Email server can retrieve POP mail and send mail

Theoretically at this point I should do a System State restore from the guest’s backup (see my article Careful with Image-Based Backup of Exchange and Active Directory). But I think we’ll call this good for now.

4:33pm Shut down test server, restart live server host and guest, and re-open port 25 on router to allow inbound email.

And a little bonus disaster recovery…

I was anticipating a quick restore of the Optiplex from the Windows Home Server backup. Here’s what happened.

4:53pm In Optiplex BIOS, set SATA Operation back to Raid On. Boot Optiplex from Windows Home Server PC Restore CD. Network driver was missing, but WHS provides an easy way to retrieve the required drivers (by logging on to the WHS console from another computer and copying a the “Windows Home Server Drivers for Restore” folder directly from the WHS backup). Unfortunately that’s the Windows 7 driver, and WHS restore needs the Vista driver.

5:43pm Vista NIC driver downloaded from Dell support site and supplied via USB stick. WHS restore started.

5:57pm Restore complete.

5:59pm Optiplex booted back to Windows 7.

Conclusions and Notes for Next Time

Obviously things did not go as smoothly as hoped. Two hours and 20 minutes on the first day plus three hours and 10 minutes on the second day to get a final logon prompt for the Hyper-V guest. Five and a half hours of solid futzing.

The good news is that it did work; in a real disaster, the client would be back in business.

The other good news is that this was a test, which re-affirms the importance of testing your disaster recovery plan. Next time, I’ll know to check a few key things during the restore process:

• In the Optiplex BIOS, set SATA Operation mode to to RAID Autodetect / ATA before starting the restore.
• In the Optiplex BIOS, set Virtualization, VT for Direct I/O Access, and Execute Disable to ON, but set Trusted Execution to OFF.
• Unlock Bitllocker-protected backup drives using manage-bde commands above.
• If just a blank screen appears after restore, use diskpart to assign correct drive letters and set the C: drive as Active.
• If BOOTMGR is missing, copy bootmgr from C:\Windows\boot\PCAT to C:\.
• Once the host starts, update the Hyper-V Virtual Network Manager to point to the new host NIC.
• If the guest references unavailable and unneeded hardware (e.g. an external backup drive), remove it from the Hyper-V configuration before starting the guest.
• If the guest won’t start because the hypervisor is not running, set it to auto-start using bcdedit command above.

Maybe I’ll try this again someday and see how fast it goes with all that in mind.

The Message

Here are the important bits of the message, along with the corresponding event IDs in the Hyper-V-Worker Admin event log:

‘VM01’ failed to start.

Microsoft Emulated IDE Controller (Instance ID {…}): failed to Power on with Error ‘General access denied error’ (0×80070005). [Event ID 12010]

IDE/ATAPI Account does not have sufficient privilege to open attachment ‘D:\Virtual Machines\VM01\Virtual Hard Disks\DRIVE01.VHD’. Error: ‘General access denied error’ (0×80070005). [Event ID 12290]

‘VM01 failed to start. (Virtual machine ID 6B78D45F5-71DF-4725-B4B2-E651800BE80EF) [Event ID 12030]

Okay, so I understand that it can’t access the VHD file. But what is the “IDE/ATAPI Account”? I see no such account in the list of available users and groups when I try to modify the VHD file’s permissions using the GUI. What I do see, when I examine the permissions of a working VHD, is what looks like a GUID as a user name with Read and Write permissions:

Virtual Machine SID

It turns out that the virtual machine’s unique identifier (SID) actually needs direct access to the file. The details are in this Microsoft knowledge base article:

Hyper-V virtual machines may not start, and you receive an error: “‘General access denied error’ (0×80070005)”
http://support.microsoft.com/kb/2249906

Update the Permissions

The solution is to give the SID Full control of the VHD using the command line.

Tip: Rather than type (or mistype) the SID by hand, cut and paste it from the event log message or from the XML file name in the Virtual Machines folder.

The command:

icacls <Path of .vhd file> /grant "NT VIRTUAL MACHINE\<Virtual Machine SID>":F

Note that there is no space before the :F. For example:

icacls "D:\Virtual Machines\VM01\Virtual Hard Disks\DRIVE01.VHD" /grant "NT VIRTUAL MACHINE\6B78D45F5-71DF-4725-B4B2-E651800BE80EF":F

Once that permission has been added, you should be able to start the virtual machine.

In the Hyper-V-Image-Management-Service Admin log, we have Event ID 15054:

Log Name:  Microsoft-Windows-Hyper-V-Image-Management-Service-Admin
Source:  Microsoft-Windows-Hyper-V-Image-Management-Service
Event ID:  15054
Level:  Error
User:  SYSTEM
Description:  ‘The system failed to compact ‘F:\<MyDrive>.VHD’. Error Code: 2424869

I thought maybe the physical drive was too fragmented, but a defrag ran quickly and made no difference.

This blog post provides the answer:  the VHD contains VSS shadow copies. (It’s a Server 2003 with Volume Shadow Copies enabled.)

Start up the virtual machine and log on to the guest. Run vssadmin list shadows to check for shadow copies. Run vssadmin delete shadows /all to delete the shadow copies. Then shut down the virtual machine guest and try compacting again.

“A required CD/DVD drive device driver is missing. If you have a driver floppy disk, CD, DVD, or USB flash drive, please insert it now.”

A bit strange to see it complaining about a CD/DVD drive when it is reading from an ISO. I spent awhile searching for drivers for the physical CD/DVD etc. until I came across this post with the answer.

Long story short, the ISO download was incomplete:  it was only about 540MB instead of 2.3GB. I re-downloaded and the installation ran fine.

Here are the event details. Note that the first one is recorded as an Information event but actually indicates an issue.

Event Type:    Information
Event Source:    W32Time
Event Category:    None
Event ID:    38
Description:
The time provider NtpClient cannot reach or is currently receiving invalid time data from SBS2008Server.mydomain.local (ntp.d|192.168.1.25:123->192.168.1.2:123).

Event Type:    Warning
Event Source:    W32Time
Event Category:    None
Event ID:    24
Description:
Time Provider NtpClient: No valid response has been received from domain controller SBS2008Server.mydomain.local after 8 attempts to contact it. This domain controller will be discarded as a time source and NtpClient will attempt to discover a  new domain controller from which to synchronize.

Event Type:    Error
Event Source:    W32Time
Event Category:    None
Event ID:    29
Description:
The time provider NtpClient is configured to acquire time from one or more time sources, however none of the sources are currently accessible.  No attempt to contact a source will be made for 15 minutes. NtpClient has no source of accurate time.

Analysis

The SBS machine is the domain controller, so it’s also the default NTP server.

I had already followed Hyper-V best practices for domain controllers by disabling Time Synchronization and setting the SBS machine to sync with an external source. That’s working fine.

Client computers in the domain, both physical and virtual, were not reporting time sync errors even though they also get time from SBS (Type is NT5DS; see how to check this). So why can’t Server 2003 sync time as well?

Microsoft KB 875424, “Time synchronization may not succeed when you try to synchronize with a non-Windows NTP server in Windows Server 2003,” describes the symptoms pretty well except for the “non-Windows” part. But even after putting the Server 2003 machine into “client mode” (/syncfromflags:MANUAL) pointed directly at the domain controller, the time sync errors continued. So I set it back to domain mode (/syncfromflags:DOMHIER).

I turned on time service logging as described in Microsoft KB 816043. Didn’t learn much here except that SBS is returning time packets with Precision: –6, so the issue I had a few years ago with very high precision servers is not the problem here.

Solution

Eventually I noticed that when starting the Windows Time Service, I would get an event indicating that time was being synchronized with the Hyper-V integration services provider:

Event Type:    Information
Event Source:    W32Time
Event Category:    None
Event ID:    35
Description:
The time service is now synchronizing the system time with the time source VM IC Time Synchronization Provider.

A little more Googling and I found this article suggesting that the solution is to disable Time Synchronization on all Hyper-V guests, not just on domain controllers. Once I disabled the Time Synchronization integration service on the Server 2003 machine and rebooted, the errors stopped.

I still don’t know why Hyper-V Time Synchronization integration service causes the standard domain-based time synchronization to fail (why not play nicely together?), but disabling Hyper-V Time Synchronization on all virtual machines seems to solve the problem.

Bonus Tip:  Time Sync on the Hyper-V Server

All the machines above are running on Windows Server 2008 R2 as the Hyper-V host. Follow Microsoft’s recommendation, the host is not joined to the domain since the domain controller is running as a guest. To keep the host’s time in sync, I set it to client mode and pointed it at the SBS domain controller. This command sequence does the trick:

sc triggerinfo w32time start/networkon stop/networkoff
w32tm /config /manualpeerlist:SBS_server_IP_Address,0x8 /syncfromflags:MANUAL
net stop w32time
net start w32time
w32tm /resync


Note:  If you get a message that the Windows Time service is not running, from services.msc, set it to Automatic and start it. For more details, see this later blog post.

Next I moved the USB drive to a machine running Windows Server 2008 R2 with Hyper-V. In Disk Management, I set the drive Offline, then in Hyper-V Manager I attached the drive to a virtual machine as a pass-through disk. Finally I booted the Hyper-V virtual machine from a ShadowProtect 3.5.2 ISO file.

The Issue

Since I was restoring a Server 2003 image, I used the ShadowProtect legacy recovery environment. I did not let it load the network since all drives were local to the VM. I started a restore of the boot partition from a ShadowProtect image file. After a minute the restore failed with a cyclic redundancy error 23. I tried to verify the various image files, and the verifies failed with a message that the files were corrupt. I rebooted from the CD several times but the results were always the same:  ShadowProtect sees the files and may even start to read or verify one, but soon the read fails.

The Network Workaround

I decided to try the workaround I remembered from Virtual PC (which does not have pass-through disks). First I removed the pass-through disk from the Hyper-V VM, then set it Online in Disk Management. Then I booted into the ShadowProtect legacy environment with networking. Finally I started the restore over the network, attaching from inside the VM to the ShadowProtect image files on the host machine (e.g. \\HOST\R$\C_VOL.spf). This time I was able to restore three partitions with no errors.

Note In order for the ShadowProtect recovery environment to see the Hyper-V network, you have to set up the Hyper-V virtual machine with a Legacy Network Adapter.

Here’s the workaround.

Step 1:  Create the Partition

When you get to this screen during the install, click on New to create a partition. I decide to leave 2GB unallocated on the first VHD in case I want to set up Bitlocker later: (Remember this is Server 2008, not R2, which automatically creates an extra system partition in advance.)

Step 2:  Click Shift-F10 to Format

Here’s the “trick”:  click on Shift-F10 to get a command prompt. Then enter the following commands to format the drive  (making sure that you select the correct partition once inside diskpart):

diskpart
list disk
select disk 0
list partition
select partition 1
format fs=ntfs label=”SBS-os” quick unit=16K
assign letter=C
exit [from diskpart]
exit [from the command prompt]

Step 3: Select the Partition and Continue

Click on Refresh or press F5 to confirm that your newly named partition is there, then click on Next to continue the installation. If you are asked about creating a system partition, click on OK.

Note:  the first time I tried this, I used the format command instead of diskpart. But for some reason that did not allow the 78GB partition to be recognized as a system partition, so the installation failed and I had to start over.

Update:  Even easier when working with a Server 2008 R2 host:  before starting the installation of the client, attach the VHD in the host and format there. The above should work if your host is Server 2008 (not R2) or for installation on (gasp!) a physical server.

Today I came in to see monitoring emails telling me that PBX had gone off- and online several times overnight. Sure enough, when I pick up the phone:  no dial tone. What happened?

I logged on to the Hyper-V host and started Core Configurator 2.0 and was a little alarmed to see that my PBX isn’t even listed as a virtual machine at all:

What? The PBX has disappeared entirely? Fortunately I also have Hyper-V Manager set up on a Windows 7 machine. This gave me the more complete picture:  the PBX, and a Vista Business test machine, were in a “Paused” state:

Lost in Space

Next I fired up Server Manager, attached to the Hyper-V host, and started wading through the Hyper-V logs. Finally I found what I was looking for under

Applications and Services Logs > Microsoft > Windows > Hyper-V-VMMS > Admin

where I saw this event:

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          6/18/2010 4:15:52 AM
Event ID:      16060
Level:         Error
User:          SYSTEM
Computer:      HYPERVSERVER01
Description: ‘PBX’ has been paused because it has run out of disk space on ‘D:\Virtual Machines\pbx\Virtual Hard Disks\’. (Virtual machine ID 8FD97E3F-90A2-45B6-8A35-58C1EEBA8B3C)

Sure enough, drive D was full. That Vista Business test machine had expanded its drive overnight during auto-updates. I moved some VHDs to another drive and was able to get the PBX running again. What I don’t fully understand is why the PBX, which has a fixed-size VHD, was paused due to space issues on the host volume.

Core Configurator Limitations

Looking more closely at the Core Configurator, I see that it only shows Running and Stopped Virtual Machines, so Paused machines are not included. It would be nice if that bottom window would show Paused and Stopped Virtual Machines.

In the same dialog, I also noticed that Core Configurator only lists one Network Switch, even though I have two installed, as seen in Hyper-V Manager:

I assume Core Configurator is not showing the second switch because I have unchecked “Allow management operating system to share this network adapter”. Again it would be nice to see the same information in Core Configurator as shown in Hyper-V Manager.

Core Configurator adds a wealth of features that would otherwise be very difficult to access on a Hyper-V Server machine. But you’ll still need a remote management machine running Hyper-V Manager to get the complete picture of the virtual machines, and remote Server Manager to see the host’s event logs.

Recently I’ve been testing Microsoft’s free bare-metal hypervisor, Microsoft Hyper-V Server 2008 R2. Especially now that Hyper-V supports Red Hat Enterprise Linux, could Hyper-V be the answer to virtualizing PiaF on a Microsoft platform?

Let’s try it and see what happens.

Setting Up CentOS under Hyper-V Server 2008 R2

I’ll assume you have Hyper-V Server installed and running. Follow these steps to set up CentOS under Hyper-V Server using the PiaF installation ISO:

1. Set up a new virtual machine with 512MB of memory, a 10GB virtual IDE hard disk (for servers, I’m told that fixed disks perform better than dynamic disks), and a connected legacy network card. Note that the fixed disk and legacy card cannot be configured through the New Virtual Machine Wizard; you’ll have to modify the settings once the virtual machine exists.
2. Download the PBX in a Flash 1.4 ISO file. Attach the ISO to the virtual machine’s DVD-ROM drive (Media > DVD Drive) and start the machine.
3. At the first install prompt, type ksalt. This starts the install in text mode, which avoids an issue with 24-bit video. It also does not use the LVM file system, which apparently adds unnecessary overhead (according to jroper). I deliberately chose a very simple root password during the setup so I could take this virtual machine to client sites one day if needed.
4. Toward the end of the installation, you’ll see that CentOS is “Installing bootloader,” then the install reboots. As soon as the reboot starts, quickly select Media > DVD Drive > Eject. If you miss this, the install will start over.
5. After the boot completes, you’ll be prompted about which PiaF “payload” to use. Choose “A” for the latest stable release via the Internet. The download goes quickly but it does take awhile to compile Zaptel, Asterisk, etc. Once that completes, the system reboots again.
6. When this reboot completes (you’ll see the “pbx login” prompt), shut down the machine by sending a Ctrl-Alt-Del from Hyper-V Server and stopping the machine when it is at the BIOS startup stage. Use Hyper-V Manager to Export a copy of your virtual machine at this point, so you can come back to this point if you encounter problems with the steps below. (A snapshot would also work but an export can be imported at another location.) I’m testing the free 7-Zip for creating a compressed copy of the exported virtual machine. (To install the 64-bit version of 7-Zip 4.65 under Hyper-V Server 2008 R2, run the installer on the server by typing msiexec /i 7z465-x64.msi  at a command prompt.) Name the compressed file something like “PiaF.1.Immediately after CentOS Install.zip”.
7. Start the virtual machine and log on to CentOS. Make sure that CentOS is connected to the network. At a command prompt:
   /sbin/ifconfig # check whether an IP address has been assigned; if so, skip the rest of this step
   system-config-network # if you need to check the network setup
   nano -w /etc/sysconfig/network-scripts/ifcfg-eth0 # view network setup in text form
   service network restart # restart with updated network setup if you made changes
   /sbin/ifconfig # check that an IP address has now been assigned
8. Add the kernel options. I tried using VMWare’s recommendations, since they have specific advice on CentOS 5.2. The CentOS bug report seems to confirm that the kernel options divider=10 clocksource=acpi_pm should take care of the clock cycle issue without the need for a “vm” kernel (compiled with a 100Hz clock instead of 1000Hz). However, after making that change, I was getting lots of messages reading, “rtc lost some interrupts at 1024hz”. So I’m going back to my original procedure posted in November 2008:
   nano -w /boot/grub/menu.lst 
   At the end of the “kernel” line, add this string:
   clocksource=pit nosmp noapic nolapic
9. Update the kernel with the CentOS Virtual Machine kernel. This is outlined on the PiaF site, but do not recompile Zaptel as instructed there. The “vm” kernels are available here. Note:  choose the version that matches your current kernel. Here are the download and install commands:
   cd /root
uname -r # Check your Linux version. Modify the next two lines to match.
   wget http://dev.centos.org/~tru/kernel-vm/5/RPMS/i386/kernel-vm-2.6.18-92.1.22.el5.i686.rpm
wget http://dev.centos.org/~tru/kernel-vm/5/RPMS/i386/kernel-vm-devel-2.6.18-92.1.22.el5.i686.rpm
rpm -ivh kernel-vm*.rpm
10. Edit grub to make the new kernel load by default.
    nano -w /boot/grub/menu.lst
    In this file, set default=0 if the .e15vm kernel is listed first. After saving the file, reboot.
    shutdown -r now

11. After the reboot, check the Linux version to make sure it ends in “vm”:
    uname -r

12. You can confirm that the clocksource is set to pit using the method suggested in this thread:
    cat /sys/devices/system/clocksource/clocksource0/current_clocksource

After these changes, I no longer have any time errors appearing on the virtual machine console, even when left on overnight.

Installing Microsoft Linux Integration Components for Windows Server 2008 Hyper-V R2

You’ll need to download and install the Linux Integration Components for Windows Server 2008 Hyper-V R2. There are some significant differences here from the old VM Additions for Linux that worked with Virtual Server. Two notable changes are that the Integration Components for Hyper-V no longer include time synchronization or shutdown integration. However, we should benefit from the enhanced network and disk drivers.

Update 9/30/2010:  This article was written using V2 of the Linux Integration Components. See the comments for links to later version(s).

One “gotcha” right out of the gate:  to extract the ISO needed to install the Integration Components, you must run the downloaded Linux_IC_v2.EXE file. In spite of its claim to be a Win32 Cabinet Self-Extractor, when I tried to run it on my 32-bit XP machine, it tells me that is is not “not a valid Win32 application.” Windows 7 (32-bit) gives a similar error. Once I copied the file to my 64-bit Hyper-V Server 2008 R2 machine, I was able to extract the ISO.

The package includes a helpful readme in PDF format. Since CentOS is based on Red Hat, follow the section titled, “Installing the Linux Integration Components Version 2 on Red Hat Enterprise Linux 5.” Here’s the trimmed-down version:

1. Normally you would need to run yum update kernel, reboot, then run yum install kernel-devel gcc to get the kernel source. However, it seems that the PiaF already includes the kernel source, so this step is not needed with PiaF.
2. From Hyper-V Manager, start your PiaF virtual machine. Then select Media > DVD Drive > Insert Disk and load the extracted LinuxIC_v2.ISO file.
3. Log on to the virtual machine. Mount the drive and copy the files to a new directory as follows:
   mkdir /media/cdrom
mount /dev/cdrom /media/cdrom
mkdir /usr/src/linux_ic_v2
cp /media/cdrom/* /usr/src/linux_ic_v2 -R # -R to recursively copy directories
   umount /dev/cdrom
   You can eject the ISO image from the virtual machine at this point.
4. Install the synthetic drivers. No reboot is required here.
   cd /root  # install won’t run if you are in the source folder, so make sure we’re at root folder
   /usr/src/linux_ic_v2/setup.pl drivers

5. Verify that the installation succeeded.
   /sbin/lsmod | grep vsc
   The output should look something like this:
   netvsc                 45616  0
storvsc                37028 0
vmbus                  56520  2 netvsc,storvsc
scsi_mod              134605  3 storvsc,libata,sd_mod
6. Configure the synthetic network card (seth0).
   Note:  do this from the Hyper-V Manager connection directly, not through a remote connection like Putty.
   So far you probably have one Legacy Network Adapter configured for the VM. From Hyper-V Manager, set that adapter to “Not connected”, then set a non-legacy network adapter to an active connection. (You can do this while the VM is online, but you’ll have to wait until shutdown to actually remove the legacy adapter.) Finally, make the corresponding changes inside the virtual machine.
   nano -w /etc/sysconfig/network-scripts/ifcfg-eth0 #
   Disable the legacy card:  set
   ONBOOT=no
After saving the file:
   service network restart # restart with updated network setup
   /sbin/ifconfig # eth0 should be gone; check that an IP address has now been assigned to seth0
7. Check the Fastpath boot setup. When you run the following command, you should see that hda=noprobe and hdb=noprobe have been added to your kernel boot options. Copy these options to the non-vm kernel line in case you ever need to boot from it:
   nano -w /boot/grub/menu.lst 

As mentioned, the Linux Integration Components don’t support shutdown from the Hyper-V host. However Hyper-V is able to save a PiaF guest within a few seconds. I plan to try that as my Automatic Stop Action in Hyper-V Settings. Restarting a saved machine may cause problems until the clock is resynchronized (see NTP below), but it seems preferable to just “pulling the plug” if, for example, the UPS needs to shut down the host due to a power outage.

Tweaking NTP

With Virtual Server 2005, I disabled NTP and relied on the time to sync to the host. The Hyper-V extensions do not include time synchronization, so follow the instructions near the bottom of this VMWare document for doing so:

Since we have conflicting advice, and nothing definitive for Microsoft Virtual Server, I’ve decided to try the following:

1. Open the NTP configuration file:
   nano -w /etc/ntp.conf
2. Add the following as the first line in the file (“instructs NTP not to give up if it sees a large jump in time”):
   tinker panic 0
3. Comment out these two lines (prevent fallback to the “undisciplined local clock”):
   #server 127.127.1.0 # local clock
#fudge 127.127.1.0 stratum 10
4. Restart the NTP daemon:
   service ntpd restart

At this point you should have a working CentOS installation under Hyper-V that can more or less tell time. Check the time by typing date at the command prompt and comparing it to the clock on your host system.

On startup I do see some NTP messages reading, “ntpd[1970]: frequency error 512 PPM exceeds tolerance 500 PPM,” but this does not seem to keep NTP from syncing the time. I’ll have to check this occasionally to make sure the clock isn’t drifting too much. This command will show the last 100 ntpd messages:

tail -100 /var/log/messages | grep ntpd --color

Moving from Physical to Virtual

Most of the remaining setup steps have been covered in other posts, but there’s a huge shortcut if you are just moving from one PiaF machine to another. Here are the steps you’ll want to follow.

Finishing the CentOS Configuration

Follow the steps in Setting Up PBX in a Flash, Part 2:  Before Running FreePBX to complete the CentOS configuration. A few tips to keep in mind:

• You may want to temporarily give the new machine its own IP address in your router so you can have both the old and new machines online during the steps below. There should be no issue giving the new machine the same host name as the old machine.
• You can use the temporary IP address to access the new machine, so you don’t need to set up a special name in your DNS server.
• If you want to save your work as you go, shut down the virtual machine and user Hyper-V Manager to either take a snapshot and/or export the virtual machine.

Copying the tftp Setup

If you’re using tftp, review Setting Up a Polycom IP 430 Phone with FreePBX for details on the tftp setup. Since my system is not open to the network, I allow writing to the tftpboot folder. Follow these steps to copy the tftp setup:

1. Open access to tftpboot:
   chmod 777 /tftpboot
2. Use WinSCP to copy the contents of the tftp configuration file, /etc/xinetd.d/tftp, from your physical your Windows desktop, then paste it into the same file on the virtual machine.
3. Use WinSCP to copy all the files in /tftpboot from the physical machine to the virtual machine.

Updating the FreePBX Configurations

The shortcut will be to copy the configuration from the old, physical machine to the new, virtual machine. To make sure this goes smoothly, let’s first get the base configuration of the two machines to match.

Update the Physical Machine

On the old, physical machine, update all installed modules to the latest versions:

1. In a browser, open the IP address or DNS name of the old PBX.
2. From the PBX in a Flash menu, in the lower left corner, click on Admin. This adds a few buttons to the screen.
3. Click on the FreePBX Administration button. Type maint as the user name. Use the password that you assigned using passwd-master near the end of part 2. This loads the main FreePBX administration screen, which will be our starting point for the rest of the tasks this article. You might want to bookmark this page so you can return here directly without going through the PiaF button top-level screen.
4. The left column has two tabs, Setup and Tools. From the Setup tab, click on Module Admin. This shows the list of currently-installed modules.
5. At the top of the Module Administration page, click on the link Check for Updates Online. Now you will see all modules that are available.
6. Click on the Upgrade All link (not Download All). This simply selects all upgradable modules; it doesn’t start the download and upgrade process.
7. Scroll to the bottom of the page. Click on Process. Scroll to the bottom of that page. Click on Confirm. An orange popup window appears where you can monitor the progress of the installations. After a few minutes, when you scroll to the bottom of the orange popup, you’ll see a Return link. Click on Return.

Update the Virtual Machine

Log on to the new, virtual machine, and follow the steps in the first section of the Step 3 article, “Updating and Adding FreePBX modules.” Install the same modules on the new machine as are installed on the old machine.

Tip:  I found it simplest to install all the modules on the new machine, then go back and uninstall the ones that are not installed on the old machine.

Now that both machines have the same FreePBX modules installed, we are ready to copy the configuration.

Copy the Configuration from Old to New

Follow these steps to use the FreePBX Backup & Restore utility to copy your entire configuration from the old to the new machine:

1. On the old machine, from the FreePBX administration screen, go to the Tools tab, Backup & Restore. Add a Backup Schedule named OnDemand. Back up all sections (Voice Mail, System Recordings, System Configuration, CDR, and Oeprator Panel). Set the Run Backup time to Now and click on Submit Changes.
2. Using WinSCP, copy the new backup file from /var/lib/asterisk/backups/OnDemand to your Windows desktop. The file will have a timestamp as its name and end in .tar.gz.
   Note:  Do not rename the backup file! I thought I would be clever and give it a more meaningful name. This caused the restore to fail, probably because the restore expects the .tar file inside the .tar.gz file to have the same name as the .tar.gz file.
3. On the new machine, from the FreePBX administration screen, also create a backup called OnDemand. This is mostly done so FreePBX will create the OnDemand directory and will know to look for backups in that directory.
4. Use WinSCP to copy the old machine’s backup file from your Windows desktop to /var/lib/asterisk/backups/OnDemand on the new machine.
5. On the new machine, from the FreePBX administration screen, , go to the Tools tab, Backup & Restore. Click on Restore from Backup and navigate down to the OnDemand directory and to the old machine’s backup file (make sure the time stamp matches the backup from the old machine.) Choose Restore Entire Backup Set, and confirm that you want to overwrite all FreePBX and Asterisk files.
6. When the restore completes, click on the orange Apply Configuration Changes. Your new system should now match your old system exactly! In the Setup tab, check some known values (Extensions, Trunks, Inbound Routes) to be sure.
7. For some reason, the configuration backup/restore does not include the /etc/amportal.conf file. If you’ve customized this file (e.g. to specify the AMPWEBADDRESS or to send backups to an FTP server), copy the file from the old machine to the new machine using WinSCP.

Cut Over from Physical to Virtual

If you manage your IP addresses through your router, this last bit is easy:

1. Shut down both your old, physical machine and your new, virtual machine.
2. Optional:  in Hyper-V Manager, Export the fully-configured virtual machine. Back up this export so you can return to this state later.
3. In your router, in the list that maps MAC addresses to IP addresses, note down the IP address of the physical machine and then delete it from the (or assign it an IP that is not in use).
4. Also in the router’s address map, update the MAC address of the virtual machine to map to the IP address that had heretofore been used by the physical machine.
5. Use Hyper-V Manager to start the new, virtual machine.
6. After the virtual machine is running, reboot or cycle power on your other telephony devices (SPAs, IP phones) to force them to register with the new machine. Note that as far as these devices are concerned, the PBX server hasn’t moved, since it has the same IP address as the old, physical machine.

Now you are finally ready to make some test calls and see how they sound! In my initial tests, I hear no echo or jittery “cell phone” audio and can detect no difference from a physical machine. Maybe Hyper-V finally allows running Asterisk in a Window virtual environment!