I blogged earlier about Microsoft Desktop Virtualization Licensing for the SMB. Mostly for licensing reasons, it doesn’t make sense to do full virtual desktops. So that leaves the option of running a terminal server to host the customer’s apps.

This is actually an attractive option for this customer. Several users, some of whom are volunteers, log in to desktops from home. Since desktop login is limited to one person at a time, they have to work when no one is at the physical desktop and no one else is logged in remotely. Giving them access to a terminal server instead would eliminate those conflicts. But that is a “nice to have,” not something that would drive this project.

Comparing the Projects

Let’s look first at what would be required to replace this desktop. I making several assumptions here:  desktops are fairly powerful Optiplex 960s from Dell Outlet; customer has Microsoft Open licenses with Software Assurance on existing server and desktops; and the customer qualifies for Microsoft non-profit pricing. What’s involved in replacing one desktop? What if it were three desktops?

Obviously the costs in the right column would be substantially higher if the customer did not qualify for non-profit pricing. But it was the fact that they do qualify that made me think we might justify this project when just replacing one desktop at first.

Audio, Video, RDP, and Choosing a Thin Client

Hewlett-Packard has a new thin client, the t5325, currently available at an introductory price of $199. Sporting a 1.2GHz processor, 512MB of SDRAM, and DVI-I with a VGA converter, it’s a pretty nice little unit. And at that price, we might justify the thin client route when replacing only one desktop. So I bought a t5325 to see what I could get for $199.

Quite a lot, actually

There are definitely some quirks in the t5325 (probably due in part to it being a brand new product) but overall the setup is quite flexible and easy, and running standard Windows apps over RDP works amazingly well.

However, if a thin client is to replace a desktop, at least in this environment, it needs to be able to handle basic audio and video as one might encounter when browsing the web or playing an AVI or WMV video clip. This, unfortunately, is where the t5325 falls short.

Multimedia Redirection Matters

After playing with the t5325, and after reading this excellent post on the Remote Desktop Team Blog, I finally “got it”:  you might as well forget about audio and video via RDP if you’re not redirecting multimedia to the client.

What is multimedia redirection? It’s passing the media stream directly to the client in its compressed form, allowing the client to format and play it. Without multimedia redirection, content must be rendered on the server and then passed as bitmaps to the client. Depending on the the power of the client and server, that may work more or less well, but it chews up CPU power on the server.

To test this, I tried playing back an AVI video while connected via Remote Desktop from three different clients:  the HP t5325 running RDP 5.2, a Windows XP SP3 machine still running RDP 6.1, and Windows 7 running RDP 7. Here are my testing notes:

HP t5325 and RDP 5.2:  Video jerky; sound awful (unintelligible, scratchy).  At one point I had somehow disabled sound and the video was okay. Server CPU 43% – 54% during playback. (I should note that I also tested connecting the t5325 via RDP to a Windows XP desktop and playing back a YouTube video. In that test, audio and video were acceptable, so the t5325 may be having issues with the way Windows Servers 2008 R2 serves audio and video.)

Windows XP SP3 and RDP 6.1:  At first, audio and video were decent but badly out of sync with each other. I set up Medium quality audio redirection as described in this blog post. After that, audio and video are okay; sync is much better but still not 100%. Server CPU utilization is 75% – 100% during playback.

Window 7 and RDP 7:  Audio and video are smooth and in sync. Server CPU utilization during playback is 4% – 12% because audio and video are being streamed, not rendered on server.

The difference with Windows 7/RDP 7 was so dramatic that I was “sold” on multimedia redirection.

Multimedia Redirection in a Thin Client

HP, Wyse, and no doubt other thin client vendors offer various kinds of add-on software to enhance audio-video playback. I did not research that in depth, but I did note that HP’s offering, Remote Graphics Software, is only available on thin clients running XP Embedded (XPe) or its replacement, Windows Embedded Standard (WES). (Click Buy Online to see pricing from $50 to a site license at $154,000!)

For completing my simple chart above, I decided to look for a thin client that supports RDP 7, with the assumption that I would get the desired multimedia redirection. No doubt RDP 7 will become a common feature of WES-based thin clients, but it looks like 10ZiG is one of the first to offer it. Their blog article describes some of the benefits. Reviewing their list of thin clients aimed at virtual desktops, the RBT-416v is the lowest-numbered model to support RDP 7. System ID Warehouse offers that model for $479, so that’s the figure I used in my comparison above. HP’s thin clients running WES (though not yet listing RDP 7) are comparable in price ($429 – $550).

Conclusion

Entry-level thin clients like the HP t5325 offer an attractive price point and may be ideal for some applications (kiosks or data entry terminals, for example). But to get the “desktop experience” when playing audio and video, yet without overly taxing the server, you’ll need a higher-end thin client that supports multimedia redirection.

The cost of higher-end thin clients is on par with basic desktops. In a very small environment, that makes it difficult to justify the time and cost involved in setting up a terminal services infrastructure to handle the RDP sessions. However as the number of desktops to be replaced increases, the economies of scale (not to mention power savings, management simplicity, and longevity) make thin clients look increasingly attractive as desktop replacements.

My test machine is a Dell PowerEdge 2900. I was able to install and access OpenManage on Hyper-V Server 2008 R2 as follows:

1. Download the latest Dell Systems Management Tools and Documentation DVD. I found version 6.1.0 A00 here. Note that this is actually two downloads that must be concatenated before burning the DVD–see the Installation Instructions on the download page.

2. Follow Dell’s article, “Installing Dell OpenManage Software On Microsoft Windows Server 2008 Core and Microsoft Hyper-V Server.” Only install the Managed System Software, not the Management Station Software.

First run the prerequisite checker by running these commands at the command prompt (assuming the DVD is in drive E):

E:
cd \SYSMGMT\srvadmin\windows\prereqchecker
runprereqchecks.exe /s

Update 10/12/2010:  Path corrected–thanks to comment from Ian N.

The instructions for the prerequisite checker tell you to take certain actions depending on the return code. Use this command to view the return code:

echo Return Code = %ERRORLEVEL%

My Return Code was -2, which, according to this article, means “The operating system is not supported. The Prerequisite Checker will not run.” My hunch is that this is because the software has not been updated to recognize the R2 version of Server 2008. I went ahead and ran the installer anyway:

cd \SYSMGMT\srvadmin\windows\SystemsManagement
msiexec /i SysMgmt.msi

After agreeing to continue without having run the prerequisite checker, the installation completed successfully.

3. Open the firewall for the OpenManage web interface default port 1311:

netsh advfirewall firewall add rule name="Dell OpenManage Server Administrator Web GUI" dir=in action=allow protocol=TCP localport=1311

Note Knowledgebase article 947709 is helpful on the netsh commands. However, do not just cut and paste its examples:  as of this writing, the quotation marks are all formatted as closing quotation marks (”), so the commands will fail. Replace those quotation marks with standard “vertical” quotation marks (“).

4. From another computer, open a browser and type in the following URL (be sure to get the “s” on the https):

https://[computer name or IP address]:1311

After adding the required security exception in your browser (due to the self-signed web server certificate), you should now see the familiar Server Administrator web interface:

There are other advantages to using the full version of Windows Server 2008 R2 as the host OS, but Dell OpenManage is available even from the free Hyper-V Server 2008 R2.

You also can’t add features remotely using Server Manager, so type these commands at the Hyper-V Server command prompt:

1. Start PowerShell:

powershell

2. Import the server manager module so you have access to those PowerShell commands:

import-module ServerManager

3. (Optional) Check available and installed features:

Get-WindowsFeature

4. (Optional) Get help on a PowerShell cmdlet:

Get-Help [cmdlet_name] -full
For example:  Get-Help Add-WindowsFeature -full

5. Install Windows Server Backup Features:

Add-WindowsFeature -Name Backup-Features -IncludeAllSubfeature

6. Quit PowerShell:

exit

7. Register the VSS writer for use with Windows Server Backup (see this TechNet article and MSKB 958662):

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\WindowsServerBackup\Application Support\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}"

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\WindowsServerBackup\Application Support\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}" /v "Application Identifier" /t REG_SZ /d Hyper-V

8. Check the registration of VSS writer for use with Windows Server Backup:

reg query "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\WindowsServerBackup\Application Support\{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}" /s

Note You can also type regedit at the server’s command prompt to launch the familiar graphical registry editor and check the setting.

Using Windows Server Backup

Once installed, it seems you are left with the Wbadmin command line tool for running backups and restores unless you have a full installation of Windows Server 2008 with the Windows Server Backup MMC plug-in. That plug-in is not available in the Remote Administration Tools available for Windows 7.

According to this TechNet article, Wbadmin has been extended for Windows Server 2008 R2 (there are some cool new backup features in R2), however as of this writing, the Wbadmin command reference has not been updated. The built-in help has been updated, so for details you can, for example, type the following at the server command line:

Wbadmin -help
Wbadmin Enable Backup -help

Here’s the command to do a one-time backup of the system drive C: and the data drive D: to an external drive F:

Wbadmin start backup -backupTarget:F: -include:C:,D: -systemState -allCritical

The -systemState flag is actually redundant since backing up the system drive C: includes the system state. The -allcritical flag means “back up everything needed to restore the operating system to bare metal,” which in this case adds the 100MB System Reserved volume used by Bitlocker.

Here are a few notes I made on setting things up so I can manage the server from Windows 7 (Professional, Ultimate, or Enterprise only):

1. On the Windows 7 machine, download and install the Remote Server Administration Tools for Windows 7. Go to Control Panel > Programs > Turn Windows features on or off. Under Remote Server Administration Tools, check (at least) Server Manager and Role Administration Tools > Hyper-V Tools.
2. (Optional) On the Hyper-V Server 2008 R2 machine, use the sconfig menu system to enable Remote Desktop so you can log on to the server without being in front of the console. Connect to the machine using its IP address.
3. On the Hyper-V Server 2008 R2 machine, use the sconfig menu system to enable all remote management options. A reboot is required in there somewhere.
4. On the Hyper-V Server 2008 R2 machine, use the sconfig menu system to join the machine to the same domain as the Windows 7 machine. According to this thread, it is apparently possible to leave the machine in a workgroup; using the Hyper-V Remote Management Configuration Utility is recommended in that case. I chose the simpler route of just joining the machine to the domain.
5. On the Window 7 machine, log on as a domain administrator. Go to Administrative Tools > Server Manager. When prompted for the Remote computer, type the name or IP address of the Hyper-V 2008 R2 server.

At this point you should be able to browse the roles, features, diagnostics (including event logs), configuration (including firewall and services) of the Hyper-V 2008 Server. However, Device Manager and Disk Management require some additional configuration before they will work.

Note You cannot add or remove features (like Windows Server Backup) remotely. See this TechNet article for what you can and cannot do remotely. I've also blogged about Setting Up Windows Server Backup on Hyper-V Server 2008 R2.

Enabling Remote Access to Device Manager

As explained in this TechNet article, to enable remote device management, you have to make a change to group policy on the Hyper-V 2008 R2 machine. (The TechNet article refers to Windows 2008 Server Core, but as I discovered, most articles on Server Core also apply to Hyper-V Server.) That article describes the process as follows:

To do this, on a computer running Windows Vista or a full installation of Windows Server 2008, open the Local Group Policy Editor MMC snap-in, connect to the computer running a Server Core installation, navigate to Computer Configuration\Administrative Templates\Device Installation, and then enable Allow remote access to the PnP interface. Restart the computer running a Server Core installation.

For some reason, when I ran gpedit.msc from Windows 7, I was not able to connect to another computer. I had to first run MMC, then select File > Add or Remove Snap-in > Group Policy Object Editor, then select the Hyper-V Server 2008 R2 computer.

Once in the Group Policy Editor, it took me a while to find the PnP setting. The correct location is:  Computer Configuration\Administrative Templates\System\Device Installation\Allow remote access to the PnP interface.

Although the TechNet article says to restart the server, I just typed “gpupdate” at the server's command prompt and I was able to connect to Device Manager remotely using Server Manager. Note that when run remotely, Device Manager is read-only–but at least you can see what devices are installed and whether any are missing drivers.

Enabling Remote Disk Management

To allow remote disk management, you need to change the firewall rules on both the Hyper-V Server and the Windows 7 machine. Run this command on both machines:

netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes

Many thanks to macnamee on this thread for that answer. The same TechNet article referenced above has some additional details, though it doesn't make clear the need to open the firewall on the Windows 7 machine as well.

Unlike remote Device Manager, Disk Management actually lets you add and remove volumes etc. from the remote computer. Very handy.

Technical Options

There are lots of options for virtualizing desktops on a server. I can run Virtual Server 2005 and create Virtual Machines (VMs) under that. Or I can run a full hypervisor like Windows Server 2008 R2 Hyper-V, VMWare ESX, or Xen and create virtual machines there.

Licensing Options

As usual, licensing is where things get tricky. I am no licensing specialist but I wanted to get down the results of some of my research.

Windows OEM licenses, like you get when you buy a PC from Dell, are tied to the physical hardware. We want to get rid of the old hardware, so that’s not an option.

A full packaged product (FPP) version of a Windows desktop OS can be run in virtual mode. For Windows 7 Professional, that means $300 MSRP per virtual machine. FPP products do not include downgrade rights so you have to actually run Windows 7.

The other option is Virtual Enterprise Centralized Desktop (VECD), formerly Vista Enterprise Centralized Desktop. For use with a thin client, VECD is like renting the desktop software:  for about $110 a year, you are allowed to access the VM from one thin client. One primary, named user of that thin client can also use the VM from home.

Unlike OEM and FPP, VECD does not include perpetual use rights:  once the agreement expires, your right to use the software also expires. Assuming a five-year desktop life span, a VECD license for the same period would run $550.

So let’s add up and compare the five-year costs, assuming $250 for a thin client:

• Hardware PC with Windows 7 Pro:  $800
• Virtualize with Windows 7 FPP:  Thin client $250 + FPP $300 = $550.
• Virtualize with VECD:  Thin client $250 + VECD $550 = $800.

Of course there are lots of other variables that speak in favor of virtualization especially in large corporate environments:  security, management, shared images, reduced power consumption, etc. But for a small business with only a few desktops those benefits are negligible and perhaps even more cumbersome than running physical desktops. One thing that could benefit a small business is that a thin client, with no moving parts, should theoretically outlast a desktop PC. But is that really the case? Many thin clients now offer dual monitor support, for example, but what additional features will be required at the desktop five years from now? USB 3? IPv6?

Another big drawback to the VECD model for this customer is that several part-time users currently share the same desktop. Often the users access the desktop from home. In the VECD scenario, only one of those users would be permitted to access the VM from home; for others, the client would have to buy additional VECD licenses. So if three users need access to the same VM from home, you need two additional VECD licenses for a total of $1650 just for the OS for five years.

OEM vs. FPP

According to this PC World article, “an OEM edition can be used to install the new operating system on a brand new, and thus blank, virtual machine….” Although OEM licenses cannot be transferred, e.g. from the old PC’s we’re retiring, the thought here would be to use an new OEM license instead of the Windows 7 FPP. I’d want to double-check with Microsoft if this is legal, but the OEM version of Windows 7 Professional is only $140 at Newegg. Theoretically the client could move its existing SA to the new OEM license and thereby obtain downgrade rights as well. See update below.

Acquiring VECD

Emma Healey, Licensing Escalation Manager at Microsoft UK, has a couple of helpful posts for understanding VECD:

• March 2008: http://ladylicensing.spaces.live.com/blog/cns!87F95F1B5B21B01E!146.entry
• October 2009: http://ladylicensing.spaces.live.com/blog/cns!87F95F1B5B21B01E!1980.entry

One clarification she posted in the comments of the first article:  “Open [licensing] cannot support the subscription method used to purchase VECD, so it’s only available in Open Value in the small and med business space, and Select/Select Plus and EA/EA Subscription in the large org space, and School/Campus in the education space.”

So for an SMB, the only option to purchase VECD is through Open Value. Another strike for my client, who qualifies for non-profit pricing, which is not available through Open Value.

It’s quite a challenge finding SKUs for VECD. They’re not listed on VECD’s page on the Microsoft Volume Licensing site. They’re not listed on the Microsoft page explaining VECD, although the VECD Licensing Guide available there is helpful. The grand overview of client virtualization in the Microsoft Client Virtualization Strategy White Paper will blow you away with options but will not give you SKUs.

Finally I resorted to searching the CDW store. Search for “OV VECD” and you’ll find 19 Open Value VECD products. Note that the products that begin with DSA- are for when you attach VECD to a thick client that already has a Windows desktop license with Software Assurance. Since I’m looking for the thin client options, I have to look at the part numbers beginning with DTA-. Search again for “DTA-00051″ and you’ll see the options.

Conclusion

Microsoft has made significant strides in supporting desktop virtualization, but their VECD model is mostly focused on the large enterprise. However the option of installing a Full Purchased Product, or perhaps an OEM version, into a virtual machine still has some appeal.

Update 9/8/2009:  Regarding the idea of installing an OEM version into a VM, I had started to think that maybe the OEM license would in that case be tied to the host server (also not desirable).  But then I found the OEM System Builder License:

http://oem.microsoft.com/downloads/Public/sblicense/2008_SB_Licenses/FY08_SB_License_English.pdf

The system builder must pre-install the license on a Customer System, which is defined as “…a fully assembled computer system that includes a CPU, a motherboard, a power supply, an internally mounted NAND or revolving magnetic-based hard drive, and a case. For Server products, a hard drive and separate power supply are not required.”  That sounds decidedly physical, not virtual.

The system would generate an occasional message “TSC appears to be running slowly. Marking it as unstable.” This issue did not keep the PBX from running.

Voice quality was sometimes jittery. In a call, it would sometimes “cut out” (like a bad cell connection). In voicemail, even listening to the pre-recorded prompts, there was frequently some garbled-sounding audio.

Workarounds

The jittery voice seemed to be a horsepower issue. It’s like it couldn’t read the virtual disk fast enough and/or it couldn’t process the audio stream fast enough.

I was running PiaF under Microsoft Virtual Server 2005. The host was an old Dell Optiplex GX260, a Pentium 4 2GHz with 1GB of RAM. I dedicated 512MB of RAM to the virtual machine.

I found that if I moved the virtual machine to my IBM T60p laptop (T5300 Dual Core), still with 512MB of RAM, the jittery voice problem went away. I still got the “TSC” message, though.

Another workaround is to get Asterisk out of the audio stream altogether. I blogged this here. That solves the problem when two endpoints are talking, but it doesn’t solve the voicemail problem, and it does limit some of Asterisk’s features (call recording and transfer, for example).

Getting Physical

I concluded that I needed to convert to a physical setup to see if I could permanently solve these issues.

I hated to convert my fully-configured Windows XP machine to a dedicated PiaF box. I noticed in the PiaF status window that it was consistently using less than 40% of the 512MB RAM. Could it possibly run on the old Dell PowerEdge 2400 that has been sitting in the garage collecting dust? That’s a Pentium III 733 MHz with a whopping 256MB of RAM. Well, doesn’t hurt to try.

Following the instructions in my earlier posts, skipping the parts relating to virtual machines, I was able to get PiaF installed on the PE 2400 in a little over an hour (it took quite a while to compile Asterisk!). Transferring my setup from the virtual to the physical machine using the FreePBX Backup & Restore functionality was a snap. I forgot the tftp setup at first, but was able to quickly add that, simply copying the contents of /tftpboot from the virtual to the physical machine.

Somewhat to my astonishment, it works great! This old clunker, destined for the free Dell recycling at Staples, now has new life. Even without bridging the audio, the jittery voice is gone. I wouldn’t try this in a large office, but for a home office, it seems fine. In fact, my Kill A Watt tells me that this old PowerEdge only uses about 75 – 90 watts of power, not much more than the Optiplex workstation (53 – 80 watts).

Conclusions

I still love the idea of using a virtual machine for a PBX, and with a strong enough host, it’s probably a viable alternative for production. It’s definitely a great way to test various configurations and features. And if my physical machine fails, I can boot the virtual machine and have phone service again in about three minutes. But based on the hardware I already have and the Asterisk features I like, it looks like I’ll be running physical for now.

Virtualization

Some random notes: 

• Nick is moving from Virutal PC to VMWare for the dual-monitor support.
• Remember to turn on hardware virtualization support in your computer's BIOS. Some (like my Thinkpad) disable that by default. 
• A quad-core processor might help performance with multiple virtual machines.
• Use a separate disk drive for a differencing volume to reduce contention.
• You can clone a PC image, but use SysPrep or (XP only) NewSID to fix the SID in the clone.

Source Control

Several products were mentioned:  SourceGear Vault, PerForce, Subversion.

Bug Tracking

More products here:  FogBugz (integrates wtih PerForce), Trac, BugTrac, Mingle.