In our tests of Vmware's Infrastructure 3 Suite, the performance overhead of VMware ESX Server, while typically less than 10 percent, spiked as high as 20 percent. We're not saying running VMware will hobble your systems--hardware virtualization simplifies server consolidation, saving money and data-center real estate while cutting power usage, and double-digit performance hits can be avoided with proper planning. But there is an undeniable cost in terms of performance.

In tests at our Boston partner labs, we found the primary benefit of running ESX Server is, not surprisingly, making the most of hardware resources by letting existing servers run more apps. In fact, virtualization may be server vendors' worst nightmare: Gartner sees a slim possibility that the technology could reduce the compound annual growth rate for x86 servers to--get this--negative 0.6 percent by 2010. We expect savings in data center power as well.

Oddly enough, then, when we recently asked readers which tech buzzword they most despise, virtualization came in a strong second, just behind SOA. One in four respondents threatened bodily harm to the next salesperson who mentioned it, and 20 percent said they didn't realize expected benefits.

Still, virtualization is here to stay, and that's a good thing despite its apparent image problem. Whether you use Microsoft Virtual Server, VMware, Xen or another package, virtualization delivers a raft of benefits, from better use of physical assets to improved management of applications to the ability to divvy up resources across machines in a way that the sum of a virtual assigned resource--such as memory--exceeds actual physical memory. Using virtual machines may reduce physical server count by moving older applications off older hardware to newer systems that are less likely to experience hardware failure or, in the event of failure, have better parts availability. We found ways to boost the payoff from virtualization technologies, and our testing highlights which areas will suffer least from performance drags.

Mixed Bag

Virtual machines let hardware run multiple OS instances without compromising application availability and stability. In contrast, running multiple applications under a single instance of an OS can at the very least reduce availability, just through maintenance demands. At worst, multiple apps running on the same OS can cause instability, resulting in unexpected downtime.

Consolidation increases risk as well, of course. Running multiple VMs on a single physical server exposes more applications to hardware failure, compared with running a single application per server. But for many, it's a risk worth taking. Reducing overall server count can help control real-estate and cooling costs by having more servers in a smaller data center. But again, consolidation in a single physical space increases the risk to all applications running in that space if the data center experiences a power failure or fire--the number of applications affected will be greater than if a company had multiple data centers servicing the same number of applications. If you use virtualization to consolidate servers, factor in redundancy to minimize risk.

When it comes to systems management, virtualization provides a layer of mobility and flexibility that can help IT handle hardware maintenance, eliminate hardware as a factor in application support and better manage application performance. A virtualization environment like VMware's provides its own virtual hardware and BIOS on which OSs and applications are installed. This makes it much easier for admins to move an application to another system--apps are less susceptible to hardware compatibility issues because the virtual hardware will always be the same.

In practical terms, this simplifies routine maintenance. If a server must go offline, simply transfer all the VMs from that server to other servers running a virtualization environment. App availability isn't interrupted.

When it comes to getting technical support for an application, so long as the software vendor supports running its app in a virtual environment, hardware-related compatibility issues can be avoided, though not entirely eliminated because of the relationship among applications and consolidated storage devices, such as iSCSI and SAN subsystems. These still require specific drivers and support in the virtualization environment.

This virtual machine mobility also makes it easier for IT to migrate an application from one server to another, should performance become a problem. This capability helps guard against changes in application performance as user demand increases or diminishes over time. Say a given server is running four VMs, but user demand on an application running in one of those VMs increases to the point that the server has difficulty supporting the demand. IT can migrate that one application to a more robust system or migrate other VMs to other systems to free up resources.

The cost of this approach is the increased overhead of managing VM images. To have a truly portable environment, in which VMs can be easily moved from one system to another--or recovered in the event of a catastrophic hardware failure--companies must invest in specialized tools, such as Altiris' Deployment Solution. In addition, they must be mindful of keeping images of VMs up-to-date, and understand network and storage topologies so that when a VM is migrated to another server, it has efficient access to storage subsystems.

By creating virtual hardware instances, ESX Server lets IT create multiple instances of OSs on the same box. The tricky part is finding the right mix of applications to run in those VM instances so that users don't see a drop in application performance, all while squeezing the most out of your hardware. The biggest challenge we saw in our tests came when we tried to determine how much application performance on one virtual machine instance suffered when we added other VMs, each running their own applications, to the mix.

Although performance is going to vary by application and the kind of resources the app demands--disk versus memory versus processor--we can make some recommendations: Anticipate the overhead of virtualization. Don't expect virtual hardware resources to deliver much benefit when compared with real hardware. Group apps with different resource requirements on the same hardware. Use VMware's utilities that prioritize resource usage. Employ VMware's and third-party tools that make it easy to migrate VMs to different hardware.

Choose Wisely

Virtualization, by its very nature, is going to introduce overhead when comparing an application running natively on hardware with the same application running in a VM on that same hardware. In our tests this rate varied by application, but it was generally less than 10 percent, with lows of 6 percent and highs of 20 percent.

We ran a number of application-based tests on two Dell PowerEdge 2850 servers. Both systems were configured identically, with dual, dual-core Intel Xeon processors, 2 GB of RAM and a three-drive RAID array. The Xeon processors included Intel VT (Virtualization Technology), which provides microcode optimizations for CPU virtualization. We ran a baseline set of tests for three applications, Microsoft's Exchange Server 2003, SQL Server 2005 and IIS (Internet Information Services), using a combination of free and commercial tools, including Microsoft's Exchange Server 2003 Load Simulator (LoadSim) and SQLIOSim as well as Borland's SilkPerformer 2006 R2 (for more on our tests, see "Tester's Notes").

Our test environment differed from what enterprises would typically run in that we didn't have dedicated storage subsystems, such as a SAN (storage area network), for data storage for specific applications. A SAN doesn't just simplify data storage and data management for applications like Exchange, it also prevents the contention for resources that arises when the OS and application need access to virtual memory as opposed to needing access to disk for data storage. With a dedicated storage subsystem, that contention is minimized.

In terms of our results, the impact of running applications against the local drive subsystem would have little bearing on our tests of the overhead of VMware's VI3 on basic application performance. The overhead of ESX Server is confined to system memory and processing power. In tests where we ran multiple VMs simultaneously, the impact again depended on whether the applications required continuous access to disk.

In all likelihood, companies looking to use virtualization to consolidate servers will still see some contention for disk resources, regardless of whether data resides on the server or a SAN. Say you consolidate multiple file and print servers on a single box using VMs and store the files in the VM or store all the data from all the servers on a single SAN. In either case, the VMs will be accessing the data from the same shared resource.

The performance overhead of virtualization can be viewed against two basic assumptions: IT is going to use virtualization to fully utilize servers the company owns, or a company is looking to consolidate existing servers onto newer hardware. For the purposes of baseline testing, we used the former assumption of maximizing server utilization.

Under the full-utilization scenario, IT divides up memory, in theory because existing memory isn't being fully used. Say you notice that an application running on dedicated hardware uses only part of the memory and processing capacity of that server--perhaps 512 MB out of 2 GB and 25 percent of 2.8 GHz. Migrating this application to VMware's ESX Server lets you dedicate that 512 MB and 700 MHz to the application and still have memory and processing power available. Therefore, in our tests, we divided our 2 GB of memory based on the needs of the applications running on the VM instances. SQL Server requires a minimum 1 GB of RAM, for example, while Exchange Server 2003 will run with 512 MB of RAM.

VMware has memory-page sharing technology that lets IT overcommit physical memory for VMs. This memory-page sharing lets VI3 allocate at least twice as much virtual memory as there is physical memory when running the same OS on VMs running on the same hardware. This is useful in situations in which you want to consolidate a number of low-use apps, such as file and print services, running on individual servers. A single server could run a dozen VMs because the likelihood of all these systems requiring processing power and memory at the same time is small, and even when requests exceed resources, slowed performance wouldn't be an issue.

Architecturally, VMware's ESX Server 2.0 uses a stripped-down version of Linux to host VMs, then abstracts each VM's hardware through a common virtual BIOS and virtual hardware drivers. In our initial tests, we measured the performance costs of these two layers, but also factored in less memory being available to each virtual machine; ESX Server requires roughly 200 MB of memory.

The good news? VMware's done a good job creating an environment that will let apps run at a high level of performance without much of a penalty caused by generic hardware drivers and a Linux-based VM hosting environment. We were able to operate relatively demanding apps, such as Exchange Server 2003 and SQL Server 2005, with just a 6 percent performance penalty. Both the LoadSim test for Exchange (see "Virtualization Hit: Exchange" in the image gallery) and the SQLIOSim test (see "Virtualization Hit: Virtual Hardware" in the image gallery) for SQL Server have heavy disk read and write components, so our results illustrate that apps that rely on drive performance may not suffer too greatly, provided there isn't contention for disk access among VMs.

The area where we saw substantially slower performance was in our intranet performance test, which used SilkPerformer 2006 to test the performance of IIS. We saw performance drop off 18 percent in hits per second, and 20 percent in kilobits per second (see "<Virtualization Hit: Intranet" in the image gallery). Unlike the SQL Server and Exchange tests, here we saw the price paid for moving from a robust system with 2 GB of RAM to a VM with only 512 MB. With less memory, system performance suffered considerably more than with more disk-bound tests.

Adding Resources

We weren't surprised that we could improve performance by adding more memory to the VM used in our intranet application test. Increasing memory from 512 MB to 1 GB improved throughput by 10 percent and reduced the performance drop compared with the baseline system from 20 percent to 12 percent.

The lesson learned: Apps with high memory demands will benefit from allocating as much physical memory as possible to the VM. In our intranet application test, for example, when we compared results of running the test on dedicated hardware with 2 GB of RAM against a VM environment with 1 GB of RAM, we saw performance drop 12 percent. When we reduced the memory in the VM environment to 512 MB, performance dropped 20 percent against the baseline hardware environment.

As part of our tests, we investigated whether adding another virtual network adapter would improve system performance, but we saw no improvement. We likewise didn't see any better performance in our SQL Server tests when we increased system memory, because the test is disk-bound.

While manipulating available resources made a difference in testing a single VM instance, VMware's resource-allocation capabilities give admins a way to allocate only a finite amount of CPUs, CPU cycles and memory.

Our baseline test hardware included two dual-core 64-bit Xeon processors running at 2.7 GHz. ESX Server let us allocate this only in terms of 32-bit processing power, meaning that while ESX Server could execute 64-bit code to manage VM environments, the VMs could run only 32-bit OSs and applications. To be clear, ESX Server does support 64-bit OSs, so you can deploy 64-bit apps under VMware. In our tests of 32-bit apps running on 32-bit versions of Windows Server 2003, however, ESX Server let us allocate processing only in terms of 32-bit processing power. When we tried installing the 64-bit version of Windows Server 2003, Windows identified our virtual processor as a 32-bit processor.

Furthermore, available resources included a representation of actual CPU MHz. In theory, we had cycles we could divide up across VMs. The number of processors is more flexible: We could create machines with more than four processors that would correspond to the four theoretical processors available in two dual-core processors.

Although we could add network cards to VMs, ESX Server doesn't support dynamically allocating drive space to a system except through a connected storage device. From a practical standpoint, this was the most problematic area we encountered in our tests. The size of a VM image, which includes disk space, could be the determinant factor when it comes to deciding the number of VMs that can run on a given system. ESX Server, by default, allocates 4 GB of disk space for a VM. On our test system, we allocated a minimum of 10 GB, just to ensure we had enough room to install apps and to allow for the impact of patches over time. On our system, with 128 GB of drive space, we had enough space for 10 VMs.

Furthermore, as app loads shift, performance characteristics change as well, and you may find it necessary to remove a VM instance from a given server. Even with that instance gone, the newly freed drive space can't be reallocated to the VMs that have increased performance demands. Unused space will sit empty on the server.

Based on our test results, it's fair to say you shouldn't expect excellent application performance when running multiple disk-bound applications on the same server, unless each application's data storage is off-loaded to a dedicated storage subsystem, such as a SAN. For example, we paired VMs running Microsoft's Exchange Server 2003 and SQL Server 2005 on the same hardware and saw dramatically different results when running tests that had both applications stressing the drive subsystem versus tests where Exchange stressed the disk subsystem and SQL Server ran largely in memory.

Bottom line, think about storage if you plan to dynamically move VMs around. You could remove a VM because an application is no longer supported, but migrating another application to that server might not be an option if the image is too big.

Balancing Act

The big question IT will likely face when moving to ESX Server: How to deploy critical apps virtually without sacrificing too much performance. Fortunately, ESX Server has tools that let you do just that--prioritize application performance by segmenting hardware resources for one VM over another. In our tests we saw that this capability can help deliver on service-level requirements.

We used SQLIOSim to measure performance of prioritized and non-prioritized loads, while using LoadSim to generate competing load. In these tests we used both the default SQLIOSim configuration, as well as three other configuration files that operated in system memory.

Initially we measured performance with the two VMs splitting the available 4.7 GHz of processing power. In this configuration, in the memory intensive test, performance dipped between 4 percent and roughly 7 percent. We then allocated 3 GHz of processing power to the SQLIOSim system and the remaining 1.7 GHz to the LoadSim test system. Performance swung back to the level we saw when running SQLIOSim without any contention for resources.

ESX Server uses the same interface and conventions for allocating CPU performance and system memory through the Virtual Infrastructure Client. We could set CPU performance as a percent of available MHz or as a set number of MHz, as well as set the number of processors available. When it came to setting memory, we could allocate a percent of available memory or a specific number of megabytes to a given VM.

Plan Well

Although VMware provides tools to help manage application performance, they can't replace the human touch. Dedicate time to assess performance profiles before a migration to ESX Server, ensuring users see comparable performance after the migration. Look not only at how many available resources a given application requires, but also at peak performance profiles.

A natural pairing may be putting a VM running an e-mail server and one running file and print services on the same box. But if a significant number of users bring mail offline at the end of the day, print large documents to read during the commute home and use offline folders as simple backup, those VMs could slow substantially at the end of the day as they compete for resources.

The Virtual Infrastructure Client used to manage VM images has a set of tools for monitoring VM performance; the one thing missing is the ability to adaptively set resources based on demand thresholds. That way, if the CPU utilization of a given VM exceeds a certain level, the ESX Server could allocate more resource from less critical VMs automatically.

One challenge you'll face with VMs is managing the changing demands users put on a given application. Some apps will become more important and experience increased load over time, while others lose importance and see load decrease. This plays into one of VMware Virtual Infrastructure 3's strengths: the ability to migrate VMs to different hardware as requirements change.

If one VM is impacting the performance of another, we found that you can, with relative ease, migrate the VM to another server. Third-party tools, such as those from Altiris, also make it easy to create images of VMs to simplify adding more computing resources as necessary.

Ultimately, virtualization using the ESX Server in VMware's Virtual Infrastructure 3 will help maximize the use of server hardware, provided you take the time to analyze application performance. Use VMs to combine applications with different performance profiles on single boxes so they don't compete for scarce resources. The performance area that will be the biggest bottleneck for VMs is disk, so use technologies such as iSCSI and SANs to help shift file storage requests off the VM hardware onto devices dedicated to a given application.

Expect to take at least a 6 percent to 10 percent performance hit just for the ESX Server environment. Potentially, performance could suffer as much as 20 percent. And, plan on additional VMs to consume an additional 6 percent to 10 percent of system performance, even if the applications in those virtual machines are little used.

Tester's Notes

We used Microsoft's Exchange Server 2003 Load Simulator (LoadSim) test to generate client load against the Exchange server and Microsoft's SQLIOSim. In these tests, we saw that the background load of an Exchange server as created by LoadSim could slow down SQL Server performance, as measured with SQLIOSim, by as much as 29 percent (see "Virtualization Hit: Virtual Hardware," page 49). During the same tests we saw Exchange server performance drop by 16 percent (see "Virtualization Hit: Exchange," page 52). These performance drops are versus the same machine running the same application solo.

It's worth noting that the Exchange Server performance results indicate the impact of SQL Server load during a fraction of the time that LoadSim is running. LoadSim is a benchmarking tool that simulates a constant and consistent performance load of MAPI clients for a set length of time, such as two hours. In contrast, the SQLIOSim utility simulates SQL Server activity on a disk subsystem, and SQLIOSim tests run on the server for as long as they take to complete.

This means that while we ran LoadSim for an hour, the 16 percent decrease in performance can be attributed to the length of time needed for the SQLIOSim default configuration test to run--10 minutes and 22 seconds. LoadSim performs a consistent set of tasks over and over again in the course of a run, such as logging users on and off, creating messages in a folder, deleting messages, and creating calendar items. Because we needed to run SQLIOSim manually from a different system, the test runs didn't begin or end while the same LoadSim tasks were being performed. Despite this, in five test runs, we saw consistent results: SQLIOSim runs finished within a 7-second range, and LoadSim finished within a 15-millisecond range.

As part of the SQLIOSim tests, we added an additional virtual machine instance, this time running only Windows Server 2003 and IIS, without any tests running. In this configuration the performance results of the SQLIOSim test dropped 129 percent. That is, while the SQLIOSim test completed in 7:59 without any other virtual machines running, the load associated with LoadSim and a relatively inactive virtual machine running during the SQLIOSim test stretched the time required to complete the test to 18:24.

In tests of Exchange Server 2003 running on a server with multiple virtual machines, we saw a fairly small decline in application performance. For example, when we added another virtual machine running IIS to the test scenario of SQLIOSim and LoadSim, performance declined roughly 6 percent. We also saw a 6 percent performance decline when we ran LoadSim on a system that included a virtual machine running SQL Server 2005, but not performing any SQLIOSim tests.

Odds And Ends

We noticed a number of small performance problems during our tests that you'll want to be aware of, both when assessing performance and when managing a virtual machine environment.

»During testing, rely on a stopwatch rather than the system clock because a virtual machine doesn't necessarily display accurate time. This issue isn't likely to break applications, but in tests where a difference of five to 10 seconds could matter, watching the system clock won't cut it. When running the SQLIOSim test, for example, we could see the system clock in the lower right corner of the screen catch up to real time as we timed tests manually.

»Be mindful of sensitivity to network performance when managing applications running on virtual machines. Although VMware has done a good job optimizing video performance of the Virtual Infrastructure Client's remote console tool, we noticed that performance could hamper relatively simple tasks substantially.

With the LoadSim test, after every run we needed to stop the storage subsystem, remove temporary user accounts, delete log files and remount the Exchange mail-store folders. On the baseline system these tasks took about 10 minutes, but on the virtual machine it could take as long as twenty minutes because of waiting for the console to refresh.

»Allocate enough time for deployment. We found the basics of setting up virtual machines equally painful to endure. Management tools that let you image virtual machines for rapid setup and deployment are money well spent. Licensing systems-management software for automating tasks on virtual machines, such as installing applications and updates, will also pay off quickly.

Michael Caton is a freelance writer with 18 years experience evaluating technology products for technology buyers at large organizations. Most recently he has been reviewing CRM and messaging and collaboration products. Write to him at mcaton@nwc.com. .

Go to the Image Gallery for this feature >>