As virtualization adoption rises in the data center, the technology could greatly diminish the importance of Windows, Linux, and other general-purpose operating systems running applications on commodity servers.

That's the opinion of some vendors who attended the LinuxWorld conference in San Francisco this week. They predict the dominant role played by the OS today will vanish as companies choose to make better use of their servers' computing power by adding a virtualization layer capable of running many applications, each with their own mini-OS like program called a virtual machine.

Rising to the status of kingpin will be the hypervisor, the system program that provides the environment for running VMs. "We think this is pretty inevitable, because there's so much pain associated with the current technology," said Billy Marshall, founder and chief executive of rPath, which was founded in 2005 and provides the components that a software maker would embed in an application to run on hypervisors built by virtualization vendors, such as VMware or XenSource.

Marshall is talking about the pain caused by the growing complexity of general purpose operating systems. With millions of lines of code, today's platforms no longer offer the highest application performance or the reliability of virtualization, Mendel Rosenblum, chief scientist and co-founder of VMware, said during Thursday's keynote at LinuxWorld.

"If you have something this complex, it's also really hard to innovate," Rosenblum said. "This is the position that Microsoft is in, even with its team of engineers, which is huge. It's hard to do anything."

For virtualization to have a chance at toppling the OS from its current perch, a number of things have to happen, supporters say. Server vendors for one are going to have to ship a preinstalled hypervisor.

That trend may have already started. Dell Chief Technology Officer Kevin Kettler said Tuesday at LinuxWorld that the computer maker was experimenting with a hypervisor that would ship with its x86 servers in order to provide customers with a high-performing virtualization environment out of the box.

Secondly, independent software vendors would have to start writing applications to run on hypervisors, instead of just general-purpose operating systems, and, most importantly, companies would have to be willing to buy into the new architecture. "Customers would have to accept this the same way they learned to accept software as a service," Marshall said.

While taking a neutral stand on whether the industry should stay with the operating system or shift toward hypervisors, chipmakers Advance Micro Devices and Intel are certainly boosting adoption of the later by building virtualization technology in their microprocessors.

Whether the hypervisor replaces the OS or becomes a part of it, a scenario Microsoft and Linux distributors would probably favor, doesn't matter to AMD. "The processor needs some type of operating system to deliver the functionality of the processor to the application," Margaret Lewis, a director of product marketing at AMD, said. "How rich or poor the OS is isn't important to the processor. From our perspective, we just want a robust platform."

Nevertheless, running multiple applications on separate virtual machines needs certain hardware hooks in the processor to boost performance. In general, these hooks improve the allocation of resources, such as memory and processing power, found in today's multi-core chips.

Barcelona, codename for AMD's first quad-core chip, will include new technology called "rapid virtualization indexing," which provides faster memory management for virtual machines than current AMD technology, Lewis said. Barcelona is scheduled to ship later this year.

In 2009, AMD plans to offer input/output technology that offers more control in dedicating specific devices, such as a printer or graphics card, to each virtual machine.

In time, AMD expects solid improvements in virtualization performance, which is already at an acceptable level for most applications. "We're picking off the bottlenecks," Lewis said.

And that may be bad news for operating system vendors.