May 1, 2007

There is no question that virtualization has captured the attention of enterprises of all shapes and sizes. And it's easy to understand why - the benefits are simply undeniable. Who wouldn't be interested in lower TCO, better resource utilization, improved reliability, increased flexibility, and rapid deployment - among other gains?

One of the biggest newsmakers in virtualization has been the open source Xen project, and for good reason. The technology is very well designed, is extraordinarily fast and scalable, and is supported by EVERY major OS, server, and silicon vendor - even Microsoft. But as Xensource CTO Simon Crosby says, "the Xen hypervisor is an engine, and not a car. A powerful engine that needs a great gearbox, shocks, tires and the body to go with it."

And that's where the vendors and the open source community come in. There are several solutions that use Xen virtualization as a base, but add functionality to it's core capabilities through management tools and enhancements. These tools and enhancements are not required - Xen is completely functional (arguably more so) on its own - they simply make it easier to manage and use. We've been using the open source version of Xen in production since July 2006 - longer if you include testing and evaluation - so we have intimate knowledge of what the "engine" can do. In recent months, however, we have had occasion to evaluate two models of the "car."

Red Hat Enterprise Linux 5

We've been a Red Hat shop for some time, so naturally we've been tracking what Red Hat has been up to for months, through a couple of Fedora releases as well as RHEL5 (short for Red Hat Enterprise Linux version 5.) In fact, we are presently running the majority of our production servers as virtual machines using Xen on Red Hat.

Red Hat's approach to implementing Xen has been fairly open, which is good if you want to utilize the full range of Xen's capabilities. Since RHEL5 is a general purpose operating system, you gain all the benefits that go along with it, such as broad hardware and software/systems support. This is particularly important when it comes to choosing a storage mechanism for your virtualized infrastructure. Running Xen on RHEL5 affords you access to any storage mechanism supported by the OS, including multipath, CLVM, raw partitions, GFS, OCFS, fiber channel or iSCSI SAN, AOE (ATA-Over-Ethernet,) and hundreds more. From basic files to highly redundant, clustered storage systems, a world of options is at your disposal.

Enabling virtualization is as simple as checking the "Virtualization" checkbox in the installer, and clicking install - it has truly become an integrated part of the OS. When the installation is complete, the system will have a variety of tools to manage the virtualized server, including the standard open source Xen command line tools, a variety virtualization management libraries and scripts, and Red Hat's GUI management console, virt-manager.

In an effort to "future proof" their tools, Red Hat's approach has been to create a virtual machine management library called libvirt, on top of which all the rest of the tools sit. The goal of this library is to provide a common interface to any current or future underlying virtualization technology through the use of simple plugins. These plugins enable libvirt to monitor and control all aspects of the underlying virtual architecture. For example, should plugins be written for VMWare or the up and coming KVM (Kernel-based Virtual Machine,) Red Hat's management tools (which all use libvirt) would be able to manage them just like any other virtual machine. In fact, Fedora Core 7 Test 4 is already using libvirt to manage KVM as well as Xen.


Red Hat has received some criticism for this approach, as it does not comply with DMTF CIM. While this concern is somewhat valid, DMTF CIM is not as extensible and does not, at present, provide access to many features of virtualization engines, which would dramatically reduce the available feature set in current and future management tools.

The GUI tool, virt-manager, is really quite solid, offering easy creation and management of virtual machines. Simple wizards guide you through the process of creating virtual machines, defining their storage mechanisms, establishing memory and processor utilization, and the like. The management tools allow you to track the state and utilization of the virtual machines, view and control their virtual screens, start, stop, pause, save, etc., etc. Overall it offers a strong set of tools for managing your virtualized system.


Click {{file:630}} for a larger view.

Of course, Red Hat's implementation isn't perfect. First and foremost, while it is capable of virtualizing unmodified OSes (such as Windows,) this release clearly focuses on Linux virtualization. Running a high I/O Windows server in the near term on RHEL5 is not recommended, as this version lacks optimized Windows drivers, which have a dramatic impact on storage and network performance. Expect a 25-35% performance hit, should you choose to do so. I believe that these optimized Windows drivers (or "paravirtualized" drivers) are on the road map for RHEL, but did not make it into this release. Long term, of course, this should not be an issue, as Microsoft has partnered with both XenSource and Novell to make sure that Longhorn (ie Vista) server will be Xen compatible.

In addition, while the full suite of open-source Xen tools are included, which enable such advanced features as live migrations and hot-add of processors and memory, these higher end features are not yet integrated into the GUI tools, so a solid command of Linux and the command line will be required to utilize them. Obviously, this could add some management complexity for those who want more fine grained control of the Xen engine.

XenEnterprise 3.2

Xensource is the commercial arm of the Xen open source project, which offers several "versions" of their tools and options for support. These "versions" are actually all the same software, they just use different license codes to enable more advanced features. Naturally, their close association and control over the Xen project offers them something of an advantage when producing tools, as they can technically steer the engine in the direction of the tools they are producing. However, since the engine is open source, the tool providers all benefit.

As their "10 minutes to Xen" tagline suggests, easing the complexity of virtualization and management are clear goals with these packages, and they really do hit a home run in that regard. Simply boot a server class machine with the Xensource CD, answer a few simple questions, and you'll have a running Xen host in a matter of minutes. All that's left is to install the Java based management console on a Windows or Linux host, and you are ready to run (actually, the console also works great on MacOS X as well, as you'll notice in the screenshots, but since it requires JRE 1.6 you'll have to download the beta from Apple's developer connection site, and set up a launcher to get it to work, which is somewhat non-trivial for someone unfamiliar with Apple Java quirks.)

The Xen console really is a thing of beauty. It can manage multiple Xen hosts, using a simple tree view at the top of the screen which displays overall CPU, memory, and I/O statistics for the host. Clicking the expansion arrow next to any host unfolds to display a list of virtual machines, with the same statistics for each. In the bottom half of the screen, you have a tabbed browser that allows you to view/set configuration options for the currently selected virtual machine, access their graphical or text consoles, view performance statistics, start, stop, reboot, and suspend. In addition, they offer handy options such as cloning of a virtual machine, as well as exporting a complete virtual machine, which allows it's entire configuration and storage to be copied/moved to another Xen(source) host.


Click {{file:632}} for a larger view

The Xen host/Xen console combination are particularly easy to use when setting up a new virtual machine. The system automatically slices and dices your available storage, which they refer to as your storage repository. The storage repository uses Linux LVM to create partitions, which allows for dynamic virtual disk sizes and provides better performance than traditional, file backed storage. Adding disk space to a virtual disk is as simple as changing its size in the console - all the complexity of dynamic partitions is completely handled by the software.

Xensource's software is quite a bit more Windows friendly than RHEL, as all versions include paravirtualized Windows drivers for disk and network I/O. Xensource based systems are able to achieve near bare-metal performance when running Win2K, XP, and/or Win2K3, as well as a variety of Linux versions. In addition to basic virtualization, the higher-end XenEnterprise license also gives you the ability to prioritize CPU and I/O utilization by virtual machine and to isolate machines by VLAN, which are nice additions to the system's functionality.

There are a few caveats to using XenEnterprise, one of which is that in order to install RHEL5 or SuSE Linux, at least one of your server class machines must have a virtualization aware processor, ie Intel VT or AMD-V (so that you can create system images on it, which you can later copy to any machines that do not contain such a processor.) This is required by the Xen engine for Windows compatibility, but is not required for Linux, so there really should be a better install template in XenEnterprise in my view.

One of the bigger limitations, especially for larger environments, is the storage repository. While it sounds cool and easy, it can be a real pain since a XenEnterprise host can only have one of them, and it can't be shared. This prevents you from, say, placing some of your VMs on internal storage while placing others on a iSCSI or fiber channel SAN - it's more of an either/or proposition. Since you can't attach multiple repositories, the only way to move virtual machines from host to host is to export them from one, and import them to another, which can be quite time consuming and results in down time. As you might imagine, this could be a particular problem in the event of a host failure in a blade environment, for example. Obviously, this also means no live migrations of virtual machines from XenEnterprise host to XenEnterprise host, which can really limit flexibility. Luckily, the next release of XenEnterprise, due by the end of the year, will support shared storage and live migrations. Still, it would be nice if it was as flexible as RHEL, allowing a user to bring whatever storage they might want.

One might say, "but the Xen engine is the same, regardless of the 'car,' so why can't you just work around the problems?" That would seem to make sense, until you discover that the Xensource version of the Xen engine is highly tweaked, so you can't work around any of the above limitations by using standard Xen tools - they aren't all there. And you can't easily transfer XenEnterprise virtual machines from a XenEnterprise host to another Xen based system, such as RHEL5, since their settings and configuration won't match up (it can be done, but it is extremely non-trivial - read painful.)

Finally, while a XenEnterprise host does support a wide-array of hardware and software, probably more than even VMWare out of the box, it doesn't hold a candle to RHEL's capabilities. Drivers for just about anything can be had for RHEL5, and most servers come with RHEL compatible management agents, which typically won't install on a XenEnterprise host (at least, not without a bunch of the aforementioned pain and suffering.) This can be especially limiting when using highly redundant hardware with multipathing or specialized drivers - neither will be supported on the Xen host (although the multipath daemons and tools are actually there - they just aren't supported.)

Limitations aside XenEnterprise is extremely powerful and easy to use, and its Windows support and performance is truly exceptional. While RHEL is far more flexible in a Linux environment, XenEnterprise is a well rounded, high-performance solution for general virtualization needs.


There is no question that both RHEL5 and XenEnterprise represent significant milestones in the maturation of Xen based virtualization in the enterprise. Which one you will choose for a specific task will depend largely on your available skill set and your open source acumen. RHEL5 will give you access to the Xen's full feature set, including 64bit, live migration, shared storage, and a wealth of OS flexibility, and it's standard Xen base is certainly more comfortable, since machines will certainly be portable to other Xen based platforms in the future. But this flexibility will come at a cost of a steeper learning curve. XenEnterprise, on the other hand, offers drop-in ease of use and rapid implementation, yet with less flexibility, a bit of vendor lock-in, and arguably less scalability (at present.) Ease of use, however, is a powerful thing, and a rapid track to virtualization can certainly revolutionize a data center.

FYI - for those of you who are wondering what Saugus uses, the answer is both. We use XenEnterprise and RHEL+Xen on HP Blades - XenEnterprise for our Windows needs, and RHEL for all of our Linux. In our view the RHEL based is far more flexible, but XenEnterprise sure is great for Windows, and man is it easy to use. Long term, we think RHEL will only get better, and we believe they and the open-source community have the opportunity to give Xensource a real run for their money. I'm sure Xensource won't rest on their laurels either - competition is good and will only result in better virtualization for all of us. The future sure looks bright for Xen based virtualization...

*Note: this post was updated on 5/10/2007 to remove a prior assertion that an Intel VT or AMD-V processor is required for XenEnterprise. While not required for XenEnterprise as a whole, at least one system with an Intel VT or AMD-V processor is required to install RHEL5 or SuSE Linux on XenEnterprise.


Post a Comment
  1. Hi Jim,I came to you blog linked from the EdTech listserv.

    I'm curious, how do you determine if you are going to run multiple services (ie DNS, DHCP, LDAP) on one single server or create, in this case 3 virtual servers for each service? Thanks for the breakdown!

  2. It's really all about load, system requirements, and reliability. In the case you listed above, for example, I would need to determine if a single vm could handle all three services, because I would want save the memory, disk space, and processing power that would be required to separate them into multiple OSes. If I believe that they can be run together, then I would need to decide if I'm OK with losing all three at once, should there be an OS problem of some sort on the one virtual server. If I didn't feel I could combine them because, say one of the services is a particularly high load application, I would probably separate it from the others on its own machine, and pin it to a different processor, essentially divvying up my hardware resources among my virtual machines. In the end, it's something of a judgment call, but all these factors would affect the decision.

  3. Jim --The answer to why XenEnteprise requires hardware assisted virtualization to run Linux is: it doesn't. An all-Linux server will run just fine without VT or AMD-V, and that's what the documentation says. (You need hardware assist to install a new RHEL 5 or SLES 10 guest directly from media, because you essentially install them using hardware assist and then switch to a paravirtualized kernel. But once you have a copy installed and paravirtualized, you can clone it, copy it, etc. and run it on a non-VT/AMD-V system just fine.)

  4. I'm sure you are right, however if you run the installer on a system without VT or AMD-V (or with it disabled) the installer hits you with an alert that makes it sound like the system won't work at all without it. The error message really should be re-written in a much less fatalistic fashion. Then again, the non-standard Xen configuration coupled with the requirement that you have VT/V to install current versions of RHEL or SuSE pretty much make it a de facto requirement - wouldn't you agree? After all, you do have to have at least one VT/V enabled system to create the VMs for any non-VT/V systems you might have.

  5. Please forgive my memory, I don't remember the exact wording - just that it sounded like the system would be severely limited if I didn't have V or VT. At the time, I was installing XenEnterprise on a blade, and had neglected to enable V in the bios. I'll have the opportunity to install it again (on another blade) in a few days and will intentionally disable V in the bios so that I can see the message again and get back to you.

  6. We have people who are generating virtual machines on one HAV (hardware-assisted virtualization, i.e., VT/AMD-V) box and running them on other non-HAV boxes, as well as users building downloadable distro appliances. But it's still a good idea to have at least one.Does the warning seem that strong to you? It explicitly says you won't be able to run *Windows* guests...

  7. Jim,You stated using VM requires much higher levels of hardware reliability. You even mentioned redundant hardware.What about transient software errors? An errant hypervisor would be disastrous.

  8. I imagine that such a problem would certainly wreak havoc, although I personally have not experienced it. Obviously, you'd want to run a full battery of tests on the system before going into production, and allow for time to develop the processes and practice of managing a virtualized environment. It's not unlike the deployment of any system - you set up a test lab, run pilots involving a steadily increasing user base, and gradually roll out services and increase workloads. This would certainly allow for time to discover and resolve any software/hardware issues that may arise.

  9. Thanks for the informative run-down. I'm writing an article on this now, and it's really helped.

Post a Comment