KVM vs. VMware: A Case Study
After a month of debate and experimentation, my employer has made the decision to use the open-source KVM virtualization infrastructure for migrating IT resources to a virtualized environment. Below, I discuss why we chose KVM over its (mostly proprietary) alternatives.
Until a couple years ago, the open-source community offered no real contender in the virtualization market. True, the qemu project has been around for a while, but qemu remains too inefficient for most production environments. Xen has also existed since 2003, but for a long time it only supported a limited set of guest operating systems, which did not include Windows until late 2005. As a result, proprietary virtualization products like VMware enjoyed a near-monopoly in the enterprise market until quite recently.
The rapid maturation of KVM, or ‘kernel-based virtual machine’, over the course of the last couple of years constituted the first open-source challenge to VMware. Integrated into the Linux kernel, KVM provides feature-rich and highly efficient virtualization.
My colleagues and I tested KVM (running on an Ubuntu 8.04 host) and ‘barebones’ VMware ESX server over the last several weeks. Ultimately, we decided KVM was a better fit for our needs based on the following considerations:
- Cost. Although our virtualization requirements are minimal–we need to run only two guest servers on a single host machine–VMware would have cost an astonishing amount of money. With features like VMware motion factored in, we were looking at a huge hit to the budget–and it didn’t help that VMware charges per CPU, not machine, regardless of whether all CPUs will actually be dedicated to virtualization. KVM is totally free, in both senses of the word, and offers functionality equivalent to VMware motion.
- Ease of deployment. Installing KVM on Ubuntu 8.04 is as simple as an apt-get. ESX server is also easy enough to install, but having to deal with licensing adds another layer of complication that we’d prefer not to face. KVM, of course, requires no license.
- Speed. Although I don’t have hard numbers, KVM-based virtual machines definitely ‘felt’ more responsive than those running on VMware. Our experience seemed to confirm Red Hat’s claim last fall that KVM can support five VMs for every three running on VMware on the same piece of hardware. It was also troubling that ESX server wasted upwards of 500 megabytes of memory–without any VMs running–on system overhead, while an Ubuntu server is a considerably more efficient host.
- Management. Apparently it never occured to VMware that systems administrators might be running Linux on their workstations. As a result, Windows is the only platform on which VMware’s graphical management infrastructure is supported. A Linux CLI client is available, but I’d like more options than that. KVM, in contrast, can be managed via the command line, via graphical interfaces (running either on the local machine or forwarded over ssh to a remote workstation) or through the Enomaly web interface.
Perhaps the single major downside of KVM is that it requires a bit more technical know-how than VMware to deploy effectively, since some features can only be configured via manual hacking of XML files. But as KVM and related tools continue to mature, expect that to change.
For IT staff interested in zero-cost, Linux-friendly, feature-rich and resource-efficient virtualization, KVM has become the way to go. If VMware wants to compete, it needs to rise from the laurels of its crumbling monopoly by innovating and lowering costs.
WorksWithU is updated multiple times per week. Don’t miss a single post. Sign up for our RSS and Twitter feeds (available now) and newsletter (coming in 2009).
WorksWithU on Identi.ca
WorksWithU on Twitter
WorksWithU's RSS Feed
Solaris Zones, Sun xVM? Free and open source.
I’m finding good documentation on using the KVM with Ubuntu so I may just give it a shot. I like the lower-resource requirements but I can tell it’s going to be a learning curve.
Want to see what’s coming next for KVM just look here:
http://docs.fedoraproject.org/release-notes/f11preview/en-US/ar01s05s03.html
Improved VNC Authentication for Virtual Machine Management
Improved Graphical Console for Virtual Machines
KVM PCI Device Assignment
KVM and QEMU merge
SVirt Mandatory Access Control
Ubuntu will most likely pick these improvements up by the next release. Well maybe not the SVirt feature as its selinux based and Ubuntu supports apparmor by default. I wonder if Ubuntu will find a way to write the equivalent apparmor policy that provides the same protection as SVirt will on a KVM server running multiple guests.
-jef
The second to last paragraph and number 2 in your list seem to contradict one another.. Was it specifically because you didn’t want to deal with the licensing that you gave VMware the disadvantage there? I’m most of the way through a VMware deployment and will admit that the licensing is as restricting as a Windows environment, but I’m also convinced that a 5 year-old could setup a new virtual machine in VMware.
Thanks for the Enomaly mention!
KVM has done really well for us in dev and production environments. It’s a heck of a lot easier to work with compared to Xen
Personally I think it’s a good thing KVM needs a bit of “know how” to get certain things working. It means the administrators involved will need to actually know what they’re doing. Something that is increasingly lacking as ICT becomes a drop-out point and click occupation.
My employers are currently implementing a new document scanning solution. Which is being provided by EMC (http://uk.emc.com/) who are clearly a Microsoft shop. There will be two scanning stations attached to Kodak scanners and eight validation/committing stations. The legacy system from Vignette will be maintained with one scanner and two committing stations for a while to come yet.
Instead of deploying a fresh drive image on each system, the guys from EMC in partnership with our own administrators have spent the last three months trying to install EMCs software onto the PCs. One PC at a time. Each of those PCs had the previous application from Vignette which seems to conflict with just about anything that does the same job.
To make matter worse, the document imaging department doesn’t get new kit very often. All of our PCs are hand-me-downs from other parts of the business. Granted the hardware is all the same. More or less. But each system is configured slightly differently. Clearly leveling the playing field and having each system configured exactly the same to begin with would be a much better approach.
It is entertaining to watch though.
But back to my point. Being forced to muck around in the guts of the system means you’ll be forced to learn and understand how it works. Which can only be a good thing.
I’m surprised OpenVZ isn’t even mentioned
http://openvz.org
[...] KVM vs. VMware: A Case Study | WorksWithU (tags: kvm vmware esx virtualization hypervisor comparison review ubuntu) This was written by andy. Posted on Wednesday, April 29, 2009, at 1:35 am. Filed under Delicious. Bookmark the permalink. Follow comments here with the RSS feed. Post a comment or leave a trackback. [...]
[...] KVM vs. VMware: A Case Study After a month of debate and experimentation, my employer has made the decision to use the open-source KVM virtualization infrastructure for migrating IT resources to a virtualized environment. Below, I discuss why we chose KVM over its (mostly proprietary) alternatives. [...]
Wyatt: what I meant to say (and could have put more clearly) is that KVM and VMware are equally easy to install, but in order to get VMware actually running with all the features we wanted required jumping through several sets of bureaucratic hoops. I think it’s a definite plus for KVM that none of the licensing nonsense applies there.
On the other hand, configuring advanced features (like bridged networking) in KVM requires familiarity with command-line tools and editing configuration files manually, while VMware, as you say, makes most things easy enough for a five-year-old (KVM is also simple if you stick to the basic features, however). But I also agree with aikiwolfie’s comment that there’s something to be said for forcing your system adminstrators to learn something about what’s actually going on in the background when they launch virtual machines, rather than teaching them unimaginatively to point and click their way through a GUI and hope nothing goes wrong.
Pat: I was blown away by how easy Enomaly was to install and how many features it offers at a low performance cost to the server. It’s proving especially useful for the Linux-phobic people in the office who need to manage Windows VMs but don’t like having to ssh or VNC into the Ubuntu host to do so. Keep up the great work.
Lyle: the last I checked, OpenVZ still doesn’t support Windows guests, which ruled it out for our purposes. But it may be useful for others; thanks for giving it the mention that I didn’t.
As a management platform for KVM, you could try OpenNebula. OpenNebula is a virtualization management tool, that provides you with an uniform view of pool of distributed hypervisors, and takes care of the VM life-cycle. OpenNebula is a sys-admin tool that can be easily integrated with the network and storage solutions already deployed in your data-center or site. It may also allocate VM’s in external clouds (Amazon EC2, and Elastic Hosts…)
OpenNebula (www.opennebula.org) is included in Ubuntu 9.04 if you want to give it a try. A quick guide can be also found here:
https://help.ubuntu.com/9.04/serverguide/C/opennebula.html
[...] The article appeared on WorksWithU may be the first documented case of a company deciding to adopt KVM instead of VMware or Citrix hypervisors. [...]
I’m not sure I understand why VMware would have cost “an astonishing amount of money” and talk about Vmotion when you then say you’re talking about 1 host machine with 2 guests on it? Vmotion isn’t relevant for a single machine installation?
Also, you talk about KVM as being the only open-source challenger to VMware, when Xen is quite a bit more mature than KVM at this point?
This is a poor article, as others have said before me.
For your tiny architecture of 2 guests/1 host you can use VMware Server or ESXi – both are *free*
I hope others reading this will smell your Linux/Open Source religious zeal poking through these flimsy arguments.
Cost: you can choose from two free VMware products
Ease of use: it takes minutes to download and install, or ESXi comes built in!
Speed: if you don’t have date, all you have is conjecture. Your “gut feel” is probably indigestion from all that Linux/Open Source kool-aid you have imbibed.
Management: you can manage VMware products vi CLI, Web and Client.
Ewan and Steve: ESXi and VMware Server are both free to install–maybe I wasn’t clear enough about that–but with VMotion and support costs factored in, they become quite expensive. My employer wants VMotion functionality so that the virtual machines can be automatically migrated to a backup server in case of failure, and wants a support contract for peace of mind. KVM support is included with the Canonical support contract, which is much cheaper than most of VMware’s offerings.
Chris:
So your employer has purchased Canonical server support contracts?
-jef
Jef: we’re in the middle of finalizing a Canonical server support agreement now. Expect to see more about our experience in coming months.
I think Chris has a very valid point. Don’t let the hater-aid hold you back from embracing open source technology to lower your initial investment costs, and TCO.
Another “open source” alternative, though not as mature as KVM perhaps, is OpenVZ.
Good read.
Fellow Open source Zealot
linuxcrypt
Chris:
Finally! someone actually willing to go on record saying they are purchasing support services directly from Canonical. Its very important for the long term health of Ubuntu that people actually value and purchase the Canonical services..or services from partners with a revenue sharing agreement with Canonical. Since you are very interesting in virtualization, see if you can get Rightscale to talk about how their partnership with Canonical works. Does Canonical derive revenue from the Rightscale’s Ubuntu platform service subscriptions to users?
Ubuntu is only going to be sustainable if and only if Canonical’s business is sustainable. Canonical customers need to be doing testimonials to build interest in Canonical services, and the much larger non-paying Ubuntu supporters need to encourage Canonical customers to speak up and start talking about their experience as a paying customer.
-jef
I think KVM still lack many features for most organisation in comparaison to VMware. DRS, HA are a few that are really useful.
But, If you dont need those features, KVM is a great solution.
Christopher,
Besides the confusion of “proprietary”, “open source”, “license” and “free”, your article is not technical sustainable.
You mix products, objectives and functionalities with license administration, which it’s not as hard to administer as you mention.
I’ve personally tried many of these, and found:
ESXi has better performance than “Ubuntu + KVM”, the issue here is hardware.
Xenserver 5 has better performance than “Ubuntu + KVM”, the issue here is hardware.
VMware Server 1.x or 2.x, on top of Ubuntu is not as good as “Ubuntu + KVM”, that is true.
But “Ubuntu + KVM” it’s very similar as VirtualBox 2.x series.
“Ubuntu + KVM” network configuration, for bridge networking, requires a savy-tech linux guy, as they mention earlier. The other products doesn’t need any special network configuration.
Backup and VM movement:
You can backup any of these solutions, just copying the vm files, the way you like/know better. Only ESXi needs a tweest to do that.
XenServer lets you administer more than one server on the same console, and move VMs too (and hw). VMware Infraestructure Client only allows you to administer one ESXi server, and the vm movement is not possible from the console (only if you have VMotion licenses and hw).
There are 3rd party and free tools to do this, too.
Support:
You have VM Communities and Xen.Org to ask anything you need, at no cost, just as in KVM.
I don’t think that Canonical KVM Support could match VMware or Citrix support, nor give your boss the peace he/she needs, speaking of virtualization -of course-.
So, I hope that in the future we read about Canonical Support “goodness” for facts and solutions they provide, not just for opinions.
Eugenio
@Eugenio
The question of value for money when it comes to support services is a very interesting one. It will be interesting to see how Canonical compares to Red Hat in that space over the next year. Red Hat’s purchase of Qumranet makes them the the defacto leader in KVM technical expertise. And that is the sort of technical expertise that people are willing to pay for as part of a support contract. Does Canonical even have the necessary technical staff to deal with critical kvm or xen issues? From a value for money proposition, how does Canonical stack up against Red Hat in virtualization support services? Where is Canonical’s technical expertise when it comes to virtualization? What can Canonical do for you exactly as a paying customer? I haven’t seen Canonical try to make its case yet as a service provider.
-jef
Moderator: can you please delete my previous post and replace by this one
(corrected some typos and remove angle brackets which made a protion of the previous post invisible)
=====================
Hi Eugenio,
Hope you still around to help for a few details I would like to know (learn):
[quote] ESXi has better performance than “Ubuntu + KVM”, the issue here is hardware.
Xenserver 5 has better performance than “Ubuntu + KVM”, the issue here is hardware.[/quote]
“the issue here is hardware”: does that mean ESXi or XenServer is capable of using better the same hardware than KVM or does that mean ESXi or XenServer require better hardware than KVM?
[quote]“Ubuntu + KVM” it’s very similar as VirtualBox 2.x series.[/quote]
Similar in performance? That is strange. I would expect that KVM would do better with the virtualization layer implemented at the level of the Linux kernel. There should be at least something superior with the KVM solution (scalability, stability?) Otherwise, if they are equivalent, VirtualBox is simpler to use. Providing that Oracle will not abandon the product.
Thanks in advance for any help.
cost???
VMware Server for Windows = Free
VMware Server for Linux = Free
VMware ESXi = Free
???
Only some components are free. The strategy is to draw customers in with the promise of “free”. Get them hook and then hammer on the charges.
http://store.vmware.com/store/vmware/ContentTheme/pbPage.buyesxi/ThemeID.1120800&src=PaidSearch_EMEA-ALL_EMEA-ALL_VI_VI_WWW_BuyESXi
Some of the comments relating to *free* and *cost* simply don’t make sense in a commercial environment. In most environments an IT person has a burdened cost of $60-$100 per hour. So if you spend more than a day trying to configure KVM then Vmware’s $995 management suite is pretty good value for money. Almost all the paid for low end virtualization products represent reasonable investments as an alternate to relying on an administrator to hack config files.
A vendor worth considering for smaller enterprises is Virtual Iron
They offer a low cost solution that is more complete than Vmware’s base offering.
I didn’t see any mention of the guest operating systems that you used. I was curious whether you used windows guests or not and whether they performed better under KVM or vmware.
Les: the guests were both Linux and Windows server 2003. I didn’t benchmark Windows on KVM vs. Windows on VMware, but its performance was fine on both.
To add, there is a really awesome virtualization project happening that is centered around KVM and OpenVZ called Proxmox VE (Virtual Environment). It’s a turn key solution that in my opinion rivals the best features in ESX and Citrix Xen such as Vmotion and live migrations. The software is built on a highly customised Debian OS and clustering software, which is one of the most crucial features as you can live migrate virtual machines from one cluster node to another. Did I mention it is all based on a really simple GUI interface that is accessible via web browser (even has Java based VNC consoles you can open to any server in the cluster from your desktop!) I have been evaluating this software for my IT department and it looks like this is going to replace VMware and save us thousands of dollars. Check it out here:
http://pve.proxmox.com/wiki/Main_Page
There are too many features to list here, and pass the word along… it seems no one is familiar with this software, it’s amazing!
Steve Chambers: get over it man. Are you some anti-open source spokesman?
Running both KVM and ESXi in tests at our data center I have determined (in a production environment serving http://www.nationalguard.com 24/7) KVM *is* faster. I achive less load average (under.8 on the app server) with kvm as compared to over 1.4 with esxi – this is on the guest side. So my conclusion is KVM surely runs the guest faster.
Perhaps ESXi *IS* faster is other situations.
Also note these two tests were of mirror images running Debian Lenny, Apache 2, and PHP.
As a former programmer, the low-down-n-dirty documentation for interfacing with KVM is less obscured than esxi’s.
Factor in that there are no Linux clients for esxi. The whole .Net thing is cheesy marketing partnership with Microsoft most likely.
Why do I run Open Source? Well hell, why do you run Windows? I run Linux because it was the closest thing to a [then-dying] Solaris I grew up on until several years back. And that was *NOT* free. We all use (at lease the non-zealous ones) what we use because it is what we are familiar with.
On pricing… Vsphere for 6 CPU’s: over $13,000. Give me a break. The only reason THEY are charging that much is because there are millions of Microsoft patriots who’s opinion is so clouded by *cough* religious zeal *cough* that they absolutely justify the cost.
Most people buying VMWare also work in a Corporate environment where they are spending someone else’s money. $13,000 is a raise for a good employee (or bonus for a couple) who is/are willing to learn a new technology IMO.
Steve Chambers, please stop. You make yourself look foolish.
I was running VMWare ESXi because it is “free”, but I was unable to get reasonable performance. On my dual core/3G machine (desktop model) with 4 VMs running (each VM has 1 CPU, 512M mem, 10G disk), a memcached-like application (no disk IO) gave me 50ms/write. I cut to 2 VMs then I got 25ms/write, which seems 2 VMs is what VMWare can support on this box.
Then (couple of days ago) I heard of proxmox, installed, created same 4 VMs, running with same application, the number was … 8ms/write.
I could not believe the number so I searched online, and this post jumped out – if you are saying VMWare/XenServer are doing much more better would you please post your numbers?
BTW, I’m using Fedora as guest, and I don’t need any fancy features other than migration.
Sorry help in address this point:
In case of maintenance (update of VMware release) for VMware I need to update virtual HW and so a downtime planned, maybe short. I can choose to avoid to do that
With KVM? Is an update with no impact on service continuity? If yes, how?
Sure some vmware solutions are free. But as soon as you ACTUYALLY use them in production, you will realize that without vmotion, you’re gonna feel some pain.
Example: in order to add a 2nd virtual ethernet to vmware, you need to run vmware-setup.pl. Guess what, you have to shut down vmware for that ! yay !
Maturity… I have seen so many weird bugs and esoteric workarounds on vmware server 2, that I don’t put much hope is vmware’s free offerings anymore.
Bottom line is, although great, vmware free offerings are designed to lure you into pricey options. If you don’t intend to eventually buy into it, look into some other product.
Thanks to everyone for sharing their experience. I work at a small auto part manufacturer and am trying to decide whether to go with MS 2008 Hyper-V or VMWare, when I decided to try to google for other options. I will certainly look in to KVM.
What I would really like is a bare-metal hypervisor that can create RAMdisks and copy virtual machine files onto them and run the VMs from RAM!
Baodad: I think you could definitely do that using KVM, if you’re willing to do a little manual configuration. KVM doesn’t care where virtual machines files are, so I don’t see why you couldn’t create a ramdisk, copy a VM image to it and then do something like “kvm /mnt/ramdisk/vm.img …”
Or actually, I guess the command would be more like “kvm -hda /mnt/ramdisk/vm.img …” Forgot the “hda” bit.
[...] are using VMware for running your virtual machines I suggest trying KVM. Here is a good post titled KVM vs. VMware: A Case Study from last year. I wrote a guide for running Win7 using KVM on Ubuntu 10.04 here. linux [...]
[...] for the inputs nix.. searched in google found out some good exp why we should think about KVM.. KVM vs. VMware: A Case Study | WorksWithU Red Hat sprints past ESX on VM running ? Channel Register __________________ Thanks, Surendra [...]
Hi Christopher,
It’s been a while since you’ve written this article. I’m curious what your experiences have been with the stability. Has the KVM platform shown itself to be reliable for production servers?
Nick: yes, we’re still happily using KVM for production–especially after the latest Ubuntu LTS, which appeared in April, brought a number of useful improvements on the KVM front. If I get a chance I may write up something more detailed about this.