It is no secret that most of the easy “low hanging fruit” has been virtualized, and what is left are the business critical applications. While the leading virtualization platform vendors (VMware and Microsoft) have made huge progress in the performance and scalability of their platforms, concerns linger about the ability of x86 virtualization platforms like vSphere and Hyper-V to handle really demanding workloads like SAP.
The SAP Performance Benchmark Wars
Fortunately for all of us, SAP has for quite some time had a standard set of benchmarks that can be run against any configuration that supports the SAP software. So SAP is one of the few enterprise applications where you can readily do an apples to apples comparison of either SAP on Hyper-V vs. SAP on vSphere, or a comparison of either of these to SAP on AIX. The SAP benchmark is also extremely useful in that you specify that response time has to be less than one second, and then you increase the load until the response time threshold is met. This is a far more realistic way of doing a benchmark than is the case for most benchmarks which give up on reality for the sake of expediency.
The news here is that Microsoft has done a new benchmark of the performance of SAP on Windows Server 2012 with Hyper-V 3.0. This is the first time that Microsoft has used Hyper-V in an SAP benchmark and it is the first time that Microsoft Hyper-V has out-performed VMware vSphere in a SAP benchmark.
|Vendor||Virtualization Platform||Operating System||Database Server||Average Response time||Number of Concurrent|
|Microsoft||Windows Server 2012 with Hyper-V||Windows Server 2012 Dataceter||SQL Server 2012||.99 Seconds||42,400||231,580|
|VMware||vSphere 5.0||Windows Server 2008 R2||SQL Server 2008|
You can find the overall set of SAP Benchmarks on the SAP Benchmarks Page.
The write-up on the Microsoft site of their results is on the MSDN SQL Server Blog Site.
The write-up of the VMware results is on the VMware VROOM Blog.
Caveats and Conclusions
As with all benchmarks, there are several very important caveats that need to be noted when drawing any conclusions about these results. The first is that the VMware one was run in November of 2011, on hardware that was two years older than the hardware that the Microsoft benchmark was run on. Newer hardware could easily have accounted for most if not all of the difference between these two results. The second is that the Microsoft benchmark was run using the 2012 versions of the Windows OS, and SQL Server, whereas the VMware one was run on the 2008 editions. We have to assume that Microsoft did not make its server OS, or is database server slower in the four years in between the 2008 and 2012 releases. So the difference here was not just between vSphere 5.0 and Hyper-V, it was also due in part to improvements that Microsoft made in its server operating systems and SQL Server. Finally, of course, the table above compares the absolute newest version of Hyper-V against the now two year old version of vSphere (the current version is 5.1 and we are probably going to get an update this fall).
The benchmark also did not explicity address the question of at what cost these results were achieved. For example, Microsoft used a total of 12 Application Servers with 2 processors, 16 cores and a total of 256 GM of main memory. VMware used a total of 10 Application Servers with 2 processors, 12 cores and a total of 96 GM of main memory. So the question of price/performance was completely ignored in these benchmarks other than by listing the hardware used, and letting the reader figure it out.
All of these caveats aside, it is likely that Microsoft has it least caught up with VMware vSphere in terms of raw hypervisor performance and scalability. If VMware were to rerun this benchmark with current hardware, Windows 2012 Datacenter, and SQL Server 2012, we would expect to see results that are either marginally better or marginally worse than the current Microsoft results.