Testing within the Virtual Environment


There has long been a debate about testing products within a virtual environment. Not just on how, but the why as well as the what to test. There are limits in some EULA’s as well on the reporting of such testing.  This was the subject of the 7/25 Virtualization Security Podcast (#112 – Virtualization Security Roundtable) held Live from NSS Labs in Austin, TX.  Where we delved into the issues of testing within a virtual environment. While the discussion was about security products, it is fairly straight forward to apply the concepts to other products within the virtual environment.First we need a testing procedure that is well thought out, documented, and supported by others. Take Virtual Desktop Testing or general Hypervisor Testing, there are certain tools associated with testing of these environments that are accepted by all as ‘defacto’ standards. Use of these tools aids in understanding the test which helps in reporting and acceptance of those reports. Even so there are several items to consider:

  • Environment – We need to be cognizant of the environment in which the test was run. Were the testing tools run as workloads on the systems to be tested? Or external? If on the system, what is there overhead? Not only that, since many of these are tests run over a network (desktop, firewall, IPS, IDS, etc.) the type of network involved comes into play so we need to understand network latency, bandwidth, burst rates, etc. A well defined environment for all tests is a must.
  • Procedures – We also need a well defined, repeatable set of procedures. It is best to have a fully automated procedure so that the human element is removed from the equation. The automated procedures should include any baseline as well as load generating tests.

With these two items covered either by documentation or fully automated systems (perhaps even the creation of a software defined data center, SDDC, for use by testing), we can proceed with our tests. As the tests proceeds the environment and procedures will change until a final environment and procedure arises. The key to both of these is to make the environment as real world as possible and to have a repeatable process. Repeatable by anyone not just your own test lab. Which also implies the use of well known tools to provide the procedure. Those tools include:

  • LoginVSILoginVSI is the defacto standard for benchmarking virtual desktop environments with its ability to automate the access to VMware Horizon View, XenDesktop, and other systems over well known protocols such as RDP (and others). In addition, while it comes stock with many existing workloads, you can easily create more workloads that run within each desktop. If you wanted for example, to determine the best settings for graphically intensive applications, it would be good to add those applications into the mix. LoginVSI has been used to test end point security tools, graphics, virtual desktop density, flash and other types of storage, and many other aspects of a virtual desktop environment. ProjectVRC has many reports on these types of tests.
  • SPECvirtSPECvirt is the standard for benchmarking virtual environments and has recently been updated to the 2013 version which includes several application tools. SPECvirt runs different tiles (workloads) across a single or cluster of hypervisors. Since all workloads on a hypervisor are unique, having a standard set of tools based on a standard gives us some measure of success. A predecessor to SPECvirt was the VMware VMmark (which also still exists).

However, these tools are not the end all, but more the beginning of testing within your virtual environment. To be a believable test however, the test must be independent, well thought out and repeatable. For security testing within the virtual environment we need to look further into what is required. Many numbers are just feeds and speeds but before we can find those numbers we need to find what is normal, so that we can further our tasks to determine the additional over head of those tools. Part of what is normal will be latency of network, storage, etc. So this implies that test labs should mimic production labs exactly as hardware makes a difference, where your data moves around the environment also makes a difference. The key is repeatability, so if we can repeat our process and recreate our environment (which includes hardware and how the data moves around the environment) we can provide a beneficial and believable result.

How do you test changes to your virtual environments?

Share this Article:

The following two tabs change content below.
Edward Haletky
Edward L. Haletky, aka Texiwill, is the author of VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment as well as VMware ESX and ESXi in the Enterprise: Planning Deployment of Virtualization Servers, 2nd Edition. Edward owns AstroArch Consulting, Inc., providing virtualization, security, network consulting and development and The Virtualization Practice where he is also an Analyst. Edward is the Moderator and Host of the Virtualization Security Podcast as well as a guru and moderator for the VMware Communities Forums, providing answers to security and configuration questions. Edward is working on new books on Virtualization. [All Papers/Publications...]
Edward Haletky

Latest posts by Edward Haletky (see all)

Related Posts:

Leave a Reply

3 Comments on "Testing within the Virtual Environment"

Sort by:   newest | oldest | most voted
3 years 2 months ago
If you have Windows VMs and want to throw some interesting activity, try PCmark For general server, storage and I/O workload generation there are many in addition to VMmark etc. For example, Bonnie, Iometer, Iorate, IOzone, Vdbench, TPC, SPC, Microsoft ESRP, SPEC and netmist, Swifttest, DVDstore and PCmark 7 among many others. Some are focused just on the storage system and IO path while others are application specific thus exercising servers, storage and IO paths. If you have Windows VMs and want to throw some interesting activity, try PCmark which you can setup to exercise secondary drives, as well as… Read more »
3 years 2 months ago

> A predecessor to SPECvirt was the VMware VMmark

Are you indicating that SPECvirt now has taken over as a the most widely used benchmark? There are still few SPECvirt results compared to VMmark results.

Note also that the test methology from VMmark is freely available while SPECvirt must be bought. Oh, wait… Looks like SPECvirt is also freely available now. Maybe that will help changing things.



[…] To read the complete article, CLICK HERE […]