Intrusion Protection and Detection within the Virtual Environment

There has been lots of debate on whether to place security tools within a virtual environment, whether such tools are needed, and how these tools should work.  Since many of these topics were covered by Hoff’s Rational Survivability blog in the past, I will not revisit them. The premise for this discussion is that yes such security tools are needed, that they do need to be redundant, and they are required to be implemented within your environment. We will answer what tools exist that provide Intrusion Protection and Detection within the virtual environment.

Intrusion Protection Systems (IPS) differ quite a bit from Intrusion Detection Systems (IDS). An IPS is designed to modify some form of security setting when an intrusion is detected, thereby preventing the intrusion from being successful. An IDS on the other hand is just the detection component used by an IPS. Like all security tools used within a virtual environment there are four major ways to implement such devices. We will discuss later some best practices for managing a security tool. We will look at what is currently shipping over products hinted at for the future such as the OpenVSwitch, Xen Instropection API.

From Domain-0 (Type 2 Hypervisors as well as KVM, Hyper-V and Xen)

This form of implementation puts an IDS/IPS device within the standard network stack of the host, domain-0, primary partition, etc. Since all network traffic must first traverse this component of the virtualization host it is a very good location to place an IDS/IPS to protect all VMs.  This placement does not protect traffic between two virtual machines, instead such a device is on the entry or egress from the virtualization host and the virtual network.

When the Xen Introspection API, or the OpenVSwitch projects are finally available for Xen, it should be possible to gain access to the network data from within Domain-0 much like VMsafe or using a SPAN port.

Currently only Catbird‘s products are integrating at this level with Xen where they are able to provide IDS/IPS capabilities. Within the Amazon EC2 cloud, which is based on Xen, Catbird’s implementation allows EC2 customers to perform internal vulnerability scans ‘at will’ in order to meet PCI audit requirements. Given the proper access by Amazon EC2, Catbird’s products could also perform IDS/IPS within EC2.

Catbird currently does provide IDS and IPS functionality for Realware and HaloFC multi-tenant clouds.

Using VMsafe (VMware ESX 4 and ESXi 4 only)

VMsafe Net provides a way for a security vendor to place a security check/firewall/policy before each vNIC attached to all VMs for a given ESX 4 host. This provides a major improvement in security capabilities as we now can monitor and protect inter-VM traffic at the vNIC and can apply per VM policy. IPS can be used at this level to detect the most egregious intrusions and squash them.

VMsafe fastpath tools currently available from Altor Networks and Reflex Systems currently do port and protocol level policy enforcement. To perform true IPS and IDS, you must use deep packet inspection which implies that the packet needs to then be sent via the slowpath to an IDS or IPS appliance. This appliance would apply the necessary rules and determine if the packet should be forwarded or not.

At the moment, there are no full IPS based VMsafe appliances, however Altor Networks VF3.0 provides two very useful IDS tools. The first is a built-in IDS and the second is a way to send data to a third-party IDS. This third party IDS could then implement an IPS.

By sitting between two virtual switches or inline devices

Another way to provide an IDS or IPS is to have an appliance that sits between two distinct virtual switches within a VMware ESX or ESXi environment. This does not apply to any of the other hypervisors as there is only one bridge or switch available.  The virtual appliance would look at all packets traversing its network adapters of which there would most likely be two, and perform IDS or IPS on those packets.

There are several products that provide inline IDS and IPS solutions. These include Catbird vSecurity as well as Altor VF2.0 (which has been superseded by the VMsafe implementation VF3.0). There are also several Open Source products that fit this mechanism: Smoothwall, IPcop, m0n0wall, and any Snort implementation rolled by the end user.

The limitation is on how many network adapters any such appliance can have, on ESX 3 it is only 4, however this has been expanded to 10 in ESX 4.

By using a SPAN or similar port (Cisco Nexus 1000v and VMware vSwitch Portgroup 4095)

The last method to implement an IDS/IPS is via a SPAN port. SPAN ports exists on most modern switches and within the virtual environment you must have a virtual switch that provides this functionality. For VMware ESX 4 there is the ability to do this using two distinct technologies. The first is the Cisco Nexus 1000V which provides SPAN and ERSPAN capability directly from the 1000V. The other is to use a specialized portgroup with a VLAN ID of 4095 on every VMware vSwitch. This last mechanism also works for ESX 3. An appliance would then be used to perform the actual IDS/IPS.

Once more there is a limitation based on how many vNICs a virtual appliance can have.


There are multiple ways to implement an IDS/IPS within the virtual environment. This crucial physical world security technology can be placed within the virtual environment by choosing the mechanism that works best for you. If you want IDS only then any of the tools mentioned could work. If you want to quarantine badly behaving VMs, then you either want to use Catbird’s, Reflex Systems, or Altor Network’s tools.  If you would rather use your existing physical IDS/IPS appliance then you will be looking at either Altor Network’s or a mechanism to forward the data from one of the SPAN port mechanisms. If however you want to protect Xen, Catbird is the solution.

Posted in SDDC & Hybrid Cloud, SecurityTagged , , , , , , ,