As a delegate for Tech Field Day 6 in Boston, I was introduced to several virtualization and performance management tools from vKernel, NetApp, Solarwinds, Embotics, and a company still in stealth mode. With all these tools and products I noticed that each was not integrated into the roles and permissions of the underlying hypervisor management servers such as VMware vCenter, Citrix XenConsole, or Microsoft System Center. This lack of integration implies that a user with one set of authorizations just needs to switch tools to gain a greater or even lesser set of authorizations. This is not a good security posture and in fact could devolve any security to non-existent.We already have issues with the base hypervisor and its many ways to access and control management functions. Now we add to it other management tools that add yet another layer and method to access and control management functions. This was addressed within my previous article, Security of Performance and Management tools within the Virtual Environment, but now there are some new wrinkles. The main premise of this previous post is that all activity that touches the management server of the hypervisor or the hypervisor directly should be proxied through something like HyTrust or some other tool. So let’s look at the new wrinkles created by the same tools.
vKernel, Solarwinds, and NetApp all use service accounts to access the virtual environment details to gather data, which is pretty standard and normal. However, each of these tools uses a directory service such as Active Directory just for authentication, not for authorization. So since the service account must have minimally read-only access to the entire environment, a user who is restricted to just one set of VMs could login to these tools and see all the VMs, Networks, Storage Devices, etc. In this way these tools lack basic multi-tenant capabilities and allow for data leakage of oft-critical details about an environment.
The tools themselves are being marketed as administrator-only, but are starting to expand to provide portlet functionality so that their output could be mashed-up with other tools. Solarwinds has this feature built in, which I find to be quite handy. Even so, the authorization restrictions built into VMware vCenter and the other management tools are completely ignored.
Tools such as Embotics and vKernel also offer the ability to perform tasks on your behalf through their interface, such as reducing memory and such. However, they do this using a delegate user. Because they have used a delegate user, you now have an audit trail correlation issue. While this correlation issue already existed within VMware vCenter and other management tools, it gets worse when you add yet another delegate user. Figure 1 shows the break in the audit trail when you just have one delegate user. You can audit data within vCenter but you just do not know who did what, when, where, or how on the host. The only recourse on a host is currently to correlate the logfiles between vCenter and the Host and if the time servers do not match this correlation is impossible.
As shown in Figure 2, even with HyTrust in the middle, once you hit vCenter the audit trail is one more broken. You can audit all actions within the vSphere Client but once a delegate user is in use within vCenter, the audit trail breaks down to the host, and you once more need to correlate timestamps within log files to determine who did what when where and how. Now there are SIEM solutions such as RSA Envision which can do this for you, it would be far better if there was some sort of tag that could be sent down with every command so that correlation is simply looking up the tag in the appropriate tool to find the real user. Unfortunately, no interface whether from VMware or a Third party supports this yet.
The last issue with delegate users, per Figure 3, is that if you now start using a tool that duplicates what vCenter does via a delegate user, you now have a broken audit trail from that tool, such as Embotics VCommander, already down to the host. You now have to correlate data across three systems ensuring of course that all three have the same time source, etc. Even with HyTrust involved when using Embotics VCommander, the audit trail will be broken before HyTrust is even as use, as HyTrust is still accessed via a delagate user.
In many cases, auditing may not be difficult, depending on the number of simultaneous actions that take place, however, when faced with a forensic study or compliance regulation that must determine who did what when where and how, such a broken audit trail makes it impossible. Tie this to the possibility of data leakage and destruction from tools having the incorrect authorizations and you end up with a system that could be heavily impacted with the possibility of no way to track back who did what when where and how. Not only would we be insecure but in violation of regulatory compliance.
I call on the tool vendors to work with the hypervisor vendors to solve this problem. There must be not only centralized authentication (which we have with directory services) but centralized authorizations in order for a multi-tenant environment to succeed. In addition, we need to be concerned about tools that span hypervisors which complexes this quite a bit. We also need audit capability all the way down the stack so that we know beyond any shadow of a doubt who did what when where and how.
Additional Info: There is at least one feature request with VMware to address their hypervisor/vCenter interactions.