On the May 30th Virtualization Security Podcast, Michael Webster (@vcdxnz001) joined us Live from HP Discover to discuss what we found at the show and other similar tools around the industry. The big data security news was a loosely coupled product named HAVEn which is derived from several products: Hadoop, Autonomy, Vertica, Enterprise Security, and any number of Apps. HAVEn’s main goal is to provide a platform on top of which HP and others can produce big data applications using Autonomy for unstructured data, Vertica for structured data, Enterprise Security for data governance and hadoop. HP has already built several security tools upon HAVEn, and I expect more. Even so, HAVEn is not the only tools to provide this functionality, but it may be the only one to include data governance in from the beginning.

There are several products that couple big data analytics with the goal of answering tough security questions as quickly as possible, with the toughest being “Was I hacked?” But why is this difficult and why do I need big data tools such as hadoop to help solve this? Consider the nature of data used for security, it is comprised from:

  • System Log files that, during an error, can grow by gigabytes per second
  • Network data captures, which can grow by 10s of gigabytes per second, which is one reason these are used sparingly
  • Intrusion Detection/Protection log files, which can show just the headers or even the packets within network traffic which can grow by 10s of gigabytes per second
  • Application Log files that, during an error, can grow by gigabytes per second
  • Application Performance Metrics that have a resolution of seconds and imply an immense amount of data

All this data can turn into terabytes per hour. I do not know about you, but I cannot investigate terabytes per hour without some help. Into this breach come the big data security tools that are detecting attacks sooner than before. While this is not the only use for HP HAVEn, it was a demonstrated one as HAVEn unlike other tools for big data security analysis is a platform on which to build tools. Here is a list of possible tools to use, there are others that I expect to be discussed at the Misti Big Data Security Conference.

HP HAVEn is a platform on which to build tools that includes HP ArcSight and the use of its logger to pull in machine data (logs) while providing the ability to handle data governance of the data pulled into the big data repository whether that is Autonomy, Vertica, or Hadoop. However, this is on ingest or access to the data, not on output of any derived content.

RSA SilverTail started as specifically design tool to ingest web log data and to find fraud but has expanded to look at nearly all machine data (logs) as well as networking data to specifically to perform behavioral analysis looking for those unknown unknowns that could point to security issues.

Splunk App for Enterprise Security has its roots in Splunk Apps used for performance management of an environment. Splunk has gone a step further to analyze this data plus machine data (logs) to determine if the behavior of an application has changed in such a way to represent a security concern. They are looking at url as they go into and out of an application for example as well as other items that could point to an unknown unknown.

These three big data analytics applications have been designed for use by security teams to find out those unknown unknown security issues that stem from looking at behavioral analysis of an application. Behavioral analysis of people and applications is the current trend in security analytics as it is nearly impossible to prevent attacks unless you can recognize them, and recognizing attacks is becoming more difficult unless we switch to using analytics to see across a wide range of non-uniform data.

As an aside, we also discussed on the podcast, a few steps for security while at conferences. More on that next time.

I would like to thank VMware for the loan of their meeting space at HP Discover and HP for inviting and paying for my trip to HP Discover.

Share this Article:

Share Button
Edward Haletky (367 Posts)

Edward L. Haletky, aka Texiwill, is the author of VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment as well as VMware ESX and ESXi in the Enterprise: Planning Deployment of Virtualization Servers, 2nd Edition. Edward owns AstroArch Consulting, Inc., providing virtualization, security, network consulting and development and The Virtualization Practice where he is also an Analyst. Edward is the Moderator and Host of the Virtualization Security Podcast as well as a guru and moderator for the VMware Communities Forums, providing answers to security and configuration questions. Edward is working on new books on Virtualization.

[All Papers/Publications...]

Connect with Edward Haletky:


Related Posts:

1 comment for “Big Data Security Tools

  1. June 23, 2013 at 8:45 PM

    HAVEn looks like a really great tool for the thousands of terabytes of logs and traffic. Seems like there is a HUGE push for more metric and analytic data to help make program decisions. I helped design a tool called the DAP Report (http://dapreport.com) that helps security analysts that perform manual security testing provide executives that big data without the extra effort.

    HAVEn and the DAP Report are strives to get away from all the security reportS and provide a security reporting solution.

    Thanks for the article, if you have any questions about the DAP feel free to ping me. I would love to hear any feedback you have.

Leave a Reply

Your email address will not be published. Required fields are marked *


six + = 7