VMware to Revolutionize Operations Management with Log Insight

VMware has announced its log management product – Log Insight. Log Insight is priced at $200 per monitored OS instance (per VM pricing) and is to be available in Q3 of this year. VMware’s own vSphere environment is the first targeted environment, and the two first use cases is Operations Management. Right now this is clearly a 1.0 offering competing with a very mature Splunk Enterprise offering – but there are some very interesting short term and long term dynamics at play.

The Log Management Opportunity in the VMware Customer Base

VMware did some market research on its own customers. The most important piece of data from the research is that most VMware customers (73%) are not using a log management solution today. The research also pointed out that customers felt that ease of use was the most important criteria for selecting a log management solution, with license costs being second as an important factor. So we have a market for log management in the VMware customer base that is wide open and that highly values a solution that is easy to use and that is concerned about licensing costs.


Log Insight Overview

When you first look at Log Insight you might simply conclude that this is VMware’s version of Splunk. And in fact there are many similarities that make using Splunk as the reference point into a great way to first understand and evaluate Log Insight. Log Insight is at is core a product designed to ingest unstructured data from a variety of sources – at scale, and to then index that data so that it can be quickly and easily searched. The first use case targeted by Log Insight is Operations Management one of the core use cases for Splunk.


But if you start to use Log Insight (and you really should download it and try it) you will quickly discover that there are some important differences with respect to Splunk, who is the clear market leader in this space. The first important difference is that you do not have to learn a query language to use Log Insight. This speaks directly to the ease of use requirement which came out of the market research that VMware did.


The second aspect of Log Insight that is important is the pricing. Log Insight is $200 per monitored operating system instance (physical or virtual), irrespective of the amount of data being indexed. This stands in stark contrast to the pricing model of Splunk which is based entirely upon the amount of MB of data being ingested by the log management solution each day.

The Strategic Implications of the VMware Log Insight Announcement

At a tactical level, Log Insight is just another component product destined to be part of the vCloud Suite. Therefore at a tactical level, Log Insight is just one more feature of the ever evolving VMware suite of management offerings.

But there are monumental forces at play here which make Log Insight into a significant entry in the operations management software business for virtualization and the cloud. Those forces and factors are:

  • By not forcing users to learn a query language, Log Insight has significant lowered the barrier to the initial adoption of a log management solution.
  • By not pricing based upon the amount of data indexed per day, Log Insight does not tax customers for their success with the Log Management solution. VMware learned this lesson the hard way with the vTax which was in effect a tax upon density, which turned out to be a tax upon customers succeeding with the VMware vSphere product.
  • If you assume that the value of a log management product is tied to the amount of data that it ingests per day, then pricing based upon that metric amounts to a tax upon customer success with the solution.
  • While Splunk has never been perceived as levying something as onerous as a vTax upon its customers, the arrival of VMware with Log Insight with a pricing model that casts Splunk in a very unfavorable light has the net effect of making Splunk look like a vendor who is taxing the success of their customers with their product.
  • If the VMware market research is right, and the two most prominent barriers to greater adoption of a log management solution in the VMware customer base are easy of use, and price, the VMware has addressed both of these issues out of the gate with the pricing, packaging and positioning of Log Insight.
  • While VMware has not said that Log Insight will be a part of one of the editions of the vCloud Suite, this is inevitable. Therefore VMware is basically saying that Log Management is a feature of Operations Management
  • By making log management part of the suite of VMware management offerings, VMware effectively revolutionizes the Operations Management market for virtualization. The reason for the revolution is that up until now, every operations management vendor was essentially constrained to the same set of data – the data from the vCenter API. This data consists of the resource utilization metrics collected by the hypervisor which is collected every 20 seconds and then made available in 5 minute rollups. Vendors who do the work can collect the data directly from the vSphere hosts every 20 seconds, but it is still the same limited data set.
  • Adding log data to the data set for operations management solves two huge problems. The first is the frequency of collection which can be a frequent as the source of the logs wants to generate the data. The second is the scope of the data. For example in the online demo for Log Insight you have access to real time SCSI latency metrics. This promises to make much more data which measures latency and response time available, and much more data which is the context for root cause available.
  • Splunk has been anticipating the arrival of Log Insight for quite some time, and has been investing heavily in its Splunk App for VMware. So while VMware is saying that log management is just a feature of Operations Management, Splunk is saying that Operations Management for VMware is just a feature of its enterprise log management solution.
  • The vCloud Suite collectively has too many databases and it is effectively impossible to search across them. Log Insight holds out the promise to be a unified and easily searchable datastore across the entire suite of VMware management solutions.
  • If both VMware and Splunk can effectively integrate Operations Management data with log data into easily searchable data stores and thereby produce more functional Operations Management solutions, this creates a huge competitive advantage for VMware and Splunk over every other operations management vendor. Any vendor that just consumes the vSphere API data and then creates a dashboard with reports and alerts is now at a huge competitive disadvantage.

The Battle Lines are Drawn – Splunk vs. vCenter Operations + Log Insight

In our Reference Architecture for the Software Defined Data Center Management Stack, we propose that a real time big data back end is essential to the ability of any set of management solutions to manage a highly dynamic and and distributed environment. One of the problems with vCenter Operations for the last couple of years has been its reliance upon infrequently collected and rolled up data from vCenter. It is clear that VMware’s strategic direction in terms of network and storage virtualization will require VMware to embark upon both real time instrumentation and the ability to ingest and analyze that data in real time. The addition of the SCSI latency metrics discussed above is an example both of what is possible now, and what will be a fertile field for innovation for both VMware and many clever ecosystem partners. Collectively VMware and the ecosytem are now poised to deliver real time instrumentation of latency across the entire stack of virtualization software.

Real time, continuous and deterministic instrumentation of latency will be a critical element in the successful deployment and management of the software defined data center. When network virtualization and storage virtualization are added to the current ability to virtualize compute, latency will be the only way to understand the actual performance of the system. For a quick picture into what network virtualization will create in terms of monitoring requirements, consider the following blog post by Bruce Davie, one of the principal engineers on the Nicira and network virtualization front with VMware.


 To see the complete post by Bruce Davie, go to the VMware CTO Blog Post here.


For Operations Management to be relevant for the SDDC and the cloud, it must be transformed from something that measures some of the things some of the time, to something that measures all of the things all of the time. This requires a fundamental rearchitecting of operations management solutions from things that collect subsets of metrics every 20 seconds to every 5 minutes to things that can collect and make sense of real time streams of operational data. Splunk has broken new ground by giving customers a way to collect these streams of machine generated data in real time. Log Insight builds upon the success of Splunk by making the adoption of the solution easier both in terms of user training and the price of adoption. The operations management business is in the process of being transformed.  Legacy solutions from vendors like IBM, BMC, CA and HP are so far out of step with the new requirements for the SDDC and the cloud that they should be immediately put on the list of products to be retired and cashiered out of the enterprise.

Posted in IT as a Service, SDDC & Hybrid Cloud, SecurityTagged , , , ,