Legacy solutions from IBM, BMC, HP and CA are not going to be the foundations upon which the management stacks for virtualized data centers, the SDDC, private clouds, hybrid clouds and public clouds will be built. Rather a new ecosystem of management vendors is going to emerge with a new set of leaders providing the core platforms around which the SDDC and clouds are managed.
Why Legacy Management Frameworks Need to Die
In the old world, large vendors (IBM, BMC, HP, CA) sold you extremely expensive management frameworks that tried to manage everything. These frameworks were not only very expensive to purchase, they proved to be very expensive to maintain, in terms of ongoing maintenance fees from the vendor, required services from the vendor, and staff from the customer dedicated to the framework. In other words, the total cost of ownership for these frameworks proved to be very high.
There are other downsides to these frameworks. Most of them have been assembled by acquisition. The acquired products all came with their own consoles, own data models, and own databases. Over the years these acquired products got partially integrated into core frameworks, but few of them got fully integrated into the frameworks. This lead to the frameworks becoming Franken-Monitors, complex suites of poorly integrated components.
It is also the case that when a large vendor like CA buys a startup, most of the great people that make that startup successful leave within two years of the acquisition of that startup. Most of the team that made Nimsoft into a successful vendor have now left CA. The people that made Wily into the market leading APM solution at the time of the acquisition of Wily by CA have mostly left. The former CEO of Wily, Lew Cirne is now the CEO of New Relic. The former VP of Development of Wily, Jyoti Bansal, is now the CEO of AppDynamics. You can be pretty sure that the talented developers who liked working for Lew and Jyoti at Wily have found their way to New Relic and AppDynamics.
The drain of development talent lead to the last pitfall with the frameworks, which is that they were and are now completely unable to keep up with the pace of innovation in the industry. The industry simply evolves to quickly for a large vendor who does not employ the best developers to be able to keep up with the entire industry and to provide a framework to manage it. Data center virtualization, the Software Defined Data Center, private clouds, hybrid clouds, and public clouds are all innovations that conspired to make the frameworks worse than useless – they consume time and money, but cannot provide the value needed to manage these modern environments.
For the above reasons many enterprises are choosing to uninstall these frameworks as they virtualize servers. Once virtualized networks and storage arrive, frameworks will cease to have a role in managing these new elements of the data center as well.
This leads to the important questions of what to replace the frameworks with, and how to do this so as to not repeat the mistakes made the last time around.
Management Platforms for the SDDC and the Cloud
In order for us to be able to properly manage the SDDC and the cloud, we are going to need a set of products that each manage their own layer or function in the SDDC or the cloud. We have described the layers and functions that need to be managed in our SDDC Reference Architecture, which was described in detail in “Building a Management Stack for the SDDC and the Cloud” The diagram for our SDDC Management Stack is shown below.
The single most important thing about this management stack is that you are highly unlikely to get all of the products that fill all of the layers and functions from one vendor. Nor should you try, as no single vendor is going to be able to do the best possible job of managing each layer and function in the stack. We covered how VMware’s management offerings fit into this architecture in “VMware’s SDDC Management Stack“, and noted that VMware is reliant upon third party solutions for the Application Performance Management layer of the stack. We covered how to assemble your management stack out of best of breed solutions in “Best of Breed SDDC Management Stack“. But no matter how you choose to assemble your management stack for your SDDC and your clouds, you will face some important issues:
- Frameworks came into being because they at least tried to integrate data from the various components of the framework. The SDDC will be unmanageable unless all of the data from all of the products managing all of the aspects of the SDDC is put into on Big Data Repository.
- All of the products that put data into this repository are going to not only have to contribute their own data, but make use of data from other products. A simple example is how Security vendors and APM vendors have mutually useful data. An unexpected increase in load or response time noticed by the APM product may signal an intrusion that the security product should go try and find. On the other hand that APM product should also query the security data in its root cause process for that slowdown in response tim
This leads to the requirement that some set of vendors offer the core platform into which every other management vendor plugs their data, and from which every other management vendor queries data to add value to their results. Management platforms, therefore, need to have the following attributes:
- From a technology perspective, a management platform must be based upon a big data back end which has open API’s which allow any vendor to freely add their data to the back end data store and query the data that the platform vendor and other vendors put in the data store. It is also critical that a mechanism exist to index data from disparate data sources so that queries and searches across these disparate sources can be easily performed.
- From a partnering perspective the platform vendor must have a go to market model which is friendly towards and embracing of vendors of complementary products. Just like Windows beat technically superior Unix operating systems by having a superior ecosystem of vendors who wrote applications to Windows, the winning management platform will have the largest and most vibrant ecosystem of third party vendors supporting their platform. The ability of the platform vendor to cooperate at a marketing and sales level with ecosystem partners will likely determine the success of each particular platform vendor. The key here is that third party ecosystem partners make money selling their product in partnership with the platform vendor.
- The platform vendor is going to have to have a business model that leaves money on the table for ecosystem partners. No management software vendor is going to drag a third party platform into all of its deals, if that platform causes the total price to the customer to go up by 50% or 100%. The winning platform vendor will have to learn to be content with taking the smallest amount of money from the largest number of deals, not the other way around.
Candidates for SDDC Management Platforms
Let us be clear about one thing. The SDDC does not exist yet. No vendor has yet even announced their SDDC management stack (here’s betting that VMware will be the first). Therefore the exercise of which vendors are candidates to be platform vendors in this upcoming market is entirely speculative in nature:
- Splunk is the only vendor of a management solution today that is based upon a big data back end, and that has a proven strategy of going to market with partners that leverage that back end. On the APM front, Splunk partners with vendors like AppDynamics and ExtraHop Networks. On the security front Splunk partners with Palo Alto Networks among other vendors. The only issue that Splunk faces is that Splunk charges customers by the amount of data indexed by their product every day. This creates incentives for third party partner vendors to minimize the amount of their data that they put into Splunk so as to not drive up the price to the customer of using Splunk.
- With Log Insight, VMware announced its competitor to Splunk. VMware has two opportunities with Log Insight. The first is to make it into the standard back end data store for all of VMware’s own management products which would be in and of itself a great accomplishment. The second would be to welcome third party vendors to add their data to Log Insight, turning Log Insight into a multi-vendor SDDC management platform. Log Insight is priced at $200 per managed operating system instance (per VM pricing) so once the customer buys Log Insight from VMware, the price of Log Insight does not go up as more people add more data per day. VMware’s biggest challenge is that VMware has never wanted to market and sell software in cooperation with other vendors, and must learn to do so if VMware wants to be a management platform vendor.
- New Relic is the market leader for APM for cloud hosted applications. New Relic is itself a cloud hosted service (APM as a Service) and New Relic has delivered a plug-in architecture that allows any customer or vendor to write a plug-in that feeds any desired set of data into the New Relic back end (which is hosted in their cloud), and then allows for that data feed to be displayed in the New Relic console. With over 50,000 organizations using New Relic already, the biggest strength of New Relic is that to many software vendors the New Relic customer base looks like an entire target market. New Relic faces the challenges of adding analytics to its platform so that data from disparate sources can be easily cross-correlated, and of ensuring that its ecosystem develops into an actual marketplace where third party vendors can make money by being ecosystem partners. New Relic currently lacks a true big data back end, but is likely to be forced to correct this merely by continuing to add customers at the current pace.
- AppDynamics has delivered its AppDynamics Exchange which allows monitoring extensions, alerting extensions, and cloud scaling extensions to be easily added to the AppDynamics APM product by customers and third party vendors. This is a very attractive alternative for third party vendors looking to target the on-premise enterprise market as AppDynamics is the most rapidly growing vendor targeting the monitoring of custom developed Java, .NET, and PHP applications (AppDynamics also offers its product as a service, and supports applications running in clouds – just most of its notable success is with on-premise applications). AppDynamics is also not yet on its own big data back end, but like New Relic will be forced by its own success in this direction. The challenge for AppDynamics will be similar to the one for New Relic which is to construct a business model that is friendly to the economic needs of third party partners.
- Pivotal (lead by former VMware CEO Paul Maritz) has the big data assets and the big data know how to produce and deliver a multi-vendor big data management platform. Martiz certainly remembers his Microsoft days when the ecosystem strategy propelled Windows to be the most successful operating system of all time based upon the strategy of open API’s and leaving almost all of the money on the table for the application providers.
- AppFirst is an APM as a Service offering that is based upon a big data back end that runs in the AppFirst cloud. AppFirst has already built collectors for most of the common data sources in use like Stats-D and Nagios. The remaining piece of work is to build an ecosystem of third party vendors whose products feed data into AppFirst, and to build a business model that encourages such an ecosystem to develop and flourish.
- CloudPhysics is a vendor of a cloud hosted operations management solution. CloudPhysics brings an impressive big data back end, an impressive set of analytics, and an impressive team of ex-VMware and ex-Google people to the table. Data collection by CloudPhysics is currently limited to VMware vSphere data collected by a virtual appliance from CloudPhysics. But CloudPhysics is certain to broaden its data sources over time, and may well choose to pursue an ecosystem strategy for both data collection and marketing and sales partnerships.
Legacy management frameworks are going to get replaced by SDDC Management Platforms that combine big data back ends, analytics and ecosystem friendly data collection and integration strategies to give customers the best of both worlds. Customers will be able to choose from among best of breed solutions, and then integrate them at the data level via a big data back end data store. This will revolutionize the management software industry, give rise to a new set of leaders in this industry, and completely destroy the legacy management frameworks.