In Management Frameworks Will Die we make the case that frameworks have failed because no one product can monitor everything, because management frameworks cannot be modernized to meet the needs of the Software Defined Data Center and the Cloud, because frameworks are too painful and expensive to maintain, and because customers prefer the “try it before you buy it” model of buying management software to the enterprise license agreement approach favored by the framework vendors.
The Modern IT Environment that Management Frameworks Cannot Manage
In order to understand how dead management frameworks are, it is useful to contrast the current and future state of what needs to be managed with what management frameworks were designed to manage. The modern IT environment has the following attributes:
- It is virtualized. CPU and memory resources are abstracted from their physical manifestations. Networking and storage resources are in the process of being abstracted from their physical manifestations.
- It is highly distributed. It is distributed across many virtual machines, many hosts, many clusters, many data centers including ones comprised of hybrid and public clouds.
- It is highly automated. DRS moves workloads between hosts. Cloud Management software provisions entire application systems automatically. Deployment tools like Puppet update hundreds and thousands of virtual machines at the same time
- The applications arrive frequently and change even more frequently
In summary, the modern IT environment consists of rapidly arriving and rapidly changing applications running on a highly dynamic and distributed infrastructure that is often distributed across data centers owned by the enterprise, and increasingly, ones that are not.
New Requirements for Managing the Modern IT Environment
The fact that the modern IT environment is so different from the static and dedicated environment for which management frameworks were designed, leads to an entirely new set of requirements for modern IT management software:
- Monitoring must be done with much greater fidelity than before. This simply means that more things have to be monitored. It used to be possible to manually list the metrics that one should monitor across a server, a network and the storage. Now the number of things that need to be monitored is so large that even trying to create such a list is pointless since by the time it is created it is out of date.
- Monitoring must be done much more frequently than before. It will simply be insufficient to collect data every 5 minutes or even every 1 minute. Data collection will need to move down into the 5 second to 15 second range in order not to miss important events in the dynamic infrastructure.
- Monitoring must be done deterministically. This means that the actual value of the metric in question needs to be collected, not an average or an approximation of that value.
- Monitoring must be done comprehensively. This means that it must be done in a manner (frequently is part of this) that ensures that that important events or spikes are not missed by the monitoring process.
- Monitoring data needs to be shared between vendors who monitor different things for different reasons. For example the application response time data, the IT Operations data, the log data and the security data all need to go into a common data store where each vendor can use other vendors’ data.
- Infrastructure performance needs to be defined as as to end infrastructure latency – not how much of the switches bandwidth is being used or by the number of storage IOPS.
- Application performance needs to be defined as response time, not how much CPU and memory the application is using.
- The communications architecture of the management solution needs to match how the applications that the business depends upon are deployed. Specifically this means that things that collect monitoring data (agents, collectors or forwarders) need to use “phone home” communications techniques over ports 80/443 into the back end management system. Legacy management systems that rely upon remote procedure calls from the management system to the collectors will not work, because it will not be possible to open port 135 inbound into all of the places where applications are running.
The above requirements create a situation that is unprecedented in the history of the business computing industry. They create a situation where due to the diversity of things that need to be monitored, the rate of innovation in new things that need to be monitored, and the rate of change throughout the entire stack of hardware and software no single vendor can possibly keep up with the pace of innovation along the requirements for fidelity and frequency.
The Rise of the Management Platform and the Management Ecosystem
If no single vendor can meet the above requirements for the diverse, distributed and rapidly changing IT environments that customers own today and will own more of tomorrow, who and what can? The answer is that only an ecosystem of cooperating vendors stands any chance of staying on top of these modern IT environments.
An example of such an ecosystem approach is depicted in the diagram below. The key attributes of such an approach are:
- There is a vendor of a common big data back end in which the data from all management vendors in the ecoystem is stored
- The vendor of the big data back end works with the vendors in the ecosystem and cooperates at the technical, marketing and sales levels with these vendors
- The data store is open allowing each vendor to put their data into the data store in a manner of their own choosing
- Each vendor makes their data and its schema available to every other vendor who is using the same data store
- The big data back end is marketed and sold in a manner so that partner vendors who use the big data back end feel comfortable recommending to their customers that the customer buy the back end as a part of the solution
The key things that need to happen for such ecosystems to come into being is that vendors of big data back ends (like Splunk) need to make it technically easy and financially feasible for partnering vendors to put most or all of their data into the back end and then re-use each others data. Splunk is currently the best example of a vendor that has both a technically correct approach in terms of the big data back end, and a well formed ecosystem of third party partners.
However, Splunk is not going to be the only vendor who comes to market with an ecosystem of partners. A year from now, customers will have a number of alternative ecosystems to choose from. And collectively those ecosystems will give customers a viable alternative to the management frameworks, hastening the death of the management frameworks.
Management frameworks are dead because they have been unable to keep up with the pace of innovation in the enterprise computing industry. Management frameworks will be replaced by ecosystems of cooperating vendors that each reuse each others unique data through a common big data back end. Splunk is the first (but not the last) vendor to offer such a back end and to pursue such an ecosystem strategy.