In Beware of the Franken-Monitor, we explained how many enterprises ended up with Franken-Monitors and the dangers associated with assuming that the present state of management tools can make the transition into the software-defined data center (SDDC) and the cloud. In Getting Rid of Your Franken-Monitor, we explained how to use green-field islands to put in place new ecosystem-based management stacks with the intent of eventually retiring your legacy management stacks. In this post, we detail how one could deploy one example of such an ecosystem of tools based upon Splunk and the vendors that comprise its ecosystem.
Comparing Frameworks and an Ecosystem-Based Approach
Before we get into the details of Splunk and its ecosystem of partners, it is important to review the differences between the old framework approach and a modern ecosystem approach. The two approaches are depicted below. At a high level, the key difference is that a Franken-Monitor is a loosely integrated collection of disparate products, each with its own databases and consoles, while a modern ecosystem approach relies upon a common big data back end as the store for all of the data from all of the various management solutions. The ecosystem approach is therefore superior to the framework in two respects: 1) The ecosystem approach recognizes that no one product can monitor everything and leaves it open to a market of cooperating vendors to solve that problem, and 2) Each vendor that participates in the ecosystem contributes its data to the data store but also can query the data from other participants as a part of adding its value. The ecosystem approach thus allows for effective cross-silo management of IT in a manner that has never before been possible.
Comparing the Franken-Monitor to the Reference Architecture for Managing the SDDC and Clouds
The Splunk Ecosystem Approach to Managing the SDDC and the Cloud
Splunk got its start collecting log data. The amount of data and the diversity of the data that one can collect in the form of logs caused Splunk to make some important and fortuitous design decisions early on. Splunk avoided the trap of relying on a relational database and instead built the first big data back end designed to handle the volume and diversity of what was then just log data.
Fast forward to today, and Splunk is much more than a log collection system with indexing and the ability to dashboard any collection of metrics that you want. Splunk has not only built out its own library of Splunk apps that collect data from a variety of platforms and products, it has also very effectively partnered with vendors who themselves collect unique data. It is in the combination of volume of commodity log data (the part you have to have as a starting point); operational data from servers, networks, and virtualization platforms; and unique data that is not available via a commodity management API that the true power and uniqueness of Splunk becomes apparent.
Understanding the Splunk Ecosystem
The table below lists some important and popular “Splunk Apps,” including apps from Splunk that collect data from a variety of environments, apps from Splunk that contain their own user interface like the apps for VMware and Citrix, and apps from third-party partners. Special attention should be paid to the quality and diversity of the apps from third-party partners, as the winning vendor of these ecosystem strategies will be the vendor with the best ecosystem, not just the vendor with the best big data back end.
|Vendor||Product||Product Description||Link in SplunkBase||Link to Vendor|
|Splunk||Splunk for Enterprise Security||Splunk App for Enterprise Security is a scalable security intelligence platform with the flexibility to make tens of terabytes of data per day security relevant through comprehensive analysis capabilities that break down organizational data silos and data collection issues.||http://apps.splunk.com/app/263/||http://apps.splunk.com/app/263/|
|Splunk||Splunk for PCI Compliance||The Splunk App for PCI Compliance is a Splunk-developed app that supports the data collection, continuous monitoring, and alerting requirements of the PCI Data Security Standard.||http://apps.splunk.com/app/1143/||http://apps.splunk.com/app/1143/|
|Palo Alto Networks||Splunk for Palo Alto Networks||Splunk for Palo Alto Networks leverages the data visibility provided by Palo Alto Networks’ firewalls with Splunk’s extensive investigation and visualization capabilities to deliver an advanced security reporting and analysis tool. This app enables security analysts, administrators, and architects to correlate application and user activities across all network and security infrastructures from a real-time and historical perspective.||http://apps.splunk.com/app/491/||Splunk Solution Brief – Palo Alto Networks|
|Cisco||Cisco Security Suite||The Splunk Cisco Security Suite includes multiple apps and add-ons that combine to create one solution running on the Splunk engine. The solution builds on the core Splunk capabilities, giving the security team the ability to search machine-generated data, perform root cause analysis and apply statistical analysis to measure adherence to key performance indicators (KPIs).||http://apps.splunk.com/app/525/||https://marketplace.cisco.com/catalog/products/4178|
|F5||Splunk for F5 Security||Splunk for F5 Security is a collection of field extractions, saved searches, reports, and dashboards for your F5 Access Security Manager and Protocol Security Manger||http://apps.splunk.com/app/815/||https://www.f5.com/products/technology/splunk/|
|FireEye||FireEye™ Malware Protection System||FireEye provides dynamic analysis of zero-day attacks within a virtual environment. This yields real-time malware security intelligence that is then used to protect the local network. This intelligence can also be shared to all subscribers of the FireEye Malware Protection Cloud.||http://apps.splunk.com/app/409/||Splunk for FireEye|
|Operations and Infrastructure Performance Management|
|Splunk||Splunk App for Microsoft Windows||The Splunk App for Windows provides examples of pre-built data inputs, searches, reports, alerts, and dashboards for Windows server and desktop management. You can monitor, manage, and troubleshoot Windows operating systems from one place.||http://apps.splunk.com/app/272/||http://apps.splunk.com/app/272/|
|Splunk||Splunk App for Unix and Linux||The Splunk App for Unix and Linux provides rapid insights and operational visibility into large-scale Unix and Linux environments. With its new prepackaged alerting capability, flexible service-based hosts grouping, and easy management of many data sources, it arms administrators with a powerful ability to quickly identify performance and capacity bottlenecks and outliers in Unix and Linux environments.||http://apps.splunk.com/app/273/||http://apps.splunk.com/app/273/|
|Splunk||Splunk App for VMware||The Splunk App for VMware provides deep operational visibility into granular performance metrics, logs, tasks and events, and topology from hosts, virtual machines, and virtual centers. It empowers administrators with an accurate real-time picture of the health of the environment, proactively identifying performance and capacity bottlenecks.||http://apps.splunk.com/app/725/||http://apps.splunk.com/app/725/|
|Splunk||Splunk App for Citrix XenApp||This app provides operational intelligence about your Citrix XenApp environment, including but not limited to details about your users, sessions, devices, licenses, and servers. It currently supports XenApp 4.5, 5.0, 6.0, and 6.5 and will be introducing support for all versions going forward.||http://apps.splunk.com/app/1020/||http://apps.splunk.com/app/1020/|
|Splunk||Splunk App for NetApp ONTAP||The Splunk App for NetApp ONTAP uses the NetApp Manageability SDK combined with read-only API access to one or more FAS controllers to provide real-time and historical visibility into the performance and configuration of your NetApp storage infrastructure.||http://apps.splunk.com/app/1293/||Splunk for Operational Intelligence|
|Splunk||Splunk App for Cisco UCS||The Splunk App for Cisco UCS combines the power and flexibility of Splunk with a tailored experience for Cisco UCS. Splunk for Cisco UCS gathers data from one or more Cisco UCS Managers.||http://apps.splunk.com/app/1126/||http://apps.splunk.com/app/1126/|
|Cisco||Splunk App for Cisco IOS||The Cisco IOS app sets different Cisco-specific fields used for identifying data from Cisco IOS, IOS-XE, NX-OS, and IOS XR devices.||http://apps.splunk.com/app/1352/||http://apps.splunk.com/app/1352/|
|InterMapper||InterMapper App for Splunk Enterprise||The InterMapper App for Splunk Enterprise is a tool for network monitoring and mapping from InterMapper, which enables the network administrator to proactively probe network hardware, software, and bandwidth utilization in real-time to create actionable knowledge within Splunk.||http://apps.splunk.com/app/1072/||http://www.intermapper.com/products/Splunk-App/|
|NetFlow Logic||NetFlow Integrator||This app provides real-time visibility into your your network traffic. It allows you to view top talkers, top destinations, and top applications, and to search network conversations. The app also shows traffic going through your network devices, allowing drill down to the the interfaces, showing percentage of usage.||http://apps.splunk.com/app/409/||https://www.netflowlogic.com/node/62|
|Application Performance Management|
|Splunk||Splunk App for Microsoft Exchange||Splunk App for Microsoft Exchange gathers performance metrics, log files, and Powershell data from all aspects of Microsoft Exchange and its underlying infrastructure, including POP3, IMAP4, ActiveSync, Exchange Audit Logs, Outlook Web Access, IIS, and the Windows Event log, and presents the data in a series of operational dashboards covering IT Operations, Security Operations, Capacity Planning, and Helpdesk functionalities.||http://apps.splunk.com/app/1660/||http://apps.splunk.com/app/1660/|
|Splunk||Splunk App for Microsoft SQL Server||The Splunk App for SQL Server provides IT monitoring for your Microsoft SQL Server 2008R2 and above servers. It covers monitoring of each instance performance, security incident analysis, index analysis, and query analysis services.||http://apps.splunk.com/app/1524/||http://apps.splunk.com/app/1524/|
|Splunk||Monitoring of Java Virtual Machines with JMX||Monitoring of Java Virtual Machines with JMX (formerly Splunk for JMX) can be used to poll local or remote JMX Management Servers running in Java Virtual Machines across your entire infrastructure, index MBean attributes and outputs from MBean operations, and listen for MBean notifications.||http://apps.splunk.com/app/668/
|Splunk||Splunk App for Hadoop Ops||Monitoring and managing Hadoop cluster operations is a big data challenge of its own. Splunk can collect and correlate events and run-time metrics from every service on every host to every job from every user. With HadoopOps, you will gain total visibility into Hadoop’s operation status, search across the entire cluster in real-time, and troubleshoot and analyze Hadoop with rich, interactive views.||http://apps.splunk.com/app/1173/||http://apps.splunk.com/app/1173/|
|AppDynamics||AppDynamics App for Splunk||The AppDynamics app allows you to mine application performance monitoring data from AppDynamics using its REST API. You can then slice and dice the data within Splunk using Search Processing Language (SPL). It also contains a notification client that can be extracted to an AppDynamics controller that will relay event and policy violation notifications in AppDynamics to Splunk and has the ability to cross-launch into AppDynamics from Splunk.||http://apps.splunk.com/app/1297/||http://www.appdynamics.com/extension/splunk|
|AppEnsure||AppEnsure 3.0||AppEnsure 3.0 identifies every application by name, maps the end-to-end topology for each application and then measures response time and throughput for each application. All of this data is then put in Splunk (no separate data store). The data already collected by Spunk is then used for automated root cause analysis when response time or throughput degrade. The user interface for AppEnsure 3.0 is a native Splunk app implemented in the new Splunk Web Framework.||http://www.appensure.com/splunk/|
|Boundary||Boundary App for Splunk Enterprise||The Boundary App for Splunk makes this data automatically available inside Splunk for users to index, search, and alert on. This makes it easier for IT operations staff to resolve application and network performance issues before end users are impacted.||http://apps.splunk.com/app/1494/||Boundary and Splunk|
|Compuware APM||Compuware App for Splunk||Compuware APM dramatically improves the depth of application data accessible within Splunk. PurePath™ technology unlocks powerful insights by automatically injecting itself inside the application, with transaction-level visibility into method arguments, query parameters, and even business context. Published events are pre-correlated end-to-end with a unique ID and streamed directly into Splunk.||http://apps.splunk.com/app/1593/||Compuware APM for Big Data Applications|
|ExtraHop||Wire Data App for ExtraHop||The Wire Data App for ExtraHop imports real-time network, web services, database, storage, and memcache metrics into Splunk that are otherwise difficult or impossible to log. To work, this app requires an ExtraHop appliance. You can download a free ExtraHop virtual appliance at http://www.extrahop.com/discovery.||http://apps.splunk.com/app/1125/||http://www.extrahop.com/partners/technology/splunk/|
|INETCO||NetStream||Download the only data streaming application that can automatically deliver full message payloads, transaction request/response timings, and network address data into Splunk. INETCO NetStream is for application support teams and IT operations and security analysts who need quick access to data on user interactions, unusual usage patterns, application performance, and business activity.||http://apps.splunk.com/app/1598/||INETCO NetStream|
|Vello||FarSight||FarSight® by Vello provides a collection of dashboards for Splunk® Enterprise that correlate network activity with applications performance monitoring, enabling administrators to fine tune networking resources in real-time for optimal performance. FarSight facilitates efficient enterprise resources allocation and cost-effective capacity planning.||http://apps.splunk.com/app/1590/||http://www.vellosystems.com/products/vellos-applications/farsight/|
|Puppet Labs||Puppet Pulse App for Splunk||The script connects to the Puppet master’s inventory web service to generate a list of known hosts that are managed by the Puppet master. Puppet uses SSL certificate authentication to authenticate Puppet agents and other clients to the Puppet master’s inventory service. By default, this script expects to run as root and thereby use the host-agent’s private key and certificate and should just work when run that way.||https://github.com/wcooley/splunk-puppet|
|Opscode||Bistro||Bistro is the Splunk App for Chef. Bistro aims to provide simple and modest visibility into your Chef deployment through simple dashboards and metrics.||http://apps.splunk.com/app/1199/||http://apps.splunk.com/app/1199/|
|Pentaho||Pentaho Business Analytics for Splunk Enterprise||Pentaho Business Analytics coupled with Splunk Enterprise provide a complete end-to-end analytics platform combining analytical data integration with a full range of traditional and advanced analytic capabilities for data discovery and insight, including reporting, dashboards, visualizations, and predictive analytics. Pentaho and Splunk bring together IT and business users for easy access, integration, visualization, and exploration of any data.||http://apps.splunk.com/app/1554/||Pentaho and Splunk Partnership|
|Prelert||Anomaly Detective||Anomaly Detective’s predictive analytics extend Splunk to enable highly accurate real-time alerts without the need to set thresholds.||http://apps.splunk.com/app/1306/||Prelert Anomaly Detective|
|Netuitive||Splunk App for Netuitive Cloud||The Splunk App for Netuitive Cloud allows users to format and stream data to a cloud instance of Netuitive’s award-winning predictive analytics solution. Netuitive replaces human guesswork with automated mathematics and analysis to visualize, isolate, and proactively address IT performance issues before they impact the business.||http://apps.splunk.com/app/1464/||Netuitive Taps into Splunk Machine Data|
|Splunk||Splunk App for AWS||The Splunk App for AWS integrates with AWS CloudTrail and offers a pre-built knowledge base of critical dashboards and reports. Using the Splunk App for AWS, you can gather important insights into security-related activity such as unauthorized access attempts, simultaneous logins from geographically disparate locations, and frequent changes to access control privileges. You can ensure security and compliance with continuous monitoring and a full audit trail of user activity.||http://apps.splunk.com/app/1274/||http://apps.splunk.com/app/1274/|
|Splunk||Splunk App for Microsoft Azure||Enterprise applications deployed in Azure typically log data into Azure diagnostic storage tables. This app enables connecting to and retrieving data from Azure diagnostic storage into Splunk for analysis and reporting purposes.||http://apps.splunk.com/app/1586/||http://apps.splunk.com/app/1586/|
Replacing Frameworks with Splunk and the Splunk Ecosystem
In most organizations, replacing one management tool with another is an intensely political exercise. People who have spent years (or decades) accumulating experience and expertise with a tool, and who may even have the name of the tool in their job title, do not take kindly to the idea that you are going to rip out the very product that justifies their existence in the company.
For the above reason, a frontal assault on an enterprise management team with Splunk and its ecosystem is not very feasible, and is in fact ill-advised. The far better approach is to find a use case that the enterprise frameworks are not handling effectively now. These include:
- Security log monitoring. You can even trot out Gartner’s Magic Quadrant showing the Splunk is in the leaders’ quadrant for Security Information and Event Management. This is a safe beachhead to attack with Splunk, since Splunk has plenty of reference customers to quote who can prove that Splunk does a great job of handling this use case.
- Once you get your security log data into Splunk, your next step should probably be to add the Operations Management data for your storage, network physical server, and virtual server environments. The good news here is that Splunk has a wealth of internally developed applications that essentially turn Operations Management into a feature of Operational Intelligence. If the enterprise framework guys get suspicious, just tell them you are trying to make sure that you can assess the impact of security issues on the operation of the environment.
- The real win will start to occur when you can get true application performance data into Splunk. This is an area in which Splunk already has a wealth of partnerships, and the list is likely to grow over time. The objective here needs to be to get the names of the applications, their response times, their throughputs, and their topologies into the Splunk data store for all applications that you care about. AppEnsure, ExtraHop and INETCO are all focused upon getting a breadth of application data into Splunk, and AppDynamics does a great job with the data for the custom developed applications that it supports.
- The icing on the cake happens when you apply self-learning big data analytics to the diverse set of data that you now have in one data store. The key here is that once you get everything wired into Splunk, it will be impossible for humans to know what to ask for before problems occur. Self-learning analytics, properly applied, can detect anomalies in streams of data coming into Splunk and let you know that there is a problem before you even launch a query into Splunk. Operational analytics solutions from vendors like Netuitive and Prelert play a critical role here.
In the spirit of not picking a fight with the blind dinosaurs and their legacy management frameworks, you might also want to focus on environments that the frameworks are late in catching up to. At the top of this list would be data center virtualization, your software-defined data center, private clouds, hybrid clouds, and public clouds.
Is This Over and Has Splunk Won?
The IT Operations Management market is measured by Gartner to be $20B in size. That means that companies worldwide spend $20B a year on IT Operations Management software and maintenance of that software. In its most recent quarter, Splunk generated almost $80M in revenues, putting it on a yearly run rate of $320M. This means that this year Splunk comprises about 1.6% of the IT Operations Management market. In other words, 98.4% of the market is in the hands of other vendors and is up for grabs. So, the answer to this question is that while Splunk is winning, this war is just getting started. While IBM, BMC, CA, and HP face difficult technical and business model challenges, their resources make it possible for them to reinvent themselves and to participate in this new approach to management software. We are also likely to see at least two initial public offerings of new management software vendors in 2014 (AppDynamics and New Relic), both of which have already taken steps to pursue ecosystem strategies of their own. In other words, by the end of 2014 there will be at least three new publicly traded companies (Splunk, AppDynamics, and New Relic) fighting it out with each other and the legacy vendors to reinvent the management software market around big data back ends and ecosystem strategies. The bottom line: this is far from over.
Management frameworks and Franken-Monitors cannot manage modern environments like the virtualized data centers, software-defined data centers, and clouds. The Splunk ecosystem is perfectly positioned to manage these new environments and allow you to keep up with the waves of change rolling through your IT environment.