Monitoring End User Experience for Virtualized and Cloud Hosted Applications

When the discussion turns to monitoring, that discussion most often focuses upon either monitoring the infrastructure for performance and availability, or monitoring the performance and availability of certain key applications.  While it is essential to monitor both the infrastructure (hardware and software) that supports the key applications in your environment, and the applications themselves, it is also critical not to overlook the single most important person in the environment – the actual end user of the application or IT service.

The History of End User Monitoring – Synthetic Transactions

While the idea of end user monitoring has been around for a while, it has not been (until very recently) very well implemented. The first attempt at understanding the performance of a an application from a user’s perspective was to implement a synthetic transaction system where software robot’s emulated the actions of a set of users by running a script that executed certain functions in the application. This was very useful for functionality and load testing, and was the basis of the early success of Mercury Interactive, who was later acquired by HP. Keynote and Gomez both built substantial businesses around a web service that did response time and availability testing on a synthetic transaction basis and Gomex was ultimately acquired by Compuware.

Emulating end users through synthetic transactions allows enterprises to be sure that applications are up, and to be sure that certain transactions are running before the users wake up or show up for the day and are very useful for these purposes. But it is impossible to write a script that does every (dumb) thing that users do inside of applications, and it is therefore impossible to know what  a real user’s experience is with an application by using just a synthetic transaction monitor.

Enter Real User Monitoring

Real User Monitoring (RUM) is all about understanding what is actually going on within the actual client side application (browser based, or proprietary thick client) of the end user. There are two core approaches to Real User Monitoring:

  • For web based applications, it is relatively straightforward to put code on a web server that inserts Javascript into the end user’s browser. This Javascript agent can then calculate how much time a transaction takes from its inception in the browser, to how long it spends going back and forth on the network between the end user and the web server, and then finally how long the local screen paint time takes.
  • For applications that are not web based like those that are based around Win32 clients, or the emerging set of applications on tablets and smartphones, a Javascript agent will not do the trick. To address either the old (Win32) case, or the newest and most rapidly growing case (tablet and smart phone apps) you need an agent on the device, and you need a profile for that agent that understands the unique transactions of that application.

RUM, Virtualization, It as a Service and the Cloud

So why should you care about RUM? The answer is that if you care about virtualization, IT as a Service (ITaaS), private clouds or public clouds, you should probably care about RUM. Here is why:

  • We have consistently pointed out the difficulty of inferring the performance of a virtualized infrastructure or the applications running on it by looking at resource utilization metrics that pertain to the infrastructure or the applications. To assess the performance of a virtual infrastructure you have to measure its latency. To assess the performance of an application running on a virtual infrastructure you have to measure its response time. Since many organizations incur resistance to virtualizing performance critical applications (due to performance concerns on the part of applications owners and end users) RUM is the only way of scientifically proving to applications owners what the actual response time of the application is to those users.
  • If you have a private cloud or IT as a Service initiative, you are constructing an environment where business requests for services (applications) will be provisioned into a shared infrastructure in a fully automated manner. If you want to performance assure applications running in this kind of environment, then you either need to provision applications monitoring along with automated provisioning of the application (BlueStripe would be an excellent choice for such an approach), or instrument the users with RUM so that you can see what the performance of the application system is (both before and after it is virtualized) from an objective point that is outside of the dynamic and shared environment that is hosting a constantly changing set of applications. The end user’s device will in some cases be the only constant point from which you can measure as you move an application from physical to virtual, and as you constantly change the mix of applications running on any particular host.
  • The degenerate case for understanding the performance of an application is a SaaS delivered application (for example SalesForce.com) where the user is using an iPad, and is sitting in a hotel room, connected to the application over a network that the IT organization never sees. However, it has already been the case that when these users experience slow performance they get their boss (the VP of Sales) to go yell at the IT department (who needless to say can do nothing about the problem).  Understanding the performance of SaaS delivered applications (even to Windows browsers) so that the organization purchasing the SaaS application can hold the provider responsible for performance can only be addressed with an agent on the client side device.
  • To build upon the point above, if your organization is using, or is planning to use SaaS delivered applications, and the users of these applications will be some of the time or all of the time remote, mobile, and travelling and therefore using the public Internet to access these applications, a client side agent is the only way to both know what SaaS applications your users are running and how they are performing.

For the above reasons, RUM is likely to take on an increasingly important role in your monitoring architecture for virtualized, private cloud, ITaaS, and public cloud environments.

The Frontier in Real User Monitoring

RUM did not used to be an exciting problem. Two years ago, you could monitor the performance of web applications reasonably well at the level of the web server as two years ago most web apps were built around a simple HTTP request/response model. Since all of the real applications code ran on the web servers and applications servers (and not in the browser), understanding the last mile to the user was of marginal benefit. For Win32 apps two years ago this problem was confined to a small set of Win32 applications (like SAP R3) for which end user experience was critical, but this was a niche problem addressed by niche vendors (Aternity and Knoa).

In the last two years much has changed. First of all functionality has moved into the browser. Rich Internet Applications written to Ajax, Adobe Flex, Adobe Flash, and Microsoft Silverlight have become much more important. In RIA applications you cannot tell what is happening to the user by watching the protocol stream at the web server (as you can with a simple HTTP web application). Furthermore we have seen the Apple and Google smart phone and tablet platforms emerge as true applications platforms for – in some cases – business critical applications. The importance of these new mobile platforms as applications platforms will only grow over time. As was mentioned above, if users are going to be accessing business critical applications with these devices, and some of these applications will be SaaS or cloud delivered, then client side agents and therefore RUM will be the only viable approach to monitoring the performance of these applications to their users.

Finally, two years ago there were two niche RUM vendors (Aternity and Knoa) both of whom specialized in the RUM or EUE market, and whose products addressed the user experience piece of the equation, but left the rest of the application system to the APM vendors. Now we have seen several of the APM vendors step up to the plate with significant RUM functionality, which goes down the road of making RUM into a feature of an APM suite – providing a true end-to-end view of applications performance to the applications owner.

So Who is Who in End User Experience and Real User Monitoring?

In the history section above we explained that synthetic transactions, while useful for certain tasks do not represent an effective method of implementing RUM, and that the modern state of the art methods require either a JavaScript agent downloaded into the browser at the time the application is accessed, or an agent installed on the end user’s desktop or laptop. Several vendors have recently jumped into this space with interesting new offerings and there have been some interesting enhancements from existing vendors so it is worth going through them and what the different offerings do.

Correlsense

Correlsense is a vendor whose primary commercial offering focuses upon tracing transactions through complex, multi-tiered and heterogeneous business critical and performance critical applications. Correlense’s instrumentation is unique in that it does not live inside of the application run-time (like a Java or a .NET APM tool), but rather that it looks at the interactions between the application and the underlying OS. This allows the back end Correlsense product (SharePath) to support a wide variety of applications and platforms.

Correlsense makes its SharePath RUM product available for free. Correlsense’s strategy in this regard is that if make it easy for you to find out of your end users are having applications performance problems, then Correlsense is then a good candidate to help you find those problems in your back end systems.

dynaTrace

dynaTrace is one of the leaders of the enterprise Java/.NET Applications Performance Management market. dynaTrace is unique in that its PurePath technology actually traces individual transactions through all of the back-end tiers of  of the application system at the Java//NET level. This allows teams that support business critical custom-developed applications to quickly find code problems in production. The premium version of the Ajax Edition covers issues with response time that are local to the Ajax code running in the browser and fully integrates with the enterprise back end dynaTrace APM solution.

Compuware

Compuware acquired Gomez (a web performance management company) and has now integrated the Gomez SaaS platform with its back-end APM Suite (formerly called Compuware Vantage and now rebranded as part of the Gomez product line) and has added RUM functionality targeted at web applications (traditional and mobile) as well as native mobile device applications. The web applications functionality is based upon tags inserted into HTML pages or injected via an F5 load balancer. The support for native mobile applications is based upon a source code agent that can be included into the mobile application by the mobile applications developer.

New Relic

New Relic pioneered the hosted delivery of APM, (Monitoring as a Service) and now has over 10,000 customers using its service. New Relic supports Java, .NET, Ruby, and PHP applications and has recently added a JavaScript based browser response time feature to its service at no additional charge. Existing and new customers simply have to check a box to turn on the the RUM features in New Relic. This makes New Relic into the easiest APM/RUM solution to deploy and get up and running as all you have to do is create an account (New Relic hosts all of the back end pieces), and then drop the New Relic agent into your application.

Aternity

Aternity is one of two vendors (Knoa is the other) that delivers a RUM solution not just for web applications but also for client-resident commercial and custom-developed Win32 applications. Aternity has recently added deep monitoring of the Outlook Win32 client on the fat client side of its offerings, and support for browser based applications that use Ajax on the web applications side of its features.

Aternity has productized and packaged the process by which transactions inside of an application are identified and instrumented – allowing customers and partners to rapidly add support for new applications and transactions. Aternity is therefore in the position of allowing for the coverage of more of the client side waterfront in terms of applications breadth than any other vendor.

Knoa

Knoa is one of the pioneers of the RUM space and focuses upon deeply monitoring every transaction for performance and compliance for the most business critical applications in use in large enterprises – including SAP, Oracle Financials, and several others.

End User Experience Solution Summary

Vendor Scope of Application Support RUM Implementation Method Support Rich Internet Applications Support for Mobile Applications Solution Delivery Model Back End APM Functionality Available
Correlsense
SharePath RUM
All HTTP/S based web applications 

 

Agent downloaded at run-time from web server 

 

No Browser based only On-Premise Solution. Can be installed in public clouds like Amazon Full Transaction Tracing available as a separate product
dynaTrace
Ajax Premium Edition
Web apps, with specific support for Ajax Agent downloaded from web server into browser Yes – specific support for Ajax Browser based only On premise software Yes – rich end to end transaction tracing and monitoring included
Compuware Web applications, mobile browser applications, and native Smartphone/tablet applications HTML Tagging for web and mobile browser applications, native code libraries for Smartphone’s and tables Yes via HTML Tags for Flash HTMl tags for browser based applications – code libraries for native Smartphone’s/tablets On premises solution or SaaS service Full back end APM Solution available as a separately sold product
New Relic Java, .NET, PHP, and Ruby Web apps Agent downloaded at run time from web server No Browser based only Entire solution (APM and RUM) delivered on a hosted Monitoring as a Service basis Robust monitoring of Web, application (Java, .NET, Ruby, and PHP) and database application system tiers included
Aternity All web apps and broad range of commercial and custom Win32 apps Agent installed on Windows PC or Laptop Yes – specific support for Ajax No – Agent is Windows specific On premise software No
Knoa SAP and selected other enterprise applications Agent installed on Windows PC or Laptop No No – Agent is Windows specific On premise software No

Summary

In a world where you are changing everything – by moving applications in whole or in part from physical to virtual, to private cloud and ultimately (at least for some applications) to a public cloud, having an external reference point from which to judge the performance of these applications is essential. That reference point might best be the end user’s workstation, laptop or mobile device. Real User Monitoring solutions are likely to play a prominent role here.

Bernd Harzog (310 Posts)

Bernd Harzog is the Analyst at The Virtualization Practice for Performance and Capacity Management and IT as a Service (Private Cloud). Bernd is also the CEO and founder of APM Experts a company that provides strategic marketing services to vendors in the virtualization performance management, and application performance management markets. Prior to these two companies, Bernd was the CEO of RTO Software, the VP Products at Netuitive, a General Manager at Xcellenet, and Research Director for Systems Software at Gartner Group. Bernd has an MBA in Marketing from the University of Chicago.

Connect with Bernd Harzog:

Tags: , , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

Please Share

Featured Solutions