TRAC's recent article "Application Performance Management - The Journey of a Technology Label" talked about how diluted the APM space has became and that it is getting more difficult for end-user organizations to distinguish between vendors' marketing messaging and the true value that these solutions can deliver. Sometimes, we would get vendor requests for a call to discuss their latest customer win, selection process that an end-user organization went though and a type of IT environment that they are managing. The most impressive part about some of these calls is not necessarily how the vendor won that deal, but how some vendors they were competing against even got to be considered in that type of usage scenario.

Recently, we had two application monitoring vendors referring to the same competitor as a company that is "having more success against us than ever before" and "gets involved in some of our deals, but we very rarely see them win". On the other hand, when talking with end-users about the same product, the feedback that we get can range anywhere from them raving about the product to complaining about all of the "blind spots" that it leaves. So, is the solution that this vendor provides good or bad? It is actually very good when used for addressing problems that it was designed to address.

The application performance monitoring market has been growing very fast and some vendors are trying to grow their presence by going after nearly any environment that lacks visibility into application performance. This typically results in them sometimes getting invited to the table in usage scenarios where they have very little chance of winning, while it makes it more difficult for companies that could actually add value in those environments to fight through marketing noise and even be considered. The reality is that even though there are a lot of vendors that play in this space, due to new market dynamics, "shortlists" of vendors that can be truly effective in certain usage scenarios are sometimes very short.

To shed light on this issue, we have been interviewing end-user organizations and trying to find out what they really care about when evaluating technology vendors. The findings came down to three key areas: 1) fit for their IT environment; 2) problems that they are trying to solve; 3) product attributes that they are looking for. We also found significant differences between preferences of different job roles and companies that fall into different categories based on the size of the organization and the industry sector that they compete in. The list of individual criteria that impacts their selection processes was fairly long and SOME of them are listed below.

Our goal was to publish a study that would help end-user organizations make the correct decisions about selecting solutions for monitoring application performance that would be the best fit for their needs. However, the variety of their goals, selection criteria, types of IT environments and company profiles created a challenge of coming up with recommendations that would be truly relevant and actionable and not just a superficial overview of the market.

We realized that the complexity of the application performance space calls for a new report format and the idea for it came straight from our research. Many organizations are reporting that a Web browser is becoming the center of their IT performance efforts, as technology initiatives such as SaaS, IaaS and PaaS are changing how computing resources are being used and information is delivered to business users. As a result, we built a Web application for publishing the research findings that will allow end-user organizations to access information they really care about in a more interactive fashion.

With all due respect to good old PDF, there is too much complexity in the application performance monitoring market to fit research findings in a "one size fits all" publishing format and still make them actionable and relevant for different profiles of end-users. We have been advocating that information only has value if it is actionable and relevant so now it is time for us to "walk the walk".

If you are an end-user of IT performance technologies, we are interested in hearing your thoughts about the new report format before it gets published. TRAC will be hosting a series of demo presentations of the new report format and If you are interested in attending, please send a note to

Industry analysts tend to classify vendors into technology "buckets" and create "labels" for each of them, as that makes it easier to compare products, capture key trends and provide context around problems that these products are addressing. This method also resonates with some technology marketers, as it allows them to partially benefit from promotions that other vendors and media are conducting around a label of a technology bucket their product was put into.

The term "application performance management" (APM) has been one of the hottest technology "labels" over the last few years. Performance of enterprise applications impacts nearly all of the key business goals, and it shouldn't come as a surprise that technology solutions for managing performance of these applications has been very high on IT agendas. With that said, it should be even less of a surprise that technology vendors, who are involved in managing the delivery of applications to business users in any way, realized this opportunity and started calling themselves "APM vendors". However, every "hot" industry term has an expiration date attached to it and sometimes it doesn't take long for a company to go from being one of the biggest promoters of an industry term to getting to the point where it doesn't even want to be associated with it.

Back in 2008, there were more than 50 technology vendors that used the term APM to position products that they provide and that number is now down to less than 30. So, had these 20+ companies gone out of business or completely changed their product portfolios? No, but they had realized that the term APM got diluted and that it is in their best interest to separate themselves from technologies that address the same problem as they do, only from a different perspective.

Emergence of new categories

Being thrown into the same technology bucket with companies that are addressing a similar problem from a different perspective could be a major challenge for many technology companies. Organizations that are in this position typically have two options: 1) wait until the market matures to the point when it becomes obvious that their solution is significantly different than other products in the same "bucket", or 2) coin a new term to describe a category in which their product belongs, promote the heck out of it and hope that it will become an industry accepted term. It took a combination of these two approaches to somewhat change the boundaries of the APM "bucket". That resulted in more market awareness about the differences between two groups of products that are also addressing challenges of managing application performance, but doing it from different perspectives: end-user experience monitoring and business transaction management (BTM).

The increased interest of end-user organizations in having visibility into how their applications are performing, not only from the perspective of their IT departments but from the perspective of business users, resulted in more market awareness about the role that end-user experience monitoring solutions are playing in managing application performance. The market matured enough to become more aware of the fact that different flavors of technologies for monitoring the quality of end-user experience, such as those provided by Aternity, Knoa Software, Coradiant or AlertSite, do not compete against, but complement vendors such as OPNET, OpTier or Quest's Foglight.

On the other side, vendors that specialize in managing application performance from a business transaction perspective also found a way to raise awareness about the differences between their solutions and many other APM products. This resulted in an increased adoption of the term BTM when describing capabilities of these solutions. These solutions are taking a different approach when addressing issues with application performance, as compared to some other APM vendors, and enable organizations to monitor the performance of applications across an entire transaction flow. Some of the vendors that fall in this group include OpTier, Nastel, INETCO, Correlsense, Precise Software, dynaTrace and AmberPoint (acquired by Oracle).

Today, Keynote Systems and dynaTrace announced a strategic partnership that would allow end-users to leverage these two solutions in an integrated fashion. This is the second strategic partnership that Keynote has created in this space over the last two months, as they formed a similar type of relationship with OPNET Technologies that was announced on July 22. These partnerships might be confusing to some, as it might seem that these three companies are essentially doing the same thing: to monitor the performance of business-critical applications. However, while Keynote is specializing in monitoring the quality of end-user experience and performance testing for Web applications from the outside of the corporate firewall, OPNET and dynaTrace provide solutions for monitoring application performance across enterprise infrastructure inside of the firewall.

The general perception of solutions for end-user experience monitoring, such as the one that Keynote is providing, is that they are very effective in identifying when business users are experiencing problems with application performance, but they are not as effective in drilling down into parts of the application delivery chain to isolate and resolve the root cause of the problem. On the other hand, tools for monitoring the performance of internal infrastructure, such as OPNET or dynaTrace, are able to monitor the transaction flow of applications across the network and into the data center, and provide a deeper dive into how applications are performing, what is causing performance problems and how they can be prevented and resolved. TRAC’s recent report “10 Things to Consider When Evaluating End-User Monitoring Solutions” revealed that the ability to integrate tools for monitoring the quality of end-user experience with tools for monitoring enterprise infrastructure is one of the key aspects of having full visibility into application performance. With that said, there is a clear value that end-user organizations can experience when products that include robust capabilities for application performance management (such as OPNET and dynaTrace) get integrated with one of the leading solutions for end-user experience monitoring from outside of the firewall (Keynote).

However, in order to evaluate a true significance of these partnerships, they should be analyzed in the context of some of the key dynamics in this market.

In October of 2009, when we launched TRAC Research, we based our approach for covering IT performance management technologies on two advises that we were given by end-users:

• Don’t evaluate products by throwing them into technology buckets, but talk about what these products can do in specific usage scenarios

• Distinguish impactful from “cool” technologies, meaning discover what are the measurable business benefits from deploying a technology solution, not how “hot” the technology is

We thought that the best approach for doing this would be to launch an end-user survey and ask folks that are using this technology what their experiences are. This is when things started to get really messy. Before we even formulated the questions, we conducted close to 150 interviews with end-users, executives of technology vendors, prominent writers and some true thought leaders in this space to make sure that the questions are spot on to what they care about. Just to clarify, none of us are rookies in this space and for me, this is the 18^th survey of this type that I’ve created. Although, this time, launching the survey was more “interesting” than usual.

One of the key questions in IT performance management is: Does an improved ability to collect more performance data generally lead to an improved performance of IT services? The answer is: No, not necessarily. Actually, a number of end-user organizations that I have spoken with reported that their ability to prevent and resolve performance issues deteriorated after they invested in additional monitoring tools. As new challenges of managing application performance “jump out”, organizations tend to deploy new point solutions that are addressing each of these problems. This does allow them to collect more information about these specific problems, but it doesn’t necessarily allow them to have  better control of the overall IT performance.

Managing application performance is one of the key IT initiatives for end-users, but there is not a single class of technology or solution provider that can address every single issue of managing the performance of business-critical applications. Some major IT management vendors are investing significant resources in acquiring companies to enhance their product offering and enable them to tackle more performance challenges. However, the capabilities needed for end-to-end management of IT performance are rapidly changing and companies that are looking to provide capabilities for addressing each of the major IT management challenges are likely to keep playing “catch-up”. End-user requirements are changing at a pace that is faster than product development cycles or times needed for acquisitions to be initiated, agreed on and completed. So, are organizations that are looking to access all relevant IT performance data through a single platform are out of luck?

Once in a while, IT management vendors pick up a theme that their customers are very interested in; they start building their marketing messaging around it, write white papers about it, and have it all over their websites. Before you know it, what originally was a legitimate request from end-user organizations for addressing challenges that they have, it becomes a marketing term that is very difficult to define for end-users. “Aligning IT with business” is becoming a very good example of that.

The fact is, the majority of end-user organizations are still struggling to come up with a set of metrics that would help them understand how their IT initiatives are contributing to their business goals. These organizations are allocating a significant part of their enterprise budgets to their IT initiatives and they need to figure out:

• How their past investments in IT are contributing to their bottom line
• What criteria they should be using when evaluating the value of new technology investments
• How to prioritize their current IT management initiatives

So the need to align IT is a true pain point for end-user organizations and they are willing to invest in technology that will help them with that. But what technology is the best fit?

Over the last 2-3 years, the term “Application Performance Management” (APM) became an integral part of marketing messaging for more than 70 technology vendors. Even though solutions provided by all of these vendors are helping to improve the speed and availability of business-critical applications, these vendors are providing solutions that are significantly different. These solutions could range anywhere from network performance monitoring to application acceleration, Web management and even managed/carrier services.

However, the APM as a general concept has become relatively easy for decision makers of end-user organizations to digest, as it hits all key pain points that IT organizations are dealing with. As a result, multiple vendors were more than happy to jump on this bandwagon and position themselves as players in this space.

Other than the language in their press releases and marketing collateral, these vendors really have nothing else in common. Technology wise, how similar are the offerings of F5, NetQoS, Keynote Systems and OpTier? They are not similar at all.