In October of 2009, when we launched TRAC Research, we based our approach for covering IT performance management technologies on two advises that we were given by end-users:

• Don’t evaluate products by throwing them into technology buckets, but talk about what these products can do in specific usage scenarios

• Distinguish impactful from “cool” technologies, meaning discover what are the measurable business benefits from deploying a technology solution, not how “hot” the technology is

We thought that the best approach for doing this would be to launch an end-user survey and ask folks that are using this technology what their experiences are. This is when things started to get really messy. Before we even formulated the questions, we conducted close to 150 interviews with end-users, executives of technology vendors, prominent writers and some true thought leaders in this space to make sure that the questions are spot on to what they care about. Just to clarify, none of us are rookies in this space and for me, this is the 18^th survey of this type that I’ve created. Although, this time, launching the survey was more “interesting” than usual.

Creating the questions was easy, but defining the terms used in the questions was not easy at all. For example, one of the questions in the survey asked about capabilities that organizations could use when moving their services to the cloud. For that question, we had a set of prepared answer choices covering things like “automated process for adding or subtracting cloud resources” or “load testing of services that are being moved to the cloud”. When we delved more into it, we realized that whatever data that we would get from a question like that would be completely flawed and unusable for feature reports. That came as a result of the fact that there is no common understanding of what cloud computing really means to begin with. How can you normalize any data collected on trends in cloud computing if for some organizations cloud computing services means “any IT services that can be accessed through public Internet”, while for others it means “using a single physical server to host multiple applications and services”?

Also, we have been hearing that making IT performance data more actionable is one of the key goals for end-users. Well, we have also learned that for some folks that means “reduce the number of false alerts (or false positives)” while for others this means “difficulty to correlate performance data collected by different tools”. The same goes for concepts such as BSM, APM or even WAN optimization. Is BSM a management concept or a class of technology? What does end-to-end management of application performance really mean? Does WAN optimization mean accelerating and reducing network traffic or fully managing applications that are transferred over the WAN? Based on hundreds of conversations that we have had, the answer to all of these questions is the same, and it depends on whom you ask.

The key takeaway from these conversations with end-user organizations is that the lines between different classes of IT performance technologies are becoming very blurry and the business value of these solutions cannot be evaluated seperately from technologies that compliment them.

The mission of TRAC Research is not to redefine any of these technology groups or to create a new set of acronyms and/or buzzwords. What we can do is use this survey as a platform for end-users to report their experiences from using different technology capabilities and, in turn, report that back to the market. From there, it should be easier to understand the true value of IT performance management products, regardless of any technology bucket that they might be thrown into.

The survey is still open and you can access it here: 2010 IT Performance Management Survey

The survey is consisted of several sections, with each of them being specific to different flavors of IT performance management, such as business service management (BSM), network and application performance monitoring, virtualization and cloud management, Web performance, business transaction management (BTM), WAN optimization and performance testing. Respondants will only be asked questions from no more than two of these sections that are the most relevant to them.

All respondents will get more than 20 reports that we’ll publish from this survey for free.