If you've ever gotten an earful from an irate business user complaining about sluggish performance, or had fingers pointed with IT when network capacity is gobbled by a particular application, you know that good APM is worth its weight in log files. The complexity of today's networks means application performance management is required for all but the smallest enterprises, yet choosing the right system is a complex undertaking.

And, those who've already dabbled in performance management with products like CA eHealth, HP OpenView Performance Insight or BMC's Performance Manager are left with a dilemma: Start fresh with a dedicated, holistic APM suite, or cobble together a system with existing tools? If you decide to start anew, get ready to justify that decision and crack open the piggy bank—APM implementations start around $80,000 and can scale well over $1 million for large companies with many applications. To sort through the wide variety of data-collection methodologies and vendor approaches in the market, and see whether they're worth the price, we decided to put APM offerings to the test and determine if one is superior for collecting and reporting on application performance issues.

Impact Assessment Click to enlarge in another window

The Basics
Most APM products gather data either by actively inserting additional traffic into the environment or by passively collecting real user data. APM products comprise a few specialized elements, though not all will support all types of monitoring:

• Synthetic transaction software, found in suites from BMC, HP Mercury, IBM Tivoli, Symantec and others, resides in key locations in the infrastructure and generates traffic that simulates use of the application, then reports performance results.

• Network probes, provided in CA Wily, NetIQ AppManager, Quest Software Foglight and others, reside as physical devices in the network and are typically connected to a SPAN port on a switch. These probes track and classify application traffic by monitoring Web application sessions or TCP traffic.

• Application server agents are widely supported; they reside on the app server and report on specific metrics to determine the cause of a performance problem. These agents may load at runtime.

• Client agents, currently offered only by Compuware and HP Mercury, live on the end-user machine and monitor application performance using APIs or TCP sockets.

• System agents look deeply into the performance of OSes and physical server hardware and report on CPU, disk performance and other critical components. BMC, Compuware, Quest and others use system agents.

• Application performance integration is universal and relies on existing hardware, systems, application and network monitoring data to create application service levels that help IT pinpoint the cause of a performance problem. Sometimes also called business-service management or service-level management software.

For deep background on this class of technology, see our in-depth APM primer.

TCO Patrol
To determine if you can justify the cost of APM, weigh how poor application performance is affecting the bottom line against the added management effort, deployment costs and maintenance fees inherent in APM. Decide whether you can stomach application server agents, which are necessary to gain deep application-performance metrics.

IT will generally hear about severe application performance problems. It's ferreting out underlying causes, then fixing the issues, that's hard. And, too often intermittent problems are never called in, even though they have a real impact on productivity. If you suspect unreported issues, consider engaging a vendor, such as BMC, Keynote or Mercury, that does assessments on an application service provider basis. These vendors simulate customers' end-to-end Web site experiences.

Solving problems requires an understanding of application complexity and the number of dependent components, such as system-level CPU utilization, network performance, database latency and client problems, all of which can have adverse affects on an application. While many organizations monitor some of these components, few have the visibility into complex interrelationships needed to decipher the impact a performance problem in one area has on the whole system. That's where APM comes in.

Making the Right Choices
Our premise of a holistic APM environment implies a configuration that functions in harmony with the surrounding network, all the pieces fitting gracefully into the bigger picture. While some vendors are working diligently to this end, most still offer disparate parts of the puzzle. We'll test how well suites collect data and see if agents, although complex to implement, are really critical, especially on application servers and supporting system components.

Synthetic-transaction-based products say they can detect performance problems in scenarios where deploying agents isn't practical. We'll find out how well they fulfill that promise. As an alternative, some organizations use appliances that track and measure end-user response times without an agent. But do you need client agents installed to really quantify when a user is experiencing unacceptable performance? If an organization is monitoring most components of its application environment, is it even worth it to deploy a holistic APM architecture? Clearly, quantifying ROI can be a real challenge.

While organizations can combine APM pieces from different vendors, the trend in the vendor community is toward using a single vendor for tight pre-built integrations. APM vendors have realized that this approach will provide the most value to organizations—while also ringing up larger sales.

Unfortunately, for many organizations this holistic approach may not realistic because of the cost, so they still need individual components that can provide greater visibility into application health than they receive today. As we review APM offerings we'll discuss ways to move forward on a budget.

Special Cases
• APM has a role to play in helping virtualization move into an operational environment. Thus far, few enterprises have enjoyed the power and space savings represented by virtual data centers because IT managers are hesitant to migrate critical business applications without proactive performance monitoring. Few organizations have a good grasp of the current state of their application landscapes, and they fear degrading performance by virtualizing.

• Telecommunications and IT service providers, facing increased industry consolidation, are looking for ways to offer more value-added services. These typically involve the application service infrastructure and APM. Without an understanding of application performance, however, they're not able to provide service level agreements and, more importantly, quality service to their customers.

• Driven by regulations such as Sarbanes-Oxley or HIPAA, or standards such as ITIL v3 or ISO 2000, IT service providers are also driven to CIO dashboards or business service management suites that depend on APM. Any dashboard requires an understanding of the applications and the underlying infrastructure that is responsible for supporting it. Fault management, while still important, is becoming less of a focus as organizations seek to be more proactive and prevent faults from occurring in the first place.

APM Architecture Click to enlarge in another window

• How does your product collect server hardware, OS, application and network-performance information?

• How does it discover the root cause of a performance problem across application, system and network layers?

• Does it require proprietary agents?

• Does it perform synthetic transactions?

• Is it a hardware-appliance or software-based product?

• What applications can it monitor?

• Does the product work with existing management tools?

• What type of historical performance reporting is available?

• Does it baseline the existing environment, then report on derivations from that baseline?

Michael Biddick is a Contributing Editor for Network Computing and executive vice president of solutions at Windward Consulting Group, a firm that helps organizations improve IT operational efficiency. Write to him at mbiddick@nwc.com.