Link: http://www.rms.net/tut_perform.

htm

Subject: IT Performance Management

Tutorial: Introduction To IT Performance Management And Measurement

Issue: IT organizations are being challenged to assure that their projects /
activities, (1) are aligned with overall strategic goals and business objectives, and
(2) deliver promised results on time and within budget. Although accustomed to
measuring and managing technical performance (e.g., network downtime), many
IT organizations have a poor track record of measuring and managing non-
technical performance (e.g., budget, project schedules, and project risk).
Purpose: This is a high-level introduction to performance management and
measurement designed for IT professionals who have little or no experience in
non-technical performance management and measurement. The major activities
involved in measuring and managing the contribution of IT projects / activities to
an organization's strategic goals and business objectives are discussed.
Objective: Our aim is to provide an understanding of the basic principles of non-
technical performance measurement and management systems applicable to IT
projects and activities. We discuss the basic issues: (1) what is performance
management, and why is it of concern to IT organizations; (2) what needs to be
done to develop and implement performance management and measurement;
and (3) things you should know before you undertake an IT performance
management initiative.
Your Comments: The goal of this web site is to provide useful information to
those who must cope with the issues; this tutorial is part of that effort. RMS would
appreciate your comments and suggestions to help us make the information as
useful as possible. Please take a few minutes to complete the evaluation form at
the end of this tutorial.








Contents Map















Part I.
What is Performance Management?
In general, performance management refers to the use of performance
measurement information to help:
define organizational goals and objectives in clear, tangible, and quantified
terms,
develop project / activity plans designed to attain the goals and objectives,
routinely monitor actual performance vs. plans,
analyze significant performance deviations,
advise (via routine performance reports) key managers of situations
requiring attention,
formulate corrective action plans, and
implement corrective actions to remedy performance deviations or modify
plans.
The goal of performance management is to assure that organizations link plans
to strategic goals and business objectives, make funding decisions in light of
project / activity benefits and outcomes that support those goals and objectives,
and actively manage projects and activities to assure that the planned benefits
are realized.
In short, it is actively and proactively managing an organization to assure that it
achieves pre-determined levels of performance. In contrast, management that
takes action only after a performance problem significantly affects the business is
engaged in crisis management and damage control, not performance
management.
What is the performance management issue for IT organizations?
Many senior managers believe that IT organizations are:
wasteful and inefficient (as evidenced by chronic project cost overruns,
late deliverables, etc.), and
not aligned with strategic goals and business objectives.
Senior management is often handicapped in its decision-making for IT because it
has difficulty determining:
if the most important strategic and business needs are being met
if they have been presented with the best IT options for decision-making
the appropriate funding levels for IT projects / activities (how much to
budget)
whether the IT organization is likely to deliver the promised benefits and
results on time and within budget
Typically, these are characteristics of organizations that have paid insufficient
attention to measuring and managing IT performance and results, and have
failed to establish an overall analytic framework for IT evaluation and decision-
making (See the tutorial "The IT Investment Management Approach" for a
description of an extended analytic framework.)
Instituting adequate performance management is an important step that CIOs
and IT organizations can take to restore their credibility. Meaningful performance
measurement and management reporting will tell IT and senior management
what is working and what is not; especially important is the ability to recognize
potential problems in time to take corrective action.
Dont IT organizations already measure and manage performance?
IT organizations typically measure, monitor, and act upon data provided by a
broad range of performance management tools and techniques; these effectively
monitor and control what we shall refer to as technical performance (e.g., through
"configuration management" and "fault management").
When asked about the performance management systems and processes, CIOs
and IT managers will typically speak at length in terms, such as:
the polling function of the topology map provides a snapshot of overall
network vitality, and it can issue alerts when nodes become unavailable or
are slow to respond. When a node is failing, it automatically alerts network
management. To get more information on a node, we drill down for critical
station statistics, and

the protocol distribution function tracks total packets, octets, percent
utilization, packet rate, and average packet size.
Management of technical performance is an important part of an IT
organizations job; there is no doubt about that. But, the IT organizations
management responsibility doesnt stop there; IT managers are responsible for
assuring that they are meeting their organizations highest priorities and business
needs to the extent possible with the available resources.
This performance information that most IT organizations lack relates to the non-
technical aspects of the job; the information that can tell IT and senior
management whether:
1. IT projects / activities are progressing as planned
2. there are telltale signs of future performance problems
3. there are troubled projects / activities that require higher managements
intervention, or
4. actions already taken to correct problems actually worked.



Part II.
IT Performance Management - What are some of the key terms and
concepts?
Before we proceed further to describe how non-technical performance
management works, some key terms and concepts should be discussed. Some
of the terms used throughout this tutorial will be interpreted differently by different
readers; therefore, a glossary of performance management terms is provided.
These are meant to be useful interpretations, not standard definitions.
Three of the concepts underlying performance management, which often confuse
novices, are an organizations "management system", performance measures,
and performance metrics:
A management system is an interconnected set of processes, and a
process is a set of activities that produce products or services (results).
For performance management and measurement purposes, products and
services are treated alike. That is, the output of a process might be a
product (such as a computer or a project deliverable) or a service (such as
PC support help desk assistance). Products and services are tangible,
measurable, and susceptible to analysis; thus,

Performance measures are indicators that can be systematically tracked
to assess progress made in achieving predetermined objectives (e.g., the
number of NT workstations deployed to replace Unix workstations) or
service levels (e.g., help desk call resolution time).

Performance metrics are standards of measurement (such as minimum
acceptable elapsed time to resolve help desk calls). A metric establishes a
benchmark target that is compared to actual performance - the difference
between the benchmark (the plan) and actual performance provides
insight into what is working as planned and what is not.
Dealing with these concepts can become complicated and are often a bit
intimidating and confusing at first. What is important is recognition that
performance management is a sophisticated management tool and that we are
introducing you to some of the basics.
How does non-technical IT performance management work?
Performance management involves the routine, and sometimes non routine,
measurement of key aspects of IT project / activity performance and making this
information available to decision-makers. The goal is to assure that the benefits
of IT are realized as planned.
Preparatory steps for performance management initiatives include: definition of
the overall analytic framework, formulation of implementation plans, assignment
of analytic responsibility, staff and management training, and conceptual design
of the necessary supporting systems applications.
Once the foundation is established, performance management involves a
number of recurring steps:
1. Gather performance data on those variables of interest to IT and senior
management.

2. Analyze the data to determine normal (baseline) levels.

3. Determine appropriate performance thresholds for each important
variable; exceeding a threshold indicates a problem worthy of attention
(e.g., achievement of a major project milestone is 30 days, or more,
behind schedule) and possibly action.

4. Periodically (often monthly) monitor performance variables for deviations
from established thresholds (a.k.a., variances). When a performance
threshold is exceeded (e.g., vacant PC support staff positions increases to
10% when the acceptable vacancy threshold is 3%), analysts identify a
variance and determine its significance. Depending on a number of factors
(e.g., severity or risk), analysts may take one of several actions, including:
identifying the project / activity as "at risk" and more closely
analyzing performance during the next reporting period
contacting the project / functional manager for a verbal variance
explanation
"margin-left: 25">investigate the cause of the variance and
formulate corrective action recommendations
5. Report on performance to senior management (at least annually), and
functional management (at least quarterly).

6. Initiate corrective action.
The foregoing steps are part of an ongoing process established to support a
performance planning, monitoring, and control system. The intent is to provide
management with a stable, recurring "early warning" mechanism to detect and
correct incipient performance problems before they actually become problems.
How does non-technical IT performance management work?(cont'd)
erformance management also employs ad hoc methods to assess and predict
performance. These include techniques such as simulations, trend analysis, and
in-depth performance reviews:
Simulation (e.g., "what if" analysis) can be used to project how changes in
certain activities might affect others.

E.g.: If the PC support group is deploying NT workstations to replace Unix
workstations very much ahead of schedule, how this will affect the
software development group that must convert Unix-based software for
use on NT. Such simulation can effectively alert management to
impending problems (e.g., the software development group cant keep up
with the accelerated NT conversion schedule; as a result, business users
would lose mission-critical functionality). This knowledge would enable
management to implement corrective measures (e.g., either put NT
deployment back on its original schedule or accelerate the software
development process by increasing development staff).
Trend analysis can be used to reveal recurring difficulties that are not
detected by ongoing performance monitoring.

E.g.: Senior management has imposed a standard employee-to-consultant
ratio for all functional areas; it is the same for accounts payable as it is for
the IT organization. A trend analysis, possibly spanning several years,
might reveal that the realities of the IT business consistently force the IT
organization to either violate the standard or shut-down important activities
because sufficient numbers of technical personnel cant be hired. The
analysis would tell management that the problems of the past are likely to
recur unless the standard for IT is changed.

Performance reviews (a.k.a., performance audits) can provide IT and
senior management with objective, often comprehensive, assessments of
the performance of a major function, activity, system, or process.
Performance reviews (audits) are not financial audits, although they may
include financial elements (e.g., a functional performance review might
include evaluation of the adequacy of budget resources to accomplish
business objectives).
These are some of the routine, and not so routine, tools and techniques used in
the practice of performance management.
IT Performance Management - How to get started
Measuring performance isn't easy; too often, performance measurement
initiatives falter because:
those charged with planning and implementing the initiative had little or no
prior experience with performance management and measurement (often,
with the best of intentions, they repeat common, predictable, and
sometimes fatal mistakes)
managements information needs were not identified in advance, and a
well-thought-out conceptual framework and implementation plan was not
developed (in one case, after a years work, the first performance report
was submitted to senior management two months late; the designers were
told "this is not the information we need to see")
too much performance information materialized too soon (the
organizations capacity to assimilate, interpret, and react to it was
overwhelmed)
To help assure the success of a performance management initiative:
understand managements information needs
have a clear picture of what you are trying to achieve and how you will do
it
involve people who have actually established and managed performance
management systems
study what other organizations are doing - learn what works and what
doesnt
manage expectations - make sure all key participants understand and
agree on what will be accomplished and when
use a limited pilot to gain experience (in other words, confine your
mistakes to a small area that you can fix quickly), and demonstrate early
results
Performance management initiatives can materially improve the way an
organization plans and conducts its business; careful preparation and a
disciplined approach can make it happen.
Next, we will briefly discuss some of the other things you need to know.















Part III.
Five Things You Should Know Before Undertaking An IT Performance
Management Initiative
Before undertaking a performance management improvement initiative, you
should be aware that:
1. they are often implemented in concert with other management
improvements

2. improvements take time and sustained management commitment

3. adequate resources must be made available

4. the IT organization will need to learn new management techniques

5. some of the approaches recommended in current literature can be risky

1. Performance management improvements are often implemented in
concert with other management improvements
IT Performance management initiatives are often implemented by organizations
that want to fundamentally shift the focus of management decision-making from a
preoccupation with staffing levels and costs to a balanced focus including key
"outcomes". Outcomes are results expressed in terms of the real difference that
an organizations work makes on the way business is conducted and what the
organization achieves.
To accomplish this, organizations sometimes undertake a radically different
approach to planning and management. This can include development of:
Strategic Plans, containing (1) a mission statement, (2) a set of strategic
goals covering the organizations major business areas / programs and
functions that are linked to the responsibility / program activity structure in
the budget, and (3) a description of how the organization intends to
achieve these goals.

Annual Performance Plans that link the strategic goals and business
objectives with the day-to-day activities. These plans are developed as
part of the budget preparation process, become key factors in making
budget decisions, and are adjusted to reflect budget decisions (e.g., if the
IT organization planned and proposed to implement 5,000 NT
workstations and senior management decided to provide funding for only
2,500, the plan would be reduced to 2,500).

Performance Reports, accompanying the annual budget request, to senior
management. This tells management what was actually accomplished
during the preceding fiscal period.

An IT Investment Management Approach is adopted to assure the "best"
IT decisions are made. (See our tutorial, "The IT Investment Management
Approach", for more information on this topic.)
2. Improvements take time and sustained management commitment
Effective organization-wide performance management initiatives take time to
design, develop, prototype, and fully implement; they also require sustained
commitment from top management.
Consideration of the following questions will give you an idea of why time is
needed to plan and implement performance management in any organization:
How well are we measuring outcomes and results now, if at all?

If it is such a good idea, why arent we doing it already?

What performance will be measured? How? When do we need to do it?

How will we "link" strategic goals and business objectives to IT projects /
activities?

How will we use performance information once developed?

How prepared is the organization to make changes based upon evaluation
of its performance?

What do our managers and staff (central as well as IT) know about
developing, analyzing, and interpreting performance measures? Do they
possess the knowledge, skills, and abilities to implement and maintain a
performance management system?

Do we have the systems and technology necessary to routinely gather,
process, and analyze performance measurement data? What is needed?
Answering these questions at the outset can be a sobering experience; yet, they
are indispensable to formulating a successful implementation plan. Once the
questions are answered, senior management buy-in is often secured by
developing a solid business case. Buy-in includes recognition and acceptance of
the fact that this is a long-term undertaking and that unwavering top management
support is essential.



3. Adequate resources must be made available
Information is rarely cheap or free. Organizations that want to reap the full
benefits of performance management must be willing to commit the necessary
resources. This includes:
design and maintenance of the process
development of management and staff knowledge and skills
dedication (eventually) of staff to perform the necessary analyses
acquisition, or development, of the technology to capture and store large
amounts of current and historical data
changing budgeting practices and procedures, along with budgeting
systems capabilities
The resources committed should be in proportion to the needs and benefits
4. The IT organization will need to learn new management techniques
Performance management can yield knowledge and information about the
organization not previously available. The ability to pinpoint potential, and actual,
performance problems will force IT management to come to terms with issues in
a new way (for example, what will the CIO do if a trend analysis reveals that his
"star" software development project managers key deliverables are always six
months, or more, late). It will also require a more businesslike approach to IT
project and activity planning, control, and reporting.
Many technically trained professionals will need to learn, or relearn, management
skills and techniques that they never viewed as integral to their jobs.
5. Some of the approaches recommended in current literature can be
risky.
As we have already noted, performance management and measurement are
sophisticated management tools that can yield significant benefits to the
organization. A review of approaches discussed in some current publications
reveals that quite a few are based more upon academic theory than solid
implementation experience.
For example: A recent article discussing how to develop performance indicators
advises readers that the best approach is to "empower" operational staff by
having them conduct "self-directed....brainstorming sessions" to identify
performance indicators, measures, and metrics.
As anyone with extensive performance management system experience knows,
the likely outcome of such a random approach will be long lists of things that
might be measured (mostly inappropriate) but little that is useful in terms of what
should be measured. That is not to say that operations management and staff
should not be involved; their participation and acceptance is essential to the
success of performance management initiatives. However, such efforts must be
carefully structured, directed, and supported by expert guidance. Random, ill
considered efforts produce little that is useful and can quickly undermine the
credibility of the initiative.
The moral: Evaluate the source before you act upon the advice.
Tutorial Recap and Conclusion
In this tutorial, we have seen that:
Senior management often believes that IT organizations are wasteful and
inefficient, based upon a track record of chronic cost overruns, late
deliverables, etc.

The purpose of Performance Management is to assure that IT projects /
activities achieve their planned level of performance and deliver intended
results on time and within budget; this is "non-technical" performance
management

IT organizations that maintain adequate performance management
systems take a significant step toward restoring credibility

Performance management improvement efforts are often part of an overall
effort including other processes, e.g., budgeting

Performance management involves the routine, and non-routine,
measurement of key aspects of IT project / activity performance

An important benefit of performance management is the ability to detect
incipient performance difficulties and correct them before they become
problems
Performance management initiatives can materially improve the way an
organization plans and conducts its business. Success requires:
time, and a sustained management commitment

adequate resources (analyst, training, process, and system)

careful planning, and a disciplined implementation approach
If you are responsible for developing and implementing a performance
management initiative:
thoroughly research the literature

carefully evaluate your sources; many suggest questionable approaches

work with someone who has actually designed, implemented, and
managed a performance management system - you can avoid years of
wasted effort.