kl

W H I T E PA P E R S E R I E S
February 2009

Security Metrics Roadmap:

A Guide for Information Security Professionals

A WHITE PAPER BY:

Jim Maloney, CISSP,
CISM, CGEIT

www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
W H I T E
February 2009
P A P E R S E R I E S
kl
Selecting and using a poor security metric is probably worse than not
using a metric at all. Care must be taken to ensure the metric drives the
correct behavior and helps identify and diagnose problems.
Security Metrics Roadmap:
reporting of relevant performance-related data.
A Guide for Information Security Professionals
The purpose of measuring performance is to
monitor the status of measured activities and
facilitate improvement in those activities by applying
corrective actions, based on observed measurements.
Information security professionals know the intrinsic
value of the work they do — how critical it is to the
The terms “metrics” and “measurements” are sometimes
continued success of organizations operating in an
used interchangeably. However, it is important to
ever-changing landscape of threats and vulnerabilities.
distinguish between the two. Measurements are
To be successful, they must not only provide leading-
typically collected at a point in time for the purpose
edge security solutions, but also communicate the
of examining a singular event. Metrics are based on
benefits of doing so in a tangible, quantitative manner
measurements collected over a period of time and are
that facilitates decision-making at many levels. In
used to support more detailed analysis and examine
addition, they need a foundation for continuous
trends. Measurements can be useful to provide a
improvement that can be applied to individual controls
snapshot of performance at a point in time; the true
or across the entire security program.
power of metrics comes from the collection of data over
a period of time, providing insight via analysis that
This information security metrics roadmap provides
highlights important trends and patterns.
guidance in the selection, collection, analysis, and
reporting of information security metrics. The focus
of this paper is on quantitative metrics; however,
similar analyses could be brought to bear on qualitative
The purpose of security metrics
There is an old saying in the field of metrics: “You
metrics such as attitude surveys, security training
get what you measure.” The phrase implies that if a
evaluations, and other “soft” measures. With a baseline
particular set of performance measures are collected
of good metrics, a security professional can leverage
and reviewed, the process or system being measured
this information to evaluate the efficiency, effectiveness
will tend to be optimized to improve those measures.
and impact of an information security program, and
If the performance measures are relevant, meaning that
help identify and diagnose security-related problems.
they are related to assessing the ability of a process
Metrics can also be a critical element in building the
or system to meet its stated objectives, improvements
business case and realizing the benefits of security
in the process or system will be aligned with meeting
activities, from a specific security solution to the entire
those objectives. Conversely, if the measures are not
security program.
relevant, the process or system may drift away from
its intended objectives. Each metric must be carefully
The original release of the National Institute of
selected to influence behaviors that are consistent with
Standards and Technology (NIST) Special Publication
the objectives of the business, and not just behaviors
(SP) 800-55 on security metrics clearly defines metrics
that satisfy the metric itself.
and describes why they are beneficial:1

When addressing information security metrics, scope,

Metrics are tools designed to facilitate decision
too, must be determined carefully. When financial
making and improve performance and
institutions and others relegate processes and services
accountability through collection, analysis, and

©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com

1
W H I T E
February 2009
P A P E R S E R I E S
to third parties, the scope of their oversight extends to
those third parties. Responsibility for oversight cannot
be outsourced. Risk management and compliance
obligations persist, just as if functions were carried out
in-house.
As a security program is deployed, it is important to
evaluate the program on a continuing basis against
a clearly defined set of control objectives. These
control objectives are typically derived from a security
framework or model and then adapted to the specific
needs of the organization. Some of the more common
frameworks include:
• The Shared Assessments Program SIG
(Standardized Information Gathering) tool and
AUP (Agreed Upon Procedures) – Standards for
assessing risks related to outsourced information
security processes that map to COBIT, ISO, and
PCI standards.2
• COBIT (Control Objectives for Information
and related Technology) – Guidance for IT
governance
• ISO (International Organization for
Standardization) 27002 – A code of practice for
information security management
• NIST SP 800-53 – Recommended security
controls for federal information systems
These frameworks’ control objectives generally fall
into one of three broad categories: people, process, or
technology, all of which can be assessed and monitored
via metrics.
The purpose of evaluating a security program against
control objectives is twofold. One purpose is to assess
the effectiveness (achieving control objectives) and
efficiency (achieving control objectives using the
minimum amount of time, money, people, or other
resources) of the program. The other purpose is to
2
kl
assist in identifying and diagnosing specific problems
with an implemented control.
In addition to using metrics related to control
objectives, metrics can be useful in understanding the
overall impact of the security program on the business.
This provides a macro perspective on the effect of
the security program with respect to the company’s
business objectives. These program-level metrics
are usually related to governance, management, and
planning of the security program.
As with any element of the security program, the metrics
program should not be static; it should be subject to the
same continuous improvement processes as the rest of the
security program.
Criteria for selection
As the previous section implies, selecting and using a
poor security metric is probably worse than not using
a metric at all. Care must be taken to ensure the metric
drives the correct behavior and helps identify and
diagnose problems.
In his book on security metrics, Andrew Jaquith
suggests the following five attributes of a good metric:3
• Consistent – The metrics can be measured
consistently without need for subjective
judgment.
• Inexpensive – The metrics can be gathered
inexpensively, and ideally in an automated
fashion.
• Numerical – The metrics can be expressed as
a cardinal number or percentage and measure
something on an absolute basis instead of a
relative basis.
©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
W H I T E
February 2009
P A P E R S E R I E S
• Unit of measure – The metrics are expressed in
one or more units of measure.
• Contextually specific – The metrics are
meaningful to the user and relevant to the
decision-making process.
The book Metrics for IT Service Management (ITSMF)
uses the SMART acronym (Specific, Measurable,
Achievable, Realistic, and Timely) more typically
applied to goal setting as another method for assessing
the quality of a metric and its intended use:4
• Specific – Is the metric measuring a specific
part of the process or a specific control?
• Measurable – Can the defined metric actually
be measured?
• Achievable – Is the target level for the metric
achievable?
• Realistic – Can the metric actually be used as a
basis for diagnosis or improvement?
• Timely – Can the metric be collected frequently
enough to be useful?
A combination of these selection and assessment
attributes is recommended for determining the quality
of a candidate security metric, and subsequently
selecting and optimizing an effective set of security
metrics.
Establishing a security metrics program
Several references suggest a similar sequence of
steps for establishing a security metrics program.
These steps, summarized below, may be used as the
basis for developing a project plan for managing the
implementation of a security metrics program.
1) Integrate metrics into the security program
design. As with any major business program
or initiative, metrics can play a key role in
determining whether a security program is
3
kl
on track to meet its objectives, and if it is
doing so efficiently. As the security program is
defined or updated, a process for continuous
improvement based on security metrics should
be incorporated. ISO 27001 provides a good
example of the central role of continuous
improvement in a security program via its plan,
do, check, act model. Monitoring of the program
for the purposes of identifying and diagnosing
problems is a control objective itself, not only
in ISO 27001, but also in many of the other
commonly adopted security frameworks.
2) Determine scope: adopt a security framework
and select relevant control objectives. Several
choices for security frameworks are available
from sources including COBIT, ISO, NIST, and
the Information Security Forum (ISF). Besides
providing a taxonomy of security controls that
can be monitored and assessed via metrics, a
security framework provides a more effective
approach to meeting security requirements
from multiple laws, regulations, and industry
standards.
3) Select metrics and targets for assessment of key
controls. Using the selection and assessment
criteria described here, an initial set of metrics
may be selected and then expanded and
improved over time. Thresholds for acceptable
performance of measured processes and systems
will also need to be established.
4) Identify data sources, collection methods and
reporting tools. Ideally, automated methods of
data collection from reliable sources will be used
to feed a reporting tool, making security metrics
reporting inexpensive and consistent.
5) Respond to analysis. The security program
must be responsive to the results of analysis
of the metrics. A combination of automated
and manual procedures should be established
that support consistent review of the metrics-
©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
W H I T E
February 2009
P A P E R S E R I E S
kl

based reports and the alerts that are triggered
based on metrics that fall outside of established
thresholds.
Candidate metrics
Three of the references used in this paper collectively
describe a wide variety of candidate security-related
metrics (see Exhibit 1).
Only a manageable number of metrics should be collected
Analysis and visualization of metrics
Unlike a single measurement, metrics are collected over
a period of time. Companies realize the true power and
benefit of collected metrics when they are analyzed to
reveal trends and patterns. As with any type of data
and data analysis, visualization can be beneficial. Many
of the standard forms of visual data display (e.g., bar
charts, line charts, pie charts, scatter diagrams, and
radar diagrams) can be applied to potentially improve
the presentation and understanding of both the raw
metrics data and the results of the data analysis.

and analyzed — those needed to ensure that the security
program is meeting its objectives and that key controls are
operating as expected. If target levels of performance are
being reached consistently, some metrics can be dropped
from the program.
Although the use of metrics specifically in the domain
of information security is relatively immature, the tools
and processes for collection, analysis, and reporting of
metrics in general are well established. Many business
intelligence and performance management software
applications include an implicit or explicit process for
managing metrics.

The importance of using a packaged application for

A limited set of initial metrics may be selected from
collection, analysis, and reporting will become more
the references listed in Exhibit 1 as a pilot program
apparent as the volume and complexity of the metrics
using the previously described metrics selection and
increase. There has been a movement toward vendor
assessment criteria. The initial metrics should cover
consolidation in this area recently; to research current
some broad security program objectives as well as a few
offerings and find an appropriate solution, consider
specific indicators of performance. They should also
reviewing analyst reports and using an RFI/RFP
be selected for simplicity of collection, analysis, and
process.
reporting format.

Critical success factors

After a few months of exercising the initial metrics,
NIST provides several critical success factors
the metrics program can be expanded and adjusted
for organizations to consider in designing and
to better meet the needs of the security program and
implementing a security metrics program. In summary,
the objectives of the business. As with any element of
these factors are:6
the security program, the metrics program should not
be static; it should be subject to the same continuous
1) Stakeholders. Information security is a
improvement processes as the rest of the security
broad, cross-functional business activity that
program.
touches many elements of the organization.
As such, it is important to involve stakeholders
Exhibit 2 lists some candidates for an initial set of
from all organizations with security-
security metrics, one per each major security category
related responsibilities in the design and
in the ISO 27002 controls framework.

©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com

4
W H I T E
February 2009
P A P E R S E R I E S
implementation process (e.g., information
security, IT operations, legal, and human
resources). If an internal organization exists that
has responsibility for business intelligence or
performance management functions, the security
metrics program should try to leverage any
existing infrastructure that may be available for
data collection and reporting.
2) Manageability. Only a manageable number
of metrics should be collected and analyzed
— those needed to ensure that the security
program is meeting its objectives and that
key controls are operating as expected. If
target levels of performance are being reached
consistently, some metrics may be dropped from
the program. New metrics may be needed in
response to changes in the security program and
its objectives.
3) Data quality. The analysis and reports based
on metrics will only be as good as the quality
of the data available to generate those reports.
Standardized and automated methods of
collection should be used along with periodic
review and validation of the collected data.
4) Relevance. An information security program
exists not for its own sake, but as an element
of helping the business meet its objectives. A
subset of the metrics and resulting analysis at the
program level should be aligned with business
objectives and relate to business impact.
Conclusion
Establishing and maturing a security metrics program
can require substantial effort. But programs that are
designed and implemented properly should not require
significant resources to maintain. Automated collection,
analysis, and reporting can further help organizations
manage their programs.
5
kl
With careful selection and use, metrics improve
security program effectiveness and efficiency, allowing
problems to be identified and diagnosed more quickly.
And security metrics can be a key element of a business
case that supports investments in the security program
today and tomorrow.
1
Security Metrics Guide for Information Technology
Systems, National Institute of Standards and Technology
Special Publication 800-55, July 2003, 9.
2
The Shared Assessments Program was developed by BITS
(www.bitsinfo.org) and is managed by The Santa Fe Group
See www.sharedassessments.org.
3
Andrew Jaquith, Security Metrics - Replacing Fear,
Uncertainty and Doubt (Upper Saddle River, NJ: Pearson
Education, 2007) 21-25.
4
Peter Brooks, Metrics for IT Service Management, The IT
Service Management Forum (Zaltbommel, Netherlands:
Van Haren Publishing, 2006) 42.
Performance Measurement Guide for Information Security,
5
NIST SP 800-55 Revision 1, July 2008.
6
Security Metrics Guide for Information Technology
Systems, 13-14.
About the Author
Jim Maloney has worked in information technol-
ogy for more than 25 years. As an expert in security
program management, Jim’s specialties include security
strategy, risk assessment, security framework selection,
policies, procedures, staffing, compliance, and technol-
ogy evaluations. He has been directly involved in the
full lifecycle of security product and services develop-
ment from both business and technology perspectives,
and is a Certified Information Systems Security Profes-
sional (CISSP) and a Certified Information Security
Manager (CISM). For more information about Jim, see
http://santa-fe-group.com/staff_jim.php.
©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
W H I T E
February 2009
P A P E R
Exhibit 1: Reference descriptions of candidate security-related metrics
Reference
NIST SP 800-55 R15
ITSMF Metrics4
ITSMF Metrics4
Security Metrics1
Security Metrics1
Exhibit 2: Candidates for an initial set of security metrics
Security Category
Risk Management and
Treatment
Security Policy
Organization of
Information Security
S E R I E S
Metrics related to
Program- and System-Level
Security Management
Incident Management
Diagnostics
Program Effectiveness
Objective
To identify and
prioritize risks and
remediate those risks
that are unacceptable
To ensure that
employees, contractors,
and third-party users
understand their
security responsibilities
To ensure that
employees, contractors,
and third-party users
understand their
responsibilities, and are
suitable for the roles
assigned to them
6
Number of candidate metrics
19
10
13
79
73
Metric
Percentage (%) of
high-risk vulnerabilities
remediated within
specified target
timeframe
Percentage (%) of
employees who
have signed an
acknowledgement that
they have read and
understood the security
policies
Percentage (%) of job
performance reviews
with evaluation of
security responsibilities
and compliance
©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
kl
Comments
For those risks for
which the decision
has been to apply
appropriate controls,
these controls should
be selected and
implemented to meet
the requirements
identified by a risk
assessment.
Employees, contractors,
and third-party
users should sign an
acknowledgement of
their security roles and
responsibilities.
Security roles and
responsibilities of
employees, contractors,
and third-party users
should be defined and
documented.
W H I T E
February 2009
P A P E R S E R I E S
Exhibit 2: Candidates for an initial set of security metrics, continued
Security Category Objective
Asset Management To achieve and maintain
appropriate protection
of organizational assets
Human Resources To ensure that
Security employees, contractors,
and third-party users
are aware of information
security threats and
their responsibilities,
and are equipped to
support security policies
in the course of their
normal work
Physical and To prevent unauthorized
Environmental Security physical access
and damage to the
organization’s premises
and information systems
Communications and To reduce risks resulting
Operations Management from exploitation of
published technical
vulnerabilities
Access Control To control access to
information
7
Metric
Percentage (%) of
critical assets with
a documented risk
assessment
Percentage (%) of
system users that have
received basic awareness
training
Percentage (%) of
physical security
incidents allowing
unauthorized entry into
facilities containing
information systems
Critical patch latency
on severs and desktops
(average days that
critical patches were
missing)
Percentage (%) of
security incidents
caused by improperly
configured access
controls
©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
kl
Comments
All critical assets should
be clearly identified
and a risk assessment
performed on those
assets.
An adequate level of
awareness, education,
and training in security
procedures should
be provided to all
employees, contractors,
and third-party users.
Critical or sensitive
information-processing
facilities should be
physically protected
from unauthorized
access, damage, and
interference.
Timely information
about technical
vulnerabilities of
information systems
being used should
be obtained, and
appropriate measures
taken to address the
associated risk.
Access to information
systems should be
controlled on the basis
of business and security
requirements.
W H I T E
February 2009
P A P E R S E R I E S
Exhibit 2: Candidates for an initial set of security metrics, continued
Security Category Objective
Information Systems To ensure that security
Acquisition, is an integral part of
Development, information systems
and Maintenance
Information Security To ensure a quick,
Incident Management effective, and orderly
response to information
security incidents
Business Continuity To protect critical
Management business processes from
the effects of major
failures of information
systems or disasters, and
to ensure their timely
resumption
Compliance To ensure compliance
of systems with
organizational security
policies and standards
(both in-house and
outsourced)
8
Metric
Percentage (%) of
system acquisition
or development
specifications that
include security
requirements
Average time from
initial incident report to
incident recovery
Percentage (%) of
critical systems
addressed in the
contingency plan
Percentage (%) of
systems that are
compliant with the
specified baseline
configuration
©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com
kl
Comments
Security requirements
should be identified at
the requirements phase
of a project.
The adverse impact of
information security
incidents on the
organization and its
business operations
should be minimized.
A business continuity
management process
should be implemented
to minimize the impact
on the organization and
recover from loss of
information assets.
The security of in-
house and outsourced
information systems
should be reviewed
regularly.
W H I T E
February 2009
P A P E R S E R I E S
kl

Other White Papers by The Santa Fe Group

The New Consumer Value: “Living Light”
Volume 1 in The Santa Fe Group’s Living Light Series
By Leslie P. Mitchell, Janey Place, and
Catherine A. Allen

Beyond Compliance: Integrating New Regulations into

Your Risk Management Practice
By Peter J. Baldassaro, Robert W. Jones,
Edward J. Potter and Jodi Pratt

Mobile Payments Get a Risk Management Reality Check

By Gary Roboff, Senior Consultant

The Future is Mobile

By Janey A. Place Ph.D., Senior Consultant

When Bad Things Happen to Good Banks: The Perils

of an Unbalanced Control Regime
By Robert W. Jones, Senior Consultant

Fraud in the ACH System: A Holistic Approach

for Financial Institutions
By The Santa Fe Group Vendor Council

Electronic Discovery 2007: A Primer for

Financial Institutions
By The Santa Fe Group Vendor Council

Internal Fraud: Surveying the Current Landscape

By The Santa Fe Group Vendor Council

Internal Fraud: Building the Business Case for Investment

By The Santa Fe Group Vendor Council

➔ Download this paper and other Santa Fe Group

titles at http://santa-fe-group.com/whats-new.php.

©
www.santa-fe-group.com | 505.466.6434
info@santa-fe-group.com