Measuring Control Effectiveness

John Mitchell PhD, MBA, CEng, CITP, FBCS, CFIIA, CISA, CGEIT, QiCA, CFE

LHS Business Control Tel: +44 (0)7774 145638

47 Grangewood
Potters Bar
Hertfordshire
EN6 1SL john@lhscontrol.com
England www.lhscontrol.com

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchell ISACA Ireland Conference – 3 October 2014
 CMM & ISO 15504 Levels

CMM ISO 15504

5 – Optimised 5 - Optimised
4 – Managed and Measurable 4 – Predictable
3 – Defined 3 – Established
________________________________________
2 – Repeatable 2 - Managed
1 – Ad Hoc 1 - Performed
0 – Non existent 0 - Incomplete

GRC 2.0 - Breaking Down The Silos

© John Mitchel rd
ISACA Ireland Conference – 3 Slide # 2 2014
October
 Components of the Control Environment

Monitoring

Control
Activity
Control
Risk Objectives
Information &
Communication Analysis

GRC 2.0 - Breaking Down The Silos

© John Mitchel rd
ISACA Ireland Conference – 3 October 2014
 Generic Risk Management Process

H
i
g E
h

L
Senior Management I
D Inherent
Attention K
E Risk
Local Management L
I
Attention C Likelihood
H
O Reduction
No Action O
D
B Residual
Risk Consequence
L Reduction
o A
w

A B C D E
Low CONSEQUENCE High

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 Which Risk
Would You Want Assurance Over?

Residual
Inherent Risk Controls
Risk

Risk 1 None

Risk 2 Some

Risk 3 Lots

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 What Is This Control Stuff?

Anything which monitors or modifies a process to ensure its

predictability

A control is basically a test against a prediction

You can only test for what you can predict

Sometimes the prediction is absolute

(gender must be ‘F’)

Sometimes the prediction is variable

(within the range of 50 to 50,000)

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
Control Classifications
Class Ability to detect the event and take recovery action Type

1 Prevents the event, or detects it as it happens and prevents further Preventive

impact
2 Detects the event and reacts fast enough to fix it well within the
specified time window
3 Detects the event and reacts just fast enough to fix it within the Detective
specified time window
4 Detects the event but cannot react fast enough to fix it within the
specified time window
5 Fails to detect the event but has a partially deployed business
continuity plan
6 Fails to detect the event but does have a business continuity plan Reactive

7 Fails to detect the event and does not have a business continuity
plan

Source: D Brewer & W List GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
Anatomy of a Control

Design
Implementation
Monitoring
Evaluation

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
Measuring Control Design

How well the control should work, in theory, if it is always

applied in the way intended:
3 – designed to reduce risk aspect entirely
2 – designed to reduce most aspects of risk
1 – designed to reduce some areas of risk
0 – very limited or badly designed, even where used correctly
provides little or no protection

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
Measuring Control Implementation

The way in which the control performs in practice:

3 – control is always applied as intended
2 – control is generally operational but on occasions is not
applied as intended
1 – control is sometimes correctly applied
0 – control is not applied or applied incorrectly

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
Measuring Control Monitoring

How we know that the control is continuing to operate

(embedded monitor):
3 – operation is always monitored
2 – operation is usually monitored, but on occasions is not
1 – operation is monitored on an ad-hoc basis
0 – operation is not monitored at all

GRC 2.0 - Breaking Down The Silos

© John Mitchel rd
ISACA Ireland Conference – 3 October 2014
Measuring Control Evaluation

How frequently control effectiveness & efficiency is

evaluated:
3 – control is regularly evaluated for effectiveness/efficiency
2 – control is occasionally evaluated for
effectiveness/efficiency
1 – control is evaluated very infrequently
0 – control is never evaluated

GRC 2.0 - Breaking Down

rd The Silos
ISACA Ireland Conference – 3 October 2014
Scoring Control Effectiveness Example
(No Weighting)

Apply DIME:

Design = 2 (3)
Implementation = 3 (3)
Monitoring = 2 (3)
Evaluation = 1 (3)

TOTAL = 8 (12) = 0.75 (75% total effectiveness)

NOTE: If either Design, or Implementation is zero then

total score becomes zero

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 Risk & Control Documentation
LHS Business Control 47 Grangewood, Potters Bar, EN6 1SL, England +44 (0)1707 851454 csa@lhscontrol.com www.lhscontrol.com

RISK & CONTROL DOCUMENTATION

Company:

Division:

Location:

Business Area/Activity: Score the Effectiveness of the

Controls in Mitigating the Risk
N/A 1 2 3 4 5
A Controls for managing the risk of

B As a minimum these should include the Is it performed? How

following standard controls Contr. Contr. Who/what performs it? Often? How is it evidenced?
Class Score
N/A Yes No

1) Control 1
2) Control 2
3) Control 3
4) Control 4

C Where the answer to a minimum requirement is Is it performed? Who/what performs it? How How is it evidenced?
NO: Contr. Contr.
Class Score
Please give details of any alternative controls N/A Yes No
providing assurance

D Where the score for control effectiveness is < 3 Contr. Proposed Pot. Who/what will perform How How will it be evidenced?
Class Implementation Score it? Often?
Please detail the control which is to be Date
implemented to improve the result

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 Assessing Overall
Control Effectiveness

Analyse each control to arrive at an overall score for all of

the controls mitigating a risk

1 = Poor level of control - management attention required

2 = Very basic control - enhancement required
3 = Adequate level of control - scope for improved
effectiveness
4 = Good control - scope for increased efficiency
5 = Excellent control - no improvement possible

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 Control Effectiveness Reporting

H
i
g E 8
h

L
I 3,4,5,6,7,9, 12) Power
D 2,18
10,11,13,14
12
K Loss
E
L 14) 3rd Party
I Support
C 16
H
O
O
D
B 1

L 15) Loss of
o A 17 15 Data Centre
w

A B C D E
Low CONSEQUENCE High

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 Graphical Representation
(Multiple Risk Areas)
M/F Ops.
Network
Changes in control over the Internet
Disaster Rec.
Change Control
4.5
Internet
3.5 EPOS
HR
2.5 EDI
Sys. Dev.
1.5
Sys. Maint.
0.5 Help Desk
2010 2012 2013 Mngt. Info.
Cap. Plan.
Tech. Support
© John Mitchel 17 GRC 2.0 - Breaking Down
rd The Silos
ISACA Ireland Conference – 3 October 2014
 Summary

 Whether you use CMM or ISO 15504 you still need to assess control
effectiveness

 Evaluation should be against the controls mitigating a risk

 Evidence must be available that the control is effectively working

 The evidence must show who/what operates the control and the
frequency of operation

 Control effectiveness can be consistently assessed by applying the

DIME method

GRC 2.0 - Breaking Down

rd The Silos
© John Mitchel ISACA Ireland Conference – 3 October 2014
 Questions?
John Mitchell
PhD, MBA, CEng, CITP, FBCS, CFIIA, CISA, CGEIT, QiCA, CFE

LHS Business Control

47 Grangewood
Potters Bar
Hertfordshire EN6 1SL
England

Tel: +44 (0)7774 145638

john@lhscontrol.com
www.lhscontrol.com

GRC 2.0 - Breaking Down The Silos

© John Mitchel rd
ISACA Ireland Conference – 3 October 2014