Using KPIs to Improve Audit Performance

Gary Armstrong
VP of Audit Services
Cherokee Nation Businesses
918.384.5829
gary.armstrong@cn-bus.com

Outline

CNB Environment
Key Performance Indicators
Catalysts for Change
Audit the Auditors
Action Plan
Results
Next Challenges
Questions

Our Environment
CNB employs more than 6,000 people and posted more than
$600 million in revenue last year.
Fast paced, highly regulated Gaming Industry
Required Regulatory Compliance Reviews

Government Contracting
Aerospace & Defense Manufacturing
Fraud Investigations
Consulting Projects
Department of 27
TeamMate Shop

Key Performance Indicators

Performance measures designed to offer
comprehensive information to assess efficiency and
effectiveness
Must establish clear performance expectations
focused on results
Must be relevant
Must be measured consistently
Must be monitored on an ongoing basis

Catalysts for Change...

Making sure we are meeting growing company needs with
current staff or determine if more staff is needed
Become more accountable for Project Budgets
More accurate and timely status updates and performance
reporting to Audit Committee
Multiple open projects across multiple disciplines
Projects took too long to complete
Delayed Audit Report Issuance
Employees didnt feel successful

Audit the Auditors...

Reflection:
Quality of Audit work and documentation high; too high for
the cost to the organization?
Are we sacrificing value for the no stone unturned
approach?
How do we become better Project Managers?
How do we better measure productivity?
How do we attain more focus?
How do we establish more meaningful and attainable
project budgets?
How do we ensure staff feel successful?
6

Audit the Auditors...

Identified Weaknesses:

Good Auditors Poor Project Managers

Department Structure
Lack of Focus
Inability to properly prioritize tasks
Lack of meaningful performance information

Action Plan...
Simplified Time Tracking
Restructured Department:
Siloed into areas of expertise
Flattened management structure to gain
efficiencies

Action Plan...
Determined metrics that would serve as KPIs:
Time
Project Milestones
Days to Completion
Review Cycle Time
Coaching Notes Cycle
Time
Days to Report
Issuance

Auditor Performance
Utilization
Average Step Time
Number of WPs
Number of Issues
Number of Coaching
Notes

Action Plan...
Determined Benchmarks and Goals:

Utilization levels for each job grade based on time tracking

Review Cycle Time < 4 Days
Coaching Note Cycle Time < 2 Days
Report Issuance < 45 days from fieldwork completion

Implemented automated tools to minimize overhead

10

Automated KPI Tools...

Weekly Time Summary Reports
Project Status Update
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Services Dashboard

11

Weekly Time Report (sample)

Em p lo yee

W E_ 0 1 0 7 2 0 1 2

W E_ 1 2 3 1 2 0 1 1

W E_ 1 2 2 4 2 0 1 1

W E_ 1 2 1 7 2 0 1 1

W E_ 1 2 1 0 2 0 1 1

W E_ 1 2 0 3 2 0 1 1

8 3 .4 4

75

65

6 7 .1 9

7 0 .6 3

8 2 .5

A n d re w B e cke tt

3 .1 3

4 .1 7

6 .2 5

6 .2 5

4 .1 7

D avid B rad le y

8 7 .5

8 2 .1 4

6 0 .4 2

7 3 .1 3

65

Fo rre st G u m p

5 4 .1 7

7 9 .5 5

50

50

4 6 .2 5

50

Jim m y D u gan

8 2 .8 1

8 7 .5

8 9 .0 6

9 3 .1 3

9 2 .5

8 2 .8 1

Jo e B an ks

6 1 .4 6

75

6 4 .0 6

5 4 .1 7

6 6 .2 5

7 8 .1 3

P e p Stre e b e ck

8 4 .3 8

4 7 .5

3 3 .3 3

3 3 .7 5

50

7 8 .1 3

R o b b ie W h e e lin g

9 2 .6 6

9 2 .8 8

8 2 .3 2

9 0 .5

8 0 .4 7

9 9 .6 3

Sam B ald w in

9 7 .6 6

9 3 .7 5

8 6 .6 7

9 0 .6 3

7 9 .3 3

9 3 .7 5

Sh e rm an M cC o y

3 2 .2 9

3 6 .1 1

3 2 .5

2 7 .6 3

25

3 3 .7 5

Ste ve n G o ld

2 5 .6

5 3 .1 3

2 0 .4 9

3 8 .7 5

6 7 .1 9

3 8 .1 3

D avid B asn e r

100

1 1 6 .6 7

9 1 .8 8

8 1 .2 5

50

A lle n B au e r

7 3 .4 4

Sco tt Tu rn e r

5 4 .6 9

6 2 .5

5 4 .6 9

6 2 .5

6 0 .6 3

6 1 .2 5

P au l Ed ge co m b

1 0 5 .0 8

9 2 .8 6

90

9 0 .8 8

9 0 .6 3

1 0 4 .3 8

35

4 1 .6 7

4 6 .8 8

4 1 .4 1

3 7 .8 8

3 3 .1 3

9 5 .3 1

7 5 .8 9

1 0 1 .2 5

9 3 .7 5

9 7 .5

C h u ck N o lan d
V ikto r N avo rski

12

Automated KPI Tools...

Weekly Time Summary Reports
Project Update Report
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Services Dashboard

13

Project Update Report

14

Automated KPI Tools...

Weekly Time Summary Reports
Project Update Report
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Services Dashboard

15

Open Steps Summary

16

Open Steps Emails

P r o je c t

F o ld e r C o d e

F o ld e r

P r o c e d u r e T itle

S ta te

G u e s t S a tis fa c tio n

A .3 .P R G

A u d it P ro c e s s

W ra p -u p C h e c k lis t

B lu e d

LM D

1 /6 /2 0 1 2 la rry.c ro w n e

LM U

Age
5

G u e s t S a tis fa c tio n

B .1 .P R G

R is k a n d C o n tro l A n a lys is

Id e n tify k e y g e n e ra l ris k s

P re p a re d

1 /9 /2 0 1 2 jo e .fo x

G u e s t S a tis fa c tio n

B .1 .P R G

R is k a n d C o n tro l A n a lys is

R o le s a n d re s p o n s ib ilitie s

P re p a re d

1 /9 /2 0 1 2 jo e .fo x

G u e s t S a tis fa c tio n

B .1 .P R G

R is k a n d C o n tro l A n a lys is

C o m m u n ic a tio n c h a n n e ls

P re p a re d

1 /9 /2 0 1 2 fo rre s t.g u m p

17

Automated KPI Tools...

Weekly Time Summary Reports
Project Update Report
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Dashboard

18

Open Notes Email

19

Automated KPI Tools...

Weekly Time Summary Reports
Project Update Report
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Services Dashboard

20

Project Status Summary

21

Automated KPI Tools...

Weekly Time Summary Reports
Project Update Report
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Services Dashboard

22

Auditor KPIs...
Project
Manager
Lead

Title 31
Larry Crowne
Forrest Gump

Auditor
Andrew Beckett
Joe Fox
Larrry Crowne
Chuck Noland
Carl Hanratty
Viktor Navorski
Michael Sullivan
Paul Edgecomb
John Miller
Forrest Gump

Issue
Issue
Notes
Notes
Steps
Steps
WPs
WPs
Avg Avg PAvg PAvg AAvg AAvg
Hours Prepared Reviewed Prepared Reviewed Prepared Reviewed Prepared Reviewed Items Item Step Item Step Item Step
77.5
0
0
0
0
2
0
2
0
4 19.38 38.75
0.64 2.63 0.68
2.74
187
7
0
16
0
14
0
44
0
81 2.31 13.36
0.64 2.63 1.88
5.92
110.8
1
18
2
42
0
123
0
394 580 0.19 0.9
0.64 2.63 0.56
1.62
120.5
0
0
31
0
33
0
98
0 162 0.74 3.65
0.64 2.63 1.41
4.96
9
0
0
0
0
0
1
0
2
3
3
9
0.64 2.63 0.24
0.88
35.75
0
18
0
8
0
128
0
8 162 0.22 0.28
0.64 2.63 0.25
0.63
171.5
0
0
8
0
22
0
88
0 118 1.45 7.8
0.64 2.63 1.05
3.97
158
2
0
14
0
17
0
118
0 151 1.05 9.29
0.64 2.63 1.8
7.26
130.2
3
0
13
0
13
0
68
0
97 1.34 10.02
0.64 2.63 1.6
6.39
247.5
17
0
4
38
28
93
31
391 602 0.41 2.05
0.64 2.63 0.99
3.51

23

Automated KPI Tools...

Weekly Time Summary Reports
Project Update Report
Daily email alerts

Open Steps Summary

Open Steps Aging
Overdue Steps
Open Coaching Notes
Overdue Coaching Notes
Project Status Summary

Auditor Performance Tracking

Automated Audit Services Dashboard

24

25

Results

Reduction in Review Cycle Times by 95% over last 2 years

Reduction in days to issue Audit Reports by 65%
Addition of 17% more coverage in audit plan
Increase in Departmental Morale of 110.7%

26

Results
Review Cycle Time

160

140

Implemented
Team Silos and
Open Notes
Emails

120

Implemented
Open Step
Emails

100

80

60

40

20

Prepared - 1st Review

1st Review - Last Review

27

Next Challenges
Remaining cognizant of the fine line between effective project
management and micro managing
Making sure we use the data as a management tool rather than
relying completely on the data
Keeping Auditors Utilized!

28

Questions?