Auditing in a Virtual World

(Team, Tools, & Techniques)

Sheronda Jeffries
TL 9000 Program Manager

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1
The workplace is evolving
In-Office Workers
In-Office Drop-In / “Co-Working”* Work from Home

35% 33% 32%

From the Future of Work Research Program Workforce Survey
*Drop-in workstations or cubicles or other facility at their company (e.g.,
conference room, training room, cafeteria, employee lounge), and co-
working spaces or third places such as cafes, libraries, airports, or hotels.

Cisco Confidential 2
 Standards are evolving

Nigel Croft – ISO TC 176 SC 2 Chair

Cisco Confidential 3
Cornish presentation to US TAGs For MSS – Liaison Group Teleconference
 ISO 9001:2015

ISO Standards Development

Status Update

Process
CD available 2013

DIS available
IAF Informative Document - Transition 05/2014
Planning Guidance for ISO 9001:2015
Published
FDIS expected
05/2015

ISO 9001:2015 to be
published 09/2015

3-Year Transition
Agreed
Cisco Confidential 4
Cisco Confidential 5
For information
visit:
questforum.org

Cisco Confidential 6
 Structure of ISO 9001:2015
PLAN DO CHECK ACT
4 Context of 9 Performance
5 Leadership 6 Planning 7 Support 8 Operation 10 Improvement
organization evaluation

Actions to Monitoring,
Understanding of
Leadership and address risk Operation planning measurement, Nonconformity and
the organization Resources
commitment and and control analysis and corrective action
and its context evaluation
opportunity

Needs & Determination of

Quality requirements for Continual
Expectations of Quality policy Competence Internal audit
interested parties objectives products/
improvement
services

Roles,
responsibilities
Planning of Design and Management
Awareness
Scope of the QMS
and authorities changes development review

Control of externally
QMS and its Communication provided products and
Processes services

Documented Production &

information service provision

Release of products
and services

Control of
(Slide Courtesy of Alan Daniels; Boeing) Nonconforming
process outputs,
products and services

(C) Nigel H Croft 2014 - All rights reserved Cisco Confidential 7

Virtual technology is critical…
§  Supports globalization
§  Drives innovation, creativity and better decisions
§  More critical now due to ‘new normal for technologies’

Cisco Confidential 8
Typical Challenges
to Virtual Auditing
•  Technology
•  Lack of trust
•  Logistics - time zones/holidays
•  Against the rules
•  Other challenges?

Cisco Confidential 9
It’s NOT against the rules…ISO 19011:2011
Guidelines for Auditing
Management Systems

Cisco Confidential 10
 It’s NOT against the rules…
IAF MD 4:2008 Use of Computer Assisted Auditing Techniques ("CAAT") for Accredited
Certification of Management Systems provides guidance
•  “Computer Assisted Auditing Techniques” (“CAAT”) may include
•  Teleconferencing
•  Web meetings
•  Interactive web-based communications
•  Remote electronic access to the management system documentation and/or management system
processes

NOTE: New MDs will include the term “Virtual Site”-

An on-line environment allowing persons from
different physical locations to execute processes. Cisco Confidential 11
 It’s NOT against the rules…
Rules may vary – for example:
•  Aerospace
•  ISO-9104-1 was revised in 2012 to include allowances in audit day rules for CAAT 

•  Information and Communication Technologies (ICT)

•  Auditor(s) and auditee(s) must be fluent in electronic data transferring
•  Rules: The following areas cannot be audited using e-audit techniques:
•  Laboratories
•  warehousing-distribution centers
•  manufacturing sites
•  repair (also know as service) centers
Cisco Confidential 12
It’s NOT against the rules…
APG Electronic-Based
Management Systems

Cisco Confidential 13
Logistics can be managed…

Cisco Confidential 14
Logistics can be managed…
Jan Feb March April May June July Aug Sep Oct Nov Dec

• Malaysia
• US • Italy • China
• Canada • India • Australia
From World Time Buddy – http://www.worldtimebuddy.com/
Cisco Confidential 15
 At
collaboration
and the use
of virtual
technology is
a our
… part of ouradvantage
competitive
culture 71K + 165+ 380+
employees countries offices
Cisco is globally certified to ISO 9001 - includes all aspects - manufacturing, sales, Cisco Services and engineering
Cisco Confidential 16
Logistics can be managed…
•  Planning is required and should consider
•  Normal business/working hours for each physical location
•  Holidays
•  Organizational protocols and norms

•  Don’t forget basic etiquette

Cisco Confidential 17
Logistics can be managed…
For a Virtual Meeting Checklist, please visit:
http://www.virtualworkswell.com/citrix

Cisco Confidential 18
What about TRUST?

Cisco Confidential 19
What about TRUST?
Use of
TRUST!
Technology!

Cisco Confidential 20
 What about TRUST?
Requires equal parts Process, Culture and Technology
Cisco uses collaboration and virtual
technology
… to change the way we work.
We use these technologies ourselves
… we are our own laboratory.

Process Technology

Culture Cisco Confidential 21

21
Virtual Team Considerations
blah
blah
blah

Cisco Confidential 22
Work is now “anytime, anywhere”

Cisco Confidential 23
Technology Options
Telepresence / Video-conferencing
l  Options
range from Webcam to TelePresence
+ Advantages:
§  Allows everyone to sit at the “same” table
Data-conferencing
l  Options
include
Cisco WebEx
Microsoft® Office Live Meeting
GoTo Meeting, etc.
Cisco Confidential 24
Technology Options
There are many FREE / Low Cost Options
l  SKYPE
l  FaceTime
l  ooVoo
l  Yahoo Messenger, Google+, ….

Cisco Confidential 25
Technology, Training & Auditing

Use technology to replicate & simulate WHICH may require more

planning but … does it really work?

Cisco Confidential 26
 Internal Quality Auditor Training
supports Cisco by:
§  Offering an effective training curriculum using an evolving blended
learning approach that supports our shareholders’ professional
development via business-focused training.
§  Explaining the basics of understanding ISO 9001, TL 9000 & ISO
14001, and effective auditing strategies for maintaining Cisco’s
certifications while promoting continual business improvement.
§  Delivering worldwide training using multiple tools (TelePresence,
WebEx Training, Vovici now known as Verint EFM…)
§  Positively impacting Auditee Satisfaction & OFI Adoption metrics
Cisco Confidential 27
 Collaboration & Internal Quality Auditor Training –
A Timeline Auditor Training
stabilized/matured to Enhanced
annual updates Learning
Update Level 3 Sessions
FLIP video used for
ISO 9001 Case Studies Training
ISO 9001 Internal Update End-
& Brown Bag Evaluation to V.2
Auditor Training ISO 9001 Internal of-Training
piloted using Level 3 Training Partnered with Survey
Auditor Training
TelePresence piloted using Evaluation V.1 LDSG to update
WebEx conducted TL 9000 GA

Q4 Q1-Q2 Q3-Q4 Q1-Q2 Q3-Q4 CY2012 - CY2014-

CY2011 2013
CY2008 CY2009 CY2009 CY2010 CY2010 2015

TL 9000 Internal On-line surveys updated

TL 9000 Internal Auditor Training piloted to comply with internal
Auditor Training using WebEx
piloted using policy changes
TelePresence WebEx Connect Spaces used to New Auditor Application
On-line survey tool store “transition” training moved to on-line survey
used to administer material, WebEx recording, tool; Quality Champion
course exams and
communicate exam url and learning path enhanced
end-of-course survey end-of-course survey url
Cisco Confidential 28
 It was a very good training course. Would
The training Great Class! Material was presented
definitely recommend to other peers
was good, in a fun and interesting way, that
Course was very good. Definitely this kept us on the ball and helped make
training helps each one of us to look at our
helpful and the information relevant
quality policy and apply that in our daily job. enjoyable.

Loved the high energy and ability to keep Here is what

the whole class engaged especially learners are
through WebEx. Never have I had an saying about
instructor make a class fun while being internal quality
effective in getting us to learn detailed auditor Training!
material. GREAT TRAINER

Very lively, kept The training was very well presented via Class was Fantastic. training
me awake till WebEx to a small group of students. The
session was very interactive with frequent Engagement was is very
after midnight pop quiz type refreshers. fantastic. good
my time :) That
Superb knowledge of subject. …a great experience
is a great
achievement.
Very good SME
I would definitely …is an expert on the subject and explained the
implementation of ISO at Cisco beautifully. I really
and fun to learn recommend this class appreciate the hard work, motivation and outstanding
with. to my colleagues.
© 2010 Cisco and/or its affiliates. All rights reserved. manner in which the training was conducted.
Cisco Confidential 29
 6months+ Post Training Survey of ISO 9001 Internal
Quality Auditors for CY2013 through 07/2014 Summary
based upon 57% survey response rate!
96%) than managers (4%) completed ISO 9001 Internal Quality Auditor training for the
•  More individual contributors (
period – versus 85% and 15% for previous year

•  The survey response rate for CY2013 – 07/2014 is 57% versus 31.4% for the previous year which shows a significant
increase

•  29% of the respondents had not participated in an internal audit as of the survey date which matches previous year
•  50% had participated in an internal audit within 6 months of training completion versus 41% for previous year
•  83% of the respondents found the training Extremely or Very Useful in their roles as Individual Contributors/Managers at
Cisco which almost matches 84% for the previous year

•  100% would continue to recommend ISO 9001 Internal Quality

Auditor training to others
Cisco Confidential 30
 Are Virtual Audits Effective?
How will you measure the effectiveness of virtual audits? 

•  Are auditors able to identify nonconformities and OFI’s? 

•  Do your internal audit results align with your Certification Body’s results? 
•  Do you get the same responsiveness from management to issues even though
the auditor may not be physically on-site? 
•  Are auditees and auditors satisfied with the use of virtual techniques?
•  Do the savings in travel balance against any extra time required in planning/
conducting virtual audits?

Cisco Confidential 31
 TIARA Dashboard
§  Operational Benefits: Management & ISO PMs can now better manage the global ISO
9001, TL 9000 and ISO 14001 audit programs.
Monitoring for Wow. that’s
aged and a lot of OFIs!
overdue audit
findings

Is the
business
improving
Progress
over time?
against plan
Nope!

Timeliness of
Tracking status audit
at a detailed completion
level

Cisco Confidential 32
 Tips & Techniques
1.  Take extra time in planning and 6.  Be familiar with the features of the
communicating - consider the length tools – sharing desktops and
of each audit interview & schedule taking screenshots is simple/
breaks effective
2.  Do as much pre-work as possible - 7.  Be prepared to familiarize others
pre-load agenda, pre-visit urls, ask with the tools
auditees to bring laptops/webcams 8.  Know how and whom to contact
3.  Allocate additional time for audit for tools help
interviews.. 9.  Have back-up plans in place!
4.  Resist inviting more people to each
audit interview Remember
5.  For large groups, use an on-site YOU are the
“host” to address issues and minimize LEADER!
“side” conversations/chats/IMs Cisco Confidential 33
Other challenges…

Cisco Confidential 34
 Questions?

Many thanks to Leading High Performing Teams in a Global and Virtual World
presented at ProjectWorld & World Congress for Business Analysts, November
2011
by Rebekah Thayer Cisco Confidential 35
 Questions?
Sheronda Jeffries
TL 9000 Program Manager

(919) 641-0608 mobile

sjeffrie@cisco.com
Cisco Confidential 36
Thank you.