Professional Documents
Culture Documents
Procedures Guide
Version 1.91
Last Updated:
December 2, 2009
Table of Contents
Introduction............................................................................................................................................................. 4 Severity Level Definitions ...................................................................................................................................... 5 Communication Checklist....................................................................................................................................... 8 Manager On Call (MOC) ................................................................................................................................ 8 IT Center ......................................................................................................................................................... 9 Incident Manager On Call IMOC List ............................................................................................................... 10 Incident Manager On-Call (IMOC) .............................................................................................................. 11 University IT Technical Staff / Technicians on Call .................................................................................... 12 University IT Director - of affected unit(s)................................................................................................... 12 Information Security ..................................................................................................................................... 13 CIOs Office.................................................................................................................................................. 14 Communication Manager and/or Other Designated University IT Employees............................................ 15 Scribe ............................................................................................................................................................ 16 University IT Office Admins ........................................................................................................................ 16 University IT Staff Members ........................................................................................................................ 16 Provost .......................................................................................................................................................... 16 President........................................................................................................................................................ 16 Other University Executives ......................................................................................................................... 17 Students......................................................................................................................................................... 17 Faculty / Departments or Divisions .............................................................................................................. 17 University Staff............................................................................................................................................. 17 University Security ....................................................................................................................................... 17 University Facilities ...................................................................................................................................... 17 Rochester Management................................................................................................................................. 17 University Legal............................................................................................................................................ 17 University HR ............................................................................................................................................... 17 Communication Call Log...................................................................................................................................... 18 Security Level Definitions .................................................................................................................................... 20 Internal Communications Template...................................................................................................................... 21 External Communications Template..................................................................................................................... 23 University IT Technician Form ............................................................................................................................ 25 External Communication Matrix .......................................................................................................................... 26 Incident Command Center Wall Charts................................................................................................................ 31 IT Alert (G2Alert) Steps to Send a Severity 3 IT Alert:.................................................................................... 40 ISD Manager On-Call - University IT (Data Center Services) Alert Notification ............................................... 41 Appendix............................................................................................................................................................... 43 Roles & Responsibilities................................................................................................................................... 44 Incident Manager On-Call (IMOC) .............................................................................................................. 44 Manager On-Call (MOC).............................................................................................................................. 45 Communications Manager ............................................................................................................................ 46 Web Content Hack Immediate Actions ......................................................................................................... 51 Debrief Procedures............................................................................................................................................ 52 Debrief Agenda Template................................................................................................................................. 53 Updating Procedures......................................................................................................................................... 54 Change Control ................................................................................................................................................. 55
3
Introduction
Leaders in the University Information Technology organization acknowledged the need to develop a wider view of incident management and communications. In the past, each University IT department utilized its own incident escalation path. Consistency in delivering incident management and expected communication levels were not meeting internal and external customer expectations, especially during high profile incidents. This Incident Management & Communication Procedures manual contains Severity 3 incident response tools. Severity 3 incidents are the highest level and most critical of events that occur within our organization. Immediate action is required by multiple people to assist in recovering services affected by the incident. By identifying scope and ownership of an incident early in the process, we can now triage to the appropriate teams, who in turn establish their communication protocols and management roles within the context of the broader incident management procedures. Incident management and communication processes that had been used independently across the organization are now merged into a single document and available across University IT. On-call escalation now has the ability to mobilize an Incident Manager On-Call (IMOC) who coordinates the Incident Command Center and communication methods to executives and customers. Each departments Manager On-Call (MOC) can now concentrate on recovering services, without the need to communicate with multiple people. Technicians will also benefit from these procedures by eliminating multiple communication paths and allowing them to concentrate on technical issues. Each Severity 3 incident will have a Communication Manager, assigned to assist with the creation of communication materials. A scribe will detail incident events. After the recovery from an incident, a mandatory debrief meeting will be scheduled to complete the Sev 3. Documentation for the debrief methodology has been finalized and is included in this manual. A coordinated University IT response is essential to our business and services. Our customers demand it, our internal resources need it, and the Information Technology Services Incident Management & Communication Procedures Guide delivers it.
SeverityLevel
ServiceImpact
3.0
Enterprisewide Immediateneedfor service
2.5
Enterprisewide Noimmediateneed forservice
2.0
Limited Singledepartment affectedbyservice interruption AgedGeneralUser Ticket ElevatedUser Ticket (Director,Manager) Triggersinformal communication plan Manager UniversityIT Manager LevelIIISupport
1.5
SingleorNone Singleuserservice impact NoServiceImpact withcomplex elevatedresolution GeneralUserTicket Resolutionby TechLead/System Lead LevelIII: TechLead,System Lead LevelIIISupport LevelIISupport CallAgent
1.0
SingleorNone Singleuserservice impact NoServiceImpact withelevated resolution GeneralUser Ticket Resolutionby SubjectMatter Expert(SME) LevelII: SubjectMatter Expert LevelIISupport CallAgent
0.0
SingleorNone Singleuser serviceimpact
Scopemaynotbedefined
Scopeisdefined
SeverityLevel Defined
VIPUserTicket Triggersformal communicationplan Director UniversityITDirector UniversityITManager IMOC AppropriateUniversity ITPersonnel ITCenter
DecisionMaker
Involvement SeniorManagement AllUniversityIT ITCenter ISD (ifDataCenteror NetworkRelated) WhoisNotifiedByWhom Immediate Notification IMOCbyDirector CIOOfficebyIMOC SeniorManagement byCIOOffice AllUniversityIT byHyperReachorEmail ITCenterbyDirector ISDbyHyperReach Communication PlanType Communication Methods Formal* DirectContact (phone,inperson) HyperReach
DirectorbyManager IMOCbyDirector ITCenterbyDirector AllUniversityIT byHyperReachor Email Formal* DirectContact (phone,inperson) HyperReach
Additional Notifications
Informal ServiceTicket
Informal ServiceTicket
Communication Checklist
This document provides a high-level overview of the communication flow that needs to take place during a declared Severity 3 (Sev 3) incident. University IT divisions will assess incidents as normal until a Sev 3 has been declared once elevated to a Sev 3, initiate this checklist.
Normal Business Hours (8:00am 5:00pm): Applies to weekdays and non-holidays.
Communication Flow Management Steps Normal Business Hours (8:00am 5:00pm) After Hours