Original Article

A Method for Evaluating Competency in Assessment and Management of Suicide Risk

Erick K. Hung, M.D., Ren L. Binder, M.D. ee Samantha R. Fordwood, Ph.D., Stephen E. Hall, M.D. Robert J. Cramer, Ph.D., Dale E. McNiel, Ph.D.
Objective: Although health professionals increasingly are expected to be able to assess and manage patients risk for suicide, few methods are available to evaluate this competency. This report describes development of a competencyassessment instrument for suicide risk-assessment (CAIS), and evaluates its use in an objective structured clinical examination (OSCE). Method: The authors developed the CAIS on the basis of the literature on suicide risk-assessment and management, and consultation with faculty focus groups from three sites in a large academic psychiatry department. The CAIS structures faculty ratings regarding interviewing and data collection, case formulation and presentation, treatment-planning, and documentation. To evaluate the CAIS, 31 faculty members used it to rate the performance of 31 learners (26 psychiatric residents and 5 clinical psychology interns) who participated in an OSCE. After interviewing a standardized patient, learners presented their risk-assessment ndings and treatment plans. Faculty used the CAIS to structure feedback to the learners. In a subsidiary study of interrater reliability, six faculty members rated video-recorded suicide risk-assessments. Results: The CAIS showed good internal consistency, reliability, and interrater reliability. Concurrent validity was supported by the nding that CAIS ratings were higher for senior learners than junior learners, and were higher for learners with more clinical experience with suicidal patients than learners with less clinical experience. Faculty and learners rated the method as helpful for structuring feedback and supervision. Conclusion: The ndings support the usefulness of the CAIS for evaluating competency in suicide risk-assessment and management.
Academic Psychiatry 2012; 36:2328

Received November 16, 2010; revised May 23, July 29, 2011; accepted August 9, 2011. From the Dept. of Psychiatry, Univ. of California San Francisco, and the Dept. of Psychology, Sam Houston State University, Huntsville, Tx (rjc). Correspondence: dalem@lppi.ucsf.edu (e-mail). Copyright 2012 Academic Psychiatry

he national emphasis on attainment of competencies in the education of healthcare professionals has created a need for methods to assess competency in specic skills (1, 2). Competency in suicide risk-assessment and management is expected of mental health professionals, including psychiatrists, psychologists, social workers, psychiatric nurse-practitioners, and others (3, 4). According to the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Psychiatry and Neurology (ABPN) (4 6), assessment and management of risk for suicide is a core competency for psychiatrists. Moreover, calls have been made for increased training of primary-care providers in recognition and management of suicide risk, in light of research showing that approximately half of American suicide victims had contact with a primary-care provider during the month before their death (7). Although many programs have been developed to teach clinicians about working with suicidal patients (8 11), our literature review identied no published methodology for assessing the competence of individual clinicians in evaluation and management of risk for suicide. This report describes development and evaluation of the reliability, validity, and acceptability of a method for assessing competency in suicide risk-assessment and management. We identied pertinent domains for a competency-assessment instrument for suicide risk-assessment (the CAIS) on the basis of a literature review and focus groups with faculty from multiple sites in a large academic psychiatry department. We applied the CAIS in the context of an objective structured clinical examination
http://ap.psychiatryonline.org 23

Academic Psychiatry, 36:1, January-February 2012

COMPETENCY TO ASSESS SUICIDE RISK

(OSCE) (12, 13), in which psychiatry residents and clinical psychology interns performed suicide risk-assessments of a standardized patient while observed by faculty members. The faculty rated learners performance and provided feedback according to the structure of the CAIS. We addressed the following questions about the CAIS: What is its internal consistency, reliability, and interrater reliability? Do senior learners perform better than junior learners? Do learners who have more clinical experience with suicidal patients perform better than trainees with less experience? How satised are faculty and learners with this method of assessing competency in suicide risk-assessment and management? Method Development of the Competency Assessment Instrument for Suicide Risk (CAIS) We developed the CAIS based on review of the literature on the standard of care in suicide risk-assessment and management (14 18), adaptation of criteria for evaluating the quality of suicide risk-assessment developed in a previous study of the effects of training on risk-assessment (11), and review of the literature on measurement of competencies in medical education (6, 12, 19 23). To further enhance the content validity of the CAIS, we incorporated feedback on drafts of the measure from focus groups composed of clinical faculty at each of three sites of a large academic psychiatry department; these were 1) a county hospital, 2) a Veterans Administration hospital, and 3) a university hospital. The focus groups discussed factors that they felt were important for trainees to master in performing a competent suicide risk-assessment, including both content-related information (e.g., asking about suicidal ideation, knowledge about local civil commitment laws) and process-related issues (e.g., establishing rapport with the patient; reviewing the medical record; and obtaining information from collateral sources, such as family members or other clinicians). The CAIS includes a checklist of 30 items concerning components of the riskassessment process, each of which is rated on a 4-point scale, from 1: Task Not Done, to 4: Advanced. The 30 items include interviewing and data-collection (sources of information: 6 items; types of information: 6 items); caseformulation and presentation (6 items); treatment-planning (11 items); and documentation (1 item). Also, the CAIS includes a rating of the overall quality of the risk-assessment for suicide, on a scale ranging from 1: Unacceptable, to 8: Advanced. The items in the CAIS are summarized
24 http://ap.psychiatryonline.org

in Table 1. (The full CAIS is available from the authors upon request.)

Application of the CAIS in an Objective Structured Clinical Examination (OSCE) Participants Participants, who are referred to here as learners, were 31 trainees who attended a 5-hour workshop on risk-assessment in July 2008; 26 were psychiatry residents (12 in the rst postgraduate year, 14 in the second postgraduate year), and 5 were clinical psychology interns. The workshop included a pretest, lectures on riskassessment for suicide and violence, an OSCE, and a course evaluation. This report concerns the components pertaining to suicide risk-assessment. Learners heard a lecture based generally on APA Practice Guidelines for the Assessment and Treatment of Patients with Suicidal Behavior (14) and received relevant materials. Another lecture covered medical-legal aspects of risk-assessment and documentation. The lectures recommended gathering information about risk and protective factors, rationally weighing the signicance of those factors to estimate the level of suicide risk, developing and implementing a plan of intervention to reduce the risk, and documenting the process. (Additional information about the content of the lectures in this model of workshop training in suicide risk-assessment is provided elsewhere (11)). Faculty Observers Thirty-one faculty (26 psychiatrists and 5 psychologists) were trained in use of the CAIS in an OSCE setting. In this training, faculty observed mock OSCE sessions in which an individual interviewed a standardized patient, and presented the suicide risk-assessment ndings to a mock examiner. Faculty observers rated the quality of the suicide risk-assessment with the CAIS, discussed their ratings, and calibrated rating differences. Standardized Patients (SPs) We trained 31 advanced trainees (27 psychiatry residents in their third or fourth postgraduate year, and 4 postdoctoral clinical psychology fellows) to be SPs in the OSCE. SPs were trained to follow a script based on a clinical vignette. The script described a young adult patient who presented to an emergency department with various risk factors for suicide and violence. The script included information about the SPs chief complaint, history of the present illness, psychiatric and medical history, family history, social history, and mental status presentation. (Additional details about the
Academic Psychiatry, 36:1, January-February 2012

HUNG ET AL. TABLE 1. Summary of Items in the Competency Assessment Instrument for Suicide Risk (CAIS)a
INTERVIEWING AND DATA COLLECTION A. Sources of Information (Process) 1. Develops rapport with patient 2. Obtains a history relevant to risk of suicide 3. Considers ndings from the mental status examination that pertain to risk of suicide 4. Considers review of records (e.g., medical chart) 5. Considers obtaining collateral information from other professionals (e.g., therapists, physicians) 6. Considers obtaining collateral information from family/signicant others B. Types of information (Content) 1. Considers current suicidal ideation, intent, and/or plan 2. Identies any risk factors for suicide 3. Evaluates past (historical) risk factors for suicide (e.g., history of suicide attempts) 4. Evaluates present (clinical) risk factors for suicide (e.g., hopelessness, depression, intoxication) 5. Evaluates future (risk-management, situational) risk factors for suicide (e.g., stress) 6. Considers protective factors against risk of suicide (e.g., social support, therapeutic alliance) CASE FORMULATION AND PRESENTATION 1. Estimates the risk of suicide (e.g., low, moderate, high) 2. Explains thought process or rationale for decision-making about risk of suicide 3. Communicates the estimate of suicide risk clearly and objectively 4. Species the timeframe to which risk estimate applies (e.g., imminent vs. long-term; acute vs. chronic) 5. Gives an organized and accurate presentation of the ndings 6. Relies on risk factors for suicide that are supported by scientic research TREATMENT-PLANNING 1. Addresses the risk of suicide 2. Describes an intervention plan that would effectively minimize the risk of suicide 3. Addresses modiable risk factors for suicide 4. Includes a risk/benet analysis (i.e., pros and cons of proposed intervention and alternatives for managing risk of suicide) 5. Considers hospitalization to reduce risk of suicide 6. Considers involuntary civil commitment for danger to self 7. Demonstrates understanding of local civil commitment statutes regarding danger to self 8. Considers involving family/signicant others 9. Considers use of psychoactive medication 10. Considers increasing the intensity of psychosocial treatment 11. Considers obtaining consultation DOCUMENTATION Communicates risk-assessment for suicide in writing OVERALL RATING Overall quality of risk-assessment for suicide
a Items are rated on a scale from 1 (Task Not Done) to 4 (Advanced), with the exception of the Overall Rating, which is rated on a scale from 1 (Unacceptable) to 8 (Advanced).

training of SPs and faculty raters are available from the authors upon request.) Objective Structured Clinical Examination (OSCE) Each OSCE team included a learner, an SP, and a faculty observer/rater. The OSCE included a 15minute interview of the SP; 15 minutes for the learner to write a progress note concerning the patient; a 10minute oral presentation by the learner, including a summary of the assessment and plan regarding the patients risk of suicide; completion of the CAIS by the faculty observer/rater; and a 25-minute discussion, during which the faculty observer gave the learner feedback based on the structure of the CAIS. At the conAcademic Psychiatry, 36:1, January-February 2012

clusion of the workshop, learners and faculty completed course evaluations. Interrater Reliability Study To evaluate the interrater reliability of the CAIS, in July 2009, we conducted a second study in which six faculty-observers (four psychiatrists and two psychologists) rated videos of three mock OSCEs. In each video, a different learner interviewed an SP and wrote a progress note containing the assessment and plan concerning the patients risk of suicide. The project was conducted in the Department of Psychiatry at the University of California, San Francisco, and was approved by UCSF Committee on Human Research.
http://ap.psychiatryonline.org 25

COMPETENCY TO ASSESS SUICIDE RISK

Participants were informed that participation in the OSCE was voluntary, and that the data would be coded in a way that removed identiers and would not become part of individual learners training les. Data Analysis We calculated Cronbachs to characterize the internal consistency reliability of the CAIS. We used t-tests and correlation analyses for continuous variables, and chisquare analyses for categorical variables, to determine whether learners who were more senior and had more experience with suicidal patients performed better on the CAIS than those who were more junior and had less experience. The subsidiary study of interrater reliability used the intraclass correlation coefcient (ICC1). We used SPSS Version 15.0 for data analysis. Results Baseline Level of Training and Experience The learners reported a mean (standard deviation [SD]: of 5.5 (9.0) hours of previous formal training in assessing and managing suicide risk, and 1.7 (2.0) years of previous experience providing mental health services. The number of suicidal patients encountered before the workshop included one learner with no patients, nine learners with 15 patients, four learners with 6 10 patients, three learners with 1120 patients, two learners with 2150 patients, and ve learners with 51100 patients (seven learners did not answer this item). Reliability The internal-consistency reliability of the CAIS was high ( 0.94), supporting the conclusion that the items on the instrument measure a common domain. In the subsidiary study of interrater reliability, the intraclass correlation coefcient (ICC) for the 30-item CAIS checklist was 0.94. The ICC for the rating of the overall quality of the suicide risk-assessment was 0.95. Validity For purposes of data analysis, we categorized the learners as Senior (second-year psychiatry residents, who had 6 months of supervised inpatient psychiatry experience that included frequent suicide risk-assessments) and junior (rst-year psychiatry residents and predoctoral psychology interns). Senior learners performed better than Junior learners on the CAIS. Mean (SD) ratings of the overall quality of the suicide risk-assessment were signicantly
26 http://ap.psychiatryonline.org

higher for Senior learners (mean: 5.9 [0.9]), than Junior learners (mean: 4.8 [1.2]; t [29] 2.67; p 0.01). Similarly, mean scores of the 30-item checklist of components of suicide risk-assessment were signicantly higher for Senior learners (mean: 89.6 [11.1]), than Junior learners (mean: 77.3 [13.0], t [29] 2.79; p 0.01). More previous experience with suicidal patients was associated with better performance on the CAIS, measured both by the rating of overall quality of the suicide risk-assessment (r 0.43; p 0.03) and the 30-item checklist (r 0.43; p 0.03). Similarly, more hours of previous training in suicide risk-assessment and management were associated with higher ratings of the overall quality of the suicide risk-assessment on the CAIS (r 0.41; p 0.04), although this positive correlation did not reach statistical signicance on the 30-item checklist (r 0.28; NS). Summative judgments of whether learners had minimal competency in suicide risk-assessment and management were calculated by dichotomizing the ratings of overall quality of the risk-assessment as either Competent (rated as 5: Competent, to 8: Advanced) or Not Competent (rated as 1: Unacceptable, to 4: Working Toward Competency). Suicide risk-assessments by Senior learners were signicantly more likely to be rated as Competent than riskassessments by Junior learners ( 2 [1] 5.28; p 0.03); 100% of risk-assessments by Senior learners (14/14) were rated as Competent, versus 59% (10/17) of risk-assessments by Junior learners. Satisfaction On course evaluations, both learners and faculty rated the CAIS as helpful for assessing competency in suicide risk-assessment. When learners were asked whether the CAIS would be helpful for getting feedback from supervisors in real patient encounters with potentially suicidal patients, the mean rating was 5.9 (1.0) on a 7-point scale, ranging from 1 (Not At All Helpful) to 7 (Extremely Helpful). Similarly, mean (SD) ratings by faculty on the same 7-point scale about whether the CAIS was helpful for rating competency in working with potentially suicidal patients was 5.5 (1.1). Discussion The results of this study support the promise of a new method for assessing the competency of individual clinicians in suicide risk-assessment and management. The CAIS showed good internal-consistency reliability and interrater reliability. Content validity was addressed by
Academic Psychiatry, 36:1, January-February 2012

HUNG ET AL.

developing the measure based on the literature on suicide risk-assessment, and by consultation from faculty focusgroups at various sites of a large academic psychiatry department. Support for concurrent validity includes the fact that senior learners in psychiatry and psychology performed better than junior learners on the CAIS in the setting of an OSCE. Learners with more experience with suicidal patients performed better than learners with less experience. Evidence of acceptability of the method includes the nding that learners and faculty found it a helpful structure for learning and feedback about competency in suicide risk-assessment. Limitations include the fact that the study was conducted in one psychiatry department and included only a modest number of participants. Also, given the cost of having faculty serve as raters in an OSCE, future research could evaluate the viability of having SPs rate the CAIS. Another limitation is that although the OSCE provides a standardized context for application of the CAIS, SPs may not show the range of problems that is comparable with actual patients encountered by learners on clinical rotations. Future research could address the applicability of the measure in clinical supervision. Historically, evaluation of the competency of individual clinicians in skills such as suicide risk-assessment was left to summative evaluations such as examinations for board certication or licensure. Given current trends toward expectations that residency and other clinical training programs will document attainment of specic competencies (2, 6), we anticipate increased demand for objective methods to assess such skills. Methods such as the one described in this report have potential to provide a framework for formatively assessing trainees in building skills in risk-assessment, measuring acquisition of these skills, and providing feedback to them in development of competency in suicide risk-assessment and management. Furthermore, use of the CAIS in an OSCE format may assist training programs in summative assessments related to this competency; for example, using it as part of the Clinical Skills Verication Examination (24).
Preliminary results were presented at the Annual Convention of the American Psychiatric Association, May 16 21, 2009, San Francisco, CA. This research was supported in part by Grant T32 MH-18261 from the NIMH. Academic Psychiatry, 36:1, January-February 2012

At the time of submission, the authors reported no competing interests.

References
1. Lurie SJ, Mooney CJ, Lyness JM: Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Acad Med 2009; 84:301309 2. Kaslow N: Competencies in professional psychology. Am Psychologist 2004; 59:774 781 3. Rudd MD, Cukrowicz, Bryan CJ: Core competencies in suicide risk assessment and management: implications for supervision. Training Educ Profess Psychol 2008; 2:219 228 4. Scheiber SC, Kramer TA, Adamowski SE: Core Competencies for Psychiatric Practice: What Clinicians Need to Know: A Report of the American Board of Psychiatry and Neurology, Inc., Washington, DC, American Psychiatric Press, 2003 5. ACGME Psychiatry Residency Requirements; available at www.acgme.org/acWebsite/downloads/RRC_progReq/ 400pr1104.pdf; accessed Feb 6, 2009 6. Andrews LB, Burruss JW: Core Competencies for Psychiatric Education: Dening, Teaching, and Assessing Resident Competence, 3rd Edition. Arlington, VA, American Psychiatric Publishing, 2004 7. Mann JJ, Apter A, Bertolote J, et al: Suicide prevention strategies: a systematic review. JAMA 2005; 292:2064 2074 8. Melton BB, Coverdale JH: What do we teach psychiatric residents about suicide? a national survey of chief residents. Acad Psychiatry 2009; 33:4750 9. Pisanti AR, Cross WF, Gould M: The assessment and management of suicide risk: state of workshop education. Suicide Life Threat Behav 2011; 4:255276 10. Shea SC, Barney C: Macro-training: a how-to primer for using serial role-playing to train complex clinical interviewing tasks such as suicide assessment. Psychiatr Clin North Am 2007; 30:e1 e29 11. McNiel DE, Fordwood SR, Weaver CM, et al: Effects of training on suicide risk assessment. Psychiatr Serv 2008; 59:14621465 12. Newble D: Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ 2004; 38:199 203 13. Goisman RM, Levin RM, Krupat E, et al: Psychiatric OSCE performance of students with and without a previous core psychiatry clerkship. Acad Psychiatry 2010; 34:141144 14. American Psychiatric Association: Practice Guidelines for the Assessment and Treatment of Patients With Suicidal Behavior. Arlington, VA, American Psychiatric Association, 2003 15. Bongar B, Berman AL, Maris RW, et al. (eds): Risk Management With Suicidal Patients. New York, Guilford, 1998 16. Simon RI, Hales RE (eds): Textbook of Suicide Assessment and Management. Arlington, VA, American Psychiatric Publishing, 2006 17. Appelbaum PD, Gutheil TG: Clinical Handbook of Psychihttp://ap.psychiatryonline.org 27

COMPETENCY TO ASSESS SUICIDE RISK atry and the Law, 3rd Edition. Philadelphia, PA, Lippincott, Williams & Wilkins, 2006 18. Simpson S, Stacy M: Avoiding the malpractice snare: documenting suicide risk assessment. J Psychiatr Pract 2004; 10:15 19. Swick S: Assessing the ACGME competencies in psychiatry training programs. Acad Psychiatry 2006; 30:330 351 20. Schuwirth LT, Van der Vleuten CM: Changing education, changing assessment, changing research? Med Educ 2004; 38:805 812 21. Hays RB, Davies HA, Beard JD: Selecting performance assessment methods for experienced physicians. Med Educ 2002; 36:910 917 22. Epstein RM, Hundert EM: Dening and assessing professional competence. JAMA 2002; 287:226 235 23. Bienenfeld D, Klykylo W, Lehrer D: Closing the loop: assessing the effectiveness of psychiatric competency measures. Acad Psychiatry 2003; 27:131135 24. Bhugra D, Malik A: Workplace-Based Assessments in Psychiatric Training. Boston, MA, Cambridge University Press, 2011

28

http://ap.psychiatryonline.org

Academic Psychiatry, 36:1, January-February 2012