Reflective Feedback in Graduate Medical Education

Adah Chung, M.Ed, University of Colorado School of Medicine Center for Advancing Professional Excellence, USA

5th Annual ASPE Conference Tucson, AZ August, 2006

Agenda

Welcome and Introductions Objectives Background and challenges Brief literature review What we did Reflective feedback Discussion Concluding remarks

Lets talk

What are your program experiences with working with residencies?

None an area were interested in Objective Structured Clinical Examinations Communications training Small group teaching Formative assessment with feedback

Objectives

Participants will understand how using reflective feedback with standardized patients in graduate medical education can help to meet the ACGME requirements. Participants will learn about the benefits of using verbal feedback for residents. Participants will learn about a subjective written patient perception tool to guide discussion with residents Participants will learn how a formative assessment using immediate reflective feedback is implemented.

A little history

ACGME Outcomes Project emphasis on educational outcomes assessment based on (6) competencies:

ACGME Toolbox:

Patient care Medical knowledge Practice based learning Communication Professionalism System Based practice

Checklist evaluation Global rating of live or recorded performance SP exam OSCEs Patient Surveys

The challenges faced

Resident competencies (skills, knowledge and attitudes) must be translated into a set of complex behaviors that result in the delivery of high-quality medical care (Whitcomb, 2002)

how to measure this??

The challenges faced

Checklist evaluation: Difficulty with consensus on essential behaviors/action and sequence Scoring behavior using checklists is more difficult when checklists assume a fixed sequence of actions because experienced physicians use various valid sequences (ACGME Toolbox, 2000) Checklists unable to capture complex and hierarchical problem-solving skills of experienced clinicians (Hodges et al, 2002)

The challenges faced

SP Exams, OSCEs: Perception of SP exams by resident directors and residents Pervasive idea that residents should have skills and behaviors at a more complex level than medical students

The challenges faced

Patient Surveys: Not enough specific behavior information for formative evaluation and performance improvement (ACGME Toolbox, 2000)

A possible solution a formative assessment (educational experience)

Allows reflection from residents (meets complexity/professional competence concerns) Non-threatening environment for learning (meets attitudes towards SPs) Trained SPs can provide more specific behavioral feedback than patients (meets patient survey concerns)

The purpose of assessment

The outcomes of assessment should foster learning, inspire confidence in the learner, enhance the learners ability to self-monitor and drive institutional selfassessment and curricular change.

Epstein, R.M. & Hundert, E.M. (2002) Defining and assessing professional competence, JAMA, 287(2):226235.

Professional competency

Epstein & Hundert (2002):

Professional competence developmental, impermanent and context dependent More than demonstration of isolated competencies A competent physician possesses the integrative ability to think, feel and act like a physician

Dimensions of competence

Barnett (1994), Talbot (2004):

Knows that knows how reflective knowing Reflective knowing: focus on dialog and argument for better practical understanding

Definition of reflection

Reflection: 6: A thought, idea or opinion formed or a remark made as a result of meditation, 7: consideration of some subject matter, idea or purpose

Merriam-Webster Dictionary

Reflection

Donald Schon (1983):

Reflection-on-action reflecting on experiences Adds to professional repertoire Difference between effective and noneffective practitioners

Reflection contd
Reflection as a way to help professionals integrate skills, knowledge, behavior and attitudes Reflection as a higher level cognitive and personal process as one develops professionally

My question:

If reflection is the key to the difference in learning & it promotes integration of skills and behaviors that residents should possess, then can it be facilitated?

The literature on SPs giving feedback

Howley & Martindale (2004):

Students who responded favorably to SP experience more likely to have received feedback from SPs Students agreed strongly that feedback from SPs are clear, non judgmental, specific and provided them with patients perspective of the encounter

The literature on SPs giving feedback

Hodder, Rivington, Calcutt & Hood (1989):

Even brief feedback encounters (2 minutes) can have significant influence on clinical competency

Feedback & Attitudes

Efficacy of feedback:

students who receive feedback regarding clinical performance report more positive attitudes towards the clinical experience

Howley, LD & Martindale, J. (2004). The efficacy of standardized patient feedback in clinical teaching. A mixed methods analysis. Med Educ Online, 9(18): 1-10.

Attitudes are an important factor in adult learning!

Questions?

Agree or disagree reflection important for graduate level medical education/professionals?

What we did

A formative assessment based on 3 aspects of the ACGME competencies:

Patient Care Communication and Interpersonal Skills Professionalism

Logistics

4 stations:

20 min. patient encounters 10 min. verbal feedback from SP/TA

Checklists were in the form of patient perception questionnaires helped to guide discussion with resident Cases: Bad news, Smoking cessation, Diabetes management, Urological exam (TA)

Logistics contd

Feedback session:

Invite resident to self assess Provide a positive statement Provide an area to improve End with an overall positive statement Use the PPQ as a tool to guide discussion Describing feelings from patients perspective

Reflective Feedback

Promotes reflection as part of a dialog (both parties involved) Focuses on observed behavior Descriptive vs. judgemental Specific vs. general Sharing information vs. advice giving

Training of SPs

Focused on feedback using a feedback manual:

Purpose of feedback Characteristics of feedback Tips for success Methods for giving feedback (the sandwich, I statements) Examples and practice scenarios Appendices

Patient Perception Questionnaire

Checklist items 1. Would you do what this doctor asks you to do? Patient Care 2. Did this doctor demonstrate compassion? 3. Was this doctor respectful? Communication and Interpersonal Skills 4. Did this doctor communicate effectively? How? ______________________________________________________________________ ______________________________________________________________________ _______________________________ 5. Did this doctor demonstrate effective listening skills? How? ______________________________________________________________________ ______________________________________________________________________ _______________________________ 6. Did this doctor demonstrate nonverbal communication skills? How? ______________________________________________________________________ ______________________________________________________________________ _______________________________ 7. Did this doctor demonstrate the ability to establish a working relationship with you? 8. Did this doctor elicit and answer your questions? Professionalism 9. Did this doctor demonstrate sensitivity and responsiveness to your needs? 10. Did you feel you could trust this doctor to be your advocate? Yes No

PPQ

Terminology and categories following (3) ACGME competencies Yes/No provide more useful, reproducible results (ACGME Toolbox, 2000) Room for comments of specific behaviors for performance improvement

PPQ guide

Guide for training focused on:

Understanding items Examples for things that could be written under HOW? Ie. Effective communication

Invited me to tell my story Summarized what I said Allowed me to ask questions

Ie. Non-effective communication

Interrupted me Used closed ended questions

An example

Hi thanks for coming in. How did you think things went? It really is uncomfortable to talk to someone when shes crying I didnt know what to do. When you put your hand on my shoulder, I felt reassured. I also liked that you gave me space to cry rather than trying to talk through my tears. Oh Im glad I actually just didnt know what to say Saying nothing at all was very comforting to me What else could I have done? Several times you tapped your foot on the floor and looked at your notes rather than me that made me feel like I was taking up too much of your time. I didnt even notice that I was doing that! Overall, your gentle demeanor and touch helped me to understand the news you were giving me. Thank you.

Resident response

Receiving feedback from standardized patients was very helpful Direct feedback Great food Good learning tool. I felt most of the feedback was useful, but there was no criticism The topics are relevant to practice Ability to ask afterward how they thought I did Immediate feedback Good feedback/Good cases. Immediate feedback was very helpful Excellent specific feedback Helpful feedback. It helps me to see my weak points. I think focusing on communication is best Very low pressure atmosphere. Real life 20 min appts

Benefits of this approach

Uses ACGME recommended toolbox ideas: combining SP encounters with patient surveys Provides structured observation Allows for professional self reflection (reflection-on-action) Checklist data for residency directors Direct feedback for residents from trained standardized patients (changing attitudes)

The purpose of assessment

The outcomes of assessment should foster learning, inspire confidence in the learner, enhance the learners ability to self-monitor and drive institutional selfassessment and curricular change.

Epstein, R.M. & Hundert, E.M. (2002) Defining and assessing professional competence, JAMA, 287(2):226235.

Discussion

What are the advantages and disadvantages of using a reflective, narrative, verbal discussion with standardized patients? What strengths and/or weaknesses do standardized patients bring to this kind of formative assessment?

Conclusion
A direct feedback method with a subjective written tool can help to demonstrate competencies while appreciating the complex learning interaction that happens as one becomes a professional

Future considerations
An added written self reflection exercise? Research on the efficacy of SP feedback to promote reflection

Final thoughts

Recognition of SP strengths and weaknesses and how to meet the needs of the residents and residency programs Challenge to think about ways we can assist residency programs in creative ways, other than our usual OSCE and SP exams Be intentional rather than accidental!

Thank you!

Please contact Adah Chung, MEd. at adah.chung@uchsc.edu or (303) 724-1209 for more information