Approaching the Limits of Knowledge: The Influence of

Priming on Error Detection in Simulated Clinical Rounds

Journal: AMIA 2011 Annual Symposium

Manuscript ID: AMIA-0892-A2011.R1

Manuscript Type: Student Paper

Date Submitted by the

15-Jul-2011
Author:

Complete List of Authors: Razzouk, Elie; UT Health Science Center at Houston, School of
Biomedical Informatics
Cohen, Trevor; University of Texas Health Science Center at
Houston, School of Biomedical Informatics
Almoosa, Khalid; Memorial Herman at Texas Medical center
Patel, Vimla

1154
Page 1 of 10

Approaching the Limits of Knowledge: The Influence of Priming on Error

Detection in Simulated Clinical Rounds

Elie Razzouk, MD1,3, Trevor Cohen, MBChB, PhD1,3, Khalid Almoosa, MD2,3, Vimla Patel
PhD, DSc,1,3
1
Center for Cognitive Informatics and Decision Making, School of Biomedical Informatics;
2
Division of Critical Care Medicine; 3University of Texas Health Science Center, Houston,
TX
Abstract

Errors are inevitable in all clinical settings, posing substantial risk to patients. Studies have shown detection and
correction are essential to error management. This paper documents the use of Opensimulator, a virtual world
development platform, to create a virtual Intensive Care Unit where error recovery can be studied in a controlled,
yet realistic environment. Subjects participated in rounds presented by computer-generated characters. Errors were
embedded in these presentations, and subjects were evaluated for their ability to detect them. Eight subjects were
asked to evaluate two cases and answer related knowledge-based questions under two conditions: primed
(forewarned of the presence of errors) and un-primed. Subjects frequently failed to detect errors despite having the
prerequisite knowledge. Priming significantly improved detection, suggesting a role for interventions that aim to
shift clinicians’ error detection toward the limits of their knowledge. Such interventions may provide means to
decrease adverse events resulting from human error.
Introduction
Advances in medicine have raised public expectations of the performance of clinicians1. Despite technological
advances, medical errors remain highly prevalent, and frequently lead to adverse events2. Consequently, over the
past decade a substantial research enterprise has examined different approaches that aim to reduce such errors3.
Another line of research investigated the role of error recovery (detection and correction) as a counterbalance to
error commission4. These studies focus on identifying the processes involved in the generation of and recovery from
preventable medical errors during the course of decision-making in critical care settings. Such studies represent a
move away from the notion of zero-defect clinical performance. Given the increasing recognition of the contribution
of error recovery to safety in other industries5, additional efforts devoted to understanding the processes involved in
decision making and error recovery are warranted. Such efforts may shed light on the factors that impede or
facilitate error detection and thus may prove valuable in future ventures toward reducing the adverse impact of
medical errors. Recent work has examined the phenomenon of error recovery, both in laboratory settings and in the
field 4, 6. These approaches complement one another, as each has different strengths and weaknesses. In particular,
there is a trade-off between strict experimental control in the laboratory and ecological validity in a naturalistic
setting7. In this paper, we attempt to find a middle ground between these two experimental approaches by studying
medical errors in the context of simulated rounds within a virtual intensive care setting. This approach allows for
experimental control, as well as a better approximation of the way in which clinical information is presented in
collaborative settings than is possible with traditional laboratory-based studies.
The purpose of our paper is to study error recovery in a context approximating the verbal presentation of cases on
clinical rounds. Therefore, we describe in some detail our methodological approach which uses virtual world
technology to approximate such settings. We are interested primarily in characterizing failure to detect errors and in
particular in distinguishing between failed error detection due to lack of knowledge and failure for other reasons.
Our experimental design, which incorporates priming of subjects to detect error and knowledge-based questions,
aims to reveal this distinction. It should be noted that the use of the term “priming” in psychology, where it refers to
a process that utilizes specific stimuli to affect implicit memory, we refer to priming in the sense that is used in
medical education8, where it refers to the process of orienting participants beforehand to the tasks and objectives
they may have9.

1155
 Page 2 of 10

Background
Clinical settings have been described as error prone complex systems where individuals’ actions are interconnected
and unpredictable10. Within such environments, it is unrealistic to expect flawless performance, as some degree of
error is to be anticipated on account of learning, conflicts and limited resources in the face of high demand11.
Furthermore, research in highly complex fields other than medicine has shown that error recovery has a pronounced
impact on safety5. However, relatively little is known about the processes that underlie error detection and recovery
in medical settings.
Clinical rounds serve as a focal point for communication, decision making, transition of care, and teaching. Frequent
rounds remain an important part of the daily routine of clinical teams: on rounds different participants rapidly
aggregate information from different sources to make clinical decisions12. Additionally, the lack of such rounds
resulted in an increase in mortality, cardiac arrests, and other adverse events in specific settings13. Furthermore,
rounds have been observed to be an important focal point for detection of, and recovery from medical errors6.
Following lengthy ethnographic observations of an intensive care clinical team, Kubose, Patel, and colleagues
determined that foci of high interactivity among physicians lead to higher incidence of error detection14. In
particular, it was observed that a higher number of errors were detected in clinical rounds than in handovers, which
involve fewer participants. Similarly, ethnographic observation revealed clinical rounds in a psychiatric emergency
department (PED) to be a source of high-yield data for incidents of error recovery. Analysis of audio recordings of
these rounds highlighted the importance of error recovery for overall patient safety and uncovered several errors
with potentially harmful consequences6. Similar work done in other emergency departments demonstrated the role
of communication and other strategies among nursing team members in detection and recovery from error15.
Retrospective data obtained from an accident reporting system used in two hospitals in Belgium demonstrated the
importance of standard checks in error detection16.
In more recent work, Patel and her colleagues employed a novel experimental paradigm using errors embedded in
paper-based case scenarios to study error detection and recovery among experts and trainees4. Overall, error
detection and recovery correlated poorly with years of experience. Additionally, no subject detected more than half
of the embedded errors, regardless of level of expertise. If this laboratory-determined limit on error detection is an
accurate reflection of the rate of detection “in the wild”, the implications for patient safety are disturbing and urgent.
As acknowledged by the authors, this study had certain limitations. Firstly, the study design did not incorporate a
method to discern whether failure to detect an error occurred on account of lack of knowledge, or for some other
reason. A deeper understanding of the mechanisms that underlie failed error detection is required if we are to
develop interventions to better equip current and future clinicians to detect and recover from potentially dangerous
errors. In particular it is necessary to distinguish between a lack of the knowledge required to detect an error, and the
failure to apply this knowledge, as these conditions suggest different intervention strategies. Moreover, information
on rounds is generally presented verbally rather than as written text, and the presentation frequently involves input
from several members of a clinical team. It is possible that the additional cognitive effort required to synthesize
verbal information from multiple sources plays a role in the process of error recovery. Ideally, error recovery and
detection would be studied in a setting that better approximates a clinical round, as it has been shown that error
recovery frequently occurs in such settings.
In this paper we apply a novel approach to study error detection and recovery in the context of clinical rounds using
an immersive virtual world to approximate the information interchange that occurs on rounds, while retaining the
control necessary to ensure the consistency of experimental conditions across subjects. A virtual world is a
computer-based simulated environment, through which users can interact with scripted programs or with one
another17. Within a virtual world, users are embodied as avatars, virtual representations of themselves that they can
direct to explore the environment and interact with it and other users. Virtual worlds are becoming increasingly
popular in medical domains and are proving to be cost-effective and reliable training methods18. In some studies,
virtual mentors have proven to be superior to traditional training and teaching methods19. However, to the best of our
knowledge no such tools have been developed to serve primarily as research instruments with the aim to capture and
study clinicians’ cognitive and decision making processes. The ability of virtual worlds to approximate real world
settings has proved important for their role as training tools, and suggests a viable alternative for the study of error
detection and recovery in a naturalistic setting without disturbing real-world clinical workflow.
There are a variety of tools that can be used to build virtual worlds. Popular tools include Opensimulator20 and
Second Life21. Opensimulator (OpenSim) is an open source project that provides a host server for virtual worlds that
can be accessed by a variety of clients. As is the case with the better-known SecondLife platform, it has the ability to

1156
Page 3 of 10

support clients that allow for the visual exploration of three-dimensional virtual environments in real time20.
However, unlike SecondLife, the platform is open source and available for download. Users can maintain their own
servers, and consequently do not need to pay fees for virtual real estate or the upload of graphics or sound files. The
platform has attracted projects from companies such as Intel and IBM, which provide examples of the commercial
prospects of virtual environments22. Scripting offers the ability to control and simulate events within virtual
environments, and can be implemented in a variety of languages such as Linden Scripting Language (LSL),
Opensim Scripting Language (OSSL), and C# using the OpenMetaverse library23. OpenSim is supported by an active
development community.
Methods
In order to focus on the processes of error detection in the Medical Intensive Care Unit (MICU), participants (N =
17) consisting of interns, residents, and fellows were recruited for the study. As part of the MICU team, all the
subjects involved were actively taking part in patient care, participating in clinical rounds, and making important
management decisions. Two case scenarios were developed to represent typical ICU cases that trainees face over
their month-long rotation at the medical intensive care unit. Our clinical collaborator and co-author, (KA), a
practicing physician who is board-certified in internal medicine, pulmonary medicine, and critical care medicine,
developed these scenarios and embedded errors with varying degrees of complexity and severity. The main focus
during the development of the cases was to maintain clinical plausibility. Therefore the number of errors included in
each case was determined by the plausibility of the clinical scenario itself. Consequently, the study design which
will be discussed further in this section, adjusts for these differences among the two cases. Examples of errors
embedded in case scenarios A and B are shown in Table 1.
Table 1. Examples of errors embedded in the case scenarios A and B.

Examples of Errors Embedded in Case Scenarios A and B

1. Coagulopathy was not corrected. This could have contributed to the bleeding. (Case B)

2. The GI team did not come in earlier to do an EGD. (Case B)

3.Paracentesis should be done before or soon after antibiotics are started. (Case B)

4. Gentamycin is not the drug of choice for someone with renal insufficiency. (Case B)

5. IV steroid dose is insufficient. (Case A)

6. Dopamine was started instead of epinephrine. (Case A)

7. No blood cultures were taken. (Case A)

8. GI stress ulcer prophylaxis was not started. (Case A)

Our work builds on the approach of Patel et al.4 in which case scenarios with embedded errors are used to study the
cognitive processes underlying error detection and recovery. However, unlike case scenarios presented on paper,
residents involved in rounds usually present and acquire patient-related information verbally. Therefore, additional
cognitive work is required to integrate this information and hypothetically this cognitive work may be important to
the processes involved in error detection. Consequently, the choice was made to present the case scenarios verbally
within Opensim. Ethnographic data gathered in the course of other related research have provided valuable insight
into the rounding process allowing us to ensure that the virtual clinical rounds approximated actual clinical rounds.
However, had such data not been available during the development process, we would have had to collect the
required data using formative evaluations during the development process. The steps required to build the virtual
world and record the rounds are depicted in Figure 1.

1157
 Page 4 of 10

Figure 1. Steps in the process of building virtual rounds.

Knowledge-based questions were added following each case to investigate the basis of undetected embedded errors.
These questions aim to test the clinical knowledge required to catch each of the errors. Errors missed due to lack of
knowledge should result in an undetected error together with an incorrect answer to the related question. However,
errors missed due to inattention or any other cause should result in an undetected error with a correctly answered
related question.
The environment simulating an Intensive Care Unit setting was developed using OpenSim due to the ease with
which scripting languages can control this platform. Functions within the OpenMetaverse library were scripted using
the C# programming language to gain control over characters representing the different participants in the
multidisciplinary rounds. These functions allow us to control the timing with which and sequence in which recorded
audio files are played. It is this strict control that allows us to successfully simulate a multidisciplinary round. The
characters representing the various participants in the rounds include two residents, a nurse, a pharmacist, a
respiratory therapist, a medical student, an attending physician, and a character representing the subject immersed in
the round. There were several steps that were required to arrange and organize the rounds (Figure 1). The first step
was to build the world using OpenSim in-world building tools. Following that step, the round was recorded and the
audio files were chopped into 10 second slices and uploaded to each character accordingly. The audio files were
later organized using a C# program that calls the audio files at 10 second intervals resulting in a coherent and well
organized round. In collaboration with a domain expert in non-verbal communication, customized gestures were
developed using the animation tool qavimator (http://www.qavimator.org), and synchronized with the audio files
using the C# script. A screenshot of the resulting round is included in Figure 2.

1158
Page 5 of 10

Figure 2. A screenshot from the rounds within Opensim.

After logging in to the world, subjects were taught to control their avatars by walking around within the virtual ICU.
The subjects were then given this set of instructions:
• Listen carefully to the first case presented
• You may take notes on the writing paper provided
• Following the presentation, summarize the case
• Comment on the management and discuss the resident’s evaluation
• All your responses will be audio recorded
For their second case, all participants were intentionally primed for increased effort focused towards error detection
and recovery by giving them the following instructions:
• Listen carefully to case scenario B
• The team caring for this patient committed several management errors
• You may take notes on the writing paper provided
• Summarize the case after presentation
• Comment on the management and discuss the resident’s evaluation
• All your responses will be audio recorded
Half of the subjects went through the steps in the order discussed above. The other half however, participated in the
simulation of case scenario B first, underwent priming, and then participated in the simulation of case scenario A.
This design accounts for any possible differences in case difficulty, allowing us to discern the effect of priming from
other factors that may influence performance. All responses were audio-recorded, transcribed, and coded for
incidents of error detection. Subjects’ written responses to knowledge-based questions related to the errors in each
case were also evaluated.

1159
 Page 6 of 10

Results and discussion

Error detection by subjects, both unprimed and primed, varied among the two case scenarios. In case scenario A,
subjects on average detected 20.3% of the embedded errors. In case scenario B, subjects generally performed better
with a mean detection rate of 37.2%. However, case scenario A contained a total of 15 embedded errors as opposed
to the six embedded in case B. In addition, there are many subject-specific factors that can affect error detection
such as differences in knowledge and vigilance.
The results from case scenario A are shown on Figure 3.

Figure 3.Number of Detected Errors and Correct Answers to Knowledge Based Questions in Case A.

Detecting errors embedded in case scenarios requires first and foremost the knowledge that the observed decision is,
in fact, erroneous. Consequently, as expected, subjects who detected embedded errors were without exception able
to answer the related knowledge-based questions correctly. However, subjects generally did not detect all of the
errors that were possible within the limits of their knowledge. As we are interested in the extent to which each
participant was able to detect errors to the best of their ability, we define a detection ratio (DR) as follows:

DR = (number of errors detected) / (number of correctly answered knowledge-based questions)

Therefore performance is calculated as the ratio of detected errors to correct answers to knowledge-
based questions related to these errors (the detection ratio). The mean detection ratio on this case
scenario overall is equal to 0.44 with a 95% confidence interval of [0.32, 0.56].

1160
Page 7 of 10

Figure 4.Number of Detected Errors and Correct Answers to Knowledge Based Questions in Case B.
For case B performance was calculated as the ratio of detected errors to correct answers to knowledge-based
questions related to these errors (the detection ratio). The mean detection ratio on this case scenario is equal to 0.47
with a 95% confidence interval of [0.33, 0.62]. Despite a difference in the mean number of errors detected across
cases, the mean detection ratio was similar across the two cases. However there was a greater variability in
performance as measured by this ratio in case B. With respect to the raw detection rate, no subject detected more
than half of the errors in case A, and only three subjects detected more than half of the errors in case B. However,
when taking knowledge into account, the mean detection ratio across all errors from all cases was 0.45. That is to
say, subjects on average detected less than half of the errors for which they possessed the prerequisite knowledge.
Of note, participants were not warned beforehand that errors were present in the first of their cases, as we were
interested in observing the extent to which they would detect errors when interpreting these cases in accordance with
their usual practice. However, for their second case, participants were primed to detect errors: they were informed
explicitly that the case contained errors, and asked to attempt to detect them. Half of the subjects interpreted case A
before being primed and subsequently interpreted case B. The other half interpreted case B before being primed, and
case A after priming. The detection ratio of both unprimed and primed participants is shown in Figure 5. In the case
of the first nine subjects, Case A was viewed first (before priming), while in the case of the remaining subjects (8
subjects), Case B was viewed first.

1161
 Page 8 of 10

1
0.9
0.8
0.7
0.6
0.5 Unprimed
0.4 Primed
0.3
0.2
0.1
0
1 3 5 7 9 11 13 15 17
Figure 5. Detection ratio in both unprimed and primed subjects.
Subjects cannot be expected to detect knowledge-based errors without possessing the prerequisite knowledge. As
illustrated in Figures 1 and 2, subjects’ knowledge acts as a ceiling for their error detection rate. Across case
scenarios, the average detection ratio was similar (Case scenario A: 0.44, Case scenario B: 0.47). However, priming
subjects had a substantial effect on their performance as depicted in Figure 3. Priming clearly shifted the
performance of subjects toward their knowledge ceiling. Furthermore, the mean detection ratio of subjects after
priming (DR = 59.9%) was significantly higher than that of subjects who were not yet primed (DR= 31.1%),(t (16)
=5.1870, p < 0.0001). This suggests that clinicians listening to clinical rounds according to their usual practice
exhibit sub-optimal error detection. This finding is encouraging, as it suggests a role for intervention. While we
cannot pro-actively correct for all possible knowledge deficits, these results show that clinicians are capable of
performance that is considerably better than their baseline, and suggest that it may be possible to develop training
programs that shift error detection toward an individual’s knowledge ceiling. The development and evaluation of
such training materials will be the focus of our future work in this area, which will also include expanding the
current study to include a larger group of representative subjects.
Limitations and future work
Our study did not draw on all of the features of virtual world technology. For example, despite being able to
navigate the virtual world, the participants were limited in their interactions in the clinical rounds to passive
observers. This limitation was intentional, as our aim was to focus on the interpretation of information only. As
such, we did not exploit the capacity that virtual worlds have to support scenarios involving multiple interacting
participants. Additionally, certain features of clinical rounds, such as frequent interruptions, were not a part of our
simulation at this time, though we intend to introduce this in our future work. Furthermore there are features of real
world rounds such as physical interaction with objects that are difficult to approximate within virtual worlds.
Finally, we are currently unable to assess the cost of priming in terms of cognitive load and assess for any resulting
errors from such load. For example, it is possible that while focused on the task of error detection, participants
neglect other cognitive tasks that are important for clinical comprehension. In future work we aim to assess the cost
of priming on the quality of the summaries generated by each participant and study the possible effects of team
interactions on error detection and recovery in general. However, this study opens new avenues to develop and

1162
Page 9 of 10

evaluate interactive tools embedded within virtual worlds to better train participants to detect errors. Despite its
importance for patient safety, clinicians are not formally trained in error detection and as with any skill it is possible
that the acquisition of expertise in error detection will be accelerated by deliberate practice24.
Conclusion
This study focused on the ability of clinicians to detect errors embedded in clinical cases presented on virtual
intensive care rounds and the effects that priming could have on this ability. These simulated virtual worlds provided
a viable and realistic setting for the study of error detection and recovery. Our subjects’ detection of embedded
errors was initially limited, in accordance with previous studies. However, priming for the detection of errors led
participants to be more vigilant, and accordingly their performance improved significantly, at times approaching the
limits of their clinical knowledge. This suggests a role for educational and training interventions that aim to shift
clinicians’ error detection closer to their potential. Such interventions may provide a means to decrease the number
of adverse events that are a result of the inevitability of human error.
Acknowledgements
This research was supported by an award from the James S McDonnell Foundation (Grant No. 220020152). We
thank the members of the clinical team for their participation in our study, as well as those who contributed toward
the realization of our virtual world. In particular, thanks go to Anay Limaye (scripting), Amy Franklin (gesture
design and audio) and Katy Carew, Zeina Razzouk and Arshag Kalanderian for contributing their vocal talents to the
two case scenarios.

References
1. Wu AW. Medical error: the second victim. BMJ 2000; 320:726–7.
2. Kohn, L.. To err is human. Washington: National academy press. 2000
3. Bleich S. Medical errors: five years after the IOM report. Issue Brief (Commonw Fund) 2005;1–15.
4. Patel VL, Cohen T, Murarka T, et al. Recovery at the edge of error: Debunking the myth of the infallible expert.
Journal of Biomedical Informatics 2011;44:413–24.
5. Amalberti R, Wioland L. Human error in aviation. In: Soekkha H, editor. Aviation safety: human factors,
system engineering, flight operations, economics, strategies, management. Utrecht: VSP; 1997.
6. Cohen T, Blatter B, Almeida C, et al. Reevaluating Recovery: Perceived Violations and Preemptive
Interventions on Emergency Psychiatry Rounds. Journal of the American Medical Informatics Association
2007;14:312–9.
7. Patel VL, Kaufman DR, Arocha JF. Emerging paradigms of cognition in medical decision-making. J Biomed
Inform 2002;35:52–75.
8. Heidenreich C, Lye P, Simpson D, et al. The search for effective and efficient ambulatory teaching methods
through the literature. Pediatrics 2000;105:231–7.
9. Grover M. Priming students for effective clinical teaching. Fam Med 2002;34:419–20.
10. Wilson T, Holt T, Greenhalgh T. Complexity science: Complexity and clinical care. BMJ 2001;323:685–8.
11. Rasmussen J. The role of error in organizing behaviour. Quality and Safety in Health Care 2003;12:377–83.
12. Irby DM. How attending physicians make instructional decisions when conducting teaching rounds. Acad Med
1992;67:630–8.
13. Pronovost PJ, Jenckes MW, Dorman T, Garrett E, Breslow MJ, Rosenfeld BA, Lipsett PA, Bass E:
Organizational characteristics of intensive care units related to outcomes of abdominal aortic surgery. JAMA
281: 1310–1317, 1999.
14. Kubose TT, Patel VL, Jordan D. Dynamic adaptation to critical care medical environment: error recovery as
cognitive activity. Proceedings of the 2002 Cognitive Science Society. 2002. p. 43.
15. Henneman EA, Blank FSJ, Gawlinski A, et al. Strategies used by nurses to recover medical errors in an
academic emergency department setting. Appl Nurs Res 2006;19:70–7.
16. Nyssen A-S, Blavier A. Error detection: a study in anaesthesia. Ergonomics 2006;49:517–25.
17. Bishop J. Enhancing the understanding of genres of web-based communities: the role of the ecological
cognition framework. IJWBC 2009;5:4.
18. Eagleson R, de Ribaupierre S, King S, et al. Medical education through virtual worlds: the HLTHSIM project.
Stud Health Technol Inform 2011;163:180–4.

1163
 Page 10 of 10

19. Henderson BA, Kim JY, Golnik KC, et al. Evaluation of the Virtual Mentor Cataract Training Program.
Ophthalmology 2010;117:253–8.
20. Main Page - OpenSim. http://opensimulator.org/wiki/Main_Page (accessed 25 Feb2011).
21. Virtual Worlds, Avatars, free 3D chat, online meetings - Second Life Official Site. http://secondlife.com/
(accessed 25 Feb2011).
22. Prendinger H, Ullrich S, Nakasone A, et al. MPML3D: Scripting Agents for the 3D Internet. IEEE Trans Visual
Comput Graphics 2011;17:655–68.
23. Open Metaverse Foundation. http://openmetaverse.org/ (accessed 25 Feb2011).
24. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition of expert
performance. Psychological Review 1993;100:363–406.

1164