Radiology Health Services Research

Appropriateness and
Imaging Utilization:
‘‘Computerized Provider Order Entry and Decision Support’’

James H. Thrall, MD

Modern imaging methods have been transformative in improving medical care. Cross-sectional imaging has largely eliminated the need for
invasive ‘‘exploratory’’ surgery and is widely used to triage acutely ill patients. However, how to best use medical imaging with ongoing
concerns related to overall costs and radiation risks remains controversial. Imaging saves lives, but overuse of imaging can add unnec-
essary costs to the health system and add to the radiation burden of the population. In this article, the American College of Radiology
Appropriateness Criteria (ACRAC) are reviewed, while the Massachusetts General Hospital experience with a computerized physician (pro-
vider) order entry system and other approaches to utilization management are discussed. There are strong evidence-based indicators of
appropriateness for a substantial percentage of common imaging applications and where this is the case, decision support systems based
on ACRAC or other criteria can and should be used. Standardize health care delivery and elimination of wasteful practice variation can be
achieved without the art of medicine being ignored or devalued.
Key Words: Appropriateness criteria; utilization; computerized physician order entry (CPOE).
ªAUR, 2014

M
odern imaging methods have been transformative
in improving medical care. Cross-sectional imag-
ing with magnetic resonance imaging (MRI) and
computed tomography (CT) has largely eliminated the need
for invasive ‘‘exploratory’’ surgery and is widely used to triage
acutely ill patients with resultant measurable positive impacts
on morbidity, mortality, and costs (1–3). The introduction
of breast cancer screening with mammography has been
associated with reduced mortality from breast cancer.
Imaging is now the guiding hand of medical practice in
innumerable settings. Nonetheless, controversy still reigns
over the correct utilization of medical imaging with
ongoing concerns related, most importantly, to overall costs
and radiation risks.
Finding the correct balance between the rising use of med-
ical imaging and its costs and risks is a central question facing
the health system at a policy level but also facing providers on a
day-to-day patient-by-patient level. In short, imaging saves
lives, but overuse of imaging can add unnecessary costs to
the health system and add to the radiation burden of the
population.
Four questions appear fundamental to achieving an ideal
balance in imaging utilization and should be addressed before
Acad Radiol 2014; 21:1083–1087
From the Department of Radiology, Massachusetts General Hospital, Harvard
Medical School, 25 New Chardon St, Suite 400, Boston, Massachusetts
20114. Received January 24, 2014; accepted February 20, 2014. Address
correspondence to: J.H.T. e-mail: jhthrall@mac.com
ªAUR, 2014
http://dx.doi.org/10.1016/j.acra.2014.02.019
any imaging examination is undertaken. 1) Does this patient
need an imaging examination in the first place? More specif-
ically, is an imaging examination likely to generate medical
information of value in caring for the patient? If the answer
to the first question is negative, the additional considerations
about imaging are moot. 2) If an imaging examination
is felt indicated, exactly what should the examination be?
3) How should the examination be performed—what are
the best modality and the best protocol?. And 4) Can the pro-
posed imaging evaluation be performed with less ionizing
radiation or no ionizing radiation? An additional important
consideration is how to create practical approaches to address-
ing these questions that fit into and mesh with the work pro-
cess or work flow of physicians and other providers that are
neither disruptive to doctor–patient relationships nor exces-
sively time consuming.
AMERICAN COLLEGE OF RADIOLOGY
APPROPRIATENESS CRITERIA (ACRAC)
All the four key questions come back to the central concept of
establishing criteria that address the relative appropriateness
of use of imaging methods in different clinical settings. The
ACRAC developed over the last 20 years define this linkage
(4,5). The ACRAC are based on clinical scenarios of disease
presentation where literature evidence and expert opinion
are used to determine how likely it is that a particular
imaging examination will yield medically meaningful
information for a given indication or set of indications. The
ACRAC use a scoring or scaling system from 1 (low

1083
THRALL
likelihood of providing useful information) to 9 (high
likelihood of providing useful information).
A more complete description and listing of the ACRAC are
available on the ACR Web site. In brief, there are now
ACRAC available for diagnostic, interventional, and radiation
oncology topics. For diagnostic imaging, there are 197 topics,
or clinical presentation scenarios, with over 900 variants (6).
Each scenario variant has been assigned both an appropriate-
ness score and an indication of the relative amount of radiation
associated with the examination. The ACRAC are updated
on an ongoing basis. There are expert teams for each part of
the body that include radiologists and nonradiologists.
COMPUTERIZED PHYSICIAN ORDER ENTRY
The emerging standard for implementing the practical use of
the ACRAC is the adoption of computer-based ‘‘decision
support’’ (DS) systems either as stand-alone systems or inte-
grated with ‘‘computerized physician order entry’’ (CPOE)
systems. The DS systems typically run on hospital information
technology platforms allowing physicians to readily access
DS data in the normal course of caring for patients (7,8). In
a typical case, once the provider has determined that an
imaging examination might be indicated, he or she selects
a proposed examination from a list of examinations for the
respective body part and then the reason or reasons for
doing it, such as headache for head CT (Figs 1, 2). The
computer-based DS system then matches this combination
of imaging examination and indication against a table contain-
ing all possible combinations of examinations and reasons for
doing them and returns an appropriateness score (Fig 3). It is
important to note that DS results are important data points,
but it is still up to each institution or practice group to deter-
mine how to proceed after the score comes back. At the Mas-
sachusetts General Hospital (MGH), scores of 7–9 are
considered to indicate a high likelihood of utility and pro-
viders can proceed on to order the test. Scores of 4–6 are
considered to be in a marginal range where a provider’s
knowledge of the patient and assessment of intangibles are
especially important. The provider is expected to consider
this and use discretion (7).
Sores of 1–3 are considered to generally indicate a low like-
lihood of utility. The provider is asked to reconsider ordering
the test but may continue if he or she feels that special circum-
stances apply that are beyond the ordinary scope of the appro-
priateness criteria. For example, there may be factors related
to unusual physical findings, family history, or even a patient’s
psychological state (7).
The MGH philosophy toward DS is that however powerful
and useful computer DS systems may be, in the end, they
should not interfere with the doctor–patient relationship or
trump the physician’s best judgment. It is simply impossible
to write enough rules to cover every variation or nuance to
reduce the practice of medicine completely to a set of
computerized rules. The importance and significance of the
way a patient looks subjectively, the timbre of a patient’s voice
1084
Academic Radiology, Vol 21, No 9, September 2014
or the changing strength of a handclasp from one point in time
to another can neither be fully addressed in a computer rule
nor even be communicated well between people but can be
pivotal in a physician’s thinking about a patient.
Rather than interceding in individual cases during the
active care process, the utilization performance of physicians
is tracked over time at MGH. Simple metrics such as imaging
utilization per patient visit and average appropriateness scores
adjusted for practice setting help identify outliers who can
then be engaged educationally. The medical leadership of
the Massachusetts Hospital Physicians’ Organization is
actively involved in reviewing physician performance data
and working with physicians to achieve reasonable utilization.
A unique feature of the MGH DS system is that it provides
a utility score not only for the requested examination and
reason for doing it but also for other protocols and other
imaging modalities aimed at the same clinical problem. This
addresses the issues of the best protocol and whether an exam-
ination can be performed with lower radiation or no radia-
tion. For example, whenever a head CT with contrast is
proposed, the DS system also provides a score for CTwithout
contrast and MRI both with and without contrast. For body
imaging, a request for a CT is often returned with a score for
ultrasound as an alternative. This is very valuable to referring
physicians who are challenged to stay current on the nuances
of the hundreds of different imaging examinations and their
variants.
For some scenarios, providers are asked for additional or
more detailed information. In the case of CT or MRI for
headache, providers are asked to specify what the nature of
the headache is. A simple long-standing headache is a lower
yield indication by far than a thunder clap headache, the worst
of a patient’s life, associated with a neurologic finding.
Likewise, to take advantage of integrating diagnostic
knowledge across disciplines, DS systems can provide different
scores depending on how much other information from other
methods has been obtained. A comparatively low cost
D-dimer test can effectively rule out pulmonary embolism.
In the MGH DS system, a request for a pulmonary CT for sus-
pected pulmonary embolism will receive a low utility score if
there is no information on D-dimer or if the test is normal.
The same request with notation that a D-dimer test was
abnormal would receive a high utility score.
The use of computer-based DS systems also offers the
opportunity to add many additional kinds of value to the
test ordering process that can help the referring physician.
In the MGH system, there is an associated search tool that
automatically retrieves information on allergies and renal
function studies when contrast media use is being considered,
saving providers the considerable effort of going through the
patient’s medical record. The system calculates an estimated
glomerular filtration rate. Another useful feature to help
reduce unnecessary duplicate procedures alerts the ordering
provider if the same or a similar test has been done recently
or has been scheduled to be performed. Safety alerts are added
especially for MRI to flag issues such as implanted devices for
Academic Radiology, Vol 21, No 9, September 2014 APPROPRIATENESS AND IMAGING UTILIZATION

Figure 1. Massachusetts General Hospital Web-based computerized physician order entry (CPOE) system with integrated decision support
(DS). On initial screen, physicians indicate what examination they would like to order. MR of the head (brain) is indicated. CT, computed tomog-
raphy; MRI, magnetic resonance imaging.

the ordering physician’s consideration. Because all these fea- OTHER APPROACHES TO UTILIZATION
tures work in concert and in essence instantaneously, the over- MANAGEMENT
all ordering process is substantially facilitated and streamlined
The use of radiology benefit management (RBM) companies
for the provider in spite of adding the DS step.
that contract with commercial insurance companies to control
In some settings, it is not uncommon for patients to request
imaging utilization is the major current alternative to CPOE
studies that are not likely to help in their care. The publicity
with DS for imaging utilization management. Typically,
surrounding star athletes undergoing imaging studies is one
a referring provider is required to call a 1-800 number to
source of influence in this regard. In selected cases, referring
receive prior approval from the RBM for a defined set of ex-
physicians have found it useful to share the information com-
aminations. Although there are data indicating that this
ing from the DS system as an objective ‘‘second opinion’’.
approach can reduce utilization, it is fraught with negatives
The response to CPOE and DS at MGH has been very
compared to CPOE and DS (9).
encouraging. Some people thought it would just become a
The two central complaints of providers about the role of
new computer game that referring physicians would quickly
RBMs are their disruption of the care process and their lack
master but then continue doing whatever they wanted. How-
of transparency. Physicians and their office staffs must stop
ever, when the dust cleared and ordering rates were examined
what they are doing in caring for patients to place phone
before and after the implementation of DS with the ACRAC,
calls to RBM representatives where they are frequently
the growth curves for major imaging were clearly impacted.
placed on hold and may need to go through several levels
CT at MGH had been growing at a compound rate of 12%
of decision makers before an approval, or disapproval, can
pre-DS. After DS, the curve flattened to <1% compound
be obtained. There is no compensation for the time and
growth rate. When overall growth in number of patients was
effort, and the process is not geared to look for alternatives
taken into account, CTwent flat to negative, and in later years,
if a particular procedure is denied. Thus, there is an element
the number of CTs and MRIs per year per each unique MGH
of cost shifting versus true savings to the health system in the
patient identification number dropped significantly. This
United States.
metric is an indicator of the intensity of use of these procedures
The lack of transparency stems from the lack of published
per patient. These data indicating the efficacy of CPOE with
criteria by RBM companies through which decisions are
DS at MGH were in hand before Congressional cuts to imag-
made and the lack of published data indicating perfor-
ing reimbursement were made and before health care reform
mance—how many requests are denied and what is the total
and the down turn in the economy made it difficult to assess
impact on utilization among other data. The lack of published
cause and effect for changes in imaging utilization.

1085
THRALL Academic Radiology, Vol 21, No 9, September 2014

Figure 2. Signs and symptoms page. Ordering physician indicates reason for requesting the examination. Headache is indicated.

Figure 3. For the combination of requested examination and indication, a low priority score is given. Note also the low scores for alternative
examinations. MRI, magnetic resonance imaging.

criteria makes the RBM approach a ‘‘black box’’ from the to request. With the lack of transparency, there is also a noto-
viewpoint of referring physicians. They cannot benchmark rious lack of consistency from one call to the next for identical
their performance or learn to do better in selecting studies scenarios.

1086
Academic Radiology, Vol 21, No 9, September 2014
Another problematic aspect of the RBM approach is the
devaluation of the role of the provider and interruption of
the doctor–patient relationship. All the accumulated empirical
wisdom of physicians and their special knowledge of their
patients is dismissed as not relevant. Physicians find themselves
backpedaling to explain to their patients that they have been
overruled, and the study they recommended was considered
inappropriate potentially undermining the patient’s trust in
their ability and authority.
CONCLUSIONS
Although the proper utilization of imaging should be a goal of
all radiologists and all physicians referring patients for imaging
evaluation, it is unlikely that ideal utilization in every clinical
setting can ever be completely defined, let alone achieved.
Yet, there are strong evidence-based indicators of appropriate-
ness for a substantial percentage of common imaging applica-
tions, and where this is the case, DS systems based on ACRAC
or other criteria can and should be used. It seems reasonable to
ask providers to endorse the use of electronic DS systems and
receive the benefits of just-in-time knowledge delivered at the
point of care.
However, in the zeal to codify and standardize health care
delivery and eliminate wasteful practice variation, the art of
medicine should not be ignored or devalued. Within limits
of reason and with indications of otherwise overall good
APPROPRIATENESS AND IMAGING UTILIZATION
performance, physicians should always have the latitude to
use their special knowledge and direct assessment of a patient
to help guide the care process including the ordering of imag-
ing examinations.
REFERENCES
1. Battle JC, Hahn PF, Thrall JH, et al. Patients imaged early during admission
demonstrate reduced length of hospital stay: a retrospective cohort study of
patients undergoing cross-sectional imaging. J Am Coll Radiol 2010 Apr;
7(4):269–276.
2. Abujudeh HH, Kaewlai R, McMahon PM, et al. Abdominopelvic CT in-
creases diagnostic certainty and guides management decisions: a pro-
spective investigation of 584 patients in a large academic medical center.
AJR Am J Roentgenol 2011 Feb; 196(2):238–243.
3. Rao PM, Rhea JT, Novelline RA, et al. Effect of computed tomography of the
appendix on treatment of patients and use of hospital resources. N Engl
J Med 1998; 338:141–146.
4. Cascade PN. Setting appropriateness guidelines for radiology. Radiology
1994; 192:50A–54A.
5. Cascade PN. The American College of Radiology. ACR Appropriateness
Criteria Project. Radiology 2000; 214(Suppl):3–46.
6. American College of Radiology. About the ACR Appropriateness Criteria.
Available at http//www.acr.org/Quality-Safety/Appropriateness-Criteria/
AboutAC. Accessed February 12, 2014.
7. Sistrom CL, Dang PA, Weilburg JB, et al. Effect of computerized order entry
with integrated decision support on the growth of outpatient procedure vol-
umes: seven-year time series analysis. Radiology 2009 Apr; 251(1):
147–155.
8. Vartanians VM, Sustrom CL, Weilburg JB, et al. Increasing the appropriate-
ness of outpatient imaging: effects of a barrier to ordering low-yield exam-
inations. Radiology 2010; 255:842–849.
9. Hendee WR, Becker GJ, Borgstede JP, et al. Addressing overutilization in
medical imaging. Radiology 2010 Oct; 257(1):240–245.
1087