Information System Concepts for Quality Measurement

Author(s): Brent James

Source: Medical Care, Vol. 41, No. 1, Supplement: The Strategic Framework Board's Design for
a National Quality Measurement and Reporting System (Jan., 2003), pp. I71-I79
Published by: Lippincott Williams & Wilkins
Stable URL: http://www.jstor.org/stable/3767730
Accessed: 12/01/2009 01:59

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://dv1litvip.jstor.org/action/showPublisher?publisherCode=lww.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.

JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with the
scholarly community to preserve their work and the materials they rely upon, and to build a common research platform that
promotes the discovery and use of these resources. For more information about JSTOR, please contact support@jstor.org.

Lippincott Williams & Wilkins is collaborating with JSTOR to digitize, preserve and extend access to Medical
Care.

http://www.jstor.org
MEDICAL CARE
Volume41, Number1, Supplement,pp I-71-I-79
02003 LippincottWilliams& Wilkins,Inc.
System Concepts for QualityMeasurement
Information
BRENTJAMES, MD, MSTAT
BACKGROUND. Health care information sys-
tems in use today frequently fall short of what
is needed to meet the demands for data and
reporting on performance. Many observers be-
lieve substantial improvements in information
systems will be necessary if the potential of a
national quality measurement and reporting
system (NQMRS) is to be realized. A shared
vision will facilitate progress in improving
information systems.
OBJECTIVES. TO articulate a set of guiding
principles and operational steps for the devel-
opment of functional information systems in
health care.
RESEARCH DESIGN.Experience in building
such systems for one health care delivery sys-
tem was used to develop an approach. This
was discussed with Strategic Framework
Board members and integrated with other con-
siderations for going from a local system to
one that could accumulate information for na-
Health care is inherently an information sci-
ence. Health care information includes formal
knowledge of disease and disease treatment, as
well as history and physical examination findings,
laboratoryand imaging results, patient preferences
and values, and outcomes of health care interven-
tions. The better information a health care profes-
sional has, the better he or she can diagnose
illness, identify health improvement opportunities,
discuss treatment options with patients, imple-
ment interventions, and achieve desired out-
comes. Similarly,information is necessary for pa-
tients to make choices consistent with their values
and preferences. Information is also key for plan-
ning, managing, and improving the health care
From Intermountain Health Care, Salt Lake City,
Utah.
Address correspondence and reprint requests to:
tional purposes.
FINDINGS.The key elements of a functional
information system include provisions that (1)
data should be collected once, (2) aggregation
of data for higher-level reports should be an-
ticipated, (3) issues related to privacy and con-
fidentiality must be addressed, and (4) mea-
surement systems should include an audit
standard. A seven-step process for developing
a functional information system is outlined.
CONCLUSIONS.A shared national measure-
ment framework is essential because the data
systems that health care delivery organizations
use are not static. A long-term vision can guide
the growth of a data system over time. An
NQMRS can be the vehicle that provides the
needed vision.
Key words: Data collection; functional infor-
mation system; national quality measurement
and reporting system; Strategic Framework
Board. (Med Care 2003;41:I-71-I-79)
delivery system. Data and information lie at the
core of any quality management system.1
In summarizing principles for data collection
and management, we looked to well-established
models from outside health care, such as bank-
ing and transportation safety. In such models, a
central agency releases specifications for report-
able data. The data are generated by businesses
as part of routine operations, then indepen-
dently reviewed to establish completeness, ac-
curacy, and reliability. Auditors examine the
structure and function of the reporting system
used within a business. They do not evaluate the
results, but only certify that the numbers gener-
ated are reasonably correct.
Brent James, MD, MStat, Intermountain Health Care, 36
South State Street, Suite 2100, Salt Lake City, UT
84111-1486. E-mail: bjames@ihc.com
1-71
JAMES
We envision a similar integrated data system for
quality measurement and reporting in health care
delivery.Under such a system, health care delivery
groups would generate information for internal
operations (direct care delivery, management, and
improvement) in a way that makes it possible to
combine those data into high-level reports for
accountability and selection. That will require
standards for data collection and reporting and an
audit system to ascertain that the data collected
are reasonably accurate.
A shared national measurement framework is
important because the manual and automated
data systems that health care delivery organiza-
tions use to manage and improve their care are not
static. A long-term vision can guide the data
system development over time. The Strategic
FrameworkBoard (SFB)viewed the national qual-
ity measurement and reporting system (NQMRS)
as providing that vision so that software vendors
and health care delivery groups can develop com-
patible internal data systems that support shared
data for external accountability. Quality measures
for specific diseases or clinical support processes
are likely to arise from a variety of different
organizations and sources. We therefore propose a
theoretic framework under which it would be
possible to combine measures generated by a
range of groups and use the measures together.
Key Elements of a Functional
Information System
For an information system to be functional, the
data contained in it must be accurate, complete,
available in a timely manner, and useful for mul-
tiple purposes. Few existing information systems
in health care have achieved this level of function-
ality and those that have generally exist within a
single organization. For an NQMRS to realize its
full potential, the nation will need all health care
organizations to have functional information sys-
tems. Some key elements of that functionality are
highlighted here.
Single-Point Data Collection
The SFB recommends single-point data collec-
tion when possible because redundancy and the
burden of data collection are minimized and data
collected at the point of origin are usually more
1-72
MEDICALCARE
accurate and complete than data generated at
secondary points. Clear, standard definitions en-
sure that consistent information is produced over
time and across groups, allowing for accurate
comparison of inputs and results.
This is perhaps easiest to envision in a com-
pletely electronic information system environ-
ment. For example, when a patient comes in for a
visit and has his or her weight and blood pressure
measured, the provider would enter those figures
directly into the clinical information system (with
a hand-held device or perhaps a scale and blood
pressure cuff that transmit results directly to the
database). The information is collected in a way
that allows the information to be used by (1) the
physician at that visit (by charting how these
values compare to the last few values measured for
this patient), (2) the patient in tracking his or her
health status over time, and (3) others to evaluate
the proportion of the physician's, medical group's,
health plan's, region's, state's, and nation's popu-
lation that is overweight or has high blood
pressure.
Combining Data for Multiple Uses
The core work of a health care delivery system
occurs at the interface between patients and
health care professionals. Those interactions gen-
erate large volumes of data (eg, a medical record),
which in turn drive other data generation and
reporting systems supporting health care delivery
operations (eg, data for billing, purchasing of
supplies, staffing, budgeting, or planning for phys-
ical facilities). Single-point, source-level data col-
lection implies that such data will first be useful to
manage work processes at the front-line interface,
but also can be "rolled up" or aggregated into
high-level reports for use throughout a manage-
ment hierarchy. For example, clinical information
generated during a patient encounter can be used
for direct patient care as well as for summary
reports at the level of individual clinicians, care
delivery teams, clinics, regions, delivery systems, or
geographic areas.
Primary data are usually obtained at the indi-
vidual patient level. Secondary roll-up data are
usually reported at a population level. Although
properly structuredprimarydata can nearly always
be combined to create secondary measures and
reports, data originally collected at a population
level often cannot be used to generate individual
Vol.41, No. 1, Supplement
patient values because most population-level data
are generated from samples (so a value does not
exist for every patient) and identifiers that would
allow users to link different pieces of information
to an individual are often removed to protect
confidentiality.
Primary data can be aggregated across groups
and conditions. Some roll-up reports combine
performance measurements across groups of pro-
viders or geographic areas within a single clinical
measure. Other roll-up reports combine measures
across conditions. Identifiable denominators are
necessary to create aggregate reports across
groups, either by identifiers for individuals or
accurate counts of those eligible to be considered
in a measure. Common metrics (eg, staging sys-
tems) are necessary to create roll-ups across con-
ditions so the groupings are consistent across
entities. Risk adjustment must be addressed in
both types of roll-up.
Aggregate reporting relies on four conceptual
underpinnings: (1) classes of outcomes, (2) a com-
mon metric, (3) an analytic method (including,
when necessary, a risk adjustment system), and (4)
patient registries (with master index systems to
create accurate denominators).
Classes of Outcomes. Nelson and colleagues2
argued that four classes of outcomes data form a
balanced "value compass"that generally applies to
health care delivery management and reporting:
(1) medical, (2) patient functional status, (3) ser-
vice, and (4) cost. Medical outcomes include com-
plications and achievement of therapeutic goals
from the clinician perspective. Patient functional
status measures the patient's perspective on treat-
ment effects. Service outcomes include dimen-
sions of the patient-clinician relationship (eg,
shared decision-making), access, and conve-
nience. Cost outcomes are the expenditures asso-
ciated with measured care processes.
Three additional classes of measures extend the
value compass to the entirety of front-line care
delivery. First, patient stratification factors are
those that care providers cannot control but that
influence outcomes (eg, demographics). Second,
referraland treatment indications are diagnosis or
treatment factors that are used to determine
whether an intervention is appropriate (eg, failure
of maximal medical therapy to control angina).
Third,key process factors are those that determine
outcomes (eg, timely administration of aspirin to
heart attack patients). Some useful breakouts
within outcomes classes can facilitate identifica-
INFORMATIONSYSTEMCONCEPTS
tion of common metrics and lead to consistent
analytic methods to combine data in roll-up
reports.
Common Metrics. Appropriate common met-
rics for these major outcomes classes must be
agreed on. Table 1 lists major outcomes classes
with examples of common metrics that allow
aggregation. For example, although the indications
that determine whether a procedure is appropriate
for a particular patient are tailored to a specific
clinical scenario, using the appropriateness metric
would allow for a report on the proportion of all
surgical procedures that are performed for clini-
cally appropriate reasons.
Medical outcomes are complex. To create aggre-
gate reports, a classification system may be neces-
sary for grouping clinically diverse outcomes. For
example, the set of complications, therapeutic
goals, and patient functional status measures one
would track for total hip arthroplasty are radically
different from those one would track for diabetes
mellitus. Having identified a particular clinical
condition as an appropriate subject for measure-
ment, a quality measurement system should "fin-
gerprint"the condition in terms of common failure
modes (complications or defects), standard thera-
peutic goals, patient functional status, evidence-
based or expert consensus referral and procedure
indications, and patient stratification factors.
When fingerprinting a particularclinical condition,
in addition to preparing a list of common defects,
a measures development team should also prepare
functional definitions to stage each defect.
One approach to summarizing defects across
conditions and organizational units is the use of a
staging system, such as the one originally sug-
gested by the US Centers for Disease Control and
Prevention (Table2). This allows diverse outcomes
to be classified on a common metric so that
summary performance scores can be reported.
Analytic Methods. Data system design
should include agreement on the analytic methods
that will be used to combine patient data across
care delivery groupings and clinical conditions. If
the data are being used to make comparisons
among entities, the methods will often include
severity-of-illness adjustment.
Patient Registries. Many useful performance
measures take the form of rates (eg, the proportion
of diabetic patients who received routine glycosy-
lated hemoglobin tests or the proportion of
women screened for breast cancer). Denominator
populations are a critical part of any such rate (eg,
1-73
JAMES MEDICAL
CARE

TABLE
1. Definitions of Common Metrics for Different Classes of Outcomes
Outcome Class Subclass Metric Descriptive Example of Metric
of
Appropriateness Indications Defectrate Patientstreatedeven thoughthey did not
care meet evidence-basedor professional
consensus indications (overuse), and
patients who did meet evidence-based or
professional consensus indications but did
not receivetreatment(underuse)
Key process factors Defectrate Patientsfor whom an evidence-basedkey
process step was not performed or where
a key process step failed.
Medical outcomes Complications Defect rate Specific complications
Therapeuticgoals Defect rate Failure to achieve treatment goals.
Patient functional Defect rate Failure to achieve functional performance
status standards; mean shift in performance
scores before and after treatment
Service outcomes Return and Whether patients preferentially return to the
recommend same care provider for future care and
recommend that provider to their family,
friends and associates
Cost outcomes Dollars Cost per unit of output (surgical procedure,
patient care year)

all diabetic patients in a practice or all women who
meet indications for mammography within a de-
fined time period). Patient registries are a means of
maintaining the data across a care delivery group
or a geographic area so that the denominators
needed to generate rates of performance can be
generated.
Disease registries have a long, positive history
within health care delivery measurement and im-
provement. For example, the American College of
Surgeons' Commission on Cancer has supported
standardized cancer registries in American hospi-
tals since 1922.3 In 1973, the National Cancer
Institute coordinated full population data for on-
cology across five states and four metropolitan
areas, and has since expanded its Surveillance,
Epidemiology, and End Results program to other
pertinent subpopulations.4 Since 1989, the Veter-
ans Administration Health System has maintained
a state-of-the-art patient registry for patients with

2. Modified Staging System For Complications: Centers for Disease Control and Prevention
TABLE

Stage Description

Stage 1 Eventoccurred,but patientwas neverat significantriskfor long-termharm.No intervention

undertaken.
Stage 2 Eventoccurred,but patientwas neverat significantriskfor long-termharm.Interventionundertakento
speed recovery or make the patient more comfortable (note that Stage 2 complications consume
extrahealthcareresourcesand so effectcost outcomes).
Stage 3 Eventoccurred,patientwas at significantriskfor long-termharm;but an interventionpreventedthat
harmfromoccurring(again,with increasedconsumptionof healthcareresourcesand higher
costs).
Stage4 The patientsuffereda long-terminjury
a. Minorlong-terminjury
b. Majorlong-terminjury
c. Death

I-74
Vol.41, No. 1, Supplement
HIV.5 Registries covering heart disease, breast
cancer, and many other acute and chronic diseases
have provided useful health services information
across the entire Swedish population for de-
cades.6-8 Many other disease-specific registries
exist. Most rely on voluntary participation of inter-
ested individual health professionals or care deliv-
ery groups.
Measurement systems should specifically iden-
tify those measures that require accurate denom-
inator patient populations, define the populations
of interest, and recommend methods by which
groups treating those populations can reasonably
identify, track, and report such patients.
Privacy and Confidentiality of Patient
Records
The Standards for Privacy of Individually Iden-
tifiable Health Information (the Privacy Rule-
part of the Health Insurance Portability and Ac-
countability Act of 1996) took effect on April 14,
2001, with compliance by most covered health care
entities required by April 14, 2003.9 The Privacy
Rule creates national standards to protect individ-
uals' personal health information and gives pa-
tients increased access to their medical records.
Two issues that might affect the functioning of an
NQMRS arise under the new regulation:
(1) Use of identifiable patient data. Although an
NQMRS would not generate reports on indi-
vidual patients, it would use individual patient
information to evaluate and report the perfor-
mance of the health care delivery system. The
Privacy Rule promotes the use of nonidentifi-
able patient health data for health services
research and lists 19 data elements which
would generally make a patient record identi-
fiable unless a competent statistical authority
judges otherwise. Those 19 fields include some
elements that could be essential to an effective
NQMRS, such as date and type of health care
service. The most recent revisions to the Rule
(August 14, 2002) removed other elements
that would be critical to the operation of an
NQMRS, such as patient zip code, city, and
age, from the proscribed list of patient identi-
fiers (see 164.514(d)2i and Modifications to
the Standards for Privacy of Individually Iden-
tifiable Health Information-Final Rule;
www.hhs.gov/ocr/hipaa/).
INFORMATIONSYSTEMCONCEPTS
It may be possible to operate an NQMRS with
use of the resultant nonidentifiable Limited Data
Sets, in conjunction with research contracts that
extend private protections to an NQMRS as re-
quired by the Privacy Rule. Even if an NQMRS
could not function with nonidentifiable data
alone-for example, if an NQMRS needed patient
identifiers to link patient records across health care
delivery sites-such use of patient data would be
covered under the Privacy Rule's health oversight
and public health sections (164.512(b) and
164.512(d)). Such an approach would require that
the NQMRS be authorized under federal
regulation.
(2) Patient access to individual NQMRS records.
The PrivacyRule gives patients broad access to
their own medical records. Maintaining logs to
enable patients to review their records and
review instances in which their records were
referenced could hopelessly burden operation
of an NQMRS. However, although an
NQMRS would necessarily rely on individual
patient information for its internal function, an
NQMRS would not use identifiable patient
data for individual patient decision-making.
Under the Privacy Rule, such uses of patient
data are explicitly excluded from patient re-
view (see section 164.524(a)).
Development of Audit Standards
The quality measurement and reporting system
that we propose rests on local data collection. We
anticipate that clinics, hospitals, and care delivery
groups will generate data as part of their routine
operations, then summarize and report those data
to regional, state, and national groups. But any
measured result represents the blending of two
subcomponents: (1) actual performance and (2)
the measurement system. It is often far easier to
manipulate the measurement system than to ac-
tually deliver excellent performance (in other
words, it is easier to look good than to be good).
Data system audits attempt to limit the role of the
measurement system as a major source of varia-
tion in reported results.
Measurement system audit standards should
focus on completeness and accuracy. Complete-
ness functions at a population level and an indi-
vidual case level. At a population level, it addresses
case finding, or patient registries: are all patients
1-75
JAMES
who comprise a denominatorpresent and ac-
countedfor?At an individuallevel, completeness
means that all individualdata fields are present
and accountedfor.Accuracyassesses the content
of each field, directly applying the definitions
included in the coding manual to insure that
recordcontent accuratelyreflectstrue patientre-
sults. Audit standardsthereforeinclude both a
structuralanalysisto assess whethera datasystem
has all necessaryfunctionalelements (complete-
ness) and a content analysis to assess whether
standardcoding definitionsare consistentlyfol-
lowed (accuracy).Audit standardsthereforein-
clude and extend the approachesused to define
core data sets.
The SFBrecommendsthat each measurement
set developed for the NQMRS include explicit
standardsfor concurrentdataaudits.Recentexpe-
rience with audit fraud in commercial,publicly
traded businesses illustrateseveral factors that
appear to be criticalwhen implementingaudit
systems.First,auditorsmust be completelyinde-
pendentfromthe caredeliverygroupstheyreview.
In particular,parallelconsultingcontractsappear
to offerstrongincentivesfor inappropriatecollu-
sion. Second,auditstandardsrequirea mechanism
for regularreview and update based on careful
observationof the audit process itself and the
ongoingevolutionof the underlyingdatasystems.
Care delivery groups that pass independent
audit will preparestandardreportsand forward
standard data to regional, state, and national
agencies. Those groups will analyze and report
performanceto the Americanpublic,health over-
sight agencies,and other interestedparties.
FunctionalSteps in Designinga
NationalData System
The foregoingdefinitionsand backgroundcon-
ceptsprovidethe foundationfora generalmethod
to add condition-specificsections to a NQMRS
that grows in breadthand depth over time. We
discussseven steps to generateindividualmodules
withinsuch a system,with sufficientdesigndetail,
check points, and transparencyto allow profes-
sionals,consumers,and healthoversightgroupsto
openlycontributeand criticizeat each point along
the way. Our recommendationsare based on
models widely used among collaborativeclinical
trials groups9'10and related to methods success-
fullyimplementedby severalclinicalimprovement
1-76
MEDICALCARE
collaboratives.The key concept is this: startwith
the desired end result, then work backwardto
front-linedata collectionand data flow.This ap-
proachis more highly structuredthan those ap-
plied to quality measurementdesign in some
other systems.l It aims to parsimoniouslycollect
only those data that lead directlyto usefulreports
while comprehensivelycoveringan entireprocess
of care.
Select High-PriorityClinical Processes
The firststep is to focuson high priorityclinical
processes.12The conceptof key clinicalprocesses
links the internalbusiness of care deliveryto the
legitimateaccountability, selection,andmotivation
needs of an NQMRS.In one health system,lead-
ers identifiedfour classes of such processes:(1)
clinical conditions (outpatient/primary care and
inpatient/specialty care), (2) clinicalsupportser-
vices, (3) service quality,and (4) administrative
supportprocesses.
Withininpatientclinicalprocesses,a group of
physiciansand nursesfurthergroupedand prior-
itized conditionsby (1) total patient volume, (2)
intensityof care(ie,cost per case),(3) case-to-case
variability,and (4) microsystems(teams of clini-
cianswho typicallyworktogether).13T14 The group
found that, among more than 600 inpatientcon-
ditions,62 accountedfor 92%of all caredelivered
on the system'shospitalcampuses.Those62 con-
ditionswere furtherclassifiedinto eightfamiliesof
clinicalprocesses (cardiovascular, neuromusculo-
skeletal,surgical,women and newborn,intensive
medicine, intensive pediatrics,intensive behav-
ioral,and oncology).
Outpatientclinicalprocesseswere categorized
by population group: (1) truly well, (2) latent
disease, (3) acute self-limited conditions, (4)
chronicconditionsincludingacute exacerbations,
and (5) terminalpatients. Fourteenpopulation-
based processesaccountedfor more than 90%of
all health maintenanceand risk managementac-
tivities for the first two groups of patients (eg,
immunization,smokingcessation,and use of seat
beltsin automobiles).Fifteenacuteand 15 chronic
outpatient conditions accounted for more than
80% of all care deliveredin communitysettings.
Manyof the priorityoutpatientclinicalconditions
linked directly to priority inpatient clinical
conditions.
Similaranalyses were undertakenfor clinical
supportservices(eg, laboratory,pharmacy,physi-
Vol.41, No. 1, Supplement
cal therapy),service quality,and administrative diagramsareflowchartsthattakeadvantageof the
supportservices.Forexample,the greatmajorityof
patients routinely list a "caringand concerned begins by generating a simple flowchart of a
as a key factorin anyhealthcaredelivery clinical process (see Fig. 1, McGlynn15).Next,
clinician"
experience.
The effort requiredto design and implement most potentialfor improvementor most strongly
qualitymeasurementand reportingsystems pre-
cludes addressing all care processes simulta- expanded as another layer of flowcharts.One
neously. It is therefore imperative that the
NQMRSinvest firstin those areasthat will have
the greatesteffect.
Generate an Explicit Conceptual Model for
a Selected Process
The second step is to generate and circulate
among key decision makersa conceptualmodel
for the high-priorityclinicalprocessesselectedin
the firststep. Physicians,nurses,technicians,and
other front-line care deliverersuse conceptual
models to organize and understandtheir work.
Suchmodelsprovidea contextforindividualtasks
and link them togetherinto a coordinatedwork-
flow. Those models very often are subconscious,
but they are always present-it is impossibleto
performcoordinatedworkwithoutthem.
Similarly,analysisandreportingrelyon concep-
tual models.A useful analysisrequiresa concep-
tual context to link measuresto work processes
and outcomes. Just as with front-line improve-
ment work, the conceptualmodels that underlie
analysisand reportingoften go unrecognized.
The measuresselectionprocessrequiresexplicit
conceptualmodels rooted in currentbest under-
standingof a clinicalconditionand associatedcare
deliveryprocesses.15Suchmodels are essentialto:
* Obtain consensus for shared improvement
work acrosscaredeliveryteams
* Prioritizeand focus within complex clinical
processes
* Provide a context for interpretablemeasure-
ment with shared understandingacross the
many groupsthat might use such measures
Currentpracticeuses two main forms of con-
ceptual models. The first is hierarchicoutcomes
chains.An outcomeschain providesan overview
of the entire disease and treatmentprocess, as
reflectedin hierarchicintermediateand finalpro-
cess steps and patientoutcomes.The secondform
is a conceptualflow diagram.Conceptualflow
SYSTEMCONCEPTS
INFORMATION
natural hierarchyfound in real processes. One
those steps in the high-orderflow that hold the
controloutcomes are identifiedand each step is
continuesuntil a decisionlayeris reached.At that
level,the conceptualflow transformsinto a tradi-
tional decisionflowchartas is commonlyused in
practiceguidelinesand protocols.
Generate a List of Reports and Test Their
Utility
Conceptual flow diagrams,when combined
with the idea of classesof data,providea practical
tool for generating balanced sets of outcomes
measuresarounda particularclinicalprocess.The
approachis simple.A conceptualflow diagramis
generatedfor a clinicalprocessdown to the deci-
sion level.Eachbox in the layerof the conceptual
flowchartimmediatelyabove the decision flow-
chart is examined while asking the question,
"what reports should we routinelygenerate to
track performanceand outcomes for this pro-
cess?"Recordthe resultinglist of reports.Out-
comes chainscan be used in a similarway.
When the potential reports are identified,
model reportsshould be built with use of either
realor simulateddata.These shouldbe circulated
to those who are expectedto use them for feed-
back.Is the informationuseful?Is it presentedat
the level of analysis needed for the decision
maker?How often should the reportsbe gener-
ated? This step is frequentlyoverlookedin the
design of reportingsystems, and identifyingthe
data elements necessaryto generatereportswill
save the time and expenseof retoolingto respond
to such demandsin the future.
Identifythe Data Elements Necessary to
Generate the Desired Reports
The fourth step is to use the report list to
determinethe data elementsthat will be required
to routinely produce informationfor decision
makers.A coding manual and self-coding data
sheets are then designed to obtain the data
elements.
1-77
JAMES
Coding Manuals. Coding manuals list every
data element in a proposed measurement set, with
(1) functional definitions, (2) complete descrip-
tions of coding schemes, (3) instructions regarding
missing and unknown data, and (4) descriptions of
primary sources for each data element.
Self-coding Data Sheets. Self-coding data
sheets reduce a coding manual to a manual data
entry sheet on which all data elements are shown
as labeled data boxes. All definitions, coding
schemes, data sources, and other related instruc-
tions necessary for accurate, complete data entry,
are included as part of the sheet. Self-coding data
sheets are shorthand coding manuals for practical
data entry use and are potent tools for designing
succinct data systems that function well.
This scheme can be pilot-tested to determine
how the necessary data collection can be inte-
grated into the flow of patient care without ham-
pering care delivery operations, and whether the
resulting reports can be produced in a timely
manner.
Plan the Flow of Data at the Level of Care
Delivery
The fifth step is to identify which data are
already automated and which must be obtained in
the course of care delivery. Converting hand-
coded data sheets into automated data acquisition
systems can substantially enhance the efficiency of
the information system. The data acquisition strat-
egy then must be designed, with those on the
front line providing input into how actual data
capture, recording, and reporting will be accom-
plished. This may require some negotiation (what
we want versus what we can get). This step can
also generate a plan for data system improvements
that can be introduced over time as the data
system goes through its regular upgrade cycle.
Data flow planning also addresses methods to
combine, preprocess, store, and report information
across different care delivery units.
Test the Final Reporting System
The sixth step is to test the resulting final
reporting system before full-scale implementation
is attempted. This iteration is necessary because it
is likely that compromises were made in the
previous step that affect the ability of the system to
1-78
MEDICALCARE
generate all the reports originally identified. Some
data elements may have accidentally fallen off or
new data elements may be required as a result of
redesign on the front lines.
Implement the Measurement and Reporting
System
The last step is to implement the system. Infor-
mation system design rests on careful attention to
these steps, or their equivalents. Experience in real
data systems has shown that shortcuts during the
planning and testing phase are likely to lead to
increased cost and decreased functionality of the
final product.
Conclusions
An efficient NQMRS should simultaneously
address measurement for accountability/selection
and measurement for improvement.1 Many others
have addressed the use of data systems for quality
measurement,16-20but the key concept we pro-
pose is that of careful data system design for the
dual purposes of local operations and national
reporting. Properly designed, a data system built
to support front-line clinical process management
and improvement can also produce data for sum-
mary reports that support accountability,selection,
and motivation. Such systems can minimize the
burden of data collection and reporting that is
often associated with quality measurement activ-
ities and ensure that the data are accurate and
complete, available when needed, and useful to
decision makers.
We identified key elements and concepts relat-
ing to the structure and function of effective data
systems. On that foundation, we propose a series
of concrete steps that all proposed clinical quality
measures included in the NQMRS should follow.
References
1. Berwick DM, James B, Coye MJ. The connec-
tions between quality measurement and improvement.
Med Care 2002;41(suppl):I-30-I-38.
2. Nelson EC, Mohr JJ, Batalden PB, et al. Im-
proving health care, part 1: the clinical value compass. Jt
Comm J Qual Improv; 1996;22:243-258.
3. American College of Surgeons. Availableat:
http://www.facs.org/dept/cancer/coc/. Accessed August
12, 2002.
Vol. 41, No. 1, Supplement
4. National Cancer Institute. Available at: http://
www.seer.cancer.gov/about/. Accessed August 12, 2002.
5. Rabeneck L, Menke T., Simberkoff MS, et al.
Using the national registry of HIV-infected veterans in
research: lessons for the development of disease regis-
tries. J Clin Epidemiol 2001;54:1195-1203.
6. Hemminki K, Granstrom C, Czene K. Attrib-
utable risks for familial breast cancer by proband status
and morphology: a nationwide epidemiologic study from
Sweden. Int J Cancer 2002;100:214-219.
7. Hammar N, Alfredsson L, Rosen M, et al. A
national record linkage to study acute myocardial infarc-
tion incidence and case fatality in Sweden. Int J Epide-
miol 2001;30(suppl 1):S30-S34.
8. Stenestrand U, Wallentin L. Early revasculari-
sation and 1-year survival in 14-day survivors of acute
myocardial infarction: a prospective cohort study. Lancet
2002;359:1805-1811.
9. Department of Health and Human Services.
Available at: http://www.hhs.gov/ocr/hipaa. Accessed
August 12, 2002.
10. Pocock SJ. Clinical Trials:A PracticalApproach.
New York,NY: John Wiley & Sons; 1983.
11. Halpern J. The measurement of quality of care
in the Veterans Health Administration. Med Care
1996;34:MS55-MS68.
12. Institute of Medicine Committee on Quality
of Health Care in America. The value of organizing
INFORMATION SYSTEMCONCEPTS
around priority conditions. In: Corrigan JM, Donaldson
MS, Kohn LT,eds. Crossing the Quality Chasm: A New
Health System for the 21st Century. Washington, DC:
National Academy Press; 2001, p. 98-110.
13. Batalden PB, Mohr JJ, Nelson EC, et al.
Continually improving the health and value of health
care for a population of patients: the panel management
process. Q Manage Health Care 1997;5:41-51.
14. Nelson EC, Batalden PB, Mohr JJ, et al.
Building a quality future. Front Health Serv Manage
1998;15:3-32.
15. McGlynn E. Selecting common measures of
quality and system performance. Med Care 2002;
41(suppl):I-39-I-48.
16. Nelson EC, Splaine ME, Batalden PB, et al.
Building measurement and data collection into medical
practice. Ann Int Med 1998;128:460-466.
17. Nelson EC, Batalden PB. Patient-based quality
measurement systems. Q Manage Health Care
1993;2:18-30.
18. Iezzoni LI. Assessing quality using administra-
tive data. Ann Int Med 1997;127(suppl):666-674.
19. McDonald CJ, Overhage JM, Dexter P, et al. A
frameworkfor capturingclinical data sets from computer-
ized sources. Ann Int Med 1997;127(suppl):675-682.
20. Palmer RH. Process-based measures of quality:
the need for detailed clinical data in large health care
databases. Ann Int Med 1997;127(suppl):733-738.
1-79