You are on page 1of 4

Education and debate

Measuring goodness in individuals and healthcare


systems
Mike Pringle, Tim Wilson, Richard Grol

All agree that we need to measure the quality of health care, including the care given by individual
doctors. Measuring goodness requires accurate data used appropriately, and it must be done
without demoralising and demotivating staff. Do current measures fulfil these requirements, and if
not, what measures should be used?

Division of General In the recent Reith lectures (broadcast annually by


Practice, University
of Nottingham,
BBC radio on issues of contemporary interest), Onora Summary points
Nottingham ONeill explored the new age of accountability. She
NG7 2RD concluded that increasing reliance on measurement
Mike Pringle Individuals and organisations constantly strive to
reduced trust in health (and other public) services and
professor define and measure quality of health care
that professionals and public servants should befree
Quality Unit, Royal
College of General
to serve the public.1 This will ring true with many. Good data on quality of care are needed to
Practitioners, However, patients, funders, commissioners, provider achieve understanding and effective change
London SW7 1PU organisations, and health professionals legitimately want
Tim Wilson to know just how good are individual doctors, teams,
director Data are often used out of context and without
and healthcare providing organisations. The tradition- taking account of natural variation
Directors Centre
for Quality of Care
ally qualitative, anecdotal approach, supplemented by
Research (WOK), trust, is being increasingly replaced by data on effective- A good measure of quality of care is appropriate
Nijmegen ness, safety, acceptability, and efficiency.
University,
to that task and is used appropriately
Netherlands PO Measurement is crucial for a range of purposes
Box 9101, 6500 HB learning, quality improvement, accountability, and Twelve attributes can be ascribed to quality
Nijmegen, regulationbut must be used appropriately. We measureshelpful when choosing indicators
Netherlands
Richard Grol
contend that measurement can be used to reinforce
professor the natural desire of healthcare staff to improve care at
Correspondence to: the same time as understanding the quality of the serv-
M Pringle ice delivered. These findings support the action of societies, gov-
mike.pringle@ However, creating meaningful information from ernments, and healthcare organisations in their moni-
nottingham.ac.uk
accurate data to facilitate rational choices is a real chal- toring of health care for expenditure, value for money,
BMJ 2002;325:7047 lenge. It must be done without distorting staff and safety. Doctors, nurses, and other health profes-
behaviour or demoralising and demotivating health sionals need to compare themselves with their peers
professionals (including managers), and it must offer and against external and self generated standards in
true comparisons. order to improve care. And crucially, patients need
information to make rational choicesis their doctor
competent, is their hospital safe, is their treatment
optimal?
Background In particular, both the public and managers want to
Recently the Institute of Medicine in the United States know which the good doctors, teams, and institutions
concluded that between health care we have and the are. They want to protect themselves from the bad and
care we could have is not just a gap but a chasm.2 An incompetent. Of course, there are many definitions and
evaluation in a large sample of general practitioners of therefore assessments of good, and any one doctor may
the use of 29 national (evidence based) guidelines and be very good in some aspects of care butpoorin others.
282 indicators for primary care in the Netherlands
showed that on average 67% of the recommended care
was provided to patients.3 Patients evaluations of
How is goodness currently measured?
primary care collected in 16 countries in Europe with The Institute of Medicine has recommended improve-
an internationally standardised questionnaire showed ments in the way that the healthcare system is
More examples of
quality measures
that 30-40% of the patients thought that the measured2 as standards, performance, and changes
can be found on organisation of services could be better (50% in the cannot be monitored effectively without secure
bmj.com United Kingdom).4 measures. However, inappropriate measures (and

704 BMJ VOLUME 325 28 SEPTEMBER 2002 bmj.com


Education and debate

there is no such thing as the perfect or right measure) We aspire to a world in which patients are
can result in perverse incentives or justification for data protected by measures of competency; failing teams
manipulation. If the person or organisation whose per- and organisations can be identified and remedial
formance is being measured feels powerless to action taken; patients and their advisers, especially
influence the indicator, inappropriate measurement family doctors, can make informed choices of services
can also lead to demotivation, dysfunction, and crisis. to use; and service commissioners can deploy
Currently, the selection of quality measures is often resources most efficiently to achieve best care. Finally,
driven by what can be measured3 rather than by a defi- the treasury can monitor improvements in care and be
nition of goodness followed by the derivation of an held to account by the electorate. This world is
appropriate measure. currently far from reality.
Many quality measures, such as revalidation,
consultation satisfaction rating, or the fellowship by
assessment quality assurance programme of the Royal
How should health care be measured?
College of General Practitioners, relate to individual Data on their own are of limited value. Information
health professionals. Others more clearly relate to that compares data with others or against standards is
teams or whole organisations. These include star more useful, but without contextand thus
ratings, assessments by the Commission for Health understandingit cannot be applied effectively (fig-
Improvement, or satisfaction with services. ure). Ideally comparisons would be like with like, but
Examples of measures from the United States this rarely happens. Local health needs, service
include a comprehensive set of quality indicators configurations, and case mix all influence data, and this
(developed by researchers at Rand, California), tested is what is meant by context. The easiest way to get
for validity and feasibility, that cover many aspects of good surgical outcomes is to admit only low risk cases;
health care,5 and a set of complex assessments to high immunisation rates are much more difficult to
accredit careused by the Joint Commission for achieve in deprived areas.
Accreditation of Health Organisations and the In the table we suggest 12 attributes to use when
National Committee for Quality Assurance. Other appraising quality measureswith examples of current
international examples can be found in box 1 on the quality measures in the United Kingdom that we
BMJ s website (bmj.com). believe have or do not have these attributes. For
Official bodies and outside commercial organisa- example, surgical waiting times have good face validity,
tions have begun placing data derived from health are usually available, and offer a comparison between
service self reporting sources in the public domain in teams and organisations. They fall down on effective-
ways that make it accessible to patients, managers, ness: they measure only part of the patients
researchers, and clinicians. For example, the 60 US experience, such as time on the waiting list for surgery;
hospitals with lowest mortality within 30 days after a their reliability is compromised by manipulation; they
myocardial infarction scored lower on patient evalua- are usually not adjusted for the context; and it is never
tions of care.6 The hope is that public access to such clear whether a long waiting list is due to inefficient
data will lead to better data, choices, and care, although surgical teams, poor management, or lack of capacity
its impact is still unproved and use by the target groups so, it is unclear if and how the problem can be resolved.
(patients, other providers, and commissioners) is still If poor quality measures as defined against the
limited.7 8 attributes in the table are applied, then perverse incen-
Even though choices are often severely constrained, tives and demoralisation of managers and health pro-
people can look at local comparisons and in many cases fessionals become a real possibility. One major English
add their own vital contextual understanding in making trustthe Royal United Hospitals, Bathhad manipu-
their choice. Prospective patients must weigh compara- lated the number it reported for patients waiting over
tive information, such as waiting times and outcomes, 15 months for surgery. This is precisely the outcome to
keeping in mind their own previous experience and the be avoided. Another powerful lesson from recent
experience of family, friends, and advisers. experience with performance indicators is that

Fictitious example Description Appropriate use Possible undesirable effects


In my hospital 41% of Reflex solutions applied
Straight data collection Local audits, clinical
Data inguinal hernias are inappropriately; data
and management
repaired as day cases quality deteriorates
This compares with a national average Normal variation ignored;
Comparisons against others;
of 43%; in the locality five other trusts League tables; performance rankings not interpreted
comparison against absolute
Information have higher percentage - management; star ratings in context; added value
or relative standards
one as high as 61% not assessed

We have fewer day case beds, Commission for Health


Information seen in context; No distinction made between
and one of our surgeons doesn't do Improvement reports; internal
problem defined; options systems and individuals;
many cases. However, our major quality review; external consultants
Understanding for change apparent resources not considered
issue relates to cancer surgery report; resource allocation

Surgical team reconfigured to allow one


Action is applied to problem
surgeon to do more serious cases
not to raw data; feedback Continuous quality Imposed change; staff
including cancer; expansion of day
is given; local targets agreed; improvement collaboratives; become defensive and
case surgery facilities when funds allow;
support for change; effective management uncooperative
Change recruitment of additional surgical
resources applied
specialist registrar

Using data for quality improvement and the levels at which they are applied

BMJ VOLUME 325 28 SEPTEMBER 2002 bmj.com 705


Education and debate

Twelve attributes and ideal descriptions of quality measures, with examples from United Kingdom
Attribute Ideal description Measures with attribute Measures without attribute
Valid Health professions, managers, and public see meeting the Waiting times; death rates from surgery; readmission Singlehanded general practice
quality measure as better quality (better patient outcomes; rates; complaints and litigation; significant event
more efficient and patient friendly services, etc) auditing
Communicable Relevance of measure can be easily explained and Prevention uptake rates (for example, cervical Star rating of NHS trusts
understood by target groups cytology or immunisation)
Effective It measures what it purports to measureso useful for Commission for Health Improvement reports Waiting times; day surgery rates;
clinicians, public, and managers in making choices and revalidation
commissioning services; free of perverse incentives
Reliable Data should be complete, accurate, consistent, and Singlehanded general practice Fellowship by assessment; availability
reproducible of general practitioner for consultation
Objective Data should be as independent of subjective judgment as Prescribing data NHS doctor appraisal; Commission for
possible Health Improvement reports
Available Data should be collected for routine clinical or Prescribing data; star rating of NHS trusts Long term effects of care; functional
organisational reasons or be available quickly with status; link between care and outcome
minimum of extra effort and cost
Contextual Measure should be context free or important context Consultant numbers per 1000 patients with disease Prevention uptake rates
effects should be adjusted for
Attributable How well measure reflects quality of individuals, teams, NHS doctor appraisal; revalidation; fellowship or Waiting times; overall patient
or organisations must be explicit; measure to be used membership by assessment; quality team satisfaction
appropriately in its presentation and interpretation development; quality practice award
Interpretation How well measure reflects health needs, capacity, Bed occupancy General practitioners referral rates;
structures, or performance should be explicit prescribing data (PACT)
Comparable Where gold standard (for example, NICE guideline, NSF Thrombolysis in myocardial infarction; aspirin in General practitioners referral rates
standard or GMC guidance) exists, measure should allow ischaemic heart disease or stroke; glycaemic and
reliable comparison with standard; otherwise comparison blood pressure control in diabetes
should be to other data in similar circumstances
Remediable Need for recognised, accepted, and feasible methods for Record keeping Effects due to deprivation and lifestyle
influencing measure and improving quality, need for (acute myocardial infarction, smoking
resources for intervening; change can be achieved if it is rates, obesity); attendance rates at
needed accident and emergency; suicide rates
Repeatable Measure should be sensitive to improvement over time Staffing levels; bed numbers and occupancy Complaints and litigation; significant
event auditing
NICE=National Institute for Clinical Excellence; NSF=national service framework; GMC=General Medical Council.

variation occurs in all systems and that detecting unac- of care or outcome adjusted for the contextteams will
ceptable or dangerous variation is the key task.9 Not all understand the context and use them wisely. As the
differences are meaningful. second of Langlands rules states that measurement
Lastly, the involvement of service users and the for improvement is not measurement for judgment.10
public in developing quality measures is an essential Teams and individuals are increasingly committed
aspect of quality improvement. Although some to improving the quality of their care as part of their
measures will continue to be technical, there is a short- professional imperative and culture.11 The adoption of
age of good measures of dimensions of care that are clinical audit and reflective practice has been slow,12 but
directly relevant to service users and which meet the the evidence of its effectiveness is mounting, particu-
attributes in the table. larly when the audit is integrated in a more
comprehensive approach to improving patient care.13
Right tool for the job One cornerstone of healthcare improvement is
continual measurement as a tool for understanding
Measurement can be used for learning and develop- systems and determining whether changes are
ing, quality improvement, making informed choices, effective.
accountability, contracting, and regulation. In choosing The way in which data are used (as in figure) is
a measure, the purpose must be explicit, alternative important. At one extreme, simply sending people data
measures appraised, and the limitations of the chosen on their performance does not create quality improve-
measure openly acknowledged. ment. We know that facilitated feedback (or
In regulation, for example, definitive judgments are academic detailing)that is, using someone trained
neededa doctor cannot beslightlyfit to practise in interpreting the data to give the feedbackis the
and this requires high face validity, reliability of data (or most effective way of giving people feedback, but it has
evidence), and attribution to an individual. Context to be supported with leadership and resources.
and interpretation may be used in mitigation, but the
key decision is a bipolar judgment: is this behaviour
acceptable or unacceptable?
However, many measures of quality are for feeding
Conclusions
into the quality improvement cycle (see box 2 on bmj. It is right that the public, health professionals, and
com). In this setting, it is much more important that the health service managers want to measure absolute and
data are understood and are interpreted within the relative performanceor goodnessas a means to
context of the performance. If a town has a high rate of improve care and support informed choices. The
ischaemic heart disease owing to local deprivation, eth- measures used will meet the ideal attributes to a
nic mix, or population behaviour, its appearance at the greater or lesser degree and require value judgments
bottom of league tables is more likely to create poor in arriving at a conclusion.
morale and apathy than improvement. If, however, the Data should be accurate, measures appropriate,
same data are used to measure health gainthe level context adjusted for, and interpretation responsible

706 BMJ VOLUME 325 28 SEPTEMBER 2002 bmj.com


Education and debate

and cautious. That way, stakeholders in health care, 2 Committee on Quality of Health Care in America, Institute of Medicine.
Crossing the quality chasm. Washington: National Academy Press, 2001.
especially service users, will be able to make informed 3 Grol R. Improving the quality of medical care. JAMA 2002;286:2578-85.
choices; good care will be identified and rewarded; and 4 Grol R. Successes and failures in guideline implementation. Med Care
2001;39: S2(II46-54).
safety will be improved. If healthcare regulators are 5 Marshall M, Shekelle P, Brook R, Leatherman S. Dying to know: public
serious about promoting quality then they must ensure release of information about quality of health care. California: Rand, 2000.
6 Kassirer J. Hospitals, heal yourselves. New Engl J Med 1999;340:309-10.
that measures of quality are not misapplied and 7 Davies HT, Marshall M Public disclosure of performance data: does the
abused,14 that natural variations in systems are public get what the public wants? Lancet 1999:353;1639-40.
8 Martin N, Marshall M, Shekelle PG, Leatherman S, Brook RH. Public dis-
recognised, and that measures are not perceived as closure of performance data: learning from the US experience. Q Health
capricious tools for shifting responsibility and blame. Care 2000;9:53-7.
9 Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and
clinical governance: Shewharts forgotten lessons. Lancet 2001;357:463-7.
Competing interests: MP is the strategic director of Primary 10 Berwick D. Looking forward: the NHS: feeling well and thriving at 75.
Care Information services (PRIMIS) and is a paid adviser to Dr BMJ 1998;317: 57-61.
11 Berwick DM. 1989. Continuous improvement as an ideal in health care.
Foster (http://home.drfoster.co.uk), a guide to local NHS and N Engl J Med 320:53-6.
private healthcare services. TW has been paid for talks and 12 Walshe K. Opportunities for improving the practice of clinical audit. Q
workshops on measurement. Health Care 1995;4:231-2.
13 Richards KF. Developments in total quality management in the United
States: the intermountain health care perspective. Q Health Care
1994;3(suppl):20-4.
1 ONeill A. Called to account. 2002 Reith lectures. www.bbc.co.uk/radio4/ 14 Scherkenbach WW. The Deming route to quality and productivity: road maps
reith2002/ (accessed June 2002). and roadblocks. Washington, DC: George Washington University, 1986.

How important are role models in making good doctors?


Elisabeth Paice, Shelley Heard, Fiona Moss

The use of teaching staff as role models for professional behaviour has long been an informal part
of medical training. The authors consider whether role models can still be an effective means of
imparting professional values, attitudes, and behaviours in a health service that is increasingly
sensitive to societys expectations

Role modelspeople we can identify with, who have London


Postgraduate
qualities we would like to have, and are in positions we Summary points Medical and Dental
would like to reachhave been shown as a way to Education,
inculcate professional values, attitudes, and behaviours University of
Students and young doctors identify enthusiasm, London, London
in students and young doctors.1 2 Because good role WC1N 1DZ
compassion, openness, integrity, and good
models are seen as important in the making of a good Elisabeth Paice
relationships with patients as attributes they seek
doctor, we need to know more about them. What are dean director
in their role models Shelley Heard
the attributes young people look for in role models?
postgraduate dean
Are these the attributes they really emulate? How do They are also drawn to senior figures who Fiona Moss
they react when they find that seniors lack these embody responsibility and status associate dean
attributes? We consider these questions and whether Correspondence to:
we should rely on role models as a mechanism for Some senior doctors show poor attitudes and E Paice epaice@
developing doctors who are more patient centred and londondeanery.
unethical behaviour, causing confusion, distress, ac.uk
ethically sensitive. and anger in young doctors and students under
their supervision BMJ 2002;325:70710

What qualities do students and young Role models may not be a dependable way to
doctors look for in role models? impart professional values, attitudes, and
behaviours
The attributes of medical role models have been the
subject of several interesting studies. Wright and Professional behaviour and ethics should be
colleagues looked at physicians who had been explicitly taught through peer group discussion,
identified as excellent role models by students and exposure to the views of people outside medicine,
residents.35 They found that the most important quali- and access to trained mentors
ties in role models were a positive attitude to junior
colleagues, compassion for patients, and integrity.
Clinical competence, enthusiasm for their subject, and
teaching ability were also important, but research socialised more with house staff, sharing professional
achievement and academic status were much less so. experiences and talking about their personal lives.
Compared with colleagues, physicians who were iden- A survey of general practitioners and their students
tified as excellent role models spent more time identified a positive attitude to teaching and excellent
teaching and conducting rounds and were more likely doctor-patient relationships as important in role mod-
to stress the importance of the doctor-patient relation- els.6 Using a different approach, other researchers
ship and psychosocial aspects of medicine. They also asked medical students to name one or two role mod-

BMJ VOLUME 325 28 SEPTEMBER 2002 bmj.com 707