MEDICAL CARE Volume 39, Number 8, Supplement 2, pp II-46II-54 2001 Lippincott Williams & Wilkins, Inc.

Successes and Failures in the Implementation of Evidence-Based Guidelines for Clinical Practice
RICHARD GROL, PHD

BACKGROUND. The development and implementation of (evidence-based) clinical practice guidelines is one of the promising and effective tools for improving the quality of care. However, many guidelines are not used after dissemination. Implementation activities frequently produce only moderate improvement. It is important to study specific guideline programs in detail to learn from their successes and failures. OBJECTIVES. Experiences with more than 10 years of development and dissemination of clinical guidelines for family medicine in the Netherlands are presented in this paper. RESULTS. More than 70 evidence-based guidelines have been set in a rigorous procedure and have been spread via a variety of strategies. Knowledge and acceptance of the guidelines in the target group is high. In particular, a multifaceted approach with written (scientific journal, support materials) and personal approaches (local consensus discussions, contact with colleagues, outreach visits by

peers) seems to be effective in the dissemination. The guideline recommendations are followed in on average 67% of the decisions, but there is a large variation between different physicians and between different guidelines. Specific strategies designed to handle possible obstacles to implementation are needed to improve adherence. Specific implementation projects showed the importance of a diagnostic analysis of the target group and target setting before the start of the implementation. CONCLUSIONS. A program to implement a guideline should be well designed, well prepared, and preferably pilot tested before use. Such a program should be built into the normal channels and structures for improving care. More research into the details of implementation is needed to better understand the critical determinants of change in practice. Key words: Clinical practice guidelines; family practice; implementation; change. (Med Care 2001;39:II-46 II-54)

Despite many years of efforts to improve the quality of patient care, research in most countries continues to show that many patients receive inappropriate or even detrimental care. American figures point at the overuse,underuse,or misuse of care.1,2 The situation in Europe cannot be expected to be much better. Many different approaches to the improvement of clinical practice have been proposed and used in the past decades, such as professional education and development, audit and feedback, evidence-based guidelines,

total quality management, economic incentives, and organizational changes. Although the number of well-designed studies examining attempts to modify clinical practices is rising, just which approaches are most effective in which settings remains unclear.35 The development and implementation of (evidence-based) clinical-practice guidelines appears to be one of the most promising and effective tools for improving the quality of care: that is, reviews of the scientific literature combined with insights from clinical practice to

From the Center for Quality of Care Research, Universities of Nijmegen and Maastricht, Nijmegen, The Netherlands. Address correspondence to: Richard Grol, PhD, Director, Center for Quality of Care Research, Universities

of Nijmegen and Maastricht, Nijmegen, The Netherlands. E-mail: r.grol@hsv.kun.nl

II-46

Vol. 39, No. 8, Supplement 2 generate concrete recommendations can help health care providers and patients decide on appropriate care, promote education and improvement of care processes, reduce unwanted variation in the delivery of health care, and perhaps help contain costs.6 The development of such recommendations or guidelines is nevertheless labor intensive and expensive. Another problem is that such guidelines do not implement themselves; they are often not used after dissemination, and implementation activities frequently produce only moderate improvement.3 6 These findings raise the question of how to gain greater insight into those factors that appear to be decisive for the effective implementation of clinical guidelines. The controlled trials currently being used within the Cochrane Collaboration on Effective Practice and Organization of Care7 do not provide all the answers. It is therefore important to study specific programs for guideline development and implementation to learn from their actual successes and failures.8 This paper reports experiences with the development and implementation of a program for national evidence-based guidelines for family medicine in the Netherlands. Approximately 7,000 family physicians care for approximately 16 million inhabitants in the Netherlands. Most of the family physicians work alone in office-based practices (approximately 50%) or with one or more partners in small group practices. Family physicians play a central gate-keeping role in the Dutch health care system because they have to approve referrals to specialists working in outpatient clinics and hospitals. After a period of regional guideline setting at the beginning of the 1980s, which led to conflicting guidelines and confusion among physicians, the Dutch College of Family Physicians took the initiative to develop national guidelines. The first set of guidelines (diabetes mellitus type 2) was published in 1987, and since that time more than 70 guidelines have been developed and disseminated. In the present paper, the lessons to be learned from more than 10 years of the dissemination of guidelines will be considered.

SUCCESSES AND FAILURES and a basis for education and local protocol setting. A good balance between the use of scientific evidence and the establishment of guidelines that are clearly feasible and acceptable for normal practice is considered crucial. A systematic and rigorous procedure with the following steps was gradually developed over time. First, a relevant topic is selected by an independent advisory board comprising 11 experienced family physicians. The advisory board further defines the topic, outlines the objectives of each guideline, and draws up a detailed working plan. Second, a working party is composed of 4 to 8 family physicians with a mixture of scientific and practical experience. A survey of the members of 52 working parties (n 243, response rate 96%) showed 50% of the participants to have an academic affiliation and 50% to be ordinary family doctors; 39% base their expertise on research on the topic and 12% have published on the topic.9,10 Participation is voluntary; a paid staff member from the Dutch College of Family Physicians provides support for the working party (literature searches, writing of draft documents, organization of meetings). The working parties meet 10 to 15 times during a period of 1 to 1.5 years to establish the guidelines. Before the first meeting, they receive a synopsis of the relevant literature on the topic; at the second meeting, a short course on critical reading and evidence-based literature analysis is provided. The different tasks are then divided among the individual group members, who scrutinize the relevant literature and draw up tentative recommendations for practice. In light of the fact that the scientific evidence is often simply lacking, not applicable, or conflicting, extensive consensus discussions often prove necessary. Next, the feasibility and acceptability of a specific set of guidelines for normal practice is evaluated with the circulation of a written survey to 50 randomly selected family physicians. A copy of the proposed guidelines is also sent to external reviewers, who are usually specialists on the topic. The comments of the physicians and experts are then used to improve the guidelines. Then an independent scientific board provides official approval of a set of guidelines. To obtain such approval, the working party must defend the guidelines to a group of experienced physicians and researchers. After approval, the final version of the guidelines is written and published in the form of a paper in a scientific journal for family physicians. This paper includes the relevant scientific

Guideline Development and Dissemination Program

The aim of the guideline development and dissemination program is to provide family physicians with a point of reference for their daily work

II-47

GROL background information. A summary of the guidelines is also printed on a plastic card accompanying the journal. Dissemination is further promoted by developing special educational programs and packages for each set of guidelines. This information is sent to approximately 100 regional and local coordinators for continuing medical education and quality improvement. Support materials in the form of summaries of the guidelines for receptionists, leaflets and letters for patients, and consensus agreements with specialist societies are also developed and disseminated among the family physicians. The guidelines are updated every 3 to 5 years, depending on the availability of new scientific evidence. This stepwise approach proved to be feasible and is more or less standardized now. Various studies have been undertaken to evaluate the impact of this comprehensive program. In the following sections, some critical results will be presented along with some lessons that have been learned about effective implementation.

MEDICAL CARE physicians.12 We concluded from these results that a program of guideline development run and owned by the profession itself and disseminated via a variety of channels can increase knowledge and acceptance of the guidelines among the target group. Differentiation of the target group is necessary for effective dissemination: for some doctors, quick access to evidence and guidelines is required and/or preferred; for others, dissemination via local networks or a more personal approach is required.13 In a controlled before-and-after study, the effects of two strategies for speeding the dissemination of the national guidelines were compared: mailing of an information package to key people versus mailing of an information package to key people with additional outreach visits by two family physicians specially trained to provide explanation and instruction on the use of the materials. The information package contained an overview of 10 sets of national guidelines (eg, guidelines for diabetes, urinary tract infection, hypertension, acute otitis media) and additional support materials, which included specially designed audit instruments pertaining to the key recommendations contained in the guidelines for use in a performance review; a computerized version of the audit instruments, providing immediate feedback after entry of the performance data; and a description of some procedures for local educational meetings addressing the guidelines. Family physicians in the Netherlands were informed about the information package via publication in family medicine media; interested doctors could order the materials at no cost. In addition, the following two strategies were undertaken to promote use of the guidelines and support materials: the information package was mailed to key people, namely all organizers of local continuing medical education and representatives of all local groups of family physicians (a district with 527 physicians), and it was mailed to key people with additional outreach visits by two family physicians specially trained to provide explanation and instruction on the use of the materials (a comparable district in terms of size, number of physicians, degree of urbanization, and organization of quality improvement, with 504 doctors). The outreach visitors approached the key people initially by telephone to motivate them to use the information package and explore the need for a visit and the provision of support for the local group. In such a manner, this dissemination strat-

Knowledge and Acceptance of the Guidelines

A survey among a random sample of 10% of all family physicians (n 453, response rate 70%) approximately 2 years after publication of the first set of guidelines showed most doctors to be well informed about the guidelines (only 7% did not know about them) and wide acceptance of the guidelines.11 A subsequent survey among a sample of 1,531 physicians (n 1,007, response rate of 67%) 3 years later showed increased acceptance: approximately 90% of the physicians surveyed considered the national guidelines very useful for local education and for translation to local protocols; only 13% of the physicians thought the guidelines could be used in contracts with health insurers. The most frequently reported sources of information on the guidelines proved to be the scientific journal (85% of the respondents), discussion of the guideline within the local group of family physicians (53%), contacts with other colleagues (43%), and course attendance (33%). Physicians reporting the scientific journal as a source of information on the guidelines, members of the Dutch College of Family Physicians, younger doctors, and physicians actively involved in the teaching of family medicine proved to be better informed about the guidelines than other

II-48

Vol. 39, No. 8, Supplement 2 egy was tailored to the needs of the key people and groups. The actual use of the information package was assessed using a written survey before and after an intervention period of 1.5 years. All of the physicians in the two study districts and a random sample of 500 doctors from the remainder of the country were asked to complete the survey. Only those physicians completing the survey on both occasions were included in the analyses. No significant differences were found between the respondents in the three groups with regard to response rate and such personal characteristics as age, type of practice, postgraduate training, and College membership. The overall response rate before intervention was 67% (1,007 doctors); the overall response rate after intervention was 76% (762 of the initial 1,007 physicians). A random sample of 80 nonresponders was interviewed by telephone. Compared with the nonresponders, the responders tended to be younger but did not differ otherwise. Comparison of the two study districts with the remainder of the country showed the outreach visits in particular to contribute to greater knowledge and use of the national guidelines and support materials (Table 1). No other factors (eg, physician characteristics) were found to influence significantly knowledge or use of the materials.

SUCCESSES AND FAILURES selection of the key recommendations and translation of these into indicators of adherence was performed with the help of a panel of five expert family physicians who were also members of the staff for guideline development. Because the guidelines are intended to steer clinical decision making by defining how to act under specific case conditions, indicators were constructed in the form of a number of if-then algorithms relating clinical actions to specific conditions. In such a manner, differences in case mix could also be accounted for. For 30 guidelines, a total of 342 specific adherence indicators were constructed. Recording forms were next developed for data collection. Specific inclusion criteria were formulated to guide physicians in the selection of the patients for recording performance. The recording forms did not provide any clues to appropriate performance with regard to concrete cases. Such self-recording was shown to be reliable in a separate study in which the agreement of self-report data with data derived from medical records and observation in the consulting room for seven family physicians proved to be good ( 0.80 0.90).14 Computer software was developed to relate actual performance (decisions, actions) to clinical conditions and thereby assess adherence to the guidelines in concrete decisions. A representative sample of 740 physicians was approached; 200 (27%) agreed to participate in the study. They differed significantly from the nonrespondents with respect to age (younger), gender (more female), and type of practice (fewer solo practitioners), but not with respect to practice location, membership in the College, and being a trainer. The family physicians recorded their performance for 1 or 2 contacts per guideline topic, resulting in a total of 36,242 decisions for 7,614 contacts allowing key recommendations to be related to actual performance: for 67% of the

Use of the Guidelines

To study the actual use of and adherence to the national guidelines, several key recommendations were selected from the guidelines to serve as indicators and review criteria. Different studies were performed during the past decade, with the most recent study involving a sample of 200 family physicians who recorded their performance to assess adherence to 30 national guidelines. The

TABLE 1. Effect of Different Strategies to Disseminate National Guidelines and Support Materials
Mailing to Key Persons and Outreach Visits (n 269) Being informed on materials Having materials in possession Having read the materials Having used the materials Difference with rest of country, *P 0.01. 66%* 49%* 35%* 25%* Mailing to Key Persons Only (n 244) 25% 14%* 9% 4% Rest of Country (n 249) 20% 6% 6% 2%

II-49

GROL decisions, actual performance was found to follow the relevant recommendations.14 However, clear differences in the adherence scores were also observed across the guidelines. For instance, 6 guidelines had an average adherence score lower than 50% and 5 had an average adherence score higher than 80% (Table 2). We also found clear differences in the adherence scores across doctors: 27% scored lower than 60% on average and 5% scored higher than 80% on average. A positive bias in the adherence scores cannot be excluded, because the physicians were only partly representative for the national population and they may have selected cases in which a high score was more easily achieved. However, the results confirm findings in earlier studies on guideline adherence in the Netherlands. Different lessons can be derived from the preceding evaluations. First, valid indicators and criteria for guideline adherence must be developed to assess normal decision making in clinical practice. The selection of such indicators and the collection of reliable performance data are a complex and labor-intensive undertaking.1517 Approximately 35% of the decisions made by physicians were not in line with the key recommendations. Given the variation in the adherence scores across the different guidelines, a failure (or predisposition) to adhere to certain recommendations may depend, at least in part, on the characteristics of the guideline recommendations themselves. In one of our studies analyzing 47 recommendations in greater detail, it was found that, in line with certain implementation theories,13 recommendations that define the desired performance very concretely, recommendations that are compatible with existing values, and recommendations that do not have major consequences for the organization of health care were better followed than others.18 Attention to the quality of a guideline in terms of being formulated as a credible and feasible message for dissemination to a particular target group is thus an important prerequisite for successful guideline development and implementation. Another lesson is that despite comprehensive guideline dissemination, many doctors still do not follow the key recommendations put forth by their own scientific organization. Better-tailored programs and strategies aimed at specific performance changes are probably needed. In yet another series of studies, different implementation strategies are thus being further tested and evaluated: for example, implementation with perfor-

MEDICAL CARE TABLE 2. Adherence Scores to Guideline Recommendation in Decision Making

Average overall clinical guidelines Guidelines with Highest Adherence Scores Micturation problems in older men Diagnosis of heart failure Cholesterol management Food allergies in neonates Guidelines with Lowest Adherence Scores Otitis externa Fluor vaginalis Abdominal complaints Diagnosis of asthma and chronic obstructive pulmonary disease in adults Adherence scores in 36,242 decisions; n physicians. 67% 100% 92.3% 84.4% 82.1% 34.4% 36.1% 48.7% 48.8% 200

mance review and feedback,19 interactive education in small groups of family physicians,20 outreach visits by trained nurses,21 organizational changes (eg, delegation of tasks to special nurses), and patient-mediated interventions (eg, the use of special passports for diabetes patients). Some of the experiences with the different implementation strategies are summarized below.

Barriers to the Implementation of Guidelines

One of the studies comparing different implementation strategies was a randomized controlled trial with two groups of family physicians and their patients. The aim of the study was to assess the effectiveness of an intensive interactive program involving small-group education and peer review (4 sessions of 2 h) for the implementation of the national guidelines regarding asthma and chronic obstructive pulmonary disease.20 The intervention program fits the model of quality improvement most commonly used in family practice in the Netherlands and Europe quite well: continuing education and performance review in local groups of physicians.22,23 The data from 433 patient questionnaires and 934 recordings of consultations, however, showed no significant differences between the control and intervention groups with regard to the provision of care (adherence to guidelines) and patient outcomes (symptoms, quality of life, smoking, exacerbation).20 A similar lack of an intervention effect was found in a study evaluating the implementation of

II-50

Vol. 39, No. 8, Supplement 2 a national guideline for cholesterol management at the beginning of the 1990s. The specific recommendations for this guideline were concentrated on the appropriate testing of cholesterol levels in patients with a positive risk profile.Twenty practices with 32 physicians were allocated to either the intervention or the control group using stratified randomization (with or without a computer system, solo or group practice, and practice size as strata). To facilitate implementation of the guideline, the intervention group received a comprehensive multifaceted program, which included small-group education, regular feedback on gaps in performance, various support tools (eg, decision trees), and outreach visits by the researcher to stimulate use of the guideline. Nevertheless, inspection of 3,950 patient records (ie, a random sample of 10% of all patients 18 years of age or older) showed no changes whatsoever in appropriate cholesterol testing after an intervention period of 1 year.24 This finding of no change prompted more detailed analysis of the obstacles to implementation for this particular guideline. Interviews with the family physicians revealed a variety of problems25: doubts about the value and scientific grounds for the key recommendations; resistance to a proactive preventive approach and efforts to motivate patients to change their lifestyles; reports that the algorithm for diagnosis and treatment determination was overly complex, difficult to understand, and not easy to use; extra workload (eg, extra testing, diet advice); and the demand for unnecessary tests by many patients. These results show the need to understand the factors playing a role in the implementation of a guideline to develop appropriate interventions and support strategies. The results are also in line with an increasing number of authors who emphasize the importance of a diagnostic analysis of the target group and target setting before initiating implementation activities.5 In most of our implementation projects, in fact, we now perform such a preliminary analysis. One example of the importance of a preliminary analysis of the target group and setting is a project on the implementation of the national Low Back Pain guideline for family practice. A prospective study on the performance of 57 family physicians in 1,640 back pain contacts showed that failure to adhere to this guideline concerned in particular a lack of patient education on staying active, prescribing nonsteroidal anti-inflammatories, and referral to a physiotherapist in the acute phase.

SUCCESSES AND FAILURES Reasons for lack of adherence provided by the physicians were perceived preferences of patients and following fixed routines.26 To explore the factors determining lack of adherence to the guideline in more detail, in-depth interviews were undertaken with patients consulting their family physician for low back pain and the physicians they consulted. Forty physicians were invited to participate in this study (a heterogeneous group); 31 (77%) agreed to participate and 20 of them recruited an eligible patient for an interview during the study time. These interviews were audiotaped, fully transcribed, and analyzed qualitatively by two researchers using a gradually developed classification scheme. Most of the patients reported consultation of the physician for diagnosis and/or simple advice.27 Among the main reasons for lack of adherence to the recommendations regarding referral to physical therapists and the prescription of nonsteroidal anti-inflammatories was a tendency on the part of the family physicians to give in to the medically inappropriate demands of patients to avoid conflict. The conclusion drawn from these interviews was that the implementation may be further promoted by training doctors to communicate better with their patients and educating patients on low back pain. Although the studies of the cholesterol and low back pain guidelines were performed with selected (motivated) groups of physicians, the results show that a variety of factors determine successful implementation in many cases. Different groups may also experience different problems with the use of the guidelines, which means that well-designed programs with different strategies addressing different factors at different levels may be required for successful and effective implementation. An example of such a program is presented below.

National Program for the Implementation of Preventive Guidelines

One of the current implementation programs is concerned with guidelines for preventive care in family practice, more specifically the flu vaccination for people older than 65 years and special risk groups, cervical cancer screening, and the detection and reduction of cardiovascular risk. Fostering prevention in family medical practice in the Netherlands has been viewed as a particularly difficult undertaking for many years. Various surveys28

II-51

GROL have shown a reluctance among family physicians toward more proactive, population-based approaches to preventive care and inadequate organization of the practices. A national program with different strategies and arrangements linked to these problems was developed and initiated in 1995. At the national level, evidence-based guidelines were determined and disseminated via the scientific journal; specific educational packages and computer software to support the identification and monitoring of patients at risk were designed. In addition, models to improve the organization of prevention were developed. Financial arrangements to reimburse doctors for the extra work involved in the prevention efforts were negotiated with the Dutch National Health Service. At the district level, educational sessions and local small-group meetings were organized to introduce the guidelines, the associated computer software, and the models for improving the organization of the prevention. Arrangements with other parties with a vested interest in prevention (eg, local health authorities) were also made. Trained outreach visitors provided support for the organization of the prevention and use of the computer software at those family practices expressing an interest. Continuous evaluation of the impact of the program was undertaken in surveys of randomly selected practices and extraction of data from electronic medical records of a smaller sample of practices. The program started with the flu vaccination guideline in 1995, the cervical cancer screening guideline was introduced in 1996, and the guideline for cardiovascular risk was introduced in 1998. The flu vaccination rate has increased considerably in recent years: from 10.5% in 1994 to 16.5% in 1997.29 Two sources of data were used: the national health surveys from the Central Office of Statistics and the data from a representative group of 58 monitoring practices. For people older than 65 years, the rate of vaccination was more than 80% in 1999. Guidelines for cervical cancer screening aim at giving family practice complete responsibility for this screening, because it has been proven to increase the attendance rate of eligible women with 10% to 15%.30 National surveys among a random sample of 1,586 practices (response rate 62%; no differences between nonresponders and national population) were performed to study changes in screening routines before and after the start of the program. The percentage of family practices completely respon-

MEDICAL CARE sible for the screening (contacting the women, reminders, taking the smears, and follow-up) increased from 7% in 1995 to 30% in 1997.30 The survey data from 1997 showed that 30% of the practices participated in local or regional education with regard to the guidelines for cervical cancer screening. One third of the surveyed practices had at least one visit by a trained nurse visitor, and almost half of the practices made use of the computer software to identify women at risk. The education of the doctors had a limited effect on their involvement in cervical screening. However, using the computer software and receiving more than two outreach visits was significantly related to following the guidelines for cervical cancer screening and active involvement of the family practice in this preventive activity (odds ratios 1.5 8.7).31 In this national project, we have learned about the importance of a well-designed and wellprepared implementation plan with a variety of interventions based on the evidence and experiences accumulated in different pilot projects. As found in other studies,32 outreach visits are found to be very useful along with the provision and use of computer software; in addition, financial reimbursement for the extra work involved in using the guidelines appears to be an important ingredient for program success. The value of involving all parties with a vested interest in prevention is another important insight. Finally, building the guideline implementation program into the normal channels and structures for contact with the physicians appears particularly important

Conclusions
Different projects on the implementation of national guidelines have been conducted in the Netherlands during the past decade with different results. To our knowledge, the present program is one of the most comprehensive programs for evidence-based guideline development and implementation in the world. Studying the acceptance and impact of such a program over time provides critical insights for the provision of adequate care. Based on the evidence accumulated to date and our experiences with the different experiments, some lessons from more than 10 years of guideline development and implementation can be presented.

II-52

Vol. 39, No. 8, Supplement 2 First, the rigorous development of clinical guidelines at a national level is both feasible and well accepted by the target group when it is owned and operatedby the profession itself. The Dutch program appeared to give family physicians greater self-confidence in relation to medical specialists on the one hand and to enhance the status of family medicine in the health care field on the other hand. The program also stimulated support from the governmental and insurance agencies without losing the support of ordinary doctors. Second, a comprehensive strategy to disseminate the guidelines via various channels, both written and personal, appears to be very important: scientific journals, local networks of peers, and colleagues trained to explain the guidelines should be part of such a strategy. In addition to a general strategy to promote the implementation of the guidelines, specific strategies designed to handle possible obstacles to adherence should also be used. This requires a diagnostic analysis of the target group and target setting: Who is interested and/or involved in implementation, and who are the stakeholders? Which aspects of care should be addressed, and which recommendations are not followed? Which subgroups of the target group appear to experience particular problems with changing practice procedures, and what are these problems? Third, a program to implement a guideline should be well designed, well prepared, and preferably pilot tested before use. All of those with a vested interest in the program should also be involved in the set-up of the program, and the program should be built into the normal channels and structures for monitoring and improving the quality of care in a specific profession or specific setting. As already stated, these lessons are only partly based on concrete evidence. More research into the details of the different implementation activities is needed to understand the critical determinants of change in clinical practice

SUCCESSES AND FAILURES

3. Grol R. Beliefs and evidence in changing clinical practice. Br Med J 1997;315:418 21. 4. Grol R, Grimshaw J. Evidence-based implementation of evidence-based medicine. Jt Commission J Quality Improvement 1999;25:50313. 5. NHS Centre for Reviews and Dissemination. Getting evidence into practice. Effective Health Care 1999;5:115. 6. Woolf S, Grol R, Hutchinson A, et al. Potential benefits, limitations, and harms of clinical guidelines. Br Med J 1999;318:52730. 7. Bero L, Grilli R, Grimshaw J, et al. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote implementation of research findings by health care professionals. Br Med J 1998;317:465 8. 8. Kanouse D, Kallich J, Kahan J. Dissemination of effectiveness and outcomes research. Health Policy 1995;34:16792. 9. Grol R, Thomas S, Roberts R. Development and implementation of guidelines for family practice: lessons from the Netherlands. J Fam Pract 1995;40:4359. 10. Grol R, Woolf S, Eccles M, Maisonneuve H. Developing clinical practice guidelines: European experiences. Diasease Management and Health Outcomes 1998;4:255 66. 11. Grol R. National standard setting for quality of care in general practice: attitudes of general practitioners and response to a set of standards. Br J Gen Pract 1990;40:361 4. 12. Grol R, Zwaard A, Mokkink H, et al. Dissemination of guidelines: which sources do physicians use in order to be informed? Int J Qual Health Care1998;10:135 40. 13. Rogers E. Diffusion of innovations. New York: Free Press; 1995. 14. Spies T, Mokkink H, Grol R. Assessing the use of national evidence-based clinical guidelines. (In press) 15. Brook R, McGlynn E, Cleary P. Quality of Care 2: Measuring quality of care. N Engl J Med 1996;335:966 70. 16. Eddy D. Performance measurement: problems and solutions. Health Aff 1998;17:725.

References
1. Bodenheimer T. The American health care system. The movement for improved quality in health care. N Engl J Med 1999;340:48892. 2. Schuster M, McGlynn E, Brook R. How good is the quality of health care in the United States? Milbank Q 1998;76:517 63.

17. McColl A, Roderick P, Gabbay J, et al. Performance indicators for primary care groups: an evidence-based approach. Br Med J 1998;317:1354 60. 18. Grol R, Dalhuijsen J, Thomas S, et al. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. Br Med J 1998;317:858 61.

II-53

GROL
19. Winkens R, Ament A, Pop P, et al. Effects of routine individual feedback over nine years on general practitioners requests for tests. Br Med J 1996;312:490. 20. Smeele I, Grol R, Van Schayck C, et al. Can small group education and peer review improve care for patients with asthma/chronic obstructive pulmonary disease? Qual Health Care 1999;8:92 8. 21. Hulscher M, Van Drenth B, Van der Wouden J, et al. Changing preventive care: a controlled trial on the effects of outreach visits to organise prevention of cardiovascular disease. Qual Health Care 1997;6:19 24. 22. Grol R, Lawrence M. Quality improvement by peer review. Oxford: Oxford University Press; 1995. 23. Grol R. Quality improvement by peer review in primary care: a practical guide. Qual Health Care 1994;3:14752. 24. Van der Weijden T, Grol R, Knottnerus J. Feasibility of a national cholesterol guideline in daily practice. A randomized controlled trial in 20 general practices. Int J Qual Health Care 1999;11:1317. 25. Van der Weijden T, Grol R, Schouten B, Knottnerus A. Barriers to working according to cholesterol guidelines. A randomized controlled trial implementation of national guidelines in 20 general practices. Eur J Public Health 1998;8:113 8.

MEDICAL CARE
26. Schers H, Braspenning, Drijver R, et al. Low back pain in general practice: reported management and reasons for not adhering to the guidelines in the Netherlands. Br J Gen Pract 2000;50:640 4. 27. Schers H, Wensing M, Huysman Z, et al. Low back trouble: a qualitative study on barriers for guideline implementation. (In press) 28. Hulscher M, Van Drenth B, Mokkink H, et al. Barriers to preventive care in general practice: the role of organizational and attitudinal factors. Br J Gen Pract 1997;4:711 4. 29. Tacken M, Braspenning J, Van Paasen J, et al. Nine years of flu vaccination in family practice. Huisarts en Wetenschap 2000;43:566 7. 30. Hermens R, Tacken M, Hulscher M, et al. Cervical cancer screening in family practices in The Netherlands. Prev Med 2000;30:35 42. 31. Hermens R, Hak E, Hulscher M, et al. Adherence to guidelines on cervical cancer screening in general practice: program elements of successful implementation. (In press) 32. Thomson MA, Oxman A, Davis D, et al. Outreach visits to impose health professional practice and health care outcomes. In: The Cochrane Library, Issue 1, 1999. Oxford: Update Software.

II-54