Professional Documents
Culture Documents
Janice M Johnston, Gabriel M Leung, Richard Fielding, Keith Y K Tin & Lai-Ming Ho
Objectives Most evidence-based practice (EBP) edu- measures of EBP activity and examined responsiveness
cational assessment tools evaluated to date have through paired t-test of the pre ⁄ post factor mean scores.
focused on specific knowledge components or tech- Results A 43-item questionnaire was developed. Four
nical skills. Other important potential barriers to the factors were identified from both student groups. The
adoption of EBP, such as attitudinal, perceptual and overall questionnaire as well as each factor had high
behavioural factors, have yet to be studied, especially construct validity (Cronbach’s alpha > 0Æ7 for each
in the undergraduate setting. Therefore, we devel- scale). No significant correlations were found between
oped and validated a knowledge, attitude and beha- the 4 factors, confirming their orthogonality. Positive
viour questionnaire designed to evaluate EBP correlations, however, resulted between factor mean
teaching and learning in an undergraduate medical scores and other EBP activities. The responsiveness of
curriculum. the questionnaire was satisfactory.
Methods We derived the questionnaire from a compre- Conclusion A reliable knowledge, attitude and beha-
hensive literature review, informed by international and viour measure of EBP teaching and learning appropri-
local experts and a Year 5 student focus group. We ate for undergraduate medical education has been
determined its factor structure and refined and valid- developed and validated.
ated the questionnaire according to the responses of a
Keywords education, medical, undergraduate ⁄ *methods;
cohort of Year 5 and a combined group of Years 2 and
evidence based medicine ⁄ *education; curriculum; atti-
3 students using principal components factor analysis
tude; questionnaires; reproducibility of results; Hong
with varimax rotation. Factor reliability was computed
Kong; literature review.
using Cronbach’s alpha coefficient. We assessed con-
struct validity by correlating the factors with other Medical Education 2003;37:992–1000
1 Knowledge of EBP13,19,20
Content: asking clinical questions, acquiring evidence, appraising evidence, applying evidence to a clinical
situation, and assessing treatment effectiveness in terms of patient outcomes
Question format: statements rated on a Likert scale (1 ¼ strongly disagree, 6 ¼ strongly agree)
2 Attitudes towards EBP6,13,20
Content: perceived need for information, willingness to practice EBP, perceived role of EBP in clinical
practice, attitude about EBP’s threat to clinical practice
Question format: statements rated on a Likert scale (1 ¼ strongly disagree, 6 ¼ strongly agree)
3 Practice of EBP6,13,19–21
Content: access and acquisition of evidence, application to patient care, influence of positive role models on
EBP adoption, barriers to adopting EBP, contribution of EBP to clinical reasoning and learning
Question format: statements rated on a Likert scale (1 ¼ never, 5 ¼ every day; 1 ¼ not at all, 6 ¼ completely)
4 Actual use of EBP
Content: current proportion of clinical activity based on EBP principles, frequency of actual use of EBP,
perceived need for EBP each day or week and for each patient encounter, overall use of EBP in the past year
Question format: mostly open-ended responses with some multiple choices
5 Future use of EBP19
Content: perceived future importance of EBP to medical practice, willingness to practise EBP in the future,
usefulness of EBP in the future, potential barriers to the adoption of EBP currently and in the future
Question format: statements rated on a Likert scale (1 ¼ very unwilling, 6 ¼ very willing; 1 ¼ completely
useless, 6 ¼ very useful; 1 ¼ not at all, 6 ¼ completely)
Cantonese-speaking context. While individual items questionnaire. Students were not specifically asked to
were not formally rated, all feedback was incorporated advise on comprehensiveness of the items as their
through an iterative process from primary to secondary limited experience was considered likely to restrict their
feedback. An additional 3 multinomial categorical appraisal of the full range of relevant issues. The
items were generated as a result of this process. resultant preliminary draft questionnaire contained 36
Fourthly, the Year 5 (n ¼ 10) undergraduate med- multinomial categorical response questions and 11
ical students were then asked to comment on the questions requiring estimates, dichotomised or other
comprehensibility and relevance of the items in the responses. This questionnaire was then returned to the
expert panel for final reconfirmation of its face and EBP activity were used to assess criterion validity.10,18
content validity. The students estimated:
1 the proportion of current medical practice that is
Questionnaire refinement based on evidence;
2 their perceived need for evidence, and
The preliminary 47-item questionnaire was adminis-
3 their perceived change in self-behaviour in looking up
tered to all Year 5 undergraduate students (n ¼ 159) at
evidence compared to that of the previous year.
the Faculty of Medicine, University of Hong Kong. The
questionnaire contained 7 items using a 5-point scale The respondents also reported how frequently they
and 29 items using a 6-point scale, adopting both Likert engaged in EBP and whether they considered them-
and adjectival scales (strongly agree)strongly disagree, selves to be EBP practitioners. As this approach
not at all)completely, never)all the time, very diffi- required multiple comparisons, only those correlations
cult)very easy, completely unprepared)completely with P £ 0Æ01 were considered significant.18
prepared, completely useless)very useful, very unwill- Responsiveness, or the extent to which the instru-
ing)very willing). Negative statements were recoded for ment can detect change,10 was assessed on the basis of a
the analysis. As none of the items were severely skewed, pre ⁄ post comparison of the Year 2 students, who had
the data can be analysed as interval data without undergone a full year of EBP teaching through a series
introducing bias.21 The 36 categorical response items of 6 learning modules during the academic year. Paired
were included in a principal components factor analysis Student’s t-test was used to compare differences
with varimax rotation. Pairwise deletion of missing between the pre ⁄ post mean factor scores.10,18 Effect
values, eigenvalues ‡ 1 and factor loading scores size was calculated to assess sensitivity to change.21
‡ 0Æ4 were used to sort items into factors.18,22 The
Scree plot was used to confirm the optimum number of
Ethics approval
factors to include in the final set. Five items did not reach
the preset factor loading threshold and were excluded This study received approval from the Undergraduate
from the questionnaire. The revised questionnaire Education Committee and the Faculty Ethics Com-
contained 42 items, 31 of which were multinomial. mittee, Faculty of Medicine, University of Hong
The 4 factors derived from the factor analysis included Kong.
EBP knowledge (9 items), attitudes towards EBP
(6 items), personal application and use of EBP (8 items)
Results
and future use of EBP (8 items). A summary score (the
mean score of all the items in each factor) and A total of 158 of 159 Year 5 students (response rate
Cronbach’s alpha coefficient were calculated for each 99%) completed the preliminary, 47-item question-
factor. naire. Individual item response rates ranged from 97%
to 100%. Students ranged in age from 21 to 29 years
but 94% were either 23 or 24 years old. The
Evaluation of the final questionnaire
male : female ratio was 2 : 1.
The final version of the questionnaire containing 43 After the refinement procedure, the confirmatory
items (with 31 categorical response items) was distri- factor analysis of the 31 categorical items in the revised
buted, completed and collected during a whole class questionnaire yielded 4 factors or subscales. The Scree
session for the Years 2 and 3 classes (n ¼ 293) at the plot supported a 4-factor solution explaining 50Æ7% of
beginning of the academic year. The Year 2 students the variance. The 4 components derived were: factor
completed the same questionnaire again at another 1 – EBP knowledge (9 items; Cronbach’s alpha ¼
whole class session at the conclusion of the school year 0Æ88); factor 2 – attitudes towards EBP (6 items;
8 months later. Cronbach’s alpha ¼ 0Æ79); factor 3 – personal applica-
The data were analysed according to the methods tion and use of EBP (8 items; Cronbach’s
described above. Following the determination of the alpha ¼ 0Æ75), and factor 4 – future use of EBP (8
factor structure, Cronbach’s alpha was used to assess items; Cronbach’s alpha ¼ 0Æ76) (Table 3). Satisfac-
the internal consistency of each factor.13,18,22,23 In tory Cronbach’s alphas were noted for each factor.
order to establish the independence of the factors and (Means, standard deviations and factor loadings for
to further examine factor construct validity, between- each item are detailed in the Appendix.)
factor correlations were calculated.10,18 Comparisons The final version of the questionnaire was completed
between the factors and estimates of actual or perceived by Years 2 and 3 students (293 ⁄ 338; response rate
Table 3 Factor analysis for Year 5 and combined Years 2 and 3 students
Variance
Number Mean explained by Cronbach’s
Factor of items scale score SD each factor alpha
Variance
Number Mean explained by Cronbach’s
Factor of items scale score SD each factor alpha
SD ¼ standard deviation.
Table 4 Correlation matrix between the combined Years 2 and 3 factor scores
Table 5 Correlation matrix between the 4 factors and other evidence-based activity for the combined Years 2 and 3 students
*Significant at P £ 0.01.
Table 6 Year 2 pre ⁄ post-assessment comparison of mean differences in mean factor scores
Paired
Mean score Mean score t-test Effect
Factor n pre-assessment* post-assessment* (T score) P-value size
combined Years 2 and 3 component structure were the first to assess these non skill-based attributes for
compared. Evidence-based practice knowledge (factor 1) EBP and also the first to assess EBP in the undergra-
was significantly different at the post-assessment and duate setting.
demonstrated the largest effect size (Table 6). Our findings indicate that the questionnaire has
satisfactory reliability and validity. The development of
the questionnaire was informed by issues identified as
Discussion
important in the literature by international and local
This paper has focused on the development and experts and as relevant by the Year 5 focus group. The
validation of a self-administered questionnaire to assess acceptability of the questionnaire to the students was
EBP knowledge, attitude, behaviour and perceptions in demonstrated by the high response rates to individual
undergraduate medical students. To our knowledge it is items and by the year groups as a whole. While the
6 Ghali WA, Saitz R, Eskew AH, Gupta M, Quan H, Hershman 16 Wayland WC, Barry HC, Farquhar L, Holzman C, White A.
WY. Successful teaching in evidence-based medicine. Med Training medical students in evidence-based medicine: a
Educ 2000;34:18–22. community campus approach. Fam Med 1999;31:703–8.
7 Ellis P, Green M, Kernan W. An evidence-based medicine 17 Schoenfeld P, Cruess D, Peterson W. Effect of an evidence-
curriculum for medical students: the art of asking focused based medicine seminar on participants’ interpretations of
clinical questions. Acad Med 2000;75:528. clinical trials: a pilot study. Acad Med 2000;75:1212–4.
8 Rosenberg WM, Deeks J, Lusher A, Snowball R, Dooley G, 18 Taylor R, Reeves B, Mears R et al. Development and valid-
Sackett D. Improving searching skills and evidence retrieval. ation of a questionnaire to evaluate the effectiveness of evi-
J R Coll Physicians Lond 1998;32:557–63. dence-based practice teaching. Med Educ 2001;35:544–7.
9 Thomas PA, Cofrancesco J Jr. Introduction of evidence-based 19 McAlister FA, Graham I, Karr GW, Laupacis A. Evidence-
medicine into an ambulatory clinical clerkship. J Gen Intern based medicine and the practising clinician. J Gen Intern Med
Med 2001;16:244–9. 1999;14:236–42.
10 Green ML, Ellis PJ. Impact of an evidence-based medicine 20 McColl A, Smith H, White P, Field J. General practitioners’
curriculum based on adult learning theory. J Gen Intern Med perceptions of the route to evidence-based medicine: a ques-
1997;12:742–50. tionnaire survey. BMJ 1998;316:361–5.
11 Sackett DL, Parkes J. Teaching critical appraisal: no quick 21 Streiner DL, Norman GR. Health Measurement Scales. A
fixes. [Editorial.] Can Med Assoc J 1998;158:203–4. Practical Guide to their Development and Use. 2nd edn. Oxford:
12 Sackett DL, Straus SE. Finding and applying evidence during Oxford Medical Publications 1995;38,168.
clinical rounds. The evidence cart. JAMA 1998;280:1336–8. 22 Cork RD, Detmer WM, Friedman CP. Development and
13 Bordley DR, Fagan M, Theige D. Evidence-based medicine: a initial validation of an instrument to measure physicians’ use
powerful educational tool for clerkship education. Am J Med of, knowledge about and attitudes toward computers. JAMA
1997;102:427–32. 1998;5:164–76.
14 Green ML. Evidence-based medicine training in graduate 23 Bland JM, Altman DG. Validating scales and indexes. BMJ
medical education: past, present and future. J Eval Clin Pract 2002;321:606–7.
2000;6:121–38.
15 Srinivasan M, Weiner M, Breitfeld PP, Brahmi F, Dickerson Received 21 August 2002; editorial comments to authors 12 December
KL, Weiner G. Early introduction of an evidence-based 2002; accepted for publication 26 March 2003
medicine course to preclinical medical students. J Gen Intern
Med 2002;17:58–65.
Appendix
Combined Years 2 and 3 factor scores for individual items included in each component