Spring 2001 Y520: 8525

Strategies for Educational Inquiry

Research Quality & Content

Educational Research: Quality of Methods
To evaluate a research article, the reader must attend to theory development, operational definitions,
instrumentation/data collection, methods/procedures, data analysis/statistical tests, writing style, and
so forth. Even though the theoretical underpinnings may be weak or not well explained in a particular
article, our goal is to identify the authors purpose and determine whether the methods used do
indeed justify the conclusions.
In this course we focus on methods. Theory development is the focus of other courses, such as H510,
Foundations of Educational Inquiry.

Overview of Quality
Beginning students often assume that research articles published in scholarly journals must by
virtue of being publish be of high quality. Several experts have investigated the quality of research
in educational journals and based on their reports, the reader should not infer that publication insures
uniformly high methodological quality.
Serious doubt has been expressed as to whether the majority of published educational research articles merit publication on the basis of their quality (Wandt, 1965). Wandt et al. reviewed 125
research articles published in educational journals and report the frequently observed problems:

Frequent Problems

Results of analysis not clearly presented.

Incorrect methods used to analyze data.
Inappropriate or defective design.
Validity and reliability of the evidence not established.
Conclusions not substantiated by the evidence.

Organization of Research Reports

The sections of a research reports are:
introduction (description of theory, statement of the problem, conceptual definitions, synthesis of
previous research, testable hypothesis),
methods/procedures (design, subjects, sampling, materials, instrumentation/data collection, reliabality of scores, operational definitions, etc.),

Educational Research: Quality of Methods

results (descriptive statistics, interpretation of statistical tests), and

conclusion (explication of results to justify problem statement/hypothesis).
Notice that the list of frequently occurring problems are found mostly in the methods/procedures
and results sections.
Nearly a decade ago the National Academy of Science evaluated educational research and characterize the body of work as methodologically weak research, trivial studies, and infatuation with jargon,
and a tendency towards fads (Atkinson & Jackson, 1992, p. 20).
Several studies have used methodology experts as judges of research quality. The judges in Wall,
Hall and Schramm (1975) reported that over 60 percent of published research exhibited serious
flaws. Hall, Ward and Comer (1988) reported that over 40 percent of the studies they examined contained serious flaws.
In an AERA invited address, Thompson (1998) stated
incorrect analyses arise from doctoral methodology instruction that teaches
research methods as series of rotely-followed routines, as against thoughtful elements of a reflective enterprise; from doctoral curricula that seemingly have less
and less room for quantitative statistics and measurement content, even while our
knowledge base in these areas is burgeoning ... (p. 4).

Erosion of Standards
One conference attendee wrote AERA president Alan Schoenfeld to complain that
At [the 1998 annual meeting] we had a hard time finding rigorous research that
reported actual conclusions. Perhaps we should rename the association the American Educational Discussion Association.... This is a serious problem. By encouraging anything that passes for inquiry to be a valid way of discovering answers to
complex questions, we support a culture of intuition and artistry rather than the
building reliable research bases and robust theories. Incidentally, theory was even
harder to find than good research (Anonymous, 1998, p. 41).
These issues are discussed at length in Thompson (1999), from which several of the preceding quotations and references were borrowed. Thompson has his own list of suggestions. He states that educational research would be improved by the elimination of five specific practices. Most of these
practices are beyond the scope of an introductory course.

Thompsons suggestions for improvement

Use of stepwise methods in regression.
When interpreting results, failure to consider the context specificity of analytical weights (e.g.,
regression beta weights).
When interpreting results, failure to interpret both weights and structure coefficients.
Failure to recognize that reliability is a characteristic of scores, not of tests
Incorrect interpretation of statistical significance and failure to report and interpret effect sizes.
Two additional methodology errors that Thompson lists:

Research Quality & Content

Educational Research: Quality of Methods

Use of univariate analyses in the presence of multiple outcomes variables and the use of univariate analyses in post hoc explorations of multivariate effects.
The conversion of intervally-scaled predictor variables into nominally-scaled data when performing anova, ancova, manova, mancova.
These suggestions focus on the content of the methods/procedures and results sections of research
reports. The first three suggestions refer to the use of regression for data analysis. If you have little
previous experience with regression these suggestions may not be meaningful at this point. We will
explore the implications of these specific suggestions when we scrutinize a research report that uses
regression. A very brief explanation of each suggestion follows.

Stepwise Methods
The stepwise method consists of letting a computer decide the order in which variables are entered
into a regression equation. Instead, the investigator should decide the order of variable entry on the
basis of theory or simple expection. Other problems with stepwise regression will be mentioned
when we study regression.

Context Specificity
The second suggestion, specify the context specificity of analytical weights such as regression beta
weights, means that instead of considering only these weights, it may be important to consider the
structure coefficients, which are correlation coefficients between a predictor variable and a composite derived by weighting and aggregating the criterion variable(s) (Thompson and Borrello, 1985, p.
205). This procedure falls in the category variously referred to as beyond the scope of this course
or elsewhere as an exercise for the interested reader. Nonetheless, it is important to know that when
predictor variables are correlated with each other this condition of collinearity can impact the interpretation of results. The inclusion of structure coefficients may improve interpretation.

Reliability of Scores
The fourth suggestion, that reliability is a characteristic of scores, not of a test, is a fundamental point
that is often reported incorrectly in published articles. An assessment instrument, aka, a test, consists
of a series of questions. As such, it is just a piece of paper or a piece of software code, if administered
via computer. The instrument is administered to individuals. The individuals respond and we calculate reliability based on the scores of these individuals. We will spend considerable time discussing
reliability and validity.

Statistical Significance
The fifth suggestion, that statistical significance is often misinterpreted and effect size is not
reported, is also a fundamental point that we will discuss in virtually every article that we analyze. In
simple terms, statistical significance indicates the likelihood that a particular result is not a chance
occurrence. We may see some authors state, incorrectly, that statistical significance indicates the
importance and/or strength of an effect. Statistical significance indicates nothing about importance.
Indeed, social science can be criticized for highly (i.e., improbable due to chance) statistically significant results that are unimportant and even meaningless (cf, Postman, 1975). The effect size, which

Research Quality & Content

Educational Research: Content

is often not reported in published research, indicates the strength of an experimental manipulation.
Wilkinson et al. (1999), an article we will read later in the semester, discusses the importance of calculating the effect size.

Univariate & Multivariate

In this course we likely will not have time to consider the implications of Thompsons sixth suggestion, that mulitvarite analysis should be used for post hoc explorations when multivaritate analysis
are used.

Collapsing Scales
Violations of the final suggestion, that one should not throw away information by converting interval scale predictor variables to nominal (e.g., categorical) variables, will be seen in some of the
research articles we read and we will discuss the implications at that time. This often happens when
an investigator creates categories in order to perform an analysis of variance. This problem can be
avoided by using regression instead (cf. Humphreys, 1978).
Before we conclude this overview of quality, note that King (1986) describes several serious methodology problems that occur frequently in the political science literature. Rosenthal (1989) and Wilkinson et al. (1999) discuss methodological problems in psychological research and similar critiques of
sociology methods can be found. In fact, all branches of social sciences use similar methodologies
and experience quality control problems in published articles.

Educational Research: Content

In addition to research quality, faculty are interested in identifying the particular skills and knowledge students need to acquire in order to evaluate and conduct research. Mundfrom, Shaw, Thomas,
Young, and Moore (1998) asked, What should graduate students know about research and statistics
after completing an initial course? (p. 2). Instructors of research courses were surveyed and Table 2
presents the mean ratings for each items importance and the depth to which it should be covered. All
of the items in this table should be included in this course, but some may not.
Rather than surveying faculty, Elmore and Woehlke (1998) state An approach to determining the
essential topics to be included in the doctoral tool sequence for students preparing to be educational
researchers is to conduct a content analysis of the methods and techniques used in published articles
in educational research journals (p. 2). Following in the footsteps of several previous investigators,
they examined the research methods used during the past 20 years in three educational research journals. They report that the methods most frequently used in published articles during that interval
were:
1. Descriptive
2. Analysis of variance and analysis of covariance
3. Multiple regression
4. Qualitative
5. Bivariate correlation
6. Multivariate

Research Quality & Content

Educational Research: Content

Unfortunately, this list confounds paradigms (qualitative), research designs (descriptive), and
statistical methods (analysis of variance, multiple regression, bivariate correlation, multivariate).
Nonetheless, it provides an additional observation about the knowledge and skills students need in
order to critique research reports and to conduct their own investigations.
Note that valicity should be validity.
Table 2. Means and Standard Deviations for Topics - All Respondents (n=80).
Importance
Topic

Depth of Coverage

Mean

Std. Dev.

Mean

Std. Dev.

Internal and external valicity

Formulation of viable research problems
Types of research
Formulation of testable hypotheses
Scientific method
Types of variables

6.09
6.00
5.85
5.84
5.62
5.59

1.00
1.24
1.17
1.33
1.27
1.23

5.63
5.42
5.41
5.32
4.47
5.02

1.41
1.48
1.32
1.66
1.77
1.43

Interpreting results (of inferential tests)

Literature sources and searching
Correlation coefficients
Types of sampling
Null and alternative hypotheses
Measures of center
Validity estimates
Statistical vs. practical significance
Types of experimental designs
Reliability estimates
Measures of dispersion

5.39
5.36
5.25
2.24
5.24
5.19
5.15
5.13
5.11
5.08
5.08

1.84
1.45
1.73
1.36
1.84
1.73
1.41
1.83
1.40
1.35
1.79

4.87
4.72
4.53
4.59
4.63
4.60
4.02
4.48
4.51
4.28
4.53

1.80
1.85
1.80
1.38
1.86
1.83
1.54
1.73
1.49
1.43
1.83

Ethical and legal issues

Type I and Type II errors
Significance level
Conducting surveys
p-values
The normal distribution
Likert scales
Graphical and tabular displays
Rating scales
Standard error of measurement
Standard scores

4.93
4.81
4.80
4.80
4.72
4.69
4.65
4.64
4.62
4.58
4.40

1.72
1.91
1.85
1.42
1.98
2.04
1.28
1.66
1.20
1.67
1.79

3.97
4.18
4.10
4.13
3.98
4.05
3.88
4.27
3.80
3.89
4.08

1.62
1.85
1.75
1.35
1.86
1.96
1.55
1.72
1.39
1.80
1.77

Conducting interviews
t-tests
Sampling error
Contingency tables
Sampling distributions
Power
Analysis of variance
Chi-square tests of association
Confidence intervals
Elements of probability
Linear regression

4.33
4.24
4.22
4.18
4.06
4.03
4.00
3.87
3.87
3.73
3.66

1.54
1.91
1.79
1.78
1.98
1.93
2.02
1.94
2.12
1.98
2.06

3.52
3.64
3.53
3.51
3.34
3.18
3.08
3.13
3.08
2.85
3.11

1.57
1.94
1.76
1.79
1.79
1.88
1.95
1.89
1.95
1.70
2.12

Research Quality & Content

References

References
Anonymous. (1998). [Untitled letter]. In G. Saxe & A. Schoenfeld, Annual meeting 1999. Educational Researcher, 27(5), 41.
Atkinson, R.C., & Jackson, G. B. (Eds.). (1992). Research and educational reform: Roles for the
Office of Educational Research and Improvement. Washington, DC: National Academy of Sciences (ERIC Document Reproduction Service No. ED 334 961).
Elmore, Patricia B., & Woehlke, Paula L. (1998). Twenty years of research methods employed in
American Educational Research Journal, Educational Researcher, and Review of Educational
Research. Paper presented at annual meeting of the American Educational Research Association,
San Diego, CA. (ERIC Document Reproduction Service No. 420 701).
Hall, B.W., Ward, A.W., & Comer, C.B. (1988). Published educational research: An empirical study
of its quality. Journal of Educational Research, 81, 182-189.
Huck, Schuyler W., Cormier, William H., & Bounds, Jr., William G. (1974). Reading Statistics and
Research. New York: Harper and Row.
Miller, D. W. (1999). The black hole of educational research. The Chronicle of Higher Education,
August 6.
Mundform, Daniel J., Shaw, Dale G., Thomas, Ann, Young, Suzanne, & Moore, Alan D. (1998).
Introductory graduate research courses: An examination of the knowledge base. Paper presented
at the annual meeting of the American Educational Research Association, San Diego, CA. (ERIC
Document Reproduction Service No. ED 419 823).
Postman, Leo (1975). Verbal learning and memory. Annual Review of Psychology, 26, 291-335.
Rosenthal, Robert, & Rosnow, Ralph L. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276-1284.
Shott, Susan (1990). Statistics for Health Professionals. Philadelphia: W. B. Sanders.
Thompson, B. (1988). Common methodology mistakes in dissertations: Improving dissertation quality. Paper presented at the annual meeting of the Mid-South Educational Research Association,
ERIC Document Reproduction Service No. ED 301 595).
Thompson, B. (1994). Common methodology mistakes in dissertations, revisited. Paper presented at
the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC
Document Reproduction Service No. ED 368 771).
Thompson, B. (1999). Common methodology mistakes in educational research, revisited, along with
a primer on both effect sizes and the bootstrap. Invited address presented at the annual meeting of
the American Educational Research Association, Montreal.

Research Quality & Content

References

Thompson, Bruce, & Borrello, Gloria M. (1985). The importance of structure coefficients in regression research. Educational and Psychological Measurement, 45, 203-209.
Wandt, Edwin, Adams, Georgia W., Collett, Dorothy M., Michael, William B., Ryans, David G., &
Shay, Carleton B. (1965) An evaluation of educational research published in journals. Report of
the Committee on Evaluation of Research, American Educational Research Association.Unpublished report.
Ward, A.W., Hall, B.W., & Schramm, C.E. (1975). Evaluation of published educational research: A
national survey. American Educational Research Journal, 12, 109-128.
Wilkinson, Leland. (1999). Statistical methods in psychology journals: Guidelines and explanations.
American Psychologist, 54(8), 594-604.

Research Quality & Content