Professional Documents
Culture Documents
T
The development of scientific and here are three options avail- institution of approximately 18,000
quantitative reasoning skills in able to faculty interested in students in Harrisonburg, Virginia and
undergraduates majoring in science, assessing the impact of un- has a strong emphasis on program as-
technology, engineering, and dergraduate education on sessment. The nationally recognized
mathematics (STEM) is an objective scientific and quantitative reasoning Center for Assessment and Research
of many courses and curricula. skills: use an existing instrument, Studies (CARS) provides significant
The Biology Department at James modify an existing instrument, or resources to the development of a
Madison University (JMU) assesses develop a new instrument. Given nationally recognized assessment
these essential skills in graduating the importance that science, tech- program (www.jmu.edu/assessment/).
biology majors by using a multiple- nology, engineering, and mathemat- Building on the need for assessment
choice exam called the Natural ics (STEM) programs and national of scientific and quantitative reason-
World-9 (NW-9). NW-9, comprised science organizations place on the ing in higher education, and more spe-
of measures of Quantitative and development of scientific and quan- cifically to inform STEM education,
Scientific Reasoning, contains titative reasoning skills, one would members of CARS in partnership with
items developed by faculty at expect to find an endless array of JMU faculty developed the Natural
JMU to assess the impact of the reliable instruments that assess World-9 (NW-9) instrument, which
General Education program on whether students graduating from contains two components: the Scien-
the development of scientific and undergraduate programs successfully tific Reasoning Test (SR-9; Sundre,
quantitative reasoning skills in acquired these essential skills (How- 2008) and the Quantitative Reason-
a content-independent manner. ard Hughes Medical Institute 1996; ing Test (QR-9; Sundre, Thelk, and
We discuss methodology we used NRC 2003). Many of the standard- Wigtil 2008). All NW-9 items were
to involve faculty in determining ized tests, such as the Graduate Re- written by James Madison University
the generalizability of NW-9 to cord Examination, include items that science and mathematics faculty to
assess the objectives of the biology assess scientific reasoning ability, but assess the objectives of the science
curriculum and setting standards for the most part research-based stan- component of the General Education
to interpret student achievement dardized tests address content knowl- program (see Table 1). Rather than
on NW-9. Student performance on edge (Bao et al. 2009). The Class- investing faculty time in developing
NW-9 identified both strong and room Test of Scientific Reasoning a new instrument, we decided to ex-
weak areas in our instruction and developed by Lawson in 1978 is still plore whether the NW-9 instrument
suggested that our biology faculty popular among STEM educators, but developed and tested by CARS could
needs to reevaluate methodology for this instrument addresses very broad assess scientific and quantitative
teaching students how to interpret areas of scientific reasoning and does reasoning skills in biology majors.
and analyze data. More important, not assess quantitative reasoning We also wanted to involve faculty
we can close the assessment loop skills (Lawson 1978). Unfortunately, in this process to enhance faculty
by allowing faculty to participate few readily accessible instruments understanding and appreciation of the
in the assessment process and are available that reliably assess both assessment process and results.
meaningfully reflect on student scientific and quantitative reasoning The Department of Biology has
assessment results. skills in undergraduates. 56 full-time and part-time faculty,
James Madison University (JMU) approximately 900 declared majors,
is a publicly funded, comprehensive and 100125 students who graduate
18
Journal of College Science Teaching
each year. The biology curriculum and skill objective 1 and General factual information (see Figure 1b).
is designed upon an explicit set of Education objective 6 both explore Based on these features of NW-9, we
content, skill, and experience learn- students ability to distinguish between determined the generalizability of the
ing objectives developed by biology association and causation. Second, NW-9 instrument to assess the skill
faculty. These objectives support the CARS has extensively tested both objectives of the biology major. We
two major goals of the curriculum: components of NW-9 to establish two did this by involving biology faculty in
insuring that biology majors are literate important measures of a meaningful a content alignment process in which
in the scientific process and integrating assessment instrument: reliability and they mapped NW-9 items to the skill
research experiences into the learn- validity. The NW-9 instrument reliabil- objectives. We also involved faculty
ing environment for all our majors. ity and validity scores suggest that the in the standard setting protocol to de-
Specifically, the skill objectives con- instrument consistently measures the termine the standards for acceptable
centrate on scientific reasoning skills scientific and quantitative reasoning performance of our graduating biology
(see Table 1, skill objectives 110), but objectives of the General Education seniors on items that mapped to the
they also include objectives related to program (Sundre 2008; Sundre, Thelk, skill objectives. Results from these en-
effective communication skills (see and Wigtil 2008). Third, NW-9 items deavors allow us to (1) evaluate senior
Table 1, skill objectives 1114) and do not test specific content knowledge. biology major students performance
the ability to use quantitative reasoning Rather, many of the items provide on the mapped items; (2) determine
skills to analyze biological phenomena content necessary to determine the whether students fell below, met, or
(see Table 1, skill objectives 7 and 14). answer (see Figure 1a), whereas other exceeded faculty standards; and (3)
Assessment of the skill objectives is items test concepts that do not rely on discuss NW-9 assessment results at
based on the results of two instruments,
a modified version of the Academic TABLE 1
Skills Inventory (ASI; Kruger and Comparison of biology major skill objectives (N = 14) with General
Zechmeister 2001) and the NW-9. The Education Cluster 3 objectives (N = 7).
ASI differs from the NW-9 instrument
in that the ASI asks students to report Biology major skill objectives
11. Discriminate between association and causation, and identify the types of
their experience level with a variety evidence used to establish causation.
of academic skills, whereas the NW-9 12. Formulate a hypothesis and identify relevant variables necessary to test that
instrument directly measures skill hypothesis.
level. Results from the ASI indicate 13. Design and execute experiments to test hypotheses.
14. Obtain data.
that students self-report behavioral 15. Organize data.
gains in skills associated with written 16. Analyze and interpret data.
and oral communication, research 17. Evaluate a statement, hypothesis, or claim using numerical or other evidence.
methodology, and statistics (Seifert et 18. Locate sources of scientific information.
19. Evaluate the reliability of sources.
al. 2009). Although the ASI provides 10. Critically evaluate a paper from the primary scientific literature.
insights regarding how well graduates 11. Use effective professional communication in posters.
of the biology major achieve some of 12. Use effective professional communication in lab reports.
the skill objectives, the NW-9 exam 13. Use effective professional communication in oral reports.
14. Use mathematics to understand and analyze biological phenomena.
provides a more direct measurement
of scientific and quantitative reason- General Education Cluster 3 objectives
ing skills. 1. Describe the methods of inquiry that lead to mathematical truth and scientific
Although the NW-9 instrument was knowledge and be able to distinguish science from pseudoscience.
2. Use theories and models as unifying principles that help us understand natural
designed to assess the General Edu- phenomena and make predictions.
cation learning objectives, there are 3. Recognize the interdependence of applied research, basic research, and
many features of NW-9 that suggest technology, and how they affect society.
this instrument will provide meaning- 4. Illustrate the interdependence between developments in science and social and
ethical issues.
ful data to assess the skill objectives 5. Use graphical, symbolic, and numerical methods to analyze, organize, and
of the biology major. First, many of interpret natural phenomena.
the General Education objectives are 6. Discriminate between association and causation, and identify the types of
similar to the biology major skill ob- evidence used to establish causation.
7. Formulate hypotheses, identify relevant variables, and design experiments to test
jectives. For example, skill objectives hypotheses.
7, 9, and 10 and General Education 8. Evaluate the credibility, use, and misuse of scientific and mathematical
objective 8 both discuss the ability of information in scientific developments and public-policy issues.
students to evaluate scientific sources,
FIGURE 1
Two examples of NW-9 items: (a) question that requires students to demonstrate proficiency in more than one
skill, and (b) question that assesses the ability of students to interpret data.
(a) Regarding the two graphical displays given below, which of the following statements is correct?
(b) Suppose a researcher wants to test the hypothesis that exposure to cadmium in childhood causes neurological damage
that reduces IQ. The researcher randomly selects 500 fourth graders, monitors their cadmium exposure for one year, and
then tests each students IQ. The researcher finds that as cadmium exposure increases, IQ declines. Can the researcher con-
clude from the observed association between cadmium exposure and intelligence that cadmium causes reduced IQ?
a. No. The researcher did not include enough persons in the study.
b. No. There may be a third variable associated with exposure to cadmium that actually causes the lowered IQ.
c. Yes. The researcher followed the scientific method.
d. Yes. An association between the amount of cadmium exposure and lowered IQ is exactly what we would predict from the
hypothesis.
20
Journal of College Science Teaching
Closing the Loop
each skill objective to provide greater Determining student the faculty standard, then students
interpretive power regarding student performance on NW-9 did not meet the faculty standards.
results (Maurer et al. 1991). The An- We administered the NW-9 instru-
goff method provides a quantitative ment to 214 graduating seniors (88
Results
benchmark to determine whether in 2008 and 126 in 2009). The mean
Content alignment of NW-9
graduating seniors are meeting fac- student scores on the suite of ques-
items to the skill objectives
ulty expectations. Biology faculty tions corresponding to each of the The stringent content alignment ac-
members (n = 15) who had no knowl- seven skill objectives were calcu- tivity we utilized revealed that 25 of
edge of student test performance ex- lated and transformed to the percent- the 66 items strongly mapped to 7 of
amined each of the NW-9 items that age correct. For each objective, the the 14 skill objectives. The objectives
mapped to the skill objectives. The faculty standards were compared for which items were successfully
faculty volunteers were asked to pro- with the performance of the gradu- aligned relate to distinguishing as-
vide a judgment of the percentage ating seniors using a Mann-Whitney sociation from causation, formulating
of graduating biology majors who U nonparametric test with sequen- and evaluating hypotheses, designing
should provide a correct response for tial Bonferroni post hoc analysis experiments, analyzing and interpret-
each item. During this exercise, facul- (see Table 2). Cohens d was used ing data, and using mathematics to
ty members were asked to not discuss to determine effect size. If the mean understand biological phenomena (see
their ratings until after completion of student score for an objective was Table 2). We found that multiple items
the entire exercise. Following Angoff significantly higher than the faculty were assigned to each of these seven
methodologies, faculty ratings for standard, students exceeded the fac- objectives. However, using the estab-
each item were grouped, on the basis ulty standard for that objective. If lished criteria, there were no items that
of the mapping data, to the appropri- the mean student scores were not mapped to skill objectives relating to
ate skill objectives. The mean of the significantly different from the fac- obtaining data; organizing data; locat-
scores for each skill objective repre- ulty standard, then students met the ing sources of scientific information;
sents the faculty standard for student faculty standard. If the mean student evaluating the reliability of sources;
success (see Table 2). score was significantly lower than critically evaluating a paper from the
TABLE 2
Number of NW-9 items mapped, faculty standard, and student performance for six skill objectives.
22
Journal of College Science Teaching
Closing the Loop
We found that the NW-9 exam can we created a customized process of formal reasoning. Journal of
be used to assess many of the JMU that we can use, as a department, to Research in Science Teaching 15 (1):
Biology Department skill objectives, analyze student performance in the 1114.
which are most likely similar to the areas of scientific and quantitative Martone, A., and S.G. Sireci. 2009.
objectives other Biology Departments reasoning. More important, we have Evaluating alignment between cur-
have for their students. Our results created a culture of assessment in our riculum, assessment, and instruction.
demonstrate that the NW-9 exam can department that reflects the goals of Review of Educational Research 79
be used to assess scientific and quanti- the curriculum, the perspective of the (4): 13321361.
tative reasoning skills in areas outside faculty, and an awareness of student Maurer, T.J., R.A. Alexander, C.M. Cal-
of the General Education curriculum. learning outcomes. This process has lahan, J.J. Bailey, and F.H. Dambrot.
Institutions interested in implementing helped us to close the loop with 1991. Methodological and psycho-
instruments, such as NW-9, should understanding and using our assess- metric issues in setting cutoff scores
map the items to their curriculum ment results. Our faculty conversa- using the Angoff method. Personal
objectives and set faculty standards, tions about assessment, our program, Psychology 44 (2): 235262.
as these will vary with student popula- and our students learning have been National Research Council (NRC).
tions, curriculum, and faculty expecta- deepened and enriched. Most impor- 2003. Biology 2010: Transforming
tions. Once student performance data tant, these results provide our faculty undergraduate education for future
is collected, faculty can identify areas with compelling evidence that the research biologists. Washington, DC:
of strength and weakness in instruc- NW-9 instrument measures many of National Academies Press.
tion and/or curriculum. the biology-major student learning Seifert, K., C.A. Hurney, C.J. Wigtil,
Overall, results from the NW-9 objectives. We were able to engage and D.L. Sundre. 2009. Using the
instrument in conjunction with the many of our faculty in the develop- academic skills inventory (ASI) to
results from the ASI (Seifert 2009) ment of a community-established assess the biology major. Assessment
suggest that the current biology expectation for student performance. Update 21 (12): 1415.
major curriculum produces students Finally, this set of student performance Sundre, D.L. 2008. The Scientific
who have met or exceeded faculty expectations gave us a new and valued Reasoning Test, Version 9 (SR-9)
expectations for most of the specified interpretive framework for our assess- test manual. Harrisonburg, VA:
curriculum skill objectives. We also ment results. n Center for Assessment and Research
noted a weak area in the curriculum Studies. www.madisonassessment.
regarding the skill of analyzing and References com/assessment-testing/scientific-
interpreting data. This suggests a need Bao, L., T. Cai, K. Koenig, K. Fang, J. reasoning-test/
for conversations to occur between Han, J. Wang, Q. Liu, et al. 2009. Sundre, D.L., A. Thelk, and C. Wigtil.
laboratory instructors in regards to Learning and scientific reasoning. 2008. The Quantitative Reasoning
this essential skill objective. Labo- Science 323 (5914): 586587. Test, Version 9 (QR-9) test manual.
ratory courses should be targeted, DAgostino, J.V., M.E. Welsh, A.D. Harrisonburg, VA: Center for Assess-
because this is where the majority of Cimetta, L.D. Falco, S. Smith, W.H. ment and Research Studies. www.
inquiry-based learning occurs, such as VanWinckle, and S.J. Powers. 2008. madisonassessment.com/assessment-
analyzing and interpreting data. This The rating and matching item- testing/quantitative-reasoning-test/
study provides a baseline measure objective alignment methods. Ap-
for the impact of the curriculum on plied Measurement in Education 21 Carol A. Hurney is an associate profes-
skill development. We will continue (1): 121. sor of biology and executive director of
to monitor our assessment results to Howard Hughes Medical Institute. the Center for Faculty Innovation, Jus-
measure the impact of changes we 1996. Beyond Bio 101: The trans- tin Brown is an assistant professor of
implement in laboratory courses to formation of undergraduate biology biology, Heather Peckham Griscom
see if these changes increase student education. Chevy Chase, MD: How- (griscohp@jmu.edu) is an associate
skill in data analysis. ard Hughes Medical Institute. www. professor of biology, and Erika Kancler
One of the most significant out- hhmi.org/BeyondBio101 is an assistant professor of biology, all at
comes we observed as we imple- Kruger, D.J., and E.B. Zechmeister. James Madison University (JMU) in Har-
mented our assessment design was 2001. A skills-experience inventory risonburg, Virginia. Clifton J. Wigtil is
an increase in faculty participation for the undergraduate psychology graduate student in gifted education at
and interest in the assessment process major. Teaching of Psychology 28 Purdue University. Donna Sundre is a
and student results. By involving (4): 249253. professor of psychology and the execu-
biology faculty in the content align- Lawson, A.E. 1978. The development tive director of the Center for Assessment
ment and standard setting activities, and validation of a classroom test and Research Studies at JMU.