ETEC 500 A2 – Research Analysis – An Informative Start - Alice Wong

Citation:
Lou, S., Chou, Y., Shih, R., & Chung, C. (2017). A study of creativity in CaC[subscript 2]
steamship-derived STEM project-based learning.​ EURASIA Journal of Mathematics, Science &
Technology Education, 13(​ 6), 2387-2404.
http://www.ejmste.com/A-Study-of-Creativity-in-CaC2-Steamship-derived-STEM-Project-based-Learni
ng,66556,0,2.html

Introduction
In response to the new curriculum changes, Lou, Chou, Shih & Chung (2017) investigated the
benefits of a specific teaching and learning model, Science, Technology, Engineering and Math –
Problem Based learning framework(STEM-PBL) in a Taiwanese middle school. Their major research
question can be simplified as inquiring about the way in which STEM-PBL framework influences
student’s creative tendencies. The authors assume that if STEM-PBL teaching framework is
employed, then students will develop stronger creative tendencies.

Lou et al. (2017) aims to add to the growing literature of research about STEM learning. In the
form of a quasi-experiment, the scholars evaluated the STEM-PBL teaching approach. This research
supports and helps rationalize the use of STEM-PBL as an educator’s primary teaching method. Their
paper informs and demonstrates the use of STEM-PBL. The results generated from this paper will
help support the goal of cultivating creativity as part of current curriculum.

After careful reading, this paper may serve as a foundational study to help build upon current
understanding about STEM-PBL. Although lacking external validity, this inquiry provides rich details
about teacher and student’s perspectives of STEM-PBL. If issues about validity and the lack of a
visible comparison group are resolved, the authors can make a more convincing claim about the
positive increase in creative tendencies with the use of STEM-PBL framework.

Research Analysis

Research Design
Notably, this quasi-experiment involves one experimental group(i.e. 60 grade 9 students) and
measured their levels of creative tendencies before and after they engaged in problem based learning
(PBL) method. As a problem-based study, two sources of data were consulted for their analysis.
Specifically, the authors choose both quantitative (i.e. scores from the Creative Tendency Scale) and
qualitative (i.e. interview responses and teacher’s observations) methods. Suter (2012) claims that
mixed methods reveals a more complete understanding about the constructs of interest. It is apparent
that the use of interview excerpts foster stronger understandings about the effects of STEM-PBL. For
example, students at the correction phrase clearly value other’s input. Participants claim that “[t]he
comments of experts and fellow students shed light on the blind spots of our design. Proper correction
made our creatively designed works perfect.” (Lou et al., 2017, p.2393)

Variables
The authors created the independent variable, STEM-PBL teaching model. STEM-PBL is
operationally defined as the specific course that the authors developed based on “the five stages of
STEM project-based learning mentioned in the literature review—preparation (for one week),
implementation (for two weeks), presentation (for one week), evaluation (for one week) and correction
(for one week)” (Lou et al., 2017, p. 2392).

Likewise, the scale used to evaluate the dependent variable, affective creative tendency
score is also developed by these scholars. The dated ‘Creativity Tendency Scale” was adapted from
Williams Creativity Assessment Packet revised by Lin & Wang (1994). These self-developed
measures may be vulnerable to rater and researcher biases(Suter, 2012). Nonetheless, the authors
assessed the scale and found that the scale items are correlated and the test-retest reliability are
statistically significant. Yet, no additional information such as sample questions were available in the
appendix.

Beyond being assessed with a scale, students also share observations and comments
through a structured interview. This qualitative data provides rich information about creativity.
However, quite possibly, participants who provide self-reported information may be less inclined to
reveal negative feelings about learning in order to appeal to the investigators.

Control procedures
In Lou et al.‘s (2017) study, the use of control techniques is lacking. Interestingly, the authors
have yet to explicitly state the way in which the scale was used (e.g. the number of raters). Although
the test items did correlate, and that the test-retest reliability is acceptable, the validity of the measure
is still questionable. Another area of doubt is that there was no control involved with possible attribute
(i.e. gender) or other extraneous variables like student’s motivation to complete the project.
Additionally, confounding variables like prior exposure to STEM-PBL learning styles may also
influence affective creative tendencies. Without strict control strategies, both internal validity and
external validity are low.

Nonetheless, the authors attempted to increase the validity of the sample. Although lacking a
control group, the use of the pre-test and post-test model allows the researchers to establish a
baseline to compare changes. Additionally, the authors completed the pair-sample t-test in
accordance to the match participant research design—i.e. purposive sampling. The authors simply
matched participants together in similar groups based on the student’s academic achievement. Lou et
al.’s (2017) choice of the paired t test is consistent with Suter (2012) claims. Specifically, when varied
groupings emerge, specific research analytical techniques should be employed.

Reliability
As mentioned previously, the researchers investigated the test-retest reliability of the
creativity tendency scale with a different sample. They found that the test was reliable half of the time
or better. Specifically, “the correlation coefficients ranged between .489 and .810” (Lou et al., 2017,
p.2393). Due to the fact that the same scale was used pre and post treatment, answers may not be
reliable due to prior exposure (i.e. carryover effects). In this case, students may be more creative in
the later tests.

Validity
Threats to both internal and external validity have not been fully neutralized. Although
investigators assessed test’s validity and found statically significant scores that “ranged between .502
and .588” (Lou et al., 2017, p.2393). This may seem convincing at first, however, in actuality,
statistically speaking, it is not surprising that the questions are valid roughly half the time. Additionally,
not only is the scale dated (i.e. developed in 1961), originally, the test was developed in non-Asian
context, thus the assessment may not be relevant for Taiwanese students. Hence, readers may not
be entirely convinced that this test is valid to fulfill the purpose of the study.

As mentioned above, external validity is also low. First, using a specific population group
limits this study’s reliability and validity. Also, without the process of random assignment, the scholars
may not be able to generalize their observations about the benefits of STEM to the public. Perhaps
the current findings exist due to Type I errors, where scholars obtained favourable results(i.e. reject
the null hypothesis) due to a misrepresentational sample. Suter (2012) suggests that these errors may
not be visible for researchers.
Contribution significance
This study may coexist with other studies as a foundational piece of scholarly work that
assesses soft skills like creativity and STEM-PBL learning. More importantly, it is a timely study that
also has societal impacts. Since the Taiwanese curriculum recently added STEM-PBL framework to
their curriculum, the study’s findings may serve as professional development tools for current
educators. Additionally, since STEM differs from culturally traditional teaching and learning practices,
this research may help advertise and enhance teacher’s acceptance of new curriculum changes.

Strengths & Weaknesses

Strengths
This paper positively contributes to field of education. There are specific benefits to
STEM-PBL educational practices. For example, it provides detailed information about implementing
STEM-PBL. This paper encourages current teaching and administrative staff to adopt recent
curriculum changes. Since Lou et al.’s (2017) paper progresses through different stages of PBL,
educators can follow the descriptions in order to implement STEM-PBL. The deliberate inclusion of
quotes enriches the reader’s understanding of the students’ perspectives and highlights areas of
consideration.

The study also may contribute to the current research community as a pioneer study. This
study helps fill the current knowledge gap about the effects of STEM curriculum. Also, the qualitative
data collected may inspire new research questions and directions. Hence, even with its flaws, this
study makes an informative paper to include for a literature review.

Weaknesses
Visibly, there are design flaws in this study. First, a comparison group is lacking. Scholars
may claim that there is positive growth in creative tendencies with PBL. However, it is difficult for
scholars to make convincing claims about how this growth compares with changes through maturation
and or with changes students experience when taught with traditional learning styles. Gallagher &
Grimm (2018) share similar conclusions as Lou et al. (2017), however, they found that there is an
interaction and main effect between time and creativity. Likewise, the specific conditions of this study
(i.e. sample and treatment) have made the conclusions less generalizable. Attribute variables like
academic achievements may influence the development of creativity tendencies. Stolaki &
Economides (2018) argue that creativity shows positive links to academic performance. Also, the
validity of the test and its ability to test the corresponding construct creativity is questionable.

Moreover, this study has little impact on current field of research. This paper falls short of
ground breaking insights. The author claim that the use of STEM-PBL framework increases
crreativitive tendencies. Additionally, one of their key findings include obvious conclusions.
Specifically, the scholars claim that with the direct teaching of STEM, students are better at using
STEM methods to learn. Similarly, as mentioned previously, the lack of a comparison group
decreases the study’s external validity. Consequently, it is difficult for the authors to generalize their
findings.

Additionally, the authors should have taken advantage of the qualitative data. The teacher
observations and participant’s interview responses may require additional analysis. Specifically, the
emerging themes from participant’s interview responses may provide additional insights about the
relationship between creativity and STEM-PBL model. Moreover, the scholars fail to includes critique
about their study’s shortcomings. The study also lacks descriptions about potentially new directions
and plausible next steps.

Conclusion

In general, this study provides an informative base for scholars investigating the ways in
which STEM-PBL effects students’ creative tendencies. Lou et al.’s (2017) study has limited external
validity and lacks tight controls. Scholars may want to repeat the study employing a revised design
that includes control and experimental groups. To strengthen their ability to generalize their findings,
scholars can randomly assign participants to different conditions – i.e. STEM-PBL or traditional
teaching methods. This will help eliminate alternative hypothesises that involve variables such as
attribute variables (i.e. age, academic achievement, learning styles, or prior experience with STEM
etc.). A factorial design will also help detect other plausible influences such as interactions and main
effects. Control strategies such as blinding and random assignment may decrease biases from
expectancy and help increase external validity. With these revisions, the study will also change from a
quasi-experimental to a true experiment. Findings from this revised design will have more persuasive
power to reject the null hypothesis.

Currently, it is appropriate for this study to claim that affective creativity tendencies may be
enhanced through the use of STEM-PBL curricula for grade 9 students, yet these findings cannot be
generalized. The two constructs, STEM-PBL and creativity, are correlated at best, yet with the
purposeful revisions, scholars can make more convincing assumptions about how the constructs
relate to each other. Collectively, this study qualifies as a piece of preliminary informative work that
may be used to develop new research projects. As STEM learning is a relatively new topic, there is
still room for improvements. Thus, scholars should read this paper with healthy amounts of criticism
and careful inspection.

Reference

Gallagher, D., & Grimm, L. g. (2018). Making an impact: The effects of game making on creativity and
spatial processing. ​Thinking Skills & Creativity,​ 28138-149. doi:10.1016/j.tsc.2018.05.001

Huang, P., Peng, S., Chen, H., Tseng, L., & Hsu, L. (2017). The relative influences of domain
knowledge and domain-general divergent thinking on scientific creativity and mathematical creativity.
Thinking Skills And Creativity,​ 251-9. doi:10.1016/j.tsc.2017.06.001

Lou, S., Chou, Y., Shih, R., & Chung, C. (2017). A study of creativity in CaC[subscript 2]
steamship-derived STEM project-based learning.​ EURASIA Journal of Mathematics, Science &
Technology Education, 13​(6), 2387-2404.

Stolaki, A., & Economides, A. A. (2018). The Creativity Challenge Game: An educational intervention
for creativity enhancement with the integration of Information and Communication Technologies
(ICTs). ​Computers & Education,​ 123195-211. doi:10.1016/j.compedu.2018.05.009

Suter, W. N. (2012). ​In Introduction to educational research: A critical thinking approach​ (2nd ed.).
Thousand Oaks, California: SAGE Publications.