Running head: ARTICLE CRITIQUE

Article Critique: Motivation and Outcomes for University Students in a

Restorative Justice Program
Nolan Theodore
University at Buffalo

Article Critique: Motivation and Outcomes for University Students in a

Restorative Justice Program
The following paper serves to critique Motivation and Outcomes for
University Students in a Restorative Justice Program, a quantitative study
authored by Meghan Gallagher Dahl, Peter Meagher, & Stacy Vander Velde
(2014). In essence, the study will be dissected and evaluated using Thomas
R. Blacks Profiling Sheet, found in Understanding Social Science Research
(Black, 2002). Blacks Profiling Sheet provides a rubric by which to evaluate a
quantitative studys structure, processes, and analyses. The following
abstract of the study is provided by the authors:
A restorative justice program (RJP) was developed at a large university
in the housing student conduct office. Students accused of misconduct

ARTICLE CRITIQUE
who participated in a restorative justice (RJ) conference completed
surveys regarding their motivations and perceived outcomes. Results
showed that students who were motivated to make reparations to
others had the best outcomes, yet these students were often
additionally focused on personal motivators. Students who felt
pressured to participate had fewer benefits. Implications and
limitations are discussed. (Dahl, Meagher, & Velde, 2014, p. 364)
Type of Study
Dahl, Meagher, & Veldes (2014) study is called an ex post facto study.
While the authors use descriptive statistics to learn about the participants in
their study, they primarily examine the group of student responses to each
motivators to the overall student response to each outcome item. Using
parametric tests, such as t tests, and non-parametric tests, such as ChiSquare tests, the authors search to find statistical differences between these
groups. To be clear, the authors did not introduce experimental variables.
Therefore, the authors focus on finding statistical differences between
groups
Questions/ Hypotheses
Dahl, Meagher, & Velde (2014) state the following questions as having
driven their study: What motivated students to participate in RJ? Did these
offenders benefit from the program? If so, how were motivations and
outcomes linked? (2014, p. 364, para. 2). However, the authors further
explain that the question they intend to investigate with their study is How
does the motivation of each student participant interact with the potential
outcomes of the program? (p. 366, para. 1). The posed question was

ARTICLE CRITIQUE
directly followed by the authors hypothesis, stating: We hypothesized
participants primarily motivated to participate for selfless reasons would
achieve better outcomes than those who were motivated by selfish concerns,
such as expunging a record (p. 366, para. 1). Prior to Dahl, Meagher, &
Veldes question and hypothesis were an exploration of the body of research
surrounding student conduct programs in higher education, student
outcomes after the student conduct process, prior use of RJ on college
campuses, offender satisfaction after undergoing RJP, and recidivism (p.
365). The authors review of the literature was thorough, relevant, and
supported the studys investigation into the authors explicitly stated
question. Blacks Profiling Sheet indicates that these researchers study had a
valid question and hypothesis based on accepted theory with well-justified
referenced support (2002).
Representativeness
Dahl, Meagher, & Velde (2014) state that their study aims to examine
the effect RJP has on university student participants (p. 366, para. 1). The
authors indicate that at a large research institution, students with cases of
misconduct could be referred to the restorative justice conference program
at the hearing officer's discretion (p. 366, para. 2).These students would then
be provided a survey to complete by the researchers, the instrument by
which data gathered from the participants (p. 366, para. 3). It is later said
that of the 220 students that completed the conference process during the
designated university semesters, 191 completed surveys were returned (p.
366, para. 5). The authors later stated: "Due to the nature of the referral

ARTICLE CRITIQUE
process, the sample was a population study as it includes student
respondents who participated during these academic terms (p. 366, para.
5). Of the completed surveys, 64.9% of students identified as Caucasian,
12.6% identified as Asian, Indian, or Indo-American, and 6.8% identified as
African-American or Black, with approximately half of all respondents
identifying as male (54.5%) (p. 366, para. 5). The researchers also found that
87.4% of respondents were first-year students with a mean age of 18.4 years
and a standard deviation of 0.6 years (page 366, para. 5). Lastly, the
researchers state that The highest reported major among participants was
Undecided (26.4%) (p. 366, para. 5). No data was gathered surrounding the
demographics of the student body at aforementioned Midwest research
institution, though such thoroughness would have been appreciated.
In terms of representativeness, all students were indeed current
university students that had completed the restorative conference program.
Since each student that completed the restorative conference program at
this institution was given the option to complete the survey (p. 366, para. 3)
and the researchers question confined their population to university student
participants of this specific program, it can be said that this study was
representative of the whole population, as indicative of Blacks Profiling
Sheet. Nonetheless, it should be mentioned that how the students are
chosen for the restorative justice program calls into question how the
population is formulated. This will be addressed further in the following
section.
Ethics and Confidentiality

ARTICLE CRITIQUE
Dahl, Meagher, & Velde (2014) state that each student was "informed
their participation was completely voluntary, anonymous, and would not
affect the outcome of their conduct cases" (p. 366, para. 3). Students were
also asked to complete a consent form when completing the survey (p. 366,
para. 3). However, it should be noted that two trained peers of the student
participants facilitated each restorative conference (p. 366, para. 2). This
could indicate some breach of confidentiality as the reader is not informed of
what degree of confidentiality the student facilitators are held to when
coming face-to-face with a peer in the restorative conference program, as
opposed to university staff. Of even more importance is the fact that the
reader is not informed of how students are sorted into the restorative
conference program. The reader is made aware that the hearing officer can
sort students into the restorative justice program at their discretion (p. 366,
para. 2), but on what grounds? Is it based on the severity of the offense, the
personality of the student? To be frank, it is the responsibility of the authors
of quantitative studies to thoroughly address issues of ethics and
confidentiality.
Lastly, the authors do not explicitly state which alpha level they will be
using to determine statistical significance. Instead, they use asterisks in their
graphed data to indicate which tests found statistical significance at levels of
p < .05, p < .01, and p < .001, as found in Table 2 (p. 369). Researchers
must decide upon which alpha level they will determine statistical
significance at in order to maintain ethical standards. The study, as
presented, does not indicate a set alpha level, leaving the reader questioning

ARTICLE CRITIQUE
the researchers motives. After all, without a set of alpha level, researchers
may have shifted their alpha levels in an attempt to falsely generate
statistical significance (e.g. researchers shifting alpha level from p < .01 to p
< .05 in order to deem more tests as finding statistical significance).
Unfortunately, Dahl, Meagher, & Velde fall short of addressing ethics and
confidentiality within their study adequately, which thereby categorizes the
study as having ethical issues not addressed or confidentiality not discussed
when it should have been on Blacks Profiling Sheet.
Data Quality I
Dahl, Meagher, & Veldes (2014) survey first asked students six
questions regarding programs. They are described as follows:
Outcome 1 (O1): How satisfied were you with how your restorative
conference went?
O2: Do you think you benefited from meeting with harm party of
parties?
O3: How likely do you think it is that you will commit another violation
like this one?
O4: I feel participating in the restorative conference strengthened my
sense of community at the University.
O5: Has your attitude toward the harmed party changed since prior to
participating in the restorative conference?
O6: If you had to do it again, would you choose to participate in the
restorative conference? (p. 366-367, para. 1).
In order to learn how outcomes were associated with motivations for
choosing the restorative conference, the authors then asked students to
indicate which motivations influenced their decision to participate in the
restorative conference program from a list of nine sub-items. They are
described as follows:

ARTICLE CRITIQUE
Motivation 1 (M1): To pay back the harmed party for their losses
M2: To help the harmed party
M3: To take direct responsibility for making things right
M4: To let the harmed party know why I did it
M5: To offer an apology
M6: To satisfy my parents
M7: To remove the offense from my record
M8: It was the best option of available choices
M9: I felt pressured to participate (p. 367, para. 1)
While there are many variables to analyze in this study, the authors are able
to clearly state the process by which they will be analyzing their data. Dahl,
Meagher, & Velde state that they will compare each motivator to the
outcome variables in order to determine any group difference, followed by a
hierarchical cluster analysis in order to see if any patterns exist in their
motivation sub-item data, and ending with the use of said clusters as
grouping variables in comparison to outcome measures (p. 367, para. 2).
Furthermore, the outcome and motivation items are clearly linked to the
researchers previous literature review on recidivism and offender
satisfaction, making the quality of the data significant to the larger body of
research in this area. Thus, it can be said that this study would be classified
as being Educationally, sociologically, psychologically, etc., significant and
having manageable number of concepts by Blacks Profiling Sheet (2001).
Data Quality II
The researchers explicitly state that this study had not been tested for
validity or reliability prior to the distribution of the survey was sued (p. 367,
para. 2). No mention is made about testing for objectivity though the authors
stated that the survey was written for college-level comprehension and that
the survey was piloted by current students (p. 366, para. 4). Additionally, it
can be assumed that the survey was designed by the authors, as no mention

ARTICLE CRITIQUE
of a professional or commercial source is made. Because the instrument
used to gather data was produced by the authors and no tests were made to
assess validity, reliability, or objectivity, Black's Profiling Sheet (2001)
classifies the studys instrument as commercially or project produced with
low V, R, O, or no other information provided.
Descriptive Statistics
Due to the multiple variables being assessed in terms of
responses to both motivations and outcomes, a graph showing the
central tendency and spread for each variable (i.e. spread for each
person that checked off M7) would be an unwise use of time. Thus, the
authors decision to use a relative frequency graph to display the
frequency of which the nine motivators were checked off by
respondents is ideal. Thanks to Table 1 (p. 368), the reader is able to
see general findings about the population in focus, such as the highest
motivators for being a part of the restorative conference program were
M7 (81.7%), M3 (78.0%), and M5 (67.5%). However, the authors rush to
begin independent t tests to compare the frequency of each motivator
to each outcome. A presentation of the central tendency and spread of
data found on student responses on the outcome variable wouldve
been appreciated and been less jarring for the reader though these data
are stated in the article for each outcome (p. 368-371).
The authors also used a cluster analysis in the article in order to
find patterns in the response of the students motivations. Considering
that the motivation sub-items are nominal variables, the authors would
be expected to describe the contents of each cluster based on the nine

ARTICLE CRITIQUE
motivators using a bar chart. Fortunately, each of the four clusters the
authors use to describe patterns in student responses to motivation are
accompanied by a bar graph (p. 372-373). The bar graph for each
cluster describes what percent of students within the cluster responded
to each of the nine motivators, accompanied by then of each cluster.
While the authors chose not to graph the outcome variable data within
the article and instead included the data in the text, each of the graphs
presented were accurate and appropriate. Thus, Blacks Profiling Sheet
would state that the study had an appropriate display of data and
results in tables and/or graphs, clearly labeled.
Inferential Statistics
Dahl, Meagher, and Velde (2014) compared the frequency of the
nine motivation items to the six outcome measures by using
independent sample t tests and, in the final two outcome items, ChiSquare tests. This resulted in 36 t tests results, of which the authors
examined significant difference in indicated motivators within each of
the six outcomes. For instance, for O3, t-tests conducted by comparing
the mean response on the designed Likert scale for all of those who
checked "yes" for O3 to the nine motivators in order to find significant
differences in each motivator group. By doing so, the authors were able
to find statistically significant difference in the group of students who
checked yes to M2, M3, M4, M5, and M9 when compared to all
students that participated in the study (p. 369, para. 1-2). This was
appropriate as using this design allowed the authors to determine how

ARTICLE CRITIQUE
each individual motivator compared to the population in each individual
outcome, find which motivator groups differed from the population, and
draw inferences about why those groups were significantly different.
Therefore, the results of the independent sample t tests in O3 allowed
the researchers to infer that the restorative motivators of M2, M3, M4,
M5 appeared to
be more committed to avoid reoffending than those who did not
select those motivations. In addition, as a group, those who felt
pressured to participate (M9) reported they would be more likely
to reoffend than those who did not feel that pressure. (p. 369,
para. 2)
Because the O5 and O6 were structured as "yes-or-no" questions,
researchers were required to compare differences in proportions
between groups as opposed to group means in order to draw inferences.
Thus, the researchers performed a Chi-Square test to compare the
proportions of those who checked "yes" of O5 and O6 to the motivator
groups. Though it is not explicitly stated, this is a Chi-Square Test of
Independence, as this test requires two samples in order to determine if
there is significant difference.
The researchers then used a cluster analysis to determine if the
patterns found in responses by allowed the students to be grouped in
such a way that each cluster would be significantly different from the
populations outcome in both mean and proportion. Clusters were
designed based on patterns in student response and given identifying
labels to infer what the prime motivators are in each cluster (e.g.

ARTICLE CRITIQUE
Cluster 1 as self-focused/ few motivators, Cluster 2 as self and
others) From Table 4, the authors found that each of the clusters they
had formed were found to be significantly different from the population
in terms of mean response to O1, O2, O3, and O4, by using an ANOVA
(p. 375). However, Table 5 indicates that the difference in proportions
between the clusters and the population to answering yes to O5 and
O6 was only statistically significant for O5 by using a Pearson ChiSquare test. Post-hoc tests were performed in order to determine
specific group differences between clusters within each outcome item
as well though it would have been helpful to add what type of post-hoc
analyses are being done.
In conclusion, the researchers appropriately used parametric tests
and non-parametric tests in order to answer their original question of
"How does the motivation of each student participant interact with the
potential outcomes of the program?, and the types of variables
corresponded correctly with the tests the authors used. To be specific,
the nominal v. interval variables correctly corresponded with the t tests
and the nominal v. nominal intervals corresponded correctly with the
Chi Square tests. Thus, Blacks Profiling Sheet would state that this
study had an appropriate choice of design and statistical tests for
resolving the null hypothesis (2002). However, it should be said that
the authors should have defended why they were using the Likert scale
as an interval variable, as it can be misconstrued as to whether data
from a Likert scale is an ordinal variable. Had they left the Likert scale

ARTICLE CRITIQUE
analyzed as an ordinal variable, the same type of tests could not have
been used. It would have also been additionally helpful to have given a
greater explanation of how the clusters were formed beyond what is
stated in the text (p. 371, para. 3)
Variable Control
On page 367 of their journal article, Dahl, Meagher, and Velde
(2014) take ample time to provide an exhaustive list of the limitations to
their study (para. 4-6). One important limitation to the study is the fact
that only post-test surveys were provided to students, which suggests
that it is possible for the original motivations of the students to
participate in the restorative conference program may have been
different from after the restorative conference program. This leaves
room for the confounding of data, as students may have forgotten their
original motivations or changed their original motivations in order to
appear. The researchers go so far as to admit that the outcomes could
not be determined as a result of the restorative conference session
(para. 2), which is most likely due to the lack of a control group and lack
of a pre-test. Even more, the researchers state that they had reason to
believe that the reverse-scoring of the first and second outcome
questions confounded their data as well (para. 3). Lastly, the
researchers failed to address how the hearing officer may refer cases
to the restorative conference program and how that decision to offer
this option is made.
Determining the variable control of this study becomes difficult as
the instrument used did not get tested for validity. This study has little-

ARTICLE CRITIQUE
to-none external validity, as making the results of this study
generalizable to a larger population would be hazardous due to the
many opportunities for confounded data. Internal validity has not been
measured, therefore it should not be assumed that the test actually
measures what it has been designed to measure. Take for instance O5,
which asks respondents Has your attitude toward the harmed party
changed since prior to participating in the restorative conference? This
question is largely ambiguous, as the question is non-directional in
nature. That is to say, should a student answer yes to this question,
the researcher should have no basis to say whether this is positive or
negative in change, as the student could have greater negative feelings
towards the harmed party. Yet, this does not stop the authors from
stating that a yes response to this question meant this was a
beneficial outcome (p. 376, para.6), thereby assuming that any
change in attitude is positive. Because there is a lack of control in both
the mediating and extraneous variables, including a lack of considering
extraneous variables, Blacks Profiling Sheet classifies this studys
variable control as having extraneous variables not considered, and that
confounding of data is likely.
Analysis and Conclusions
It cannot be helped that a better choice of design would have
been of greater value. A design with a control group of, for instance,
students that attend administrative hearings to test how restorative
conference sessions may affect students would be of great importance.

ARTICLE CRITIQUE
Additionally, a pre-test and post-test would allow for researchers to
compare how students felt about their misconduct, the harmed party,
and the community prior to the restorative conference session before
and after, thereby providing the opportunity to draw inferences about
any significant changes between the tests. Even more, testing the
validity and reliability of the survey instrument would improve the
constitution of the study as well, such that other researchers could
replicate the study knowing that the instrument measured what it is
intended to measure and could, therefore, be generalized to a larger
population.
However, the authors analysis of the results indicates a clear
understanding that correlation does not necessarily mean causation. For
instance, the authors state that Both the individual sub-item analyses
and the cluster analyses indicated that satisfaction (O1) rates were
higher for students who were motivated by restorative reasons
compared to those who were motivated solely by personal goals or
external pressure (p. 376, para. 2). Consistently, the authors use
language that acknowledges that while the inferential statistics provides
greater insight into the population examined, causality cannot be
determined without further investigation. The authors also refer back to
the body of research from which their question originates when relevant
to the results found. This is evident when the authors relate the finding
that participants in their study generally benefitted from meeting with
the harmed party to previously cited research that found similar results.

ARTICLE CRITIQUE
The future directions of studying RJP advised by the authors are
wise and relevant to the findings of their study. For instance, the
authors acknowledge that they did not study the demographics of their
population (e.g. race, sex) (p. 378, para. 2). The authors explain that
such factors could play a role in how students experience restorative
conference process and that such research could inform how to make
restorative conferences more culturally appropriate. The authors also
state that the development of a prescreening instrument in regards to
evaluating if a student is community-oriented or self-oriented, going so
far as to say that such an instrument could inform practitioners on
whether a restorative conference would be appropriate (p. 378, para. 3).
This insight is especially keen, as the clusters designed around
motivators were shown to be, for the most part, statistically significant
from the general population in terms of outcome means and difference
in proportions. Therefore, Black's Profiling Sheet classifies the analysis
and conclusion of this study as having "some lack of justification" or "its
conclusions poorly defended" (2001). However, it must be stated that
this study could have obtained the desired rank of "appropriate design
for the question(s) posed and conclusions justifiably drawn from data
shown" had the researchers made serious edits to their study's design.
Conclusion
Overall, this study had qualities of quantitative research that were
both desirable and flawed. While many aspects of the study were
appropriately executed, such as the questions and hypotheses and the

ARTICLE CRITIQUE
inferential statistics, the flaws of a study endanger the quality of its
entire existence and calls into question whether its findings can
justifiably be added to the body of research from which it originates. In
the end, the lack of variable control and the absence of validity,
reliability, and objectivity tests on the studys instrument render the
findings largely inadmissible to its larger body of research.
References
Black, T. R. (2002). Understanding social science research. London, UK:
SAGE.
Dahl, M.G., Meagher, P., & Velde, S.V. (2014). Motivation and outcomes
for university students in a restorative justice program. Journal of
Student Affairs Research and Practice, 51(4), 364-379.