Automated Writing Evaluation of Justifications.

Jos Rafael Hernndez Tadeo

Advisor: Dr. Aurelio Lpez Lpez
INAOE

Abstract

Prewrite

The justification section of an academic paper

should cover certain parts and content to be
considered complete. The parts are called
components and describe why its important or
who will benefit from the research. This
research proposes a method to classify if
these categories are found within a
justification.

Submit

Introduction

An academic paper should express why a

study is important and demonstrate why it
should be performed.
I propose to research a method to analyze if a
justification can be considered complete by
examining if components are located in a
text.
Components are defined as follows [UPH,
2009]:
I. Importance
II. Necesity.
III. Benefits.
IV. Convenience.

Related Work

Edit

Revise

General objective:
Design a method that uses text features that
allow to identify key components in a
justification of an academic text.
Build a tagged corpus with each component
to analyze.
Examine and define text features to extract.
Examine and define features based on
dictionaries.
Design a plan to evaluate the utility of each
component

( + )( + )( + )( + )

RNNLM

PROM

NBM

Lema

0.7666

0.7727

0.7727 0.130435

Lema+
POS

0.7492

0.7727

0.7650

Table 1. Weighted f-measure for importance.

SRILM

RNNLM

PROM

NBM

Lema

0.2804

0.2582

0.2577 -0.26015

Lema+
POS

0.3113

0.2577

0.3113

Table 2. Matthew correlation coeff. for importance.

Expected Results

Methodology
Evaluation will be made on sentence
level.
The corpus will be tagged manually for
at least 2 experts and agreement
measures will be applied.
Selected features will be extracted for
each sentence.
A classifier for each component will be
trained.
Feedback will be delivered.

A method that uses a subset of the features

proposals to reliably identify the components
of a justification in an academic paper

Timetable

References

Preliminary Results

The first drafts of a paper tend to make many

mistakes or need multiple revisions due to
inexperience or lack of information of students,
this work proposes a tool to facilitate the
writing process, specifically the justification
section.

SRILM

Getting feedback can be beneficial in

learning to write in a foreign language.
[Wang, 2013]
Most of the work in AWE is focused on
essay scoring. [Elliot et al., 2013]
Discourse analysis can be used to identify
thesis and conclusion sentences in essays.
[Burstein and Marcu, 2013]
[Relan et al., 2013] proposes POS tagging
and readability measures to identify
sentences to improve.
Weak sentences are identified with Lexical
Richness and other measures. [Gonzalez et
al., 2014]

Problematic

Draft

Diagram 1. Writing Process

Key Words: Text classification, automated

writing evaluation, academic writing, text
features

A preliminary test was conducted with the use

of language models, which assign a probability
to a sequence of words and were compared
against a bag of words model and a Nave
Bayes Multinomial classifier.

Burstein, J., & Marcu, D. (2003). A machine learning approach

for identification of thesis and conclusion statements in student
essays. Computers and the Humanities, 37(4), 455467.
Elliot, N., Gere, A. R., Gibson, G., Toth, C., Whithaus, C., &
Presswood, A. (2013). Uses and Lmitations of Automated
Writing Evaluation Software. WPA-CompPile Research
Bibliographies, 23(23), 126.
Lpez, S. G., Bethard, S., & Lpez-Lpez, A. (2014).
Identifying Weak Sentences in Student Drafts: A Tutoring
System. In Methodologies and Intelligent Systems for
Technology Enhanced Learning (Vol. 292, pp. 7785).
Relan, M., Khurana, S., & Singh, V. (2013). Qualitative
Evaluation and Improvement Suggestions for eBooks using
Text Analytics Algorithms. Proceedings of Second
International , 324335.
Universidad Politecnica Hispano Mexicana. (2009). Manual
para la Elaboracin de Tesisi y Trabajos de Investivacin, 1
91.
Wang, P. (2013). Can Automated Writing Evaluation Programs
Help Students Improve Their English Writing? International
Journal of Applied Linguistics & English Literature, 2(1), 6
12

Coordinacin de Ciencias Computacionales

Instituto Nacional de Astrofsica ptica y Electrnica, Luis Enrique Erro # 1, Tonantzintla, Puebla, Mxico
Apartado Postal 51 y 216, 72000 Tel: (222) 266-31-00