You are on page 1of 17

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No.

4, July 2016

A DECADE OF USING HYBRID INFERENCE


SYSTEMS IN NLP (2005 2015): A SURVEY
Ali Almiman
Department of Computer Science, University of Manchester, Manchester, UK

ABSTRACT
In todays world of digital media, connecting millions of users, large amounts of information is being
generated. These are potential mines of knowledge and could give deep insights about the trends of both
social and scientific value. However, owing to the fact that most of this is highly unstructured, we cannot
make any sense of it. Natural language processing (NLP) is a serious attempt in this direction to organise
the textual matter which is in a human understandable form (natural language) in a meaningful and
insightful way. In this, text entailment can be considered a key component in verifying or proving the
correctness or efficiency of this organisation. This paper tries to make a survey of various text entailment
methods proposed giving a comparative picture based on certain criteria like robustness and semantic
precision.

KEYWORDS
Natural Language Processing, Textual Entailment, Semantic inference, Textual inference, Textual
hypothesis,

1. INTRODUCTION
In todays world of textual information everywhere, and days of high amounts of information
being generated every day, in the form of emails, chats, discussion forums and comments on
articles, it has become a herculean task to make sense of the text and categorise them
meaningfully. And the text generated here is generated by human interactions and is in natural
language.

1.1. THE PROBLEM OF INFERENCE IN NLP


There has always been a barrier in communications between human beings and devices, ever
since the advent of computers. We solved this by creating syntactically robust languages.
However, computer still fails to understand human beings, who beyond using multiple languages
(problem of translation), make sense of a same sentence in different way in different contexts.
There is a semantic element that difficult to extract from a given text. This precisely is the
problem of Natural Language Processing. However as noted by Liddy [1], this area of research
being active and young has no single agreed-upon definition. This has drawn interest of various
researchers in the areas of computer science, artificial intelligence and linguistics equally in
deciding how to make a machine make sense of human natural language. Early attempts to
handle this problem involved direct hand-coding the rules, which in a way was very tedious. Most
of the modern ways use methods similar to the machine learning, which are highly statistical in
nature.[2]

DOI: 10.5121/ijaia.2016.7405

59

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

1.2. TEXTUAL ENTAILMENT.


In the process of Natural Language Processing, when organising texts, we make various
hypotheses which one could infer from the text. This inference has a directional relation to the
given text. It is essentially to determine if meaning of one sentence T entails the hypothesis H.
The RTE Challenge guidelines[3]
[3] defines the textual entailment in the following way - our
applied notion of textual entailment
ment is defined as a directional relationship between pairs of text
expressions, denoted by T - the entailing Text, and H - the entailed Hypothesis. We say that
T entails H if, typically, a human reading T would infer that H is most likely true.

2. THE PROBLEM
Looking at the brief history of NLP[4],
NLP , where since the advent of computers, we have been trying
ways to communicate with the machine effectively, we see an interesting trajectory of events.
Earlier a lot importance was given in extracting meaning from each word that a human utters or
seeks a computer as much information as possible from each of the words. This meaning making
or inference involved processing information at different levels ranging
ranging from syntactic, lexical,
semantic, discourse/pragmatic. But only later have we as a community started realising that this
processing of each word often is not needed, instead one must focus on extracting as much sense
from the sentence as possible as a whole. Thus the problem of NLP can be understood as
extracting as much meaning as possible from the given sentence.

Figure.1: Table showing various NLP tasks as seen as TE problem [3]

However, broadly from the early 21st century, focus has been semantic inference and the area of
Recognising Textual Entailment popularly known as RTE is a child of this pursuit. A typical
system followed for any Natural Language processing is shown below[4].
below
60

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

Figure.2: Pipeline view of a typical system of NLP [4]

Various tasks that are typically encountered include Question Answering (QA), Information
Extraction (IE), (multi-document)
document) summarization, and machine translation (MT) evaluation
[3].RTE
.RTE is proposed as a generic task capturing various earlier existing tasks of QA, IE, MT, etc.
RTE Challenge guidelines [3] specifically look for 7 such tasks namely Information Retrieval
(IR), Comparable Documents (CD), Reading Comprehension (RC), Question Answering (QA),
Information Extraction (IE), Machine Translation
Translation (MT), Paraphrase Acquisition (PP). These are
proven with some strong empirical studies and arguments[5][7].
arguments
The efficacy of any given approach is understood by 2 key measures namely accu
accuracy and a
parameter defined as Confidence-Weighted
Confidence
score (cws).
). These are also used to compare various
approaches. This is a measure to grade how well a system correctly entails the T-H
T H pair. This is
weighted based on the classification of the TE into positive
positive (T entails H), negative (T does not
entail H) and non-TE
TE (T neither entails nor contradicts H) categories[3],
categories
[8].. A correct judgement
gives higher confidence than the wrong judgements. It should work well in identifying and
explaining both successful and failed T
T-H matches. cws is score between 0 and 1 and higher the
score better is the system in meeting the requirement
requirement of RTE. A system with a good overall
cws(on all the 7 tasks IR - - PP) is deemed better at TE.

3. SOME HYBRID METHODS TO SOLVE THE ENTAILMENT PROBLEM


Textual Entailment problem was solved collectively using platforms like RTE.Ever
RTE Ever since, the
launch of the RTE as a regular annual activity; it has gained momentum in both academic circles
as well as corporate research houses, as this problem has a very high significance in linking
human understanding with the high computational power of the machines (computers).
(computers). Later,
other data sets were are provided to test the efficacy of various approaches. FraCas Suite has a
new set of data based on the formal semantics.
We shall make a review of various studies in the context of the textual entailment. The key
difference
ifference among various approaches that we discuss below lies in the structure and the steps that
are followed to arrive at a conclusion between the dependency relations the text and the
hypothesis pair. Broadly, it can be seen that they mainly employ a middle
ddle way between deep
(semantic) analysis and shallow (word overlap like) analysis. Further, it can be seen that the
semantic inferences can be of 3 kinds: propositional logic, first order predicate logic and
others[4].. We will see in the following, some of these three approaches being used in isolation or
in a mixture to accomplish the goal of RTE.

61

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

3.1. ATOMIC PROPOSITIONS [9]


One of the very early studies of the RTE age was this study, where the authors propose a textual
entailment based on atomic propositions. The text and the hypothesis pairs are firstly parsed to
extract atomic propositions. These atomic propositions are later compared using OTTER which
uses the Knowledge rules. If for every atomic proposition in H has a counterpart in T, then the
entailment holds, else the entailment breaks. This algorithm is shown in the picture below.

Figure.3: A system picture of the RTE using atomic propositions approach [9]

The performance is low with an accuracy of 0.5188 and overall cwsof


of 0.5067, which the authors
attribute to lack of good knowledge rules database. Also, more work was needed to be done on
proposition extraction and logical representation.

3.2. ABDUCTIVE THEOREM PROVING [10]


Here the text hypothesis pair is first converted to a logical representation using some already
proven conventions [11].. This translates the textual inference problem to a logical inference
problem.
em. Then various abductive assumptions are analysed using a specially defined cost model.
The assumption cost model needs to be minimised so that the entailment is more robust. Further
the theorem prover (proving the Text Hypothesis relation) is allowed to learn good assumption
costs iteratively. This among others compared gave better results. The Theorem prover which was
trained separately for each class gave the best results with an accuracy of 0.57 and a cws of 0.651,
significantly better than the simple atomic propositions approach.

3.3. COGEX [12]


COGEX was used earlier as a logic prover for a specific task of Question Answering ((QA) [11].
However,, here it is extended to the Textual entailment. Very much similar to the abductive
theorem prover, the system here starts with converting text into logical form [10][12]
[12] along with
the semantic relations. NLP axioms (used as linguistic rules) are then generated to connect the
62

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

concepts of T-H
H pairs. Some (310) knowledge axioms are also incorporated. Then WordNet is
used to create lexical chains. The above processes are used
used to define the arguments of a predicate.
This will make the logic prover, COGEX later not to miss any possible connections between the
T-H.
H. For, each case, a prover score is calculated iteratively till a refutation
refutation is reached, relaxing the
arguments of thee predicate step by step. This did very well with the CD with accuracy of 0.78 and
cws of 0.822. But with other tasks the performance was not very satisfactory. The overall
accuracy was 0.551 andcwswas
was 0.56.

3.4. MODEL BUILDING [13]


As can be seen the above studies were not giving entirely satisfactory results. This approach gives
a special focus on the semantic analysis. It performs deep semantic analysis using CGG Parser
[14] to characterise the text and hypothesis pair individually. The figure.4 below shows an
example. This is done along with several simple shallow semantic interpretations, or simple
shallow word overlaps.

Figure.4: Semantic ana


analysis of Text and Hypothesis pairs [13].

Further, model building approach if employed in contrast to the theorem proving ones, which will
handle the negation well. Here theorem prover [10] and model builder are used in tandem to
handle and account for both the negative and positive results effectively. This hybrid approach
very clearly shows an advantage over the previous studies, with an accuracy of 0.61 and cws of
0.65. The same authors have later tried with logical inferences too, with no major improvements
[15]

3.5. LCCS GROUNDHOG SYSTEM[16]


The LCCs Groundhog
roundhog system unlike the other systems is more robust and produced good results.
It is based on the classifications and lexico-semantic
lexico
information.

63

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

Figure.5: GroundHog System

In thee GroundHog system, as shown in the above schematic, the T-H


T H pair is first pre-processed
pre
and is annotated with four key sets of information (lexical, syntactic and semantic
tic parse, co
coreference, semantic). Then the system is trained with a large amount of test
test data acquired from
World Wide Web (www). Later a lexical alignment probability is computed and paraphrase
acquisition from top 500 documents of Google is done using a statistical approach similar to
parallel corpora. Then the entailment is assessed by classifying using the four features as
described in the figure.5 above (alignment, dependency, paraphrase and semantic). The
entailment results were very positive with a good accuracy (>0.652) was reported, which
increased with more training.

3.6. NATURAL LOGIC (LOGIC RUNNING OVER NATURAL LANGUAGE)[17]


As can be seen in the early examples studies running purely on lexical basis had little semantic
precision [18], [19],, but those based on the first order logic[9],
logic
[15] were less robust as we saw in
the earlier studies. Here [17],, we see a middle approach being proposed, where the logic is
overlapped over the natural language, leading to something called Natural Logic. The system
(known as NatLog)) architecture is pretty similar to the groundhog [16]] with three key stages
namely pre-processing,, alignment and
a entailment classification. This
his approach can be seen to
have produced good precision of about 68.06%, which is a significant improvement. The hybrid,
which the authors claim to be the middle path showed good accuracy and recall in both
development (74.03%,
74.03%, 69.63%) and test stages (67.32%, 63.62%). These were tested on RTE3
test sets.[17] An improved study of Natural logic was later proposed [20] later, after the next
approach bringing semantic and lexical
lexical-syntactic levels.

3.7. SEMANTIC INFERENCE AT LEXICAL-SYNTACTICAL LEVEL [21]


Semantic inference is something that is at the heart of the problem of textual entailment. However
all the previous studies have had a mixed story in term of meeting semantic criteria and some
nascent efforts can be seen to push this limit ahead slowly. Here,
Here, a new approach is proposed
altogether,, where the semantic inference is overlapped directly on the syntactical tree, so that the
robustness of the shallow lexical-syntactical
lexical
representations is taken advantage of. To do this, one
transcends from text t, to the hypothesis h,, in small incremental steps. That is the prover will try
to generate h from tthrough
through a series of parse trees. This entire series of translations of t to h is
guided by a set of entailment rules. The figure below give a picture of what a translation of the
above kind would look like.

64

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

Figure.6: (a) t to h translation through intermediate parse trees


(b)the entailment rule

A set of entailment rules like L--matching,


matching, R instantiation, Alignment copying. Other Annotation
rules, generic-linguistic rules, lexical-syntactic
lexical
rules are also manually created to guide the
process. RTE data was not directly usable here due to the small sample size of the test set.
However a different test with Base line and Proof shows a significant precision of 78.5%, which
is quiet impressive. This also opens up a new framework for integrating semantic inferences
along with lexical-syntactic
syntactic level. This very clearly reveals the flaws in the approximate
entailments allowed by RTE.

3.8. IMPROVED NATURAL LOGIC [20]


The previous study of Natural Logic [17] proposing a middle approach between the brittleness
brittle
of
semantic approaches and the robust approaches, which lack semantic precision. He the previous
work is extended to incorporate semantic exclusion and impilcativity. It is essentially similar to
the previous study using intermediate trees [21] arriving at the hypothesis in small edits. Here, a
statistical classifier is used at each edit to predict the lexical entailment and then feeds the
relations upwards to higher level edits through the syntax trees as per the semantic properties of
the intermediate nodes. In the process the entailment relation is obtained across the edits. Though
this does not answer the entire problem of Natural language inference, it definitely works very
well for the everyday patterns.. The performance on RTE3 shows improvement when
when any existing
RTE system is hybridized with the Natural logic system [20].

3.9. LOGICAL INFERENCE [22]


This is another approach that needs to be explored. Though has not proved its performance
significantly better than others, this can be considered as an important approach in giving
explanations to the entailment decisions. After the initial language processing using BLUE
BLUE[23],
[24],, the entailment is verified using WordNet and DIRT paraphrases as knowledge sets.
Following these entailment decisions, the explanations are provided. RTE4 is used aas the test set
of data. This opens
pens up a new area of providing explanations to the entailment decisions in the TE
problem.
65

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

3.10. Inference Rules[25]


Entailments rules were being used earlier studies to guide the incremental edits towards
hypothesis[21]. Here we see how inference rules can be used to improve text entailment. This
again falls into the group of TE approaches focussing on the semantic inferences. Using DIRT,
the inference rules are acquired. DIRT algorithm includes determining the inference rules from
Text. Then a few missing
ng rules are identified, and some of them are acquired from the WordNet.
A combination of DIRT
RT and WordNet is applied on the RTE data. The figure.7 below shows a
dependency tree skeleton is developed in the process.

Figure.7: Dependency structure of text. Tree Skeleton

3.11.. Distributional and Logical Semantics


Semantics[17]
This involves mixing both the conventional semantics mapping to logical representations, but
here the relational constants are given by distributional clustering
clustering at the level predicate
predicateargument. This used FraCas and has shown impressive results beyond the existing studies. The
figure.8 below shows the flow of steps in this approach. After the initial semantic analysis, the
word typing (like identifying the
the types of the nouns) is performed. Certain logical forms are
employed in this step. Then these are clustered using the distributional relations using statistics
based on semantic relations. The last step is the semantic parsing using the relation clusters
generated in the previous step. Experiments on FraCas suite shows a very impressive accuracy
rate of 89% on single premise and 80% on multiple premises. Their study shows the first order
logic approaches dont affect precision much. Also in terms of robustness
robustness of syntax, they admit
their work being inferior to the work on Natural Logic [17].

Figure.8: Flow of steps in Distributional Logical Semantics


66

International Journal of Artificial Intelligence and Applications


Applications (IJAIA), Vol. 7, No. 4, July 2016

3.12.. Dependency based Compositional Semantics (DCS)


(DCS)[26]
This work is similar to the ones trying to transform the trees in the process of reaching hypothesis
from text. DCS is a very goof framework with simple structures as well as tight semantics. This
approach merges the logical inference with the DCS giving
giving promising results both on RTE and
FraCas datasets.

Figure.9: Figure showing the RTE implementation

Though the results are not good in comparison with the earlier studies on RTE2 and RTE3
datasets, the results are promising with RTE4 and RTE5 datasets. Further,
Further on-the-fly
implementations have also shown good accuracies.

3.13. Higher Order Logic [27]


This is the beginning of the higher order logic used in TE
TE. [27]It
It is largely believed that higher
order logic is not suitable for logical inferences; this study shows some promise in this direction.
This opens up another approach to the text entailment bridging
bridgin the parser and the semantic
composition. The following picture below shows the CCG derivation of the semantic
representation of a sample sentence.

Figure.10 A CCG derivation of the semantic representation for a sentence[27]


sentence

Most of the results here very preliminary yet show promise in the fact that they out perform state
stateof the art first order logic system in the tests done on the
th FraCas datasets.

4. COMPARATIVE OBSERVATIONS
BSERV
We can see that each of the above methods have different advantages and disadvantages from the
logical point of view. Mixing up Semantic inferences with the lexical and syntactical inferences is
something that the area is seems to struggle with. The
The Table in the appendix at the end shows a
comparative overview of all the above mentioned approaches their performances quantitatively
wherever is available. Also, most of these approaches still being tested using more and more data
sets, as the knowledge base is also developing in parallel, as can be seen with the various versions
of RTEs and the likes of FraCas.
67

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016

5. FUTURE SCOPE
There has been an attempt to solve the Natural Language processing problem for many decades.
However as can be seen in the above attempts to solve text entailment, a very concerted effortis
being put a large community of researchers to make the machine process and understand natural
language.
In the case of Text entailment, many algorithms have been proposed but, most of them are still
not able to crack the problem or give a breakthrough. A revolution in the approach of solving this
problem, (paradigm shift in the lines of historian and philosopher of Science Thomas Kuhn[28],
is very badly needed. We can be more confident than ever, with the amount of computational
prowess current technological age has.
With wider applications these days, with expanding reach of the technology, thanks to the smart
phone revolution, the natural language processing is becoming a much sought after technology.
Moreover, the latest studies have also started looking at languages other than English like this
recent study in Arabic[29] Also, there are studies going on for speech recognition in the same
flow of things. So, the approaches of higher order logical inference and semantic inferences
which we ended with will be key thread to pursue in future directions.
There are two key aspects that could be realised to be hurdles to the TE problem. As shown in the
above review, the first is the algorithm. The approach has slowly transformed from a mere
syntactic one to more rigorous ones. We have stopped at the initial stages of higher order logic
inference. The second key issue is the availability of sufficient data base of the resources like
lexical resources and the more rigorous test data. Both these problems need to be handled
together. The second issue however, could be easier, in the context of everyday interaction these
days using digital media producing big data [30][32]. With more solid resources and potential
test data, a crucial breakthrough is needed in the better algorithms, the second major problem.
Attempts like Mannings Foundations of Statistical NLP [33] outline an endeavour in this very
direction to chalk out the lookout for an effective algorithm.The latest handbook of semantic
theory by Lappin[34] is another great guide book to start the pursuit of a breakthrough. Some
latest approaches in this direction are the Bi-direction LSTM model and Inter-attention[35] and
various other new models[36] addressing entailment using limited data[37] which is one of the
key issues with the NLP problem.

6. CONCLUSIONS
This area of research as we can see is a very great meeting place for the people from Artificial
intelligence, computer sciences, linguistics and philosophy. Also, practically, with the amount of
data that is being generated and the information and insight these could carry in them, makes the
problem even more relevant to todays requirements. Further, the communication with the
machine is various interesting problem in the context of the entire human evolution and the
scientific pursuit. The results of this research can be very productive in enabling intelligent
machine and the robots to be more user-friendly and have a smooth transition with the natural
world. Machines understanding the human language as it is a going to break through in the entire
history of science and technology.

68

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016

REFERENCES
[1]
[2]

[3]

[4]
[5]

[6]

[7]
[8]

[9]
[10]
[11]

[12]

[13]

[14]

[15]
[16]

[17]

[18]
[19]

E. D. Liddy, Natural language processing, 2001.


Wikipedia contributors, Natural language processing, Wikipedia, The Free Encyclopedia, 21-Jun2016.
[Online].
Available:
https://en.wikipedia.org/w/index.php?title=Natural_language_processing&oldid=726293558.
[Accessed: 24-Jun-2016].
I. Dagan, O. Glickman, and B. Magnini, The PASCAL Recognising Textual Entailment
Challenge, in Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object
Classification, and Recognising Tectual Entailment, J. Quionero-Candela, I. Dagan, B. Magnini,
and F. dAlch-Buc, Eds. Springer Berlin Heidelberg, 2006, pp. 177190.
M. Bates, Models of natural language understanding, Proc. Natl. Acad. Sci. U. S. A., vol. 92, no.
22, pp. 99779982, Oct. 1995.
C. Condoravdi, D. Crouch, V. de Paiva, R. Stolle, and D. G. Bobrow, Entailment, Intensionality
and Text Understanding, in Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning Volume 9, 2003, pp. 3845.
L. Vanderwende and W. B. Dolan, What Syntax Can Contribute in the Entailment Task, in
Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and
Recognising Tectual Entailment, J. Quionero-Candela, I. Dagan, B. Magnini, and F. dAlch-Buc,
Eds. Springer Berlin Heidelberg, 2006, pp. 205216.
C. Monz and M. de Rijke, Light-weight entailment checking for computational semantics, in Proc.
of the third workshop on inference in computational semantics (ICoS-3), 2001.
Wikipedia contributors, Textual entailment, Wikipedia, The Free Encyclopedia, 26-Nov-2015.
[Online].
Available:
https://en.wikipedia.org/w/index.php?title=Textual_entailment&oldid=692504148. [Accessed: 25Jun-2016].
E. Akhmatova, Textual entailment resolution via atomic propositions, in Proceedings of the
PASCAL Challenges Workshop on Recognising Textual Entailment, 2005, vol. 150.
R. Raina, A. Y. Ng, and C. D. Manning, Robust textual inference via learning and abductive
reasoning, in AAAI, 2005, pp. 10991105.
D. Moldovan, C. Clark, S. Harabagiu, and S. Maiorano, COGEX: A Logic Prover for Question
Answering, in Proceedings of the 2003 Conference of the North American Chapter of the
Association for Computational Linguistics on Human Language Technology - Volume 1, Edmonton,
Canada, 2003, pp. 8793.
A. Fowler, B. Hauser, D. Hodges, I. Niles, A. Novischi, and J. Stephan, Applying COGEX to
recognize textual entailment, in Proceedings of the PASCAL Challenges Workshop on Recognising
Textual Entailment, 2005, pp. 6972.
J. Bos and K. Markert, Recognising Textual Entailment with Logical Inference, in Proceedings of
the Conference on Human Language Technology and Empirical Methods in Natural Language
Processing, Vancouver, British Columbia, Canada, 2005, pp. 628635.
J. Bos, S. Clark, M. Steedman, J. R. Curran, and J. Hockenmaier, Wide-coverage Semantic
Representations from a CCG Parser, in Proceedings of the 20th International Conference on
Computational Linguistics, Geneva, Switzerland, 2004.
J. Bos and K. Markert, When logical inference helps determining textual entailment (and when it
doesnt), in Proceedings of the Second PASCAL RTE Challenge, 2006, p. 26.
A. Hickl, J. Williams, J. Bensley, K. Roberts, B. Rink, and Y. Shi, Recognizing textual entailment
with LCCs GROUNDHOG system, in Proceedings of the Second PASCAL Challenges Workshop,
2006.
B. MacCartney and C. D. Manning, Natural Logic for Textual Inference, in Proceedings of the
ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic, 2007,
pp. 193200.
L. Romano, M. Kouylekov, I. Szpektor, I. Dagan, and A. Lavelli, Investigating a Generic
Paraphrase-Based Approach for Relation Extraction, in EACL, 2006.
V. Jijkoun, M. Rijke, and Others, Recognizing textual entailment using lexical similarity, 2005.

69

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016
[20]

[21]
[22]

[23]

[24]

[25]

[26]
[27]
[28]
[29]
[30]
[31]
[32]
[33]
[34]
[35]
[36]

[37]

B. MacCartney and C. D. Manning, Modeling Semantic Containment and Exclusion in Natural


Language Inference, in Proceedings of the 22Nd International Conference on Computational
Linguistics - Volume 1, Manchester, United Kingdom, 2008, pp. 521528.
R. Bar-Haim, I. Dagan, I. Greental, and E. Shnarch, Semantic inference at the lexical-syntactic
level, in Proceedings of the National Conference on Artificial Intelligence, 2007, vol. 22, p. 871.
P. Clark and P. Harrison, Recognizing textual entailment with logical inference, in Text Analysis
Conference (TAC 2008) Workshop-RTE-4 Track. National Institute of Standards and Technology
(NIST), 2008.
P. Clark and P. Harrison, Boeings NLP System and the Challenges of Semantic Representation,
in Proceedings of the 2008 Conference on Semantics in Text Processing, Venice, Italy, 2008, pp.
263276.
P. Clark, C. Fellbaum, J. R. Hobbs, P. Harrison, W. R. Murray, and J. Thompson, Augmenting
WordNet for Deep Understanding of Text, in Proceedings of the 2008 Conference on Semantics in
Text Processing, Venice, Italy, 2008, pp. 4557.
G. Dinu and R. Wang, Inference Rules and Their Application to Recognizing Textual Entailment,
in Proceedings of the 12th Conference of the European Chapter of the Association for Computational
Linguistics, Athens, Greece, 2009, pp. 211219.
R. Tian, Y. Miyao, and T. Matsuzaki, Logical Inference on Dependency-based Compositional
Semantics, in ACL (1), 2014, pp. 7989.
K. Mineshima, P. Martnez-Gmez, Y. Miyao, and D. Bekki, Higher-order logical inference with
compositional semantics, in Proceedings of EMNLP, 2015, pp. 20552061.
T. S. Kuhn, The Structure of Scientific Revolutions. Chicago: Third Edition, 1996.
F. AL-Khawaldeh and V. Samawi, Lexical cohesion and entailment based segmentation for arabic
text summarization (lceas), World ComputSciInfTechnol J, 2015.
M. Chen, S. Mao, and Y. Liu, Big Data: A Survey, Mobile NetwAppl, vol. 19, no. 2, pp. 171209,
Jan. 2014.
L. Manovich, The Language of New Media. MIT Press, 2001.
D. V. Shah, J. N. Cappella, and W. R. Neuman, Big data, digital media, and computational social
science possibilities and perils, Ann. Am. Acad. Pol. Soc. Sci., vol. 659, no. 1, pp. 613, 2015.
C. D. Manning and H. Schtze, Foundations of Statistical Natural Language Processing. MIT Press,
1999.
S. Lappin and C. Fox, The Handbook of Contemporary Semantic Theory. Wiley, 2015.
Y. Liu, C. Sun, L. Lin, and X. Wang, Learning Natural Language Inference using Bidirectional
LSTM model and Inner-Attention, arXiv [cs.CL], 30-May-2016.
B. Magnini, Recognizing Textual Entailment: Models and Applications Ido Dagan1, Dan Roth2,
Mark Sammons2, and Fabio Massimo Zanzotto3 (1Bar-Ilan University, Israel, 2University of
Illinois, Urbana, IL, and 3University of Rome Tor Vergata, Italy) Morgan & Claypool (Synthesis
Lectures on Human Language Technologies, edited by Graeme Hirst, volume 23), 2013, xx+ 200 pp;
paperbound, ISBN 978-1-59829-834-5; e-book, ISBN 978-1-59829-835-2, Comput. Linguist.,
2015.
C. Shivade, P. Raghavan, and S. Patwardhan, Addressing Limited Data for Textual Entailment
Across Domains, arXiv [cs.CL], 08-Jun-2016

70

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016

APPENDIX: TABLE

S.
No

overall cws
Approach Key Claim/ task (only for
RTE)

Describing a
computer tool
which can
extract text
entailments by
comparing
atomic
propositions.

Textual
Entailment
Resolution
via Atomic
Propositio
ns

Robust
Textual
Inference
Via
Learning
and
Abductive
Reasoning

Applying
COGEX to
Recognize
Textual
Entailment

Recognisin
g Textual
Entailment
with
Logical

Parsing
sentences into
logical-formula
like
representations.
Then a
minimum cost
set of
assumptions are
realised using
an abductive
theorem prover.
If one sentence
follows the
other low cost
set of
assumptions.
This marries
precision of
logical
reasoning with
the robustness
of machine
learning.
Transforming
T-H pair into
logic form
representation
with semantic
relations. The
system then
generates
axioms as
linguistic
rewriting rules
and lexical
chain axioms to
connect T-H.
Using Model
building and
machine
learning

0.5067

0.651

0.56

0.65

accuracy/
precision
(when
mentioned)

0.5188

0.57

0.551

0.61

FraCas Advantages/
Authors
score Disadvantages

Year

Simple.
Inadequate
Knowledg
e rule
database
for a
robust
Entailment
.

Elena
Akhmat
ova

2005

More
adaptive
and
flexible.
Highly
syntactical,
Semantic
aspects
still weak.

RajatRai
na,
Andrew
Y.Ng
and
D.Manni
ng

2005

Has better
semantic
connectivit
y. Creating
knowledge
base could
be tedious.

Abraha
m
Fowler
et al.

2005

Better
semantic
inference
and hybrid
model

Johan
Bos and
KatjaMa
rkert

2005

71

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016
Inference

Recognisin
g Textual
Entailment
using
LCC's
Ground
Hog

When
logical
inference
helps
determinin
g textual
entailment
(and when
it doesnt)

Natural
Logic for
Textual
Inference

Semantic
Inference
at the
LexicalSyntactic
Level

leading to
more
robust
entailment.
Introducing a
new system for
recognizing
textual
entailment
(known as
GROUNDHO
G), which
utilizes a
classificationbased approach
to combine
lexico-semantic
information
derived from
text processing
applications
with a large
collection of
paraphrases
acquired
automatically
from the
WWW.

Comparison of
logical
inference
(shallow
method) in its
efficacy in text
entailment.

Most
approaches
sacrifice
semantic
precision for
robustness. But
those based on
first order logic
and theoremproving are
highly brittle.
This is a middle
way
Classical
approaches to
semantic
inference rely
on complex
logical
representations.
However,

>0.652

0.616
(shallow)
and
0.606
(both)

0.673

78.5
(precision)
RE
(Relation
Extraction)
setting

Machine
learning
based
approach
and high
accuracies
possible.
Requires
large
amount of
training
examples.

Hicks, et
al.

2006

No
significant
improvem
ent in
results
using the
logical
inference
(except for
a few
tasks)

Johan
Bos and
KatjaMa
rkert

2006

(accura
cy)
59.56%

Tries to
use both
deep and
shallow
using
Natural
logic
(NatLog),
among the
first to use
FraCas

Bill
MacCart
ney and
Christop
her
D.Manni
ng

2007

Does not
use
standard
RTE or
FraCas
setting.
More
precise and

BarHaim,
Dagon,
Greental
, and
Shnarch

2007

72

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016
practical
applications
usually adopt
shallower
lexical or
lexicalsyntactic
representations,
but lack a
principled
inference
framework. We
propose a
generic
semantic
inference
framework that
operates
directly on
syntactic trees.
New trees are
inferred by
applying
entailment
rules, which
provide a
unified
representation
for varying
types of
inferences.

10

Modeling
Semantic
Containme
nt and
Exclusion
in Natural
Language
Inference

Using a new
method to
reduce the error
incorporating
semantic
inference, by
using a
sequence of
atomic edits
between T-H

Recognizi
ng Textual
Entailment
with
Logical
Inference

First the
semantic
interpretation
of the sentence
is performed
and then check
if the logic for
the H is implied
by some
inference
elaborated

less
approxima
te than the
other
studies. A
new
framework
to
incorporat
e semantic
as well as
lexicalsyntactic
levels

64.5
(accuracy
RTE3)

~ 65%
correctness

70.49
(accura
cyFraCas
)

Hybrid
method.
More
reliable
results
than earlier
NatLog.
Using
semantic
containme
nt,
exclusion
and
impilcativi
ty explains
many
everyday
patterns.
Provides
explanatio
ns.
Produces
preliminar
y positive
results to
start with.
Deductive
style of
reasoning.

Bill
MacCart
ney and
Christop
her
D.Manni
ng

2008

Peter
Clark
and Phil
Harrison

2008

73

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016

11

12

13

Inference
Rules and
their
Applicatio
n to
Recognizi
ng Textual
Entailment

Combined
Distributio
nal and
Logical
Semantics

Logical
Inference
on
Dependenc
y-based
Compositi
onal
Semantics

version of T.
The system
also tries to
produce
explanations
for the
entailments,
sometimes also
erroneous.
Starting with
automatically
acquired
collection and
refine it and
obtain more
rules using
hand crafted
lexical
resource. Then
produce a
dependency
based structure
representation
from texts, with
an aim to
provide proper
base for the
inference rule
application.
Following
formal
semantics in
mapping
language to
logical
representations
but differ in
that the
relational
constants used
are induced by
offline
distributional
clustering at the
level of
predicateargument
structure.
Equipping the
DCS
framework with
logical
inference, by
defining
abstract
denotations as
an abstraction
of the
computing
process of

Use of
noisy
knowledge

high
precision
(>55)
across
tasks

Flexible
combinato
rial
approach.
Tedious
inference
rules
writing.

89%
(accura
cy
single
premis
e) 80%
(accura
cy
multipl
e
premis
es)

Distributio
nal logical
Semantics.
Advanced
clustering
techniques
deployed
for textual
entailment

M Lewis
and
Mark
Steedma
n

2013

accuracies
of above
59%

79.5%
(accura
cy
single
premis
e) 80%
(accura
cy
multipl
e
premis
es)

DCS and
logical
inferences.
On the fly
implement
ations

Tian,
Miyao,
and
Matsuza
ki

2014

Georgia
na Dinu
and Rui
Wang

2009

74

International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 4, July 2016

14

Higherorder
logical
inference
with
compositio
nal
semantics

denotations in
original DCS.
An inference
engine is built
to achieve
inference on
abstract
denotations.
Furthermore,
we propose a
way to generate
on-the-fly
knowledge in
logical
inference, by
combining our
framework with
the idea of tree
transformation.
Developing a
bridge between
parser and
semantic
composition.
higher order
logic used for
logical
inference

69%
(accura
cy)

Opens
gates for
higher
order logic
inferences.

Koji
Mineshi
ma, et
al.

2015

75

You might also like