3.dist Semantics

Distributional Semantics
Pawan Goyal
CSE, IIT Kharagpur
October 4th, 2016
Pawan Goyal (IIT Kharagpur)
October 4th, 2016
1 / 24
Introduction
1, 3, 4, . . .
October 4th, 2016
2 / 24
Introduction
1, 3, 4, . . .
I, III, IV, . . .
October 4th, 2016
2 / 24
Introduction
1, 3, 4, . . .
I, III, IV, . . .
What is Semantics?
October 4th, 2016
2 / 24
Introduction
1, 3, 4, . . .
I, III, IV, . . .
What is Semantics?
The study of meaning: Relation between symbols and their denotata.
October 4th, 2016
2 / 24
Introduction
1, 3, 4, . . .
I, III, IV, . . .
What is Semantics?
The study of meaning: Relation between symbols and their denotata.
John told Mary that the train moved out of the station at 3 oclock.
October 4th, 2016
2 / 24
Conceptual Graph Representation
Finding the underlying functional relations among various entities and events
Sentence: John told Mary that the

train moved out of the station at 3
oclock.
October 4th, 2016
3 / 24
Computational Semantics
October 4th, 2016
4 / 24
The study of how to automate the process of constructing and reasoning with
meaning representations of natural language expressions.
October 4th, 2016
4 / 24
Methods in Computational Semantics generally fall in two categories:

Formal Semantics: Construction of precise mathematical models of the
relations between expressions in a natural language and the world.
October 4th, 2016
4 / 24

John chases a bat x[bat(x) chase(john, x)]
October 4th, 2016
4 / 24

John chases a bat x[bat(x) chase(john, x)]
Distributional Semantics: The study of statistical patterns of human
word usage to extract semantics.
October 4th, 2016
4 / 24
Distributional Hypothesis
October 4th, 2016
5 / 24
Distributional Hypothesis: Basic Intuition

The meaning of a word is its use in language. (Wittgenstein,
1953)
You know a word by the company it keeps. (Firth, 1957)
October 4th, 2016
5 / 24

1953)
Word meaning (whatever it might be) is reflected in linguistic distributions.
October 4th, 2016
5 / 24

1953)

Words that occur in the same contexts tend to have similar
meanings. (Zellig Harris, 1968)
October 4th, 2016
5 / 24

1953)

Words that occur in the same contexts tend to have similar
meanings. (Zellig Harris, 1968)
Semantically similar words tend to have similar distributional patterns.
October 4th, 2016
5 / 24
Distributional Semantics: a linguistic perspective
If linguistics is to deal with meaning, it can only do so through

distributional analysis. (Zellig Harris)
October 4th, 2016
6 / 24

If we consider words or morphemes A and B to be more
different in meaning than A and C, then we will often find that the
distributions of A and B are more different than the distributions of A
and C. In other words, difference in meaning correlates with
difference of distribution. (Zellig Harris, Distributional Structure)
October 4th, 2016
6 / 24

If we consider words or morphemes A and B to be more
different in meaning than A and C, then we will often find that the
distributions of A and B are more different than the distributions of A
and C. In other words, difference in meaning correlates with
difference of distribution. (Zellig Harris, Distributional Structure)
Differential and not referential
October 4th, 2016
6 / 24
Distributional Semantics: a cognitive perspective
Contextual representation
A words contextual representation is an abstract cognitive structure that
accumulates from encounters with the word in various linguistic contexts.
October 4th, 2016
7 / 24
We learn new words based on contextual cues
October 4th, 2016
7 / 24

He filled the wampimuk with the substance, passed it around and we all drunk
some.
October 4th, 2016
7 / 24

He filled the wampimuk with the substance, passed it around and we all drunk
some.
We found a little wampimuk sleeping behind the tree.
October 4th, 2016
7 / 24
Distributional Semantic Models (DSMs)
Computational models that build contextual semantic repesentations from

corpus data
October 4th, 2016
8 / 24

corpus data
DSMs are models for semantic representations
I
I
The semantic content is represented by a vector

Vectors are obtained through the statistical analysis of the linguistic
contexts of a word
October 4th, 2016
8 / 24

corpus data
DSMs are models for semantic representations
I
I
The semantic content is represented by a vector

Vectors are obtained through the statistical analysis of the linguistic
contexts of a word
Alternative names
I
I
I
I
I
corpus-based semantics
statistical semantics
geometrical models of meaning
vector semantics
word space models
October 4th, 2016
8 / 24
Distributional Semantics: The general intuition
Distributions are vectors in a multidimensional semantic space, that is,

objects with a magnitude and a direction.
The semantic space has dimensions which correspond to possible
contexts, as gathered from a given corpus.
October 4th, 2016
9 / 24
Vector Space
October 4th, 2016
10 / 24
Vector Space
October 4th, 2016
10 / 24
Vector Space
In practice, many more dimensions are used.

cat = [...dog 0.8, eat 0.7, joke 0.01, mansion 0.2,...]
October 4th, 2016
10 / 24
Word Space
Small Dataset
An automobile is a wheeled motor vehicle used for transporting passengers .
A car is a form of transport , usually with four wheels and the capacity to carry around
five passengers .
Transport for the London games is limited , with spectators strongly advised to avoid
the use of cars .
The London 2012 soccer tournament began yesterday , with plenty of goals in the
opening matches .
Giggs scored the first goal of the football tournament at Wembley , North London .
Bellamy was largely a passenger in the football match , playing no part in either goal .
Target words: hautomobile, car, soccer, footballi

Term vocabulary : hwheel, transport, passenger, tournament, London, goal,
matchi
October 4th, 2016
11 / 24
Constructing Word spaces
Informal algorithm for constructing word spaces

Pick the words you are interested in: target words
October 4th, 2016
12 / 24

Define a context window, number of words surrounding target word
October 4th, 2016
12 / 24

I
The context can in general be defined in terms of documents, paragraphs

or sentences.
October 4th, 2016
12 / 24

I

or sentences.
Count number of times the target word co-occurs with the context words:
co-occurrence matrix
October 4th, 2016
12 / 24

I

or sentences.
Count number of times the target word co-occurs with the context words:
co-occurrence matrix
Build vectors out of (a function of) these co-occurrence counts
October 4th, 2016
12 / 24
Constructing Word spaces: distributional vectors
distributional matrix = targets X contexts
automobile
car
soccer
football
wheel
1
1
0
0
transport
1
2
0
0
passenger
1
1
0
1
tournament
0
0
1
1
London
0
1
1
1
goal
0
0
1
2
October 4th, 2016
match
0
0
1
1
13 / 24
2.5
football (0,2)
goal
1.5
soccer (0,1)
0.5
automobile (1,0)
car (2,0)
0
0
0.5
1.5
2.5
transport
October 4th, 2016
14 / 24
Computing similarity
automobile
car
soccer
football
wheel
1
1
0
0
transport
1
2
0
0
Using simple vector product

automobile . car = 4
automobile . soccer = 0
automobile . football = 1
passenger
1
1
0
1
tournament
0
0
1
1
London
0
1
1
1
goal
0
0
1
2
match
0
0
1
1
car . soccer = 1
car . football = 2
soccer . football = 5
October 4th, 2016
15 / 24
Vector Space Model without distributional similarity
Words are treated as atomic symbols
October 4th, 2016
16 / 24
Vector Space Model without distributional similarity
Words are treated as atomic symbols
One-hot representation
October 4th, 2016
16 / 24
Distributional Similarity Based Representations
You know a word by the company it keeps
October 4th, 2016
17 / 24
October 4th, 2016
17 / 24
These words will represent banking
October 4th, 2016
17 / 24
Building a DSM step-by-step

The linguistic steps
Pre-process a corpus (to define targets and contexts)
Select the targets and the contexts
October 4th, 2016
18 / 24
Building a DSM step-by-step

The linguistic steps
Pre-process a corpus (to define targets and contexts)
Select the targets and the contexts
The mathematical steps

Count the target-context co-occurrences
Weight the contexts (optional)
Build the distributional matrix
Reduce the matrix dimensions (optional)
Compute the vector distances on the (reduced) matrix

October 4th, 2016
18 / 24
Many design choices
October 4th, 2016
19 / 24
Many design choices
General Questions
How do the rows (words, ...) relate to each other?
How do the columns (contexts, documents, ...) relate to each other?
October 4th, 2016
19 / 24
The parameter space
A number of parameters to be fixed

Which type of context?
Which weighting scheme?
Which similarity measure?
...
A specific parameter setting determines a particular type of DSM (e.g. LSA,
HAL, etc.)
October 4th, 2016
20 / 24
Documents as context: Word document
October 4th, 2016
21 / 24
Words as context: Word Word
October 4th, 2016
22 / 24
Words as contexts
Parameters
Window size
Window shape - rectangular/triangular/other
October 4th, 2016
23 / 24
Words as contexts
Parameters
Window size
Consider the following passage

Suspected communist rebels on 4 July 1989 killed Col. Herminio Taylo, police
chief of Makati, the Philippines major financial center, in an escalation of street
violence sweeping the Capitol area. The gunmen shouted references to the
rebel New Peoples Army. They fled in a commandeered passenger jeep. The
military says communist rebels have killed up to 65 soldiers and police in the
Capitol region since January.
October 4th, 2016
23 / 24
Words as contexts
Parameters
Window size
5 words window (unfiltered): 2 words either side of the target word

October 4th, 2016
23 / 24
Words as contexts
Parameters
Window size
5 words window (filtered): 2 words either side of the target word

October 4th, 2016
23 / 24
Context weighting: documents as context
October 4th, 2016
24 / 24
Indexing function F: Essential factors

Word frequency (fij ): How many times a word appears in the document?
F fij
Document length (|Di |): How many words appear in the document?
1
|Di |
Document frequency (Nj ): Number of documents in which a word

appears. F N1
j
October 4th, 2016
24 / 24
Indexing function F: Essential factors

Word frequency (fij ): How many times a word appears in the document?
F fij
Document length (|Di |): How many words appear in the document?
1
|Di |
Document frequency (Nj ): Number of documents in which a word

appears. F N1
j
Indexing Weight: tf-Idf

fij log( NNj ) for each term, normalize the weight in a document with
respect to L2 -norm.
October 4th, 2016
24 / 24

3.dist Semantics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3.dist Semantics

Uploaded by

Copyright:

Available Formats

Distributional Semantics

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Conceptual Graph Representation

Sentence: John told Mary that the

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Methods in Computational Semantics generally fall in two categories:

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Methods in Computational Semantics generally fall in two categories:

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Methods in Computational Semantics generally fall in two categories:

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Hypothesis: Basic Intuition

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Hypothesis: Basic Intuition

Word meaning (whatever it might be) is reflected in linguistic distributions.

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Hypothesis: Basic Intuition

Word meaning (whatever it might be) is reflected in linguistic distributions.

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Hypothesis: Basic Intuition

Word meaning (whatever it might be) is reflected in linguistic distributions.

Semantically similar words tend to have similar distributional patterns.

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Semantics: a linguistic perspective

If linguistics is to deal with meaning, it can only do so through

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Semantics: a linguistic perspective

If linguistics is to deal with meaning, it can only do so through

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Semantics: a linguistic perspective

If linguistics is to deal with meaning, it can only do so through

Pawan Goyal (IIT Kharagpur)

October 4th, 2016

Distributional Semantics: a cognitive perspective

Pawan Goyal (IIT Kharagpur)