Supervised Learning

Lecture 4

K-Nearest Neighbor (KNN) Classifier and application

Dr Mustansar Ali Ghazanfar Mustansara.ali@uettaxila.edu.pk
06/03/2014

Supervised Learning
Lecture 6

K-Nearest Neighbor (KNN) Classifier and application

Dr Mustansar Ali Ghazanfar Mustansara.ali@uettaxila.edu.pk
*Note: Not part of the course
06/03/2014 2

Objects, Feature Vectors, Points

i

x1

x1(i)

x2(i)

X(15) X(1) X(3) X(7) X(16) X(8) 9

1 4 6

3 5 7

10

X(25) X(9) X(4) X(14)

X(12) 14

12

11

X(13) X(6) X(10) X(11)

13 15

16

Elliptical blobs (objects)

x2
06/03/2014 3

Nearest Neighbours
x1
X(j)=(x1(j), x2(j), ,xn(j))

D(i, j ) xk i xk j
k 1

X(i)=(x1(i), x2(i), ,xn(i))

x2

06/03/2014

Nearest Neighbour Algorithm

Given training data (X(1),D(1)), (X(2),D(2)), , (X(N),D(N))

Define a distance metric between points in inputs space. Common measures are:

Euclidean Distance
D(i, j ) xk i xk j
k 1 n 2

06/03/2014

K-Nearest Neighbour Model

Given test point X Find the K nearest training inputs to X Denote these points as
(X(1),D(1)), (X(2), D(2)), , (X(k), D(k))
x

06/03/2014

K-Nearest Neighbour Model

Classification The class identification of X
Y = most common class in set {D(1), D(2), , D(k)} x

06/03/2014

K-Nearest Neighbour Model

Example : Classify whether a customer will respond to a survey question using a 3-Nearest Neighbor classifier
Customer John Rachel Hannah Tom Nellie David Age 35 22 63 59 25 37 Income 35K 50K 200K 170K 40K 50K No. credit cards 3 2 1 1 4 2 Response No Yes No No Yes ?

06/03/2014

K-Nearest Neighbour Model

Example : 3-Nearest Neighbors
Customer John Rachel Hannah Tom Nellie David Age 35 22 63 59 25 37 Income 35K 50K 200K 170K 40K 50K No. credit cards 3 2 1 1 4 2 Response No Yes No No Yes ?
15.74 122 152.23 15.16

15

06/03/2014

K-Nearest Neighbour Model

Example : 3-Nearest Neighbors
Customer John Rachel Hannah Tom Nellie David Age 35 22 63 59 25 37 Income 35K 50K 200K 170K 40K 50K No. credit cards 3 2 1 1 4 2 Response No Yes No No Yes ?
15.74 122 152.23 15.16

15

Three nearest ones to David are: No, Yes, Yes

06/03/2014

10

K-Nearest Neighbour Model

Example : 3-Nearest Neighbors
Customer John Rachel Hannah Tom Nellie David Age 35 22 63 59 25 37 Income 35K 50K 200K 170K 40K 50K No. credit cards 3 2 1 1 4 2 Response No Yes No No Yes ? Yes
15.74 122 152.23 15.16

15

Three nearest ones to David are: No, Yes, Yes

06/03/2014

11

K-Nearest Neighbour Model

Example: For the example we saw earlier, pick the best K from the set {1, 2, 3} to build a K-NN classifier
Customer John Rachel Hannah Tom Nellie David Age 35 22 63 59 25 37 Income 35K 50K 200K 170K 40K 50K No. credit cards 3 2 1 1 4 2 Response No Yes No No Yes ?

06/03/2014

12

k-Nearest Neighbor Pseudo code

Training Algorithm:
Store all training examples <x, f(x)> Find best value for K

Classification (Testing) Algorithm:

Given a query instance xq to be classified,
Let x1, xk denote the k instances from the list of training examples k Return f ( xq ) argmax d (c, f ( xi )) where (a,b)=1 if a=b and where (a,b)=0 otherwise (C = class)
06/03/2014 13

i=1

Nearest Neighbour Rule

Consider a two class problem where each sample consists of two measurements (x,y).

For a given query point q, assign the class of the nearest neighbour.

k=1

Compute the k nearest neighbours and assign the class by majority vote.
06/03/2014

k=3
14

k-Nearest Neighbor Examples

(discrete-valued target function)

k=1 k=5

06/03/2014

15

KNN Flavors

06/03/2014

16

Distance-Weighted Nearest Neighbor Algorithm

Distance-weighted function (For Discrete-valued function)

f argmax w d (c, f ( x ))
i i

where

i=1

wi

1 d ( xq , xi )2

weights are proportional to distance; d(xq, xi) is Euclidean distance.

06/03/2014

17

How many neighbors, K?

K
Fixed constant Determines number of elements to be included in each neighborhood.
Neighborhood determines classification Different k values can and will produce different classifications

06/03/2014

18

K-Nearest Neighbour Model

Picking K

Use N fold cross validation Pick K to minimize the cross validation error
For each of N training example
Find its K nearest neighbours Make a classification based on these K neighbours Calculate classification error Output average error over all examples

Use the K that gives lowest average error over the N training examples

06/03/2014

19

Nearest Neighbour Complexity

Expensive for high dimensional data (d>20?)

O(Nd) complexity for both storage and query time

N is the number of training examples, d is the dimension of each sample

06/03/2014

20

Advantages/Disadvantages
Advantages:
Training is very fast Learning complex target functions Dont lose information

Disadvantages:
Slow at query Easily fooled by irrelevant attributes

06/03/2014

21

Nearest Neighbour Issues

Expensive
To determine the nearest neighbour of a query point q, must compute the distance to all N training examples
Pre-sort training examples into fast data structures (kd-trees) Remove redundant data (condensing)

Storage Requirements

Must store all training data D_tr

High Dimensional Data

Remove redundant data (condensing) Pre-sorting often increases the storage requirements

Curse of Dimensionality

Required amount of training data increases exponentially with dimension Computational cost also increases dramatically

06/03/2014

22

Condensing
Aim is to reduce the number of training samples For example, Retain only the samples that are needed to define the decision boundary

*Note: Not part of this lecture

06/03/2014 23

06/03/2014

24

Applications of KNN

Predicting Unknown movie for a movie fan

06/03/2014

25

06/03/2014

26

KNN in Collaborative Filtering (CF)

Item1 User1 Item2 4 Item3 2 Item4 1 Item5 5 Item6

User2
User3 User4

5
4 3

5
4

5
4 3

1
1 5

06/03/2014

27

KNN in CF
Item1 User1 User2 5 Item2 4 5 Item3 Item4 1 1 Item5 5 Item6

x
5

User3
User4

4
3

4
3

1
5

06/03/2014

28

KNN in CF
Item1 User1 User2 5 Item2 4 5 Item3 Item4 1 1 Item5 5 Item6

x
5

User3
User4

4
3

4
3

1
5

06/03/2014

29

KNN in CF
Item1 User1 User2 5 Item2 4 5 Item3 Item4 1 1 Item5 5 Item6

x
5

User3
User4

4
3

4
3

1
5

Sim (u1,u2) = {-1,+1} 0.5 Sim (u1,u3) = {-1,+1} -0.1 Sim (u1,u4) = {-1,+1} 0.3
06/03/2014

Sum(|sim|) = 0.5 + 0.1 + 0.3 = 0.9

30

KNN in CF
Item1 User1 User2 5 Item2 4 5 Item3 Item4 1 1 Item5 5 Item6

x
5

User3
User4

4
3

4
3

1
5

Prediction = {(Sim (u1,u2) * rating_User2) + (Sim (u1,u3) * rating_User3) + (Sim (u1,u4) * rating_User4) } / Sum(sim) Prediction = {(0.5 * 5) + (-0.1 * 4) + (0.3* 3)} /(0.9) = 3.05 31 06/03/2014

How does KNN work here?

Find similar users. Similarity measures.
Vector

Pearson correlation

Select set of K most similar users User their votes for prediction
06/03/2014 32

Pearson Correlation

06/03/2014

33

06/03/2014

34

Questions?

06/03/2014

35