●

●
○
○
○

2
●
●
𝑎Ԧ ∙ 𝑏 = 𝑎,
Ԧ 𝑏 = 𝑎 𝑏 cos 𝜃 = ෍ 𝑎𝑖 𝑏𝑖
𝑖
●
● ( 𝑎Ԧ cos 𝜃) 𝑏Ԧ 𝑎Ԧ 𝑏Ԧ
𝑏Ԧ
𝑎Ԧ

𝜃 𝑏

3
•

4
●

5
●

6
●

?
?

7
●

?
?

?
8
●

9
●

10
●
○
○
○
○

11
● 𝑥Ԧ ℎ

𝑥Ԧ

12
● 𝑥Ԧ ℎ

𝑤⊥ℎ

𝑥Ԧ

𝑤 ℎ

13
● 𝑥Ԧ ℎ

𝑤⊥ℎ
𝑥,
Ԧ 𝑤 = ( 𝑥Ԧ cos 𝜃) 𝑤
𝑥Ԧ 𝑤 𝑥Ԧ

𝜃 𝑤 ℎ

𝑥Ԧ cos 𝜃 14
● 𝑥Ԧ ℎ

𝑤⊥ℎ
𝑥,
Ԧ 𝑤 = ( 𝑥Ԧ cos 𝜃) 𝑤
𝑥Ԧ 𝑤 𝑥Ԧ
⟹
∃𝑐 ∈ ℝ
𝑥,
Ԧ 𝑤 ≥𝑐 𝑥Ԧ ℎ
𝑥Ԧ 𝜃 𝑤 ℎ

𝑥Ԧ cos 𝜃 15
● 𝑥Ԧ ℎ

𝑤⊥ℎ
𝑥,
Ԧ 𝑤 = ( 𝑥Ԧ cos 𝜃) 𝑤
𝑥Ԧ 𝑤 𝑥Ԧ
⟹
∃𝑐 ∈ ℝ
𝑥,
Ԧ 𝑤 ≥𝑐 𝑥Ԧ ℎ
𝑥Ԧ 𝜃 𝑤 ℎ
𝑏 = −𝑐
𝑥Ԧ cos 𝜃 16
 𝑥,
Ԧ 𝑤 + 𝑏 > 0 ⟹ 𝑥Ԧ ℎ
𝑥,
Ԧ 𝑤 + 𝑏 < 0 ⟹ 𝑥Ԧ ℎ ℎ
𝑥Ԧ 𝑤 b
𝑥,
Ԧ 𝑤 + 𝑏 = 0 ⟹ 𝑥Ԧ ℎ

𝑤 ℎ 𝑥,
Ԧ 𝑤 +𝑏 ℎ

𝒙 𝒙, 𝒘 + 𝒃 ≥ 𝟎
𝒙

● 𝑤 b
17
● 𝐸= 𝑥 1
,𝑦 1
, 𝑥 2
,𝑦 2
,…, 𝑥 𝑚
,𝑦 𝑚
, 𝑥Ԧ 𝑖
∈ ℝ𝑛 , y 𝑖
∈ +1, −1

18
● 𝐸= 𝑥 1
,𝑦 1
, 𝑥 2
,𝑦 2
,…, 𝑥 𝑚
,𝑦 𝑚
, 𝑥Ԧ 𝑖
∈ ℝ𝑛 , y 𝑖
∈ +1, −1

●
𝑖 𝑖
𝑥+ , 𝑤 + 𝑏 ≥ 0 ∀𝑥+ ∈ 𝑥Ԧ 𝑦 = +1
𝑖 𝑖
𝑥− , 𝑤 + 𝑏 < 0 ∀𝑥− ∈ 𝑥Ԧ 𝑦 = −1

19
● 𝐸= 𝑥 1
,𝑦 1
, 𝑥 2
,𝑦 2
,…, 𝑥 𝑚
,𝑦 𝑚
, 𝑥Ԧ 𝑖
∈ ℝ𝑛 , y 𝑖
∈ +1, −1

●
𝑖 𝑖
𝑥+ , 𝑤 + 𝑏 ≥ 0 ∀𝑥+ ∈ 𝑥Ԧ 𝑦 = +1
𝑖 𝑖
𝑥− , 𝑤 + 𝑏 < 0 ∀𝑥− ∈ 𝑥Ԧ 𝑦 = −1

20
 𝑥+ , 𝑤 + 𝑏 ≥ 0 ∀𝑥+ ∈ 𝑥Ԧ 𝑖 𝑦 𝑖 = +1
𝑥− , 𝑤 + 𝑏 < 0 ∀𝑥− ∈ 𝑥Ԧ 𝑖 𝑦 𝑖 = −1

21
 𝑥+ , 𝑤 + 𝑏 ≥ 𝟏 ∀𝑥+ ∈ 𝑥Ԧ 𝑖 𝑦 𝑖 = +1
𝑥− , 𝑤 + 𝑏 ≤ −𝟏 ∀𝑥− ∈ 𝑥Ԧ 𝑖 𝑦 𝑖 = −1

22
● 𝐸= 𝑥 1
,𝑦 1
, 𝑥 2
,𝑦 2
,…, 𝑥 𝑚
,𝑦 𝑚
, 𝑥Ԧ 𝑖
∈ ℝ𝑛 , y 𝑖
∈ +1, −1

𝑖 𝑖
𝑥+ , 𝑤 + 𝑏 ≥ 𝟏 ∀𝑥+ ∈ 𝑥Ԧ 𝑦 = +1
𝑖 𝑖
𝑥− , 𝑤 + 𝑏 ≤ −𝟏 ∀𝑥− ∈ 𝑥Ԧ 𝑦 = −1

23
● 𝐸= 𝑥 1
,𝑦 1
, 𝑥 2
,𝑦 2
,…, 𝑥 𝑚
,𝑦 𝑚
, 𝑥Ԧ 𝑖
∈ ℝ𝑛 , y 𝑖
∈ +1, −1

𝑖 𝑖
𝑥+ , 𝑤 + 𝑏 ≥ 𝟏 ∀𝑥+ ∈ 𝑥Ԧ 𝑦 = +1
𝑖 𝑖
𝑥− , 𝑤 + 𝑏 ≤ −𝟏 ∀𝑥− ∈ 𝑥Ԧ 𝑦 = −1

●
𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0

24
● 𝐸= 𝑥 1
,𝑦 1
, 𝑥 2
,𝑦 2
,…, 𝑥 𝑚
,𝑦 𝑚
, 𝑥Ԧ 𝑖
∈ ℝ𝑛 , y 𝑖
∈ +1, −1

𝑖 𝑖
𝑥+ , 𝑤 + 𝑏 ≥ 𝟏 ∀𝑥+ ∈ 𝑥Ԧ 𝑦 = +1
𝑖 𝑖
𝑥− , 𝑤 + 𝑏 ≤ −𝟏 ∀𝑥− ∈ 𝑥Ԧ 𝑦 = −1

●
𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0
● 𝑥Ԧ 𝑖 , 𝑦 𝑖

𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 = 0
25
● 𝑥Ԧ 𝑖 , 𝑦 𝑖

𝒊
𝒚 𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 = 𝟎
𝑤

26
● 𝑥Ԧ 𝑖 , 𝑦 𝑖

𝒊
𝒚 𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 = 𝟎
𝑤
●

27
●

28
●

𝑥−

𝑤 𝑥+

29
●
● 𝑥+ − 𝑥−

𝑥−

𝑤 𝑥+

30
●
● 𝑥+ − 𝑥−

● 𝑢⊥ℎ 𝒖
𝑥−

𝑤 𝑥+

31
●
● 𝑥+ − 𝑥−
𝑔

● 𝑢⊥ℎ 𝑢

𝑤 𝑥−
𝑔 = 𝑥+ − 𝑥− ∙ 𝑢
𝑤
𝑤 𝑥+

32
●
● 𝑥+ − 𝑥−
𝒘
● 𝑢⊥ℎ 𝒘

𝑤 𝑥−
𝑔 = 𝑥+ − 𝑥− ∙
𝑤
𝑤 𝑥+

33
●
● 𝑥+ − 𝑥−
𝑤
● 𝑢⊥ℎ 𝑤

𝑥+ , 𝑤 − 𝑥− , 𝑤 𝑥−
𝑔=
𝑤
𝑤 𝑥+

34
●
● 𝑥+ − 𝑥−
𝑤
● 𝑢⊥ℎ 𝑤

𝑥+ , 𝑤 − 𝑥− , 𝑤 𝑥−
𝑔=
𝑤
● 𝑦 𝑥,
Ԧ 𝑤 +𝑏 −1=0 𝑥Ԧ 𝑤 𝑥+

35
●
● 𝑥+ − 𝑥−
𝑤
● 𝑢⊥ℎ 𝑤

𝑥+ , 𝑤 − 𝑥− , 𝑤 𝑥−
𝑔=
𝑤
● 𝑦 𝑥,
Ԧ 𝑤 +𝑏 −1=0 𝑥Ԧ 𝑤 𝑥+

1 − 𝑏 − −1 − 𝑏
𝑔=
𝑤 𝑔

36
●
● 𝑥+ − 𝑥−
𝑤
● 𝑢⊥ℎ 𝑤

𝑥+ , 𝑤 − 𝑥− , 𝑤 𝑥−
𝑔=
𝑤
● 𝑦 𝑥,
Ԧ 𝑤 +𝑏 −1=0 𝑥Ԧ 𝑤 𝑥+

𝟐
𝒈=
𝒘 𝑔

37
●
2
𝑔=
𝑤

2
𝑤

38
●
2
𝑔=
𝑤
●
2
⇒ 𝑤 ⇒
𝑤
𝑤

𝒘 𝟐
2
𝟐 𝑤

39
●

𝒙 𝒙, 𝒘 + 𝐛 ≥ 𝟎

● 𝑤 𝑏

𝒘 𝟐
𝟐
𝒚 𝒊 𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 ≥ 𝟎

40
41
●
𝒘 𝟐
● 𝟐
𝒚 𝒊
𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 ≥ 𝟎

42
●
𝒘 𝟐
● 𝟐
𝒚 𝒊
𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 ≥ 𝟎
●
2
𝑤 𝑖
𝐿 𝑤, 𝑏, 𝛼 = − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 𝛼𝑖 ≥ 0
2 𝑖

𝛼𝑖

43
●
𝒘 𝟐
● 𝟐
𝒚 𝒊
𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 ≥ 𝟎
●
2
𝑤 𝑖
𝐿 𝑤, 𝑏, 𝛼 = − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 𝛼𝑖 ≥ 0
2 𝑖

𝜕𝐿
= 𝑤 − ෍ 𝛼𝑖 𝑦 𝑖 𝑥Ԧ 𝑖
= 0 ⟹ 𝒘 = ෍ 𝜶𝒊 𝒚 𝒊 𝒙 𝒊
𝜕𝑤 𝑖 𝒊 𝛼𝑖

𝜕𝐿 𝑖 𝒊
= − ෍ 𝛼𝑖 𝑦 = 0 ⟹ ෍ 𝜶𝒊 𝒚 =𝟎
𝜕𝑏 𝑖 𝒊

44
●
𝒘 𝟐
● 𝟐
𝒚 𝒊
𝒙 𝒊 ,𝒘 + 𝒃 − 𝟏 ≥ 𝟎
●
2
𝑤 𝑖
𝐿 𝑤, 𝑏, 𝛼 = − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 𝛼𝑖 ≥ 0
2 𝑖

𝜕𝐿
= 𝑤 − ෍ 𝛼𝑖 𝑦 𝑖 𝑥Ԧ 𝑖
= 0 ⟹ 𝒘 = ෍ 𝜶𝒊 𝒚 𝒊 𝒙 𝒊
𝜕𝑤 𝑖 𝒊 𝛼𝑖

𝜕𝐿 𝑖 𝒊
= − ෍ 𝛼𝑖 𝑦 = 0 ⟹ ෍ 𝜶𝒊 𝒚 =𝟎
𝜕𝑏 𝑖 𝒊

45
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖

𝑤 = ෍ 𝛼𝑖 𝑦 𝑖 𝑥Ԧ 𝑖
𝑖
𝑖
෍ 𝛼𝑖 𝑦 =0
𝑖

46
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖
1
𝑤 = ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 𝑖 𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 𝑖 2 𝑖 𝑗
𝑖
෍ 𝛼𝑖 𝑦 =0
𝑖

47
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖
1
𝑤 = ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 𝑖 𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 𝑖 2 𝑖 𝑗
𝑖
෍ 𝛼𝑖 𝑦 =0
𝑖

48
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖
1
𝑤 = ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 𝑖 𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 𝑖 2 𝑖 𝑗
𝑖
෍ 𝛼𝑖 𝑦 =0
𝑖

● 𝑥Ԧ 𝑥,
Ԧ 𝑤 +𝑏≥0
𝑖
now it becomes: ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0
𝑖

49
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖
1
𝑤 = ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 𝑖 𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 𝑖 2 𝑖 𝑗
𝑖
෍ 𝛼𝑖 𝑦 =0
𝑖

● 𝑥Ԧ 𝑥,
Ԧ 𝑤 +𝑏≥0
𝑖
now it becomes: ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0
𝑖
●

50
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖
1
𝑤 = ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 𝑖 𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 𝑖 2 𝑖 𝑗
𝑖
෍ 𝛼𝑖 𝑦 =0 𝛼𝑖
𝑖

● 𝑥Ԧ 𝑥,
Ԧ 𝑤 +𝑏≥0
𝑖
now it becomes: ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0
𝑖
●

51
 2
𝑤 𝑖
𝐿= − ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1
2 𝑖
1
𝑤 = ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 𝑖 𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 𝑖 2 𝑖 𝑗
𝑖
෍ 𝛼𝑖 𝑦 =0 𝛼𝑖
𝑖

● 𝑥Ԧ 𝑥,
Ԧ 𝑤 +𝑏≥0 𝑏
𝑖
𝑖 ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ+ + 𝑏 = 1
now it becomes: ෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 𝑖
𝑖
● 𝑥Ԧ+

52
●

𝒊
𝒙 ෍ 𝜶𝒊 𝒚 𝒙 𝒊 ,𝒙 + 𝒃 ≥ 𝟎
𝒊

● 𝛼𝑖 b

𝟏
෍ 𝜶𝒊 − ෍ ෍ 𝜶𝒊 𝜶𝒋 𝒚 𝒊 𝒚 𝒋 𝒙 𝒊 ,𝒙 𝒋
𝒊 𝟐 𝒊 𝒋
𝜶𝒊 ≥ 𝟎 ∀𝒊

53
 𝑤 2
𝑥,
Ԧ 𝑤 +𝑏 ≥0 2
𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0
1
𝑖 𝑖 ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
෍ 𝛼𝑖 𝑦 𝑥Ԧ , 𝑥Ԧ + 𝑏 ≥ 0 𝑖 2 𝑖 𝑗
𝑖
α𝑖 ≥ 0

• • 𝑚≫ 𝑛
𝑛
• • 𝑛≈𝑚
𝑚
•
𝑏
54
●

55
●

●
●

56
57
58
●

59
𝑥Ԧ ∈ ℝ, 𝑥Ԧ = 𝑥1 , 𝑥2
𝜙: ℝ2 ⟶ ℝ3 , 𝜙 𝑥1 , 𝑥2 = 𝑥1′ , 𝑥2′ , 𝑥3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2

𝑥2

𝑥1 60
 𝑥Ԧ ∈ ℝ, 𝑥Ԧ = 𝑥1 , 𝑥2
𝜙: ℝ2 ⟶ ℝ3 , 𝜙 𝑥1 , 𝑥2 = 𝑥1′ , 𝑥2′ , 𝑥3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2

𝜙
𝑥2

𝑥1 61
 𝑥Ԧ ∈ ℝ, 𝑥Ԧ = 𝑥1 , 𝑥2
𝜙: ℝ2 ⟶ ℝ3 , 𝜙 𝑥1 , 𝑥2 = 𝑥1′ , 𝑥2′ , 𝑥3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2

𝜙
𝑥2

𝑥1 62
●
1
𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 2 𝑖 𝑗

𝑖
෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 ⇒ 𝑥Ԧ is a +
𝑖

● 𝜙

1
𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝜙 𝑥Ԧ 𝑖
, 𝜙 𝑥Ԧ 𝑗
𝑖 2 𝑖 𝑗

𝑖 𝑖
෍ 𝛼𝑖 𝑦 𝜙 𝑥Ԧ , 𝜙 𝑥Ԧ + 𝑏 ≥ 0 ⇒ 𝑥Ԧ is a +
𝑖

63
●
1
𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 2 𝑖 𝑗

𝑖
෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 ⇒ 𝑥Ԧ is a +
𝑖

● 𝜙

1
𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝜙 𝑥Ԧ 𝑖
, 𝜙 𝑥Ԧ 𝑗
𝑖 2 𝑖 𝑗

𝑖 𝑖
෍ 𝛼𝑖 𝑦 𝜙 𝑥Ԧ , 𝜙 𝑥Ԧ + 𝑏 ≥ 0 ⇒ 𝑥Ԧ is a +
𝑖

64
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

65
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

𝑥 ′, 𝑦 ′ = 𝑥1′ , 𝑥2′ , 𝑥3′ , 𝑦1′ , 𝑦2′ , 𝑦3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2 , 𝑦12 , 𝑦22 , 𝑦1 𝑦2 2

66
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

𝑥 ′ , 𝑦 ′ = 𝑥1′ , 𝑥2′ , 𝑥3′ , 𝑦1′ , 𝑦2′ , 𝑦3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2 , 𝑦12 , 𝑦22 , 𝑦1 𝑦2 2 =
= 𝑥12 𝑦12 + 𝑥22 𝑦22 + 2𝑥1 𝑥2 𝑦1 𝑦2

67
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

𝑥 ′ , 𝑦 ′ = 𝑥1′ , 𝑥2′ , 𝑥3′ , 𝑦1′ , 𝑦2′ , 𝑦3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2 , 𝑦12 , 𝑦22 , 𝑦1 𝑦2 2 =
= 𝑥12 𝑦12 + 𝑥22 𝑦22 + 2𝑥1 𝑥2 𝑦1 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2 2

68
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

𝑥 ′ , 𝑦 ′ = 𝑥1′ , 𝑥2′ , 𝑥3′ , 𝑦1′ , 𝑦2′ , 𝑦3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2 , 𝑦12 , 𝑦22 , 𝑦1 𝑦2 2 =
= 𝑥12 𝑦12 + 𝑥22 𝑦22 + 2𝑥1 𝑥2 𝑦1 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2 2 = 𝑥, Ԧ 𝑦Ԧ 2

69
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

𝑥 ′ , 𝑦 ′ = 𝑥1′ , 𝑥2′ , 𝑥3′ , 𝑦1′ , 𝑦2′ , 𝑦3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2 , 𝑦12 , 𝑦22 , 𝑦1 𝑦2 2 =
= 𝑥12 𝑦12 + 𝑥22 𝑦22 + 2𝑥1 𝑥2 𝑦1 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2 2 = 𝑥, Ԧ 𝑦Ԧ 2
The dot product in the transformed
space is the square of the dot
product in the original space.

70
Ԧ 𝑦Ԧ ∈ ℝ2 , 𝜙 𝑥Ԧ = 𝑥 ′ , 𝜙 𝑦Ԧ = 𝑦 ′
𝑥,

𝑥,
Ԧ 𝑦Ԧ = 𝑥1 , 𝑥2 , 𝑦1 , 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2

𝑥 ′ , 𝑦 ′ = 𝑥1′ , 𝑥2′ , 𝑥3′ , 𝑦1′ , 𝑦2′ , 𝑦3′ = 𝑥12 , 𝑥22 , 𝑥1 𝑥2 2 , 𝑦12 , 𝑦22 , 𝑦1 𝑦2 2 =
= 𝑥12 𝑦12 + 𝑥22 𝑦22 + 2𝑥1 𝑥2 𝑦1 𝑦2 = 𝑥1 𝑦1 + 𝑥2 𝑦2 2 = 𝑥, Ԧ 𝑦Ԧ 2
The dot product in the transformed
space is the square of the dot
product in the original space.
𝐾: ℝ2 × ℝ2 ⟶ ℝ, 𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ 2 = 𝜙 𝑥Ԧ , 𝜙 𝑦Ԧ

The function 𝐾 is defined in the

original space, but computes the dot
product in the transformed space!
71
●

72
●

73
●

2
𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ = 𝜙 𝑥Ԧ , 𝜙 𝑦Ԧ

74
●

2
𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ = 𝜙 𝑥Ԧ , 𝜙 𝑦Ԧ
●
1
𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 2 𝑖 𝑗

෍ 𝛼𝑖 𝑦 𝑖 𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 ⇒ 𝑥Ԧ is a +
𝑖

75
 The decision boundary is no longer
● linear in the original space.

2
𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ = 𝜙 𝑥Ԧ , 𝜙 𝑦Ԧ
●
1
𝐿 = ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
𝑖 2 𝑖 𝑗

෍ 𝛼𝑖 𝑦 𝑖 𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 ⇒ 𝑥Ԧ is a +
𝑖

76
 The decision boundary is no longer
● linear in the original space.

77
●

𝒙 ෍ 𝜶𝒊 𝒚 𝒊 𝑲 𝒙 𝒊 , 𝒙 + 𝒃 ≥ 𝟎
𝒊

● 𝛼𝑖 b

𝟏
෍ 𝜶𝒊 − ෍ ෍ 𝜶𝒊 𝜶𝒋 𝒚 𝒊 𝒚 𝒋 𝑲 𝒙 𝒊 , 𝒙 𝒋
𝒊 𝟐 𝒊 𝒋
𝐬𝐮𝐛𝐣𝐞𝐜𝐭 𝐭𝐨 𝜶𝒊 ≥ 𝟎 ∀𝒊

78
𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ

𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ + 𝑐 𝑑

Ԧ
𝑥−𝑦 2
−
𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑒 2𝜎2

𝐾 𝑥,
Ԧ 𝑦Ԧ = tanh 𝛾 𝑥,
Ԧ 𝑦Ԧ + 𝑐

79
 𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ

𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑥,
Ԧ 𝑦Ԧ + 𝑐 𝑑

Ԧ
𝑥−𝑦 2
−
𝐾 𝑥,
Ԧ 𝑦Ԧ = 𝑒 2𝜎2

𝐾 𝑥,
Ԧ 𝑦Ԧ = tanh 𝛾 𝑥,
Ԧ 𝑦Ԧ + 𝑐

Kernel functions need to be symmetric and positive

semidefinite (a.k.a. Mercer’s condition), otherwise the
existence of the underlying mapping 𝜙 is not guaranteed.
80
 𝑤 2
𝑥,
Ԧ 𝑤 +𝑏 ≥0 2
𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0

1
𝑖 ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
෍ 𝛼𝑖 𝑦 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 𝑖 2 𝑖 𝑗
𝑖
α𝑖 ≥ 0

1
෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
෍ 𝛼𝑖 𝑦 𝑖 𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ + 𝑏 ≥ 0 𝑖 2 𝑖 𝑗
𝑖
α𝑖 ≥ 0

81
 𝑤 2
𝑥,
Ԧ 𝑤 +𝑏≥0 2
𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0

1
𝑖 𝑖 ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗
𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
෍ 𝛼𝑖 𝑦 𝐾 𝑥Ԧ , 𝑥Ԧ + 𝑏 ≥ 0 𝑖 2 𝑖 𝑗
𝑖
α𝑖 ≥ 0

82
83
●

84
●

85
●

86
●

87
●

88
●

89
●

90
●
● ξ𝑖 ≥ 0 𝑥Ԧ 𝑖

𝝃𝒊

91
●
● ξ𝑖 ≥ 0 𝑥Ԧ 𝑖

𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0 𝑦 𝑖
𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ −ξ𝑖
𝝃𝒊

92
●
● ξ𝑖 ≥ 0 𝑥Ԧ 𝑖

𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0 𝑦 𝑖
𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ −ξ𝑖
𝝃𝒊
●
𝐰 𝟐
+ 𝐂 ෍ 𝛏𝐢
𝟐 𝒊
𝒊
𝒚 𝒙 𝒊 , 𝒘 + 𝒃 ≥ 𝟏 − 𝛏𝐢

93
●
● ξ𝑖 ≥ 0 𝑥Ԧ 𝑖

𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0 𝑦 𝑖
𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ −ξ𝑖
𝝃𝒊
●
𝐰 𝟐
+ 𝐂 ෍ 𝛏𝐢
𝟐 𝒊
𝒊
𝒚 𝒙 𝒊 , 𝒘 + 𝒃 ≥ 𝟏 − 𝛏𝐢
●

94
 2
𝑤
+ 𝐶 ෍ 𝜉𝑖
2 𝑖
𝑖 𝑖
𝑦 𝑥Ԧ , 𝑤 + 𝑏 ≥ 1 − 𝜉𝑖
𝜉𝑖 ≥ 0

95
 2
𝑤
+ 𝐶 ෍ 𝜉𝑖
2 𝑖
𝑖 𝑖
𝑖 ⟹ 𝜉𝑖 = max 0, 1 − 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏
𝑦 𝑥Ԧ , 𝑤 + 𝑏 ≥ 1 − 𝜉𝑖
𝜉𝑖 ≥ 0

96
 2
𝑤 𝑖
+ 𝐶 ෍ max 0, 1 − 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏
2 𝑖

97
 2
𝑤 𝑖
+ 𝐶 ෍ max 0, 1 − 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏
2 𝑖

𝓛 𝒙 ≝ 𝐦𝐚𝐱 𝟎, 𝟏 − 𝒚 𝒙, 𝒘 + 𝒃

98
 2
𝑤 𝑖
+ 𝐶 ෍ max 0, 1 − 𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏
2 𝑖

𝓛 𝒙 ≝ 𝐦𝐚𝐱 𝟎, 𝟏 − 𝒚 𝒙, 𝒘 + 𝒃

𝑤 2
⟹ 𝐶 ෍ ℒ 𝑥Ԧ 𝑖 +
𝑖 2

What the soft-margin SVM is actually

doing is minimizing the hinge loss
with Tikhonov regularization.
99
●
𝒙 𝒙, 𝒘 + 𝐛 ≥ 𝟎

● 𝑤 𝑏

𝒘 𝟐
+ 𝐂 ෍ 𝛏𝐢
𝟐 𝒊
𝒚𝒊 𝒊
𝒙 , 𝒘 + 𝒃 ≥ 𝟏 − 𝝃𝒊

𝟐
𝒘 𝒊
+ 𝑪 ෍ 𝐦𝐚𝐱 𝟎, 𝟏 − 𝒚 𝒙 𝒊 ,𝒘 + 𝒃
𝟐 𝒊

100
●
𝒊
𝒙 ෍ 𝜶𝒊 𝒚 𝒙 𝒊 ,𝒙 + 𝒃 ≥ 𝟎
𝒊

● 𝛼𝑖 𝑏
𝟏
෍ 𝜶𝒊 − ෍ ෍ 𝜶𝒊 𝜶𝒋 𝒚 𝒊 𝒚 𝒋
𝒙 𝒊 ,𝒙 𝒋
𝒊 𝟐 𝒊 𝒋
𝟎 ≤ 𝜶𝒊 ≤ 𝑪 ∀𝒊

The only difference from the hard-margin is the

upper-bound on the Lagrange multipliers.

101
 𝑤 2
𝑥,
Ԧ 𝑤 +𝑏≥0 2
𝑖
𝑦 𝑥Ԧ 𝑖 , 𝑤 + 𝑏 − 1 ≥ 0

1
𝑖 𝑖 ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
෍ 𝛼𝑖 𝑦 𝑥Ԧ , 𝑥Ԧ + 𝑏 ≥ 0 𝑖 2 𝑖 𝑗
𝑖
α𝑖 ≥ 0

𝑤 2
+ 𝐶 ෍ 𝜉𝑖
𝑥,
Ԧ 𝑤 +𝑏 ≥0 2 𝑖
𝑖 𝑖
𝑦 𝑥Ԧ , 𝑤 + 𝑏 ≥ 1 − 𝜉𝑖

1
𝑖 𝑖 ෍ 𝛼𝑖 − ෍ ෍ 𝛼𝑖 𝛼𝑗 𝑦 𝑖 𝑦 𝑗 𝐾 𝑥Ԧ 𝑖 , 𝑥Ԧ 𝑗
෍ 𝛼𝑖 𝑦 𝑥Ԧ , 𝑥Ԧ + 𝑏 ≥ 0 𝑖 2 𝑖 𝑗
𝑖
0 ≤ α𝑖 ≤ 𝐶

102
103
●

104
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

105
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

106
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

107
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

𝑷 𝒚=𝟏
=

𝟏
𝟏 + 𝒆𝑨𝒇 𝒙 +𝑩

• 𝑓 𝑥Ԧ ∈ ℝ
• 𝐴, 𝐵 ∈ ℝ 108
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

𝑷 𝒚=𝟏
=

𝟏
𝟏 + 𝒆𝑨𝒇 𝒙 +𝑩

• 𝑓 𝑥Ԧ ∈ ℝ
• 𝐴, 𝐵 ∈ ℝ 109
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

𝑷 𝒚=𝟏
=

𝟏
𝟏 + 𝒆𝑨𝒇 𝒙 +𝑩

• 𝑓 𝑥Ԧ ∈ ℝ
• 𝐴, 𝐵 ∈ ℝ 110
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

𝑷 𝒚=𝟏
=

𝟏
𝟏 + 𝒆𝑨𝒇 𝒙 +𝑩

• 𝑓 𝑥Ԧ ∈ ℝ
• 𝐴, 𝐵 ∈ ℝ 111
●
● 𝒇 𝒙 = 𝒙, 𝒘 + 𝒃

𝑷 𝒚=𝟏
=

𝟏
𝟏 + 𝒆𝑨𝒇 𝒙 +𝑩

• 𝑓 𝑥Ԧ ∈ ℝ
• 𝐴, 𝐵 ∈ ℝ 112
●

● 𝑓 𝑥Ԧ
1
𝑃 𝑦=1 = 𝑥Ԧ +𝐵
1 + 𝑒 𝐴𝑓
●

113
114
●

115
●

●
○ 𝑛
○

●
○ 𝑛(𝑛 − 1)/2
○

116
●

●
○ 𝑛
○

●
○ 𝑛(𝑛 − 1)/2
○

○
■

117
118
● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

119
 ● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

•
𝑤

• 𝑤
𝑥,
Ԧ 𝑤 +𝑏+𝜖
±𝜖
𝝐
𝒘 𝟐
𝑥,
Ԧ 𝑤 +𝑏
𝟐
𝒊 𝒊
𝒚 − 𝒙 ,𝒘 − 𝒃 ≤ 𝝐
𝑥,
Ԧ 𝑤 +𝑏−𝜖

𝑤 =0 120
● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

•
𝑤

• 𝑤

±𝜖
𝝐
𝒘 𝟐
𝟐
𝒊 𝒊
𝒚 − 𝒙 ,𝒘 − 𝒃 ≤ 𝝐

𝝐 𝒘 𝒃
121
● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

•
𝑤

• 𝑤

𝝐 ±𝜖

𝒘 𝟐
𝟐
𝒊 𝒊
𝒚 − 𝒙 ,𝒘 − 𝒃 ≤ 𝝐

𝝐 𝒘 𝒃
122
● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

•
𝑤

• 𝑤

𝝐 ±𝜖

𝒘 𝟐
𝟐
𝒊 𝒊
𝒚 − 𝒙 ,𝒘 − 𝒃 ≤ 𝝐

⟹
𝝐 𝒘 𝒃
123
● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

•
±ϵ

124
● 𝑦ො = 𝑥,
Ԧ 𝑤 +𝑏
● 𝜖

•
𝝃𝒊 ±ϵ
𝝐 •

𝒘 𝟐
+ 𝑪 ෍ 𝝃𝒊
𝟐 𝒊
𝒊 𝒊
𝒚 − 𝒙 , 𝒘 − 𝒃 ≤ 𝝐 + 𝝃𝒊

𝓛 𝒙 ≝ 𝐦𝐚𝐱 𝟎, 𝒚 − 𝒙, 𝒘 − 𝒃 − 𝝐 𝜖

𝟐
𝒊
𝒘
⟹ minimize 𝑪 ෍ 𝓛 𝒙 +
𝒊 𝟐

125
1 from sklearn.svm import SVC, SVR
2
3 clf = SVC(C = 100, kernel = ‘linear’, probability = True) # soft-margin SVM classifier
4 clf.fit(X, y)
5 clf.predict([x]) # predicted class for x
6 clf.decision_function([x]) # actual value of the decision function at point x
7 clf.predict_proba([x]) # predicted probability for x (only works if probability was True at init)
8
9 clf.n_support_ # number of support vectors per class
10 clf.support_ # indices of support vectors
11 clf.support_vectors_ # support vectors (selected rows from X: X[clf.support_])
12 clf.dual_coef_ # lagrange multiplers multipled by labels (𝛼𝑖 ∙ y𝑖 ) Obs: 𝑦𝑖 = ±1, not your labels
13 clf.intercept_ # b
14 clf.coef_ # 𝑤 = clf.dual_coef.dot(clf.support_vectors_), only available for linear kernel
15
16 reg = SVR(C = 10, epsilon = 0.2, kernel = ‘rbf’) # svm regression
126
1 from sklearn.svm import LinearSVC, LinearSVR
2 # LinearSVC implements the svm classifier with linear kernel, but it is more flexible than SVC.
3 # It can optimize either the primal or dual
4 # It can use 𝐿2 or 𝐿1 for 𝑤 minimization
5 # It can minimize the hinge or squared-hinge losses (squared-hinge is the default)
6 # For multi-class problems, it can also use the crammer-singer loss
7
8 clf = LinearSVC(C = 100, dual = False, penalty = ‘l2’, loss = ‘hinge’)
9 clf.fit(X, y)
10 clf.predict([x]) # predicted class for x
11 clf.decision_function([x]) # actual value of the decision function at point x

127
●
○

●
○

128
●
○
○
●
○
○
●
○
●
●
●
129
130