Lecture 9

EE531 Statistical Learning Theory

Contents
Regression : Definition
Linear Basis function
Linear regression
ML
Ordinary least squares (OLS)

Regularization methods of linear regression

Ridge regression, Lasso
Graphical interpretations

Bayesian linear regression

Polynomial Curve Fitting
Example

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Regression: Definition

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Basis Function

1
0.9
0.8
0.7

sigma(a)

0.6
0.5
0.4
0.3
0.2
0.1
0
-10

EE531 Statistical Learning Theory

Spring, 2016

-8

-6

-4

-2

0
a

10

Korea Advanced Institute of Science and Technology

Linear Basis Function: Purpose

We don't assume that this model is actually true. We choose this model
because either it is computationally expedient or we have too little data to use
a more complex model (non-linear model) without overfitting.

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Regression: min Risk

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Regression: ML

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

OLS: Residual Square Sum (RSS)

N
data
items

EE531 Statistical Learning Theory

0
1
1
.
.
.

Spring, 2016

Korea Advanced Institute of Science and Technology

OLS: interpretation

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

OLS: limitation

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Ridge Regression

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Regularization methods of linear

regression :

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Regularization methods of
linear regression: Graphical Interpretation

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression: Intro.

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting

Goal = given predict on the basis of
corresponding labels (target values)
where

and
. e.g.

(1) Simple approach (fit with polynomial)

Define error function

Obtain
Evaluate generalization performance using test data separate from training data

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

Issue: What is an appropriate value for

Plots of polynomials having various orders

data set
EE531 Statistical Learning Theory

Spring, 2016

? Over-fitting can occur.

, shown as red curves, fitted to the

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

Graphs of the root-mean-square error, evaluated on the training set and on an

independent test set for various values of
EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

Table of the coefficients

for polynomials of various order. Observe how the
typical magnitude of the coefficients increases dramatically as the order of the
polynomial increases.
EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

(2) More Advanced Approach (Regularization)

Plots of
polynomials fitted to the data set using the regularized error
function for two values of the regularization parameter corresponding
to
and
. The case of no regularizer, i.e.,
,
corresponding to
, is shown at the bottom right of slide 16.
EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

(3) Probabilistic Approach (ML, MAP)
Assume

Schematic illustration of a Gaussian conditional distribution

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

For i.i.d. data

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

Assume

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

(4) Bayesian Approach
Predictive distribution

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

Given a marginal Gaussian distribution for
distribution for
given
in the form

the marginal distribution of

given by

and a conditional Gaussian

and conditional distribution of

given

are

where

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Polynomial Curve Fitting: cont'd

polynomial regression model

corresponding to left figure

Joint distribution

Predictive distribution

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression: Example 1

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression: Example 1

1st data

0.8
0.6
0.4
0.2

2nd data

1
0.8
0.6
0.4
0.2

3rd data

-1
-1
1

1
0.8
0.6
0.4
0.2

0
-1
-1

0.3

0.2

-1
-1
1

-1
-1
1

0.1
0

1
1.4
1.2
1
0.8
0.6
0.4
0.2

0
-1
-1
1

-1
-1
1

2
1

-1
-1
1

3
2

0
-1
-1
1

-1
-1
1

1
-1
-1

EE531 Statistical Learning Theory

-1
-1

Spring, 2016

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression: Example 1

1
5th data

0.8
0.6
0.4
0.2

0
-1
-1
1

1
0.8
0.6
0.4
0.2

7th data

0
-1
-1
1

1
0.8
0.6
0.4
0.2

9th data

0
-1
-1
1

-1
-1
1

-1
-1
1

10
8
6
4
2

0
-1
-1
1

0
-1
-1
1

10
0
-1
-1
1

5
0

-1
-1
1

10

-1
-1

11th data

0
-1
-1

0.5
0

EE531 Statistical Learning Theory

0
-1
-1

Spring, 2016

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression: Example 1

1

1
12th data

0
-1
-1
1

0.5
0

0.8
0.6
0.4
0.2

14th data

0
-1
-1
1

1
0.8
0.6
0.4
0.2

17th data

20th data

-1
-1
1

1
0.8
0.6
0.4
0.2

0
-1
-1

EE531 Statistical Learning Theory

0
-1
-1
1

-1
-1
1

20

0
-1
-1
1

10

10

-1
-1
1

20
0
-1
-1
1

10
0

0
30
20

0
-1
-1
1
0

10
-1
-1

Spring, 2016

-1
-1

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression:

Examples of the predictive distribution(3.58) for a model consisting of 9

Gaussian basis functions of the form(3.4) using the synthetic sinusoidal data
set.
(3.58)
(3.4)

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Bayesian Linear Regression: cont'd

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Summary
Regression : Definition
Linear Basis function
Linear regression
ML
Ordinary least squares (OLS)

Regularization methods of linear regression

Ridge regression, Lasso
Graphical interpretations

Bayesian linear regression

Polynomial Curve Fitting
Example
Bayesian treatment of linear regression avoids the over fitting problem of ML
and leads to automatic methods of determining model complexity using training
data alone.

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology