CS189/CS289A

Introduction to Machine Learning

Lecture 1: Overview
Alexei Efros and Peter Bartlett

January 20, 2015

1 / 37

Organizational Issues

2 / 37

Organizational Issues

Instructors: Peter Bartlett and Alyosha Efros.

2 / 37

Organizational Issues

Instructors: Peter Bartlett and Alyosha Efros.

GSIs: Peter Gao, Yun Park, Faraz Tavakoli, Kevin Tee, Pat Virtue,
Christopher Xie, Daniel Xu, Yuchen Zhang.

2 / 37

Organizational Issues

Instructors: Peter Bartlett and Alyosha Efros.

GSIs: Peter Gao, Yun Park, Faraz Tavakoli, Kevin Tee, Pat Virtue,
Christopher Xie, Daniel Xu, Yuchen Zhang.
Discussion sections: You choose. If the room is full, please go to
another one. (If necessary, we may offer some specialty
sectionswatch website for announcements.)

2 / 37

Organizational Issues

Instructors: Peter Bartlett and Alyosha Efros.

GSIs: Peter Gao, Yun Park, Faraz Tavakoli, Kevin Tee, Pat Virtue,
Christopher Xie, Daniel Xu, Yuchen Zhang.
Discussion sections: You choose. If the room is full, please go to
another one. (If necessary, we may offer some specialty
sectionswatch website for announcements.)
Office hours: see web site.

2 / 37

Organizational Issues

Instructors: Peter Bartlett and Alyosha Efros.

GSIs: Peter Gao, Yun Park, Faraz Tavakoli, Kevin Tee, Pat Virtue,
Christopher Xie, Daniel Xu, Yuchen Zhang.
Discussion sections: You choose. If the room is full, please go to
another one. (If necessary, we may offer some specialty
sectionswatch website for announcements.)
Office hours: see web site.
http://www-inst.eecs.berkeley.edu/cs189
bCourses (+ piazza, kaggle), office hours, syllabus, assignments,
readings, lecture slides, announcements.

2 / 37

Organizational Issues

Assessment:
CS189

Homework 40%
Implementation and application of methods. (Kaggle)
Mathematical/reinforcement of concepts.
Seven total.
Late policy: 5 slip days total. Thats it.

Midterm 20%
(Thursday, March 19, in the lecture slot.)
Final Exam 40%

3 / 37

Organizational Issues

Assessment:
CS289A Plus a project:
Homework 40%
Midterm 20%
Final Exam 20%
Final Project 20%
(due Friday, May 1. Proposal due Friday, April 3.)

4 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).

5 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).
No screens in lectures. (To see why, google laptops in class.)

5 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).
No screens in lectures. (To see why, google laptops in class.)
Ethics:

5 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).
No screens in lectures. (To see why, google laptops in class.)
Ethics:
Discussion of homework problems with other students is encouraged.

5 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).
No screens in lectures. (To see why, google laptops in class.)
Ethics:
Discussion of homework problems with other students is encouraged.
All homeworks must be written individually (including programming
components).

5 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).
No screens in lectures. (To see why, google laptops in class.)
Ethics:
Discussion of homework problems with other students is encouraged.
All homeworks must be written individually (including programming
components).
Please read the department policy on academic dishonesty. We will be
actively checking for plagiarism.

5 / 37

Organizational Issues

(Real) Prerequisites:
Math53 (vector calculus); Math54 (linear algebra); CS70 (discrete
math, probability); CS188 (more probability, decision theory).
No screens in lectures. (To see why, google laptops in class.)
Ethics:
Discussion of homework problems with other students is encouraged.
All homeworks must be written individually (including programming
components).
Please read the department policy on academic dishonesty. We will be
actively checking for plagiarism.

Questions: Use piazza. Public and private.

5 / 37

Texts
Springer Series in Statistics

Trevor Hastie
Robert Tibshirani
Jerome Friedman

The Elements of
Statistical Learning
Data Mining, Inference, and Prediction
Second Edition

6 / 37

CS189: Introduction to Machine Learning

7 / 37

CS189: Introduction to Machine Learning

Machine Learning
Systems that learn to solve
information processing problems.

7 / 37

CS189: Introduction to Machine Learning

Machine Learning
Systems that learn to solve
information processing problems.

Learn
Use experience to improve performance:
data, queries, interaction, experiments
Statistical issues are central.

7 / 37

CS189: Introduction to Machine Learning

Machine Learning
Systems that learn to solve
information processing problems.

Learn
Use experience to improve performance:
data, queries, interaction, experiments
Statistical issues are central.

Systems
Computational issues are also central.
Algorithms, optimization.
7 / 37

An Overview of Machine Learning

1
2
3

8 / 37

An Overview of Machine Learning

Problems

2
3

8 / 37

An Overview of Machine Learning

Problems

Methods

8 / 37

An Overview of Machine Learning

Problems

Methods

Concepts

8 / 37

An Overview of Machine Learning

Problems

Methods

Concepts

8 / 37

Classification Problems (Homework)

Email

9 / 37
ESL

Classification Problems (Homework)

ESL

10 / 37

Classification

11 / 37

Classification

microsoft.com

12 / 37

Classification

apple.com

ESL
13 / 37

Classification

ISLR
14 / 37

Classification

ISLR
15 / 37

Classification

ESL
16 / 37

Regression

ESL
17 / 37

Regression

ESL
18 / 37

Regression

ESL
19 / 37

Regression

ESL

20 / 37

Regression

ESL

21 / 37

Density Estimation

ESL

22 / 37

Density Estimation

ESL
23 / 37

Dimensionality Reduction

ESL
24 / 37

Dimensionality Reduction

ESL

25 / 37

Dimensionality Reduction

ESL
26 / 37

Clustering

ESL
27 / 37

Clustering

28 / 37

Clustering

ESL

29 / 37

Clustering

ESL

30 / 37

Machine Learning Problems

Classification

31 / 37

Machine Learning Problems

Classification
Regression

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking
Collaborative filtering

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking
Collaborative filtering
Sequential decision
problems:

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking
Collaborative filtering
Sequential decision
problems:
bandits

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking
Collaborative filtering
Sequential decision
problems:
bandits
contextual bandits

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking
Collaborative filtering
Sequential decision
problems:
bandits
contextual bandits
dynamic pricing

31 / 37

Machine Learning Problems

Classification
Regression
Density estimation
Dimensionality reduction
Clustering
Ranking
Collaborative filtering
Sequential decision
problems:
bandits
contextual bandits
dynamic pricing
reinforcement learning

31 / 37

An Overview of Machine Learning

Problems

Methods

Concepts

32 / 37

Methods
Linear classifiers: Perceptron
Support vector machines
Gaussian class conditionals
Logistic regression
Naive Bayes
Linear discriminant analysis
Linear regression
Decision trees, regression trees
Ensemble methods
Neural networks
Nearest neighbor
Principal components analysis
k-means clustering
33 / 37

Methods
Linear classifiers: Perceptron
Support vector machines
Gaussian class conditionals
Logistic regression
Naive Bayes
Linear discriminant analysis
1

Classification

Regression

Linear regression
Decision trees, regression trees
Ensemble methods
Neural networks
Nearest neighbor
Principal components analysis
k-means clustering

33 / 37

Methods
Linear classifiers: Perceptron
Support vector machines
Gaussian class conditionals
Logistic regression
Naive Bayes
Linear discriminant analysis

Probabilistic
modeling.

Prediction; not based

on a model.

Linear regression
Decision trees, regression trees
Ensemble methods
Neural networks
Nearest neighbor
Principal components analysis
k-means clustering

33 / 37

An Overview of Machine Learning

Problems

Methods

Concepts

34 / 37

Concepts
1

Prediction versus probabilistic modeling.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:
Bias-variance/approximation-estimation trade-off.

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:
Bias-variance/approximation-estimation trade-off.
Regularization

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:
Bias-variance/approximation-estimation trade-off.
Regularization
Priors

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:
Bias-variance/approximation-estimation trade-off.
Regularization
Priors

Practical issues:

35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:
Bias-variance/approximation-estimation trade-off.
Regularization
Priors

Practical issues:
Train/validate/test. Over-fitting.
35 / 37

Concepts
1
2

Prediction versus probabilistic modeling.

Probabilistic modeling:
Generative versus discriminative models.
Maximum likelihood estimation.
Bayesian inference.

Optimization.
Convexity.
(Stochastic) gradient methods.
Newtons method.

Controlling complexity:
Bias-variance/approximation-estimation trade-off.
Regularization
Priors

Practical issues:
Train/validate/test. Over-fitting.
Resampling methods.
35 / 37

Overview (Part I: Bartlett)

Linear classification

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression
Optimization

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression
Optimization
Linear Classification revisited

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression
Optimization
Linear Classification revisited
Logistic regression

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression
Optimization
Linear Classification revisited
Logistic regression
Linear Discriminant Analysis

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression
Optimization
Linear Classification revisited
Logistic regression
Linear Discriminant Analysis
Support vector machines

36 / 37

Overview (Part I: Bartlett)

Linear classification
Statistical learning background
Decision theory
Generative and discriminative models
Controlling complexity.
Resampling, cross-validation.
The multivariate normal distribution.

Linear regression
Optimization
Linear Classification revisited
Logistic regression
Linear Discriminant Analysis
Support vector machines

Statistical learning theory

36 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

Boosting

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets
Examples and applications

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets
Examples and applications

Unsupervised methods

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets
Examples and applications

Unsupervised methods
Clustering

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets
Examples and applications

Unsupervised methods
Clustering
Density estimation

37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets
Examples and applications

Unsupervised methods
Clustering
Density estimation
Dimensionality reduction
37 / 37

Overview (Part II: Efros)

1

Memory-based/Instance-based learning
k-nearest-neighbor
Properties of high-dimensional spaces
distance learning
Efficient indexing and retrieval methods

Decision trees
Classification and regression trees
Random Forests

3
4

Boosting
Neural networks / Deep Learning
Multilayer perceptrons
Variations such as convolutional nets
Examples and applications

Unsupervised methods
Clustering
Density estimation
Dimensionality reduction
Applications: Collaborative filtering, etc.
37 / 37