Math 459

Applied Statistical Learning

Homework 1

Work out the following problems by hand.

1) Some of the problems below are best addressed using a supervised learning algorithm, and the
others with an unsupervised learning algorithm. Which of the following would you apply supervised
learning to? In each case, assume some appropriate dataset is available for your algorithm to learn
from.
a. Examine the statistics of two football teams, and predicting which team will win tomorrows
match (given historical data of teams wins/losses to learn from). (2 pts)
b. Examine a large collection of emails that are known to be spam email, to discover if there
are sub-types of spam mail. (2 pts)

2) Suppose you are working on stock market prediction. You would like to predict whether or not
a certain company will declare bankruptcy within the next 7 days (by training on data of similar
companies that had previously been at risk of bankruptcy). Would you treat this as a classification
or a regression problem? (2 pts)

3) Consider the following training set of m = 4 training examples:

x
1
2
4
0

y
0.5
1
2
0

If we fit the linear regression model h (x) = 0 + 1 x, what are the values of 0 and 1 that you
would expect to obtain upon running gradient descent on this model? (Linear regression will be
able to fit this data perfectly.) (5 pts)

4) Find the partial derivative of J(0 , 1 ) with respect to 1 where J(0 , 1 ) and h (x(i) ) are given
below. (5 pts)
m
1 X
J(0 , 1 ) =
(h (x(i) ) y (i) )2
2m
i=1

h (x(i) ) = 0 + 1 x(i) .

5) Perform 3 iterations of gradient descent to get a better estimate of where

J() = 2
where initially = 2 and = 0.2. (5 pts)
1

6) For each part, calculate the cost using the appropriate cost function. The appropriate cost
function is one of the following:
m

1 X
J(0 , 1 ) =
(h (x(i) ) y (i) )2
2m
i=1

#
m
1 X (i)
J(0 , 1 ) =
y log(h (x(i) )) + (1 y (i) ) log(1 h (x(i) )) .
m
"

i=1

a. Regression problem with the following response and hypothesis: (3 pts)

y
3
1
3.5
1

h
2.5
1.5
3
0

b. Classification problem with the following response and hypothesis: (3 pts)

y
0
1
0
1

h
0.1
0.9
0.9
0.9

Helpful hints for working in Matlab.

It is a good practice to start your scripts by clearing all the variables and the command line
window which is easily done with the commands clear all and clc


a b
can be created by the
Creating matrices is simple in matlab, the matrix A =
c d
Matlab command A=[a b; c d]
The % denotes a commented line
We can perform element by element operations by putting a . before the operator (i.e. c =
a./b performs right-array division by dividing each element of a by the corresponding element
of b. If inputs a and b are not the same size, one of them must be a scalar value)
If you want the result of a variable assignment to be displayed then end your line of code
with a ;
The transpose of a matrix A can easily be done by typing A
See http://www.mathworks.com/help/matlab/ref/function.html for information on how
to create a function
2

Use Matlab to answer the following questions.

7) Open the file hw1 prob7.m and fill in the unfinished code for each part.
a. Given the coefficients of a linear regression model () and the feature values (X), calculate
the resulting estimate of y (
y ). (2 pts)

1 4.6
3
1 5.4
2

=
3 , X = 1 5.1
1 5.9
2

6.5 4.3
6.1 4.2

6.3 4
6.8 3.9

b. Given the actual value of y below, calculate a vector of residuals. (2 pts)

40
35

y=
45 .
40

c. Calculate the residual sum of squares. From the residuals in part b, square each value and
sum all of the values. You may use the sum function and element by element operation to square
each value. (3 pts)
d. Calculate the cost function for linear regression. (3 pts)
8) Open the file hw1 prob8.m and simple function.m in Matlab.
a. Fix the simple function.m code so that it returns a = bcT . If you did this correctly, then
after running the code in hw1 prob8.m the variable A will take on the value of 14. (2 pts)
b. From scratch, create a function called half mse.m which calculates half of the MSE as
calculated in problem 7 part d. This function will take in values of , y, and x then return a cost
value J. (4 pts)
c. From scratch, create a function called normal linear regression.m which calculates the
coefficients of linear regression using the normal equation. This function will take in values of y,
and X then return . (4 pts)
9) Open the file hw1 prob9.m in Matlab.
a. Compute the cost of the model using as calculated in problem 8 part c with X and y as
specified in problem 7. (2 pts)
b. Which model appears to be better, the model specified in problem 7 b, or the model we
generated in problem 8 c? (2 pts)
c. Now consider the test data set Xtest and ytest as given. Calculate the cost of using each
model with the test data. (3 pts)

1 4.7 5.9 4.3

39
1 5.8 6.2 4.1
41

Xtest = 1 5.1 6.3 4.1 , ytest =

40
39
1 5.9 5.4 3.9
1 5.0 7.5 4.1
50
3

d. Which model will give us the lowest cost using the test set data from problem 9 c, the model
specified in problem 7 b, or the model we generated in problem 8 c? (2 pts)
10) Open the file hw1 prob10.m in Matlab.
a. The code for part a will open up the file ex1data1.txt and plot the data. It is your job to
modify the plotData.m file so that the data is plotted correctly. (4 pts)
b. The code for part b will format the data so that it is ready to be processed by the function
gradientDescent.m. Modify the gradientDescent.m file so that the algorithm runs correctly.
(5 pts)
c. Provide a description of the plots given in the figures generated from the code for part c. (2
pts)
11) Open the file hw1 prob11.m in Matlab.
a. When using learning algorithms, it is often useful to normalize your feature values. Normalizing features has several advantages, it ensures that your feature values are on the same scale
(usually between -3 and 3) and also makes the resulting linear regression coefficients on the same
scale. Additionally, gradient descent and other algorithms benefit from this scaling making the
algorithm typically run faster. If i and i are the mean and standard deviation of feature i
respectively, then we can get a new feature set
(j)

(j)

zi

x i i
i

(j)

from the original feature value xi . Modify the code in featureNormalize.m so that we take in
X and get the normalized feature matrix Z. (4 pts)
b. First make sure that your functions - half mse.m and gradientDescent.m already work
with this starter code and support multiple variables. In other words, you should now be able to
run the file hw1 prob11.m in Matlab without error. Now choose a value of alpha that will make
the cost function converge (as can be seen in the plot). (2 pts)
c. Estimate the price of a 1650 sq-ft, 3 br house using your gradient descent algorithm. (4 pts)
d. Estimate the price of a 1650 sq-ft, 3 br house using the normal equation. (4 pts)
12) Open the file hw1 prob12.m in Matlab.
a. Visualize the data by running the code for part a. From the code for plotData2.m what
features are accessed when we use the code X(y == 1, :)? (2 pts)
b. We use the function mapFeature.m in this part. What is this function doing? (2 pts)
c. Modify the function sigmoid.m so that it computes the sigmoid of each value of z (z can
be a matrix, vector or scalar). Recall that the sigmoid is
g(z) =

1
.
1 + ez

(3 pts)
d. Modify the function logisticCost.m so that it computes the cost function for logistic
regression. (8 pts)
e. Modify the function logisticCostReg.m so that it computes the cost function for logistic
regression with a regularization term. (7 pts)
f. Run the code for part f. Is the decision boundary appropriate? (2 pts)
4

Name:
Date:
Submission Directions
1) Keep all of the original Matlab files in one directory, and zip the entire directory after completion.
2) Rename the zipped file with your first name (ex: if your name is Dan then the file will be
Dan.zip). Email me the zipped file to dpmcgibney@wm.edu.
3) Complete the following Matlab answer sheet to accompany your Matlab code.
4) Staple this sheet to your hard copy of problems 1 through 6 and submit this stapled set to Prof.
McGibney on the due date.
Answers:
9b

9d

10c

12a

12b

12f