PREDICTIVE ANALYTICS

USING REGRESSION
Sumeet Gupta
Associate Professor
Indian Institute of Management Raipur

Outline
Basic Concepts
Applications of Predictive Modeling
Linear Regression in One Variable using OLS
Multiple Linear Regression
Assumptions in Regression
Explanatory Vs Predictive Modeling
Performance Evaluation of Predictive Models
Practical Exercises
Case: Nils Baker
Case: Pedigree Vs Grit

BASIC CONCEPTS

Predictive Modeling: Applications

Predictive customer activity on credit cards from their

demographic and historical activity patterns

Predicting the time to failure or equipment based on
utilization and environment conditions
Predicting expenditures on vacation travel based on
historical frequent flyer data
Predicting staffing requirements at help desks based on
historical data and product and sales information
Predicting sales from cross selling of products from
historical information
Predicting the impact of discounts on sales in retail outlets

Basic Concept: Relationships

Examples of relationships:
Sales and earnings
Cost and number produced
Microsoft and the stock market
Effort and results

Scatterplot
A picture to explore the relationship in bivariate data
Correlation r
Measures strength of the relationship (from 1 to 1)
Regression
Predicting one variable from the other

Basic Concept: Correlation

r = 1
A perfect straight line
tilting up to the right
r = 0
No overall tilt
No relationship?
r = 1
A perfect straight line
tilting down to the right

X
Y

X
Y

X
Y

X
Y

Basic Concepts: Simple Linear Model

Linear Model for the Population
The foundation for statistical inference in regression
Observed Y is a straight line, plus randomness

Y = + X +
Randomness of individuals
Population relationship, on average
Y

Basic Concepts: Simple Linear Model

Time Spent vs. Internet Pages Viewed
Two measures of the abilities of 25 Internet sites
At the top right are eBay, Yahoo!, and MSN

Correlation is r = 0.964
Linear relationship
Straight line

with scatter
Increasing relationship
Tilts up and to the right

Minutes per person

Very strong positive association (since r is close to 1)

90
eBay
Yahoo!

60
MSN

30
0

100
200
Pages per person

Basic Concepts: Simple Linear Model

Dollars vs. Deals
For mergers and acquisitions by investment bankers
244 deals worth $756 billion by Goldman Sachs

Correlation is r = 0.419
Positive association
Straight line

with scatter
Increasing relationship
Tilts up and to the right

$1,000
Dollars (billions)

Linear relationship

$500

$0

100

200 300
Deals

400

10

Basic Concepts: Simple Linear Model

Interest Rate vs. Loan Fee
For mortgages
If the interest rate is lower, does the bank make it up with a higher loan

fee?
Correlation is r = 0.890
Linear relationship
Straight line

with scatter
Decreasing relationship
Tilts down and to the right

Interest rate

Strong negative association

6.0%
5.5%
5.0%
0%

1%

2%
3%
Loan fee

4%

11

Basic Concepts: Simple Linear Model

Todays vs. Yesterdays Percent Change
Is there momentum?
If the market was up yesterday, is it more likely to be up today? Or is

each days performance independent?

3%

Correlation is r = 0.11
No relationship?
Tilt is neither

up nor down

2%

Today's change

A weak relationship?

1%
0%
-1%
-2%
-3%
-3% -2% -1% 0% 1% 2% 3%
Yesterday's change

12

Basic Concepts: Simple Linear Model

Call Price vs. Strike Price
For stock options
Call Price is the price of the option contract to buy stock at the

Strike Price
The right to buy at a lower strike price has more value
A nonlinear relationship

A curved relationship
Correlation r = 0.895
A negative relationship:

Higher strike price goes

with lower call price

$100

Call Price

Not a straight line:

$75
$50
$25
$0
$450

$500

$550
Strike Price

$600

$650

13

Basic Concepts: Simple Linear Model

Output Yield vs. Temperature
For an industrial process
With a best optimal temperature setting

A nonlinear relationship
Not a straight line:

Correlation r = 0.0155
r suggests no relationship

But relationship is strong

It tilts neither

up nor down

Yield of process

A curved relationship
160
150
140
130
120
500

600 700 800

Temperature

900

14

Basic Concepts: Simple Linear Model

Circuit Miles vs. Investment (lower left)
For telecommunications firms
A relationship with unequal variability
More vertical variation at the right than at the left
Variability is stabilized by taking logarithms (lower right)

r = 0.957
Log of miles

Circuit miles
(millions)

Correlation r = 0.820

2,000
1,000
0
0

1,000 2,000
Investment
($millions)

20

15

15
20
Log of investment

15

Basic Concepts: Simple Linear Model

Price vs. Coupon Payment
For trading in the bond market
Bonds paying a higher coupon generally cost more

Two clusters are visible

Ordinary bonds (value is from coupon)
Inflation-indexed bonds (payout rises with inflation)
for all bonds

Correlation r = 0.994
Ordinary bonds only

Bid price

Correlation r = 0.950

$150
$100
0%

5%
10%
Coupon rate

16

Basic Concepts: Simple Linear Model

Cost vs. Number Produced
For a production facility
It usually costs more to produce more

An outlier is visible
A disaster (a fire at the factory)

Cost

High cost, but few produced

r = 0.623

10,000

20
40
60
Number produced

Cost

5,000

Outlier removed:
More details,
r = 0.869

4,000
3,000
20
30
40
50
Number produced

17

Basic Concepts: OLS Modeling

Salary vs. Years Experience
For n = 6 employees
Linear (straight line) relationship
Increasing relationship
higher salary generally goes with higher experience

Experience
15
10
20
5
15
5

Salary
30
35
55
22
40
27

Salary ($thousand)

Correlation r = 0.8667

60
50
40
30
20
0

10

20 Experience

18

Basic Concepts: OLS Modeling

Summarizes bivariate data: Predicts Y from X
with smallest errors (in vertical direction, for Y axis)
Intercept is 15.32 salary (at 0 years of experience)
Slope is 1.673 salary (for each additional year of experience, on
average)

Salary (Y)

60
50
40
30
20
10
0

10

20

Experience (X)

19

Basic Concepts: OLS Modeling

Predicted Value comes from Least-Squares Line
For example, Mary (with 20 years of experience)
has predicted salary 15.32+1.673(20) = 48.8
So does anyone with 20 years of experience

Residual is actual Y minus predicted Y

Marys residual is 55 48.8 = 6.2
She earns about $6,200 more than the predicted salary for a person

with 20 years of experience

A person who earns less than predicted will have a negative residual

20

Basic Concepts: OLS Modeling

Marys residual is 6.2
60

Mary earns 55 thousand

50
Marys predicted value is 48.8
Salary

40
30
20
10
0

10

Experience

20

21

Basic Concepts: OLS Modeling

Standard Error of Estimate

Se = SY

(1 r ) nn 12
2

Approximate size of prediction errors (residuals)

Actual Y minus predicted Y:
Y[a+bX]
Example (Salary vs. Experience)

Se = 11.686 (1 0.86672 )

6 1
= 6.52
62

Predicted salaries are about 6.52 (i.e., $6,520) away from actual
salaries

22

Basic Concepts: OLS Modeling

Interpretation: similar to standard deviation
Can move Least-Squares Line up and down by Se
About 68% of the data are within one standard error of estimate
of the least-squares line
(For a bivariate normal distribution)

Salary

60
50
40
30
20
0

10 Experience20

23

Multiple Linear Regression

Linear Model for the Population
Y = ( + 1 X1 + 2 X2 + + k Xk) +
= (Population relationship)
+ Randomness
Where has a normal distribution with mean 0 and constant

standard deviation , and this randomness is independent from one

case to another
An assumption needed for statistical inference

24

Multiple Linear Regression: Results

Intercept: a
Predicted value for Y when every X is 0
Regression Coefficients: b1, b2, bk
The effect of each X on Y, holding all other X variables constant
Prediction Equation or Regression Equation
(Predicted Y) = a+b1 X1+b2 X2++bk Xk
The predicted Y, given the values for all X variables
Prediction Errors or Residuals
(Actual Y) (Predicted Y)

25

Multiple Linear Regression: Results

t Tests for Individual Regression Coefficients
Significant or not significant, for each X variable
Tests whether a particular X variable has an effect on Y, holding the
other X variables constant
Should be performed only if the F test is significant
Standard Errors of the Regression Coefficients

Sb1 , Sb2 ,!, Sbk

(with n k 1 degrees of freedom)

Indicates the estimated sampling standard deviation of each

regression coefficient
Used in the usual way to find confidence intervals and hypothesis
tests for individual regression coefficients

26

Multiple Linear Regression: Results

Predicted Page Costs for Audubon
= a + b1 X1 + b2 X2 + b3 X3
= $4,043 + 3.79(Audience) 124(Percent Male)
+ 0.903(Median Income)
= $4,043 + 3.79(1,645) 124(51.1) + 0.903(38,787)

= $38,966
Actual Page Costs are $25,315
Residual is $25,315 38,966 = $13,651
Audubon has Page Costs $13,651 lower than you would expect for
a magazine with its characteristics (Audience, Percent Male, and
Median Income)

27

Standard Error
Standard Error of Estimate Se
Indicates the approximate size of the prediction errors
About how far are the Y values from their predictions?
For the magazine data
Se = S = $21,578
Actual Page Costs are about $21,578 from their predictions for this

group of magazines (using regression)

Compare to SY = $45,446: Actual Page Costs are about $45,446 from
their average (not using regression)
Using the regression equation to predict Page Costs (instead of simply
using Y ) the typical error is reduced from $45,446 to $21,578

28

Coeff. of Determination
The strength of association is measured by the square of the multiple
correlation coefficient, R2, which is also called the coefficient of
multiple determination.

R2 =

SS reg
SS y

R2 is adjusted for the number of independent variables and the sample

size by using the following formula:
Adjusted

R2

= R2

k(1 - R 2)
n-k-1

29

Coeff. of Determination
Coefficient of Determination R2
Indicates the percentage of the variation in Y that is explained by
(or attributed to) all of the X variables
How well do the X variables explain Y?
For the magazine data
R2 = 0.787 = 78.7%
The X variables (Audience, Percent Male, and Median Income) taken

together explain 78.7% of the variance of Page Costs

This leaves 100% 78.7% = 21.3% of the variation in Page Costs
unexplained

30

The F test
Is the regression significant?
Do the X variables, taken together, explain a significant amount of
the variation in Y?
The null hypothesis claims that, in the population, the X variables
do not help explain Y; all coefficients are 0
H0: 1 = 2 = = k = 0
The research hypothesis claims that, in the population, at least

one of the X variables does help explain Y

H1: At least one of 1, 2, , k 0

31

The F test
H0 : R2pop = 0
This is equivalent to the following null hypothesis:

H0: 1 = 2 = 3 = . . . = k = 0
The overall test can be conducted by using an F statistic:
F=

SS reg /k
SS res /(n - k - 1)

R 2 /k
(1 - R 2 )/(n- k - 1)

which has an F distribution with k and (n - k -1) degrees of freedom.

32

Performing the F test

Three equivalent methods for performing F test; they

always give the same result

Use the p-value
If p < 0.05, then the test is significant
Same interpretation as p-values in Chapter 10

Use the R2 value

If R2 is larger than the value in the R2 table, then the result is significant
Do the X variables explain more than just randomness?

Use the F statistic

If the F statistic is larger than the value in the F table, then the result is

significant

33

Example: F test
For the magazine data, The X variables (Audience, Percent
Male, and Median Income) explain a very highly significant

percentage of the variation in Page Costs

The p-value, listed as 0.000, is less than 0.0005, and is therefore

very highly significant (since it is less than 0.001)

The R2 value, 78.7%, is greater than 27.1% (from the R2 table at
level 0.1% with n = 55 and k = 3), and is therefore very highly
significant
The F statistic, 62.84, is greater than the value (between 7.054
and 6.171) from the F table at level 0.1%, and is therefore very
highly significant

34

t Tests
A t test for each regression coefficient
To be used only if the F test is significant
If F is not significant, you should not look at the t tests

Does the jth X variable have a significant effect on Y, holding the

other X variables constant?

Hypotheses are
H0: j = 0,
H1: j 0
Test using the confidence interval

b j tSb j

use the t table with n k 1 degrees of freedom

Or use the t statistic

tstatistic = b j / Sb j

compare to the t table value with n k 1 degrees of freedom

35

Example: t Tests
Testing b1, the coefficient for Audience
b1 = 3.79, t = 13.5, p = 0.000
Audience has a very highly significant effect on Page Costs, after

adjusting for Percent Male and Median Income

Testing b2, the coefficient for Percent Male

b2 = 124, t = 0.90, p = 0.374
Percent Male does not have a significant effect on Page Costs, after

adjusting for Audience and Median Income

Testing b3, the coefficient for Median Income

b3 = 0.903, t = 2.44, p = 0.018
Median Income has a significant effect on Page Costs, after adjusting

for Audience and Percent Male

36

Assumptions in Regression
Assumptions underlying the statistical techniques

should be tested twice

First for the separate variables
Second for the multivariate model variate, which acts

collectively for the variables in the analysis and thus must

meet the same assumption as individual variables. Differs for
different multivariate technique

Assumptions in Regression
Linearity
The independent variable has a linear relationship with the dependent
variable
Normality
The residuals or the dependent variable follow a normal distribution
Multicollinearity
When some X variables are too similar to one another
Homoskedasticity
The variability in Y values for a given set of predictors is the same
regardless of the values of the predictors
Independence among cases (Absence of correlated errors)
The cases are independent of each other

38

Assumptions in Regression
Normality
The residuals or the dependent variable follow a normal

distribution
If the variation from normality is significant then all
statistical tests are invalid
Graphical Analysis
Histogram and Normal probability plot
Peaked and Skewed distribution result in non-normality

Statistical Analysis

If Z value exceeds critical value, then the distribution is non-

normal
Kolmogorov Smirnov Test; Shapiro-Wilks Test

39

Assumptions in Regression
Normality

40

Assumptions in Regression
Homoskedasticity
Assumption related primarily to dependence

relationships between variables

Assumption that the dependent variable(s) exhibit
equal levels of variance across the range of predictor
variable(s).
The variance of the dependent variable should not
be concentrated in only a limited range of the
independent values
Source
Type of variable
Skewed distribution

41

Assumptions in Regression
Homoskedasticity
Graphical Analysis
Analysis of residuals in case of Regression
Statistical Analysis
Variances within groups formed by non-metric variables
Levene Test
Boxs M Test
Remedy
Data Transformation

42

Assumptions in Regression
Homoskedasticity
Graphical Analysis

43

Assumptions in Regression
Linearity
Assumption for all multivariate techniques based on

correlational measures such as

multiple regression,
logistics regression,
factor analysis, and
structural equation modeling

Correlation represents only the linear association

between variables
Identification

Scatterplots or examination of residuals using regression

Remedy
Data Transformations

44

Assumptions in Regression
Linearity

45

Assumptions in Regression
Absence of Correlated Errors
Prediction errors should not be correlated with each

other
Identification
Most possible cause is the data collection process, such as

two separate groups in the data collection process

Remedy
Including the omitted causal factor into the multivariate analysis

46

Assumptions in Regression
Multicollinearity
Multicollinearity arises when intercorrelations among the predictors

are very high.

Multicollinearity can result in several problems, including:
The partial regression coefficients may not be estimated precisely.
The standard errors are likely to be high.
The magnitudes as well as the signs of the partial regression
coefficients may change from sample to sample.
It becomes difficult to assess the relative importance of the
independent variables in explaining the variation in the dependent
variable.
Predictor variables may be incorrectly included or removed in
stepwise regression.

47

Assumptions in Regression
Multicollinearity
The ability of an independent variable to improve the prediction of the

dependent variable is related not only to its correlation to the

dependent variable, but also to the correlation(s) of the additional
independent variable to the independent variable(s) already in the
regression equation
Collinearity is the association, measured as the correlation,

between tow independent variables

Multicollinearity refers to the correlation among three or more
independent variables
Impact
Reduces any single IVs predictive power by the extent to which it is

associated with the other independent variables

48

Assumptions in Regression
Multicollinearity
Measuring Multicollinearity
Tolerance
Amount of variability of the selected independent variable not explained

by the other independent variables

Tolerance Values should be high
Cut-off is 0.1 but greater than 0.5 gives better results
VIF
Inverse of Tolerance
Should be low (typically below 2.0 and usually below 10)

49

Assumptions in Regression
Multicollinearity
Remedy for Multicollinearity
A simple procedure for adjusting for multicollinearity consists of
using only one of the variables in a highly correlated set of
variables.
Omit highly correlated independent variables and identify other
independent variables to help the prediction
Alternatively, the set of independent variables can be transformed into
a new set of predictors that are mutually independent by using
techniques such as principal components analysis.
More specialized techniques, such as ridge regression and latent root
regression, can also be used.

50

Assumptions in Regression
Data Transformations
To correct violations of the statistical assumptions

underlying the multivariate techniques

To improve the relationship between variables
Transformation to achieve Normality and
Homoscedasticity
Flat Distribution Inverse transformation
Negatively Skewed Distribution Square Root

Transformation
Positively Skewed Distribution Logarithmic Transformation
If the residuals in regression are cone shaped then
Cone opens to right Inverse transformation
Cone opens to left Square root transformation

51

Assumptions in Regression
Data Transformations
Transformation to achieve

Linearity

52

Assumptions in Regression
Data Transformations

53

Assumptions in Regression
General guidelines for transformation
For a noticeable effect of transformation the ratio of a variables
mean to the standard deviation should be less than 4.0
When the transformation can be performed on either of the two
variables, select the one with smallest ratio of mean/sd.
Transformation should be applied to independent variables
except in case of heteroscedasticity
Heteroscedasticity can only be remedied by transformation of
the dependent variable in a dependent relationship
If the heteroscedastic relationship is also non-linear the
dependent variable and perhaps the independent variables must
be transformed
Transformations may change the interpretation of the variables

54

Issues in Regression
Variable Selection
How to choose from a long list of X variables?
Too many: waste the information in the data
Too few: risk ignoring useful predictive information

Model Misspecification
Perhaps the multiple regression linear model is wrong
Unequal variability? Nonlinearity? Interaction?

EXPLANATORY VS
PREDICTIVE MODELING

Explanatory Vs Predictive Modeling

Explanatory models fits the data closely, whereas a good

predictive model predicts new cases accurately

Explanatory models uses entire dataset for estimating the
best-fit model and to maximize explanatory variance (R2).
Predictive models estimate the model on training set and
assess it on the new, unobserved data
Performance measures for explanatory models measures
how close the data fit the models, whereas in predictive
models performance is measured by predictive accuracy

Performance Evaluation
Prediction Error for observation i= Actual y value

predicted y value
Popular numerical measures of predictive accuracy
MAE or MAD (Mean absolute error / deviation)

Average Error

MAPE (Mean Absolute Percentage Error)

Performance Evaluation
RMSE (Root mean squared error)

Total SSE (total sum of squared errors)

CASE

Case: Pedigree Vs Grit

Why does a low R2 does not make the regression useless?
Describe a situation in which a useless regression has a high R2.
Check the validity of the linear regression model assumptions.
Estimate the excess returns of Bobs and Putneys funds. Between them, who

is expected to obtain higher returns at their current funds and by how much?
If hired by the firm, who is expected to obtain higher returns and by how
much?
Can you prove at the 5% level of significance that Bob would get higher
expected returns if he had attended Princeton instead of Ohio State?
Can you prove at the 10% level of significance that Bob would get at least 1%
higher expected returns by managing a growth fund?
Is there strong evidence that fund managers with MBA perform worse than
fund managers without MBA? What is held constant in this comparison?
Based on your analysis of the case, which candidate do you support for
AMBTPMs job opening: Bob or Putney? Discuss

Case: Nils Baker

Is the presence of a physical Bank Branch creating

demand for checking accounts?

Thank You