Is Your Model Good Enough?

Michael Conerly,
Winston Choi,
Michael Hardin
University of Alabama

Copyright © 2007, SAS Institute Inc. All rights reserved.

Outline of Presentation
ƒ Goodness of Fit (GOF) Procedures
ƒ Why are these necessary?
ƒ Review of Hosmer and Lemeshow
ƒ Adapting H-L to data mining
ƒ Selection of groups for GOF
ƒ Application of New test to real data
ƒ What if test fails?

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Goodness of Fit (GOF) Procedures
ƒ Consider a simple Contingency table
Group Low Med High Total
Goods 12 18 30 60
Bads 32 20 8 60

ƒ Note that rows are inversely related

ƒ Are category and row related?
ƒ Focus on either Goods or Bads,
we will use Goods.

Copyright © 2007, SAS Institute Inc. All rights reserved.

Goodness of Fit (GOF) Procedures

ƒ Test that Observed and Predicted
Good values are in agreement
Group Low Med High Total
Observed 12 18 30 60
Predicted 8 18 34 60

ƒ Here χ2 = 2.3 with p = .313

ƒ Model seems accurate

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Another example
ƒ Group Low Med High Total
Observed 12 18 30 60
Predicted 6 16 38 60

ƒ Here χ2 = 7.9 with p = .019,

Model does not seem accurate
ƒ We are more accurate in Med group,
less in Low, High groups.
ƒ The propensity of being correct is
related to the prediction???

Copyright © 2007, SAS Institute Inc. All rights reserved.

Why use GOF?

ƒ Are predictions consistent?
For Linear models we look for residuals to be
centered at 0 with similar dispersion.
ƒ Graphs for (0/1) data are not informative
ƒ Binning the data allows us to compare counts.
ƒ If we predict better for some categories than
others:
• Is our model consistent?
• Are there unexplained data issues?
• Are we overfitting?

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Predictive Models
ƒ Generally predict Pr(Good) based on:
• Logistic Regression
• Neural Networks
• Decision Trees
• Or some combination of these
• Or other techniques
ƒ In all cases, we predict Pr(Success),
with 0 < Pr(Success) < 1.

Copyright © 2007, SAS Institute Inc. All rights reserved.

GOF for Predictive Models

ƒ Review Hosmer & Lemeshow

procedure for Logistic Regression
See Applied Logistic Regression ,
second edition, by David Hosmer and
Stanley Lemeshow, Wiley
ƒ Extend this to Other Estimation
Techniques

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Hosmer and Lemeshow Procedure
ƒ First divide the data into categories.
These need to be homogeneous.
ƒ H&L suggest deciles based on predicted
probabilities, i.e.,
0 - .1, .1 - .2, …
The resulting categories are equal in size
ƒ With large data sets, may use 50 or more
groups.
ƒ Need Expected #s > 5 for almost all
categories
Copyright © 2007, SAS Institute Inc. All rights reserved.

Hosmer and Lemeshow Procedure

ƒ Tests focus on Goods or Bads, but not

both, since these are related.
ƒ We can construct Expected # by
averaging the predicted probabilities
for each category to obtain
ƒ Expected # = n x

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Tabulated Data
Category Observed Pr(Success) Expected

1 Obs1 π1 n×π1
2 Obs2 π2 n×π2
3 Obs3 π3 n×π3

…
k Obsk πκ n×πk
Total n 1.0 n

Copyright © 2007, SAS Institute Inc. All rights reserved.

Chi-square statistic
ƒ The statistic

2 (Obsi − Expi )2
χ =∑
nπi (1 − πi )
is approximately Chi-square with k-2
degrees of freedom.
ƒ Large values Little or no
agreement between Observed and
predicted counts

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Generalizing to Predictive Modeling
ƒ Given categories, this H-L procedure easily
adapts to Predictive Models
ƒ To choose categories:
• Bin data based on the Predicted probs as
in H-L.
• This is problematic if you consider multiple
models (Logit, Tree, NN) since different
predictions yield different bins
• Need to compare competing models on
same categories.

Copyright © 2007, SAS Institute Inc. All rights reserved.

Generalizing to Predictive Modeling

ƒ Bin data using Cluster Analysis to identify
homogeneous groups.
ƒ Many forms are available to cluster similar
observations
ƒ Perform cluster analysis then do K-Means
to ensure homogeneity of categories
(clusters)
ƒ Store the cluster # in column to create
summary tables for comparing models.

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Generalizing to Predictive Modeling
H-L also suggest applying this GOF
test to “hold-out” data for validation.
This is a straightforward application
of the same process. Apply Cluster
analysis first, then create a summary
table on these clusters and compute
Chi-Square statistic.

Copyright © 2007, SAS Institute Inc. All rights reserved.

Examples

ƒ Using the Home Equity data, without

missing values, available in JMP 7 we
performed the following GOF tests.
We will compare a Logistic, Decision
Tree and Neural Network fit to these
data to predict P(Good Loan).

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Continuation of Example here

ƒ To be added later

Copyright © 2007, SAS Institute Inc. All rights reserved.

What if Test Fails?

ƒ Use the Chi-square calculation for each category
to determine which ones are problematic.
ƒ Do these categories contain aberrant data?
ƒ Fit these categories separately to see where
model(s) change.
ƒ Use Predictive Models to predict these problem
categories (1) versus other categories(0) to find
problem variables.
ƒ Adjust the overall models accordingly.

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Conclusion

ƒ Fitting Predictive Models to large data

sets makes discovering unusual data
problems difficult.
ƒ GOF is part of the validation and
model checking process.
ƒ It may help to identify serious flaws
and suggest corrections leading to
improved models.

Copyright © 2007, SAS Institute Inc. All rights reserved.

Questions?

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.

 Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved.