Predictive Analytics and Data Mining

- Evaluation And Assessment of Data Mining Models

- Most of the figures in this slide deck are

- Data Science for Business, Provost and Fawcett, Wiley

- Applied Predictive Analytics, Dean Abbott, Wiley
Overfitting
Over-fitting in Decision Trees
Which is the best Model?
Which is the best Model?
 Training Set

Sample
Test Set
 Training Set validation data

Sample
Test Set
Evaluating Classifiers: Plain Accuracy

Number of correct decisions made

Accuracy=
Total number of decisions made

=1error rate

Too simplistic..
What do Businesses Care About?

RMSE, R-squared error, Gini Index, Entropy, Minimum distance OR

Return on Investment, Expected Profit, False alarm cases for the next
1000 cases

Training Data OR Test Data

No One Right way or evaluation metric

Its important to link the Data Mining Goals/ objectives (and

consequently DM evaluation and assessment criteria) to Business
Goals/ Outcomes
Typical Binary Classification Accuracy Metrics
Percent Correct Classification (PCC)
PCC = (t_n + t_p)/(t_p + t_n + f_p +
f_n)
Predicted Class

Confusion Matrix
0 1
Total Actual
(predicted value is (predicted value is
(down)
negative) positive)

0 fp Total actual
tn
(actual value is (false positive, negatives
(true negative)
negative) false alarm) tn + fp
Actual Class
1 fn Total actual
tp
(actual value is (false negative, positives
(true positive)
positive) false dismissal) tp + fn
Total
Total negative Total positive
Total Predicted Examples
predictions predictions
(across) tp + tn +
tn + fn tp + fp
fp + fn

11 Abbott Analytics, 2001-2017

Computing Quadrants of a Confusion Matrix

Actual Predicted
Probability Confusion Matrix
Target Target
Target = 1 Quadrant
Value Value
0 0.641 1 false alarm
1 0.601 1 true positive
We first must populate all
0 0.587 1 false alarm
four quadrants of the
1 0.585 1 true positive confusion matrix.
1 0.575 1 true positive
0 0.562 1 false alarm
0 0.531 1 false alarm AT left, the probability
1 0.504 1 true positive threshold is 0.5
0 0.489 0 true negative
1 0.488 0 false dismissal
0 0.483 0 true negative
0 0.471 0 true negative
0 0.457 0 true negative
1 0.418 0 false dismissal
0 0.394 0 true negative
0 0.384 0 true negative
0 0.372 0 true negative
0 0.371 0 true negative
0 0.341 0 true negative
1 0.317 0 false dismissal

12 Abbott Analytics, 2001-2017

Accuracy Metrics Computed from Confusion Matrices

13 Abbott Analytics, 2001-2017

Which is a better Model

A no. of issues to be considered eg

Unbalanced datasets

Expected Value (Cost-Benefit analysis)

Evaluating Classifiers: The Confusion Matrix
Expected Value Framework in Use Phase

Online marketing:

Expected benefit of targeting = + 1

Product Price: $200

Product Cost: $100

Targeting Cost: $1

$99 1 $1 > 0
> 0.01
Using Expected Value to Frame Classifier Evaluation
A cost-benefit matrix
A cost-benefit matrix for the marketing example
Using Expected Value to Frame Classifier Evaluation

Expected profit
= , , + , ,
+ , , + , ,

Expected profit
= , + ,
+ (|) () (, ) + (|) () (, )

Expected profit
= , + ,
+ [ , + , ]
Using Expected Value to Frame Classifier Evaluation
= 110
= 61 = 49
() = 0.55 () = 0.45
(|) = 56/61 = 0.92 (|) = 7/49 = 0.14
(|) = 5/61 = 0.08 (|) = 42/49 = 0.86

Expected profit = , + ,
+ , + ,

= 0.55 0.92 , + 0.08 ,

= 0.55 0.92 99 + 0.08 0
= 50.1 0.063 $.
Three Key Rank-Ordered Metrics

Sorted Predictions

IMAGINE ALL Actual Predicted Target

Measure What it is Record ID INPUT COLS Target Value
HERE Variable (probability)
% of target records found at each sorted 185436 | 0 0.0731
Gain depth 14279 | 1 0.0715
727 | 1 0.0699
ratio of target % target found at the 24610 | 1 0.0683
sorted depth to average target % (all 22645 | 1 0.0668
Lift data) 82943 | 1 0.0653
108412 | 0 0.0639
Receiver Operating Characteristic 190313 | 1 0.0624
(ignore this name!) 48804 | 0 0.0611
True alerts vs. false alarms for every 123822 | 1 0.0597
94039 | 0 0.0583
ROC threshold
47605 | 0 0.0570
25641 | 1 0.0558
47476 | 0 0.0545
6023 | 0 0.0533
47784 | 0 0.0521
148569 1 0.0509
171099 0 0.0497

23 Abbott Analytics, 2001-2017

Lift Curve
Gains

25 Abbott Analytics, 2001-2017

Perfect Gains Chart

Perfect means in the rank-

ordered list, every record is
classified correctly. In this
example, the rate is about 7%

If you get a perfect gain, you

know you did something
wrong!

26 Abbott Analytics, 2001-2017

Lift Chart

Cumulative Lift Chart Lift Chart by Demi-Decile

27 Abbott Analytics, 2001-2017

Lift Value: Cumulative
Lift Value: Non- Cumulative
% Captured Response: Cumulative
Ranking Instead of Classifying
Profit Curves
ROC Graphs and Curves (Tradeoffs between benefits true
positives and costs false positives Accommodates uncertainties)
ROC Graphs and Curves
Typical Binary Classification Accuracy Metrics
Percent Correct Classification (PCC)
PCC = (t_n + t_p)/(t_p + t_n + f_p +
f_n)
Predicted Class

Confusion Matrix
0 1
Total Actual
(predicted value is (predicted value is
(down)
negative) positive)

0 fp Total actual
tn
(actual value is (false positive, negatives
(true negative)
negative) false alarm) tn + fp
Actual Class
1 fn Total actual
tp
(actual value is (false negative, positives
(true positive)
positive) false dismissal) tp + fn
Total
Total negative Total positive
Total Predicted Examples
predictions predictions
(across) tp + tn +
tn + fn tp + fp
fp + fn

35 Abbott Analytics, 2001-2017

Computing Quadrants of a Confusion Matrix

Actual Predicted
Probability Confusion Matrix
Target Target
Target = 1 Quadrant
Value Value
0 0.641 1 false alarm
1 0.601 1 true positive
We first must populate all
0 0.587 1 false alarm
four quadrants of the
1 0.585 1 true positive confusion matrix.
1 0.575 1 true positive
0 0.562 1 false alarm
0 0.531 1 false alarm AT left, the probability
1 0.504 1 true positive threshold is 0.5
0 0.489 0 true negative
1 0.488 0 false dismissal
0 0.483 0 true negative
0 0.471 0 true negative
0 0.457 0 true negative
1 0.418 0 false dismissal
0 0.394 0 true negative
0 0.384 0 true negative
0 0.372 0 true negative
0 0.371 0 true negative
0 0.341 0 true negative
1 0.317 0 false dismissal

36 Abbott Analytics, 2001-2017

Accuracy Metrics Computed from Confusion Matrices

37 Abbott Analytics, 2001-2017

ROC Curves

ROC Curves are plots

of every confusion
matrix threshold of the
posterior probability

38 Abbott Analytics, 2001-2017

Comparison of Three Models using ROC Curves

39 Abbott Analytics, 2001-2017

Cumulative Response curve
NEXT TWO SESSIONS: DATA
UNDERSTANDING, DATA PREPARATION
and PREDICTIVE ANALYTICS USING
LOGISTIC REGRESSION