Today

COM 578  Dull organizational stuff

Empirical Methods in Machine – Course Summary
Learning and Data Mining – Grading
– Office hours
– Homework
Rich Caruana – Final Project
 Fun stuff
http://www.cs
http://www.cs..cornell.
cornell.edu/Courses/cs578/2007fa
edu/Courses/cs578/2007fa – Historical Perspective on Statistics, Machine Learning,
and Data Mining

Staff, Office Hours, … Topics

Rich Caruana
Tue 4:30-5:00pm
caruana@
caruana@cs.
cs.cornell.
cornell.edu
TA: Daria Sorokina
TBA
daria@cs.cornell.edu
TA: Ainur Yessenalina
TBA
ainur@cs.cornell.edu
TA: Alex Niculescu-
Niculescu-Mizil
TBA
alexn@cs.cornell.edu
alexn@cs.cornell.edu
Admin: Melissa Totman
M-F 9:00am-4:00pm
Upson Hall 4157
Wed 10:30-11:00am  Decision Trees
 K-Nearest Neighbor
Upson Hall 5156  Artificial Neural Nets
 Support Vector Machines
Upson Hall 4156  Association Rules
 Clustering
 Boosting/Bagging
Upson Hall 5154
 Cross Validation
Upson Hall 4147
~30% overlap with CS478
 Performance Metrics
 Data Transformation
 Feature Selection
 Missing Values
 Case Studies:
– Medical prediction
– Protein folding
– Autonomous vehicle
navigation

1
 Grading Homeworks
 4 credit course  short programming and experiment assignments
– e.g., implement backprop and test on a dataset
 25% take-home mid-term (late-October) – goal: get familiar with a variety of learning methods
 25% open-book final (????)  two or more weeks to complete each assignment
 30% homework assignments (3 assignments) C, C++, Java, Perl,
 Perl, shell scripts, or Matlab
 20% course project (teams of 1-4 people)
 must be done individually
 late penalty: one letter grade per day  hand in code with summary and analysis of results
 90-100 = A-, A, A+  emphasis on understanding and analysis of results,
 80-90 = B-, B, B+ not generating a pretty report
 70-80 = C-, C, C+  short course in Unix and writing shell scripts

Project Text Books

 Data Mining Mini Competition  Required Text:
 Train best model on problem(s) we give you – Machine Learning by Tom Mitchell
– decision trees
– k-nearest neighbor  Optional Texts:
– artificial neural nets – Elements of Statistical Learning: Data Mining, Inference, and
– SVMs Prediction by Hastie,
Hastie, Tibshirani,
Tibshirani, and Friedman
– bagging, boosting, model averaging, ... – Pattern Classification,
Classification, 2nd ed., by Richard Duda,
Duda, Peter Hart, &
 Given train and test sets David Stork
– Have target values on train set – Pattern Recognition and Machine Learning by Chris Bishop
– No target values on test set – Data Mining: Concepts and Techniques by Jiawei Han and
– Send us predictions and we calculate performance Micheline Kamber
– Performance on test sets is part of project grade
 Due before exams & study period  Selected papers

2
 Statistics, Machine Learning,
Fun Stuff
and Data Mining

Past, Present, and Future Once upon a time...

3
 Pre-Statistics: Ptolmey-1850
 First “Data Sets”
Sets” created
– Positions of mars in orbit: Tycho Brahe (1546-1601)
– Star catalogs
before statistics  Tycho catalog had 777 stars with 1-2 arcmin precision
– Messier catalog (100+ “dim fuzzies”
fuzzies” that look like comets)
– Triangulation of meridian in France
 Not just raw data - processing is part of data
– Tychonic System: anti-Copernican, many epicycles
 No theory of errors - human judgment
– Kepler knew Tycho’
Tycho’s data was never in error by 8 arcmin
 Few models of data - just learning about modeling
– Kepler’
Kepler’s Breakthrough: Copernican model and 3 laws of orbits

Pre-Statistics: 1790-1850 Statistics: 1850-1950

 The Metric System:  Data collection starts to separate from analysis
– uniform system of weights and measures
 Hand-collected data sets
 Meridian from Dunkirk to Barcelona through Paris
– Physics, Astronomy, Agriculture, ...
– triangulation
– Quality control in manufacturing
 Meter = Distance (pole to equator)/10,000,000
– Many hours to collect/process each data point
 Most accurate survey made at that time
 1000’
1000’s of measurements spanning 10-20 years!  Usually Small: 1 to 1000 data points
 Data is available in a 3-volume book that analyses it  Low dimension: 1 to 10 variables
 No theory of error:  Exist only on paper (sometimes in text books)
– surveyors use judgment to “correct data”
data” for better consistency  Experts get to know data inside out
and accuracy!
 Data is clean: human has looked at each point

4
 Statistics: 1850-1950
 Calculations done manually
– manual decision making during analysis
– Mendel’
Mendel’s genetics
– human calculator pools for “larger”
larger” problems
 Simplified models of data to ease computation
– Gaussian, Poisson, …
– Keep computations tractable
 Get the most out of precious data
– careful examination of assumptions
– outliers examined individually
Statistics would look very
different if it had been born after
the computer instead of 100
years before the computer
Statistics: 1850-1950
 Analysis of errors in measurements
 What is most efficient estimator of some value?
 How much error in that estimate?
 Hypothesis testing:
– is this mean larger than that mean?
– are these two populations different?
 Regression:
– what is the value of y when x=xi or x=x
x=xj?
 How often does some event occur?
– p(fail(part1)) = p1; p(fail(part2)) = p2; p(crash(plane)) = ?
Statistics meets Computers
5
 Machine Learning: 1950-2000...
 Medium size data sets become available
– 100 to 100,000 records
– Higher dimension: 5 to 250 dimensions (more if vision)
– Fit in memory
 Exist in computer, usually not on paper
 Too large for humans to read and fully understand
 Data not clean
– Missing values, errors, outliers,
– Many attribute types: boolean,
boolean, continuous, nominal, discrete,
ordinal
– Humans can’
can’t afford to understand/fix each point
Machine Learning: 1950-2000...
 Computers can do very complex calculations on medium
size data sets
 Models can be much more complex than before
 Empirical evaluation methods instead of theory
– don’
don’t calculate expected error, measure it from sample
– cross validation
– e.g., 95% confidence interval from data, not Gaussian model
 Fewer statistical assumptions about data
 Make machine learning as automatic as possible
 Don’
Don’t know right model => OK to have multiple models
(vote them)

Machine Learning: 1950-2000... ML: Pneumonia Risk Prediction

 Regression Pneumonia
Risk
 Multivariate Adaptive Regression Splines (MARS)
 Linear perceptron
 Artificial neural nets
 Decision trees
 K-nearest neighbor

Chest X-Ray

RBC Count
Blood Pressure

Albumin
Blood pO2
White Count
Age
Gender
 Support Vector Machines (SVMs
(SVMs))
 Ensemble Methods: Bagging and Boosting
 Clustering Pre-Hospital In-Hospital
Attributes Attributes

6
ML: Autonomous Vehicle Navigation
Steering Direction
Can’t yet buy cars that drive
themselves, and few hospitals use
artificial neural nets yet to make
critical decisions about patients.

Machine Learning: 1950-2000...

 New Problems:
– Can’
Can’t understand many of the models Machine Learning Leaves the Lab
– Less opportunity for human expertise in process
– Good performance in lab doesn’
doesn’t necessarily mean Computers get Bigger/Faster
good performance in practice
– Brittle systems, work well on typical cases but often but
break on rare cases Data gets Bigger/Faster, too
– Can’
Can’t handle heterogeneous data sources

7
 Data Mining: 1995-20??
 Huge data sets collected fully automatically
– large scale science: genomics, space probes, satellites
– Cornell’
Cornell’s Arecibo Radio Telescope Project:
 terabytes
per day
 petabytesover life of project
 too much data to move over internet -- they use FedEx!

Protein Folding

8
 Data Mining: 1995-20?? Data Mining: 1995-20??
 Huge data sets collected fully automatically  Data exists only on disk (can’
(can’t fit in memory)
– large scale science: genomics, space probes, satellites  Experts can’
can’t see even modest samples of data
– consumer purchase data  Calculations done completely automatically
– web: > 500,000,000 pages of text – large computers
– clickstream data (Yahoo!: terabytes per day!) – efficient (often simplified) algorithms
– many heterogeneous data sources – human intervention difficult
 High dimensional data  Models of data
– “low”
low” of 45 attributes in astronomy – complex models possible
– 100’
100’s to 1000’
1000’s of attributes common – but complex models may not be affordable (Google
(Google))
– linkage makes many 1000’
1000’s of attributes possible  Get something useful out of massive, opaque data
– data “tombs”
tombs”

9
 Data Mining: 1990-20?? Data Mining: 1995-20??
 What customers will respond best to this coupon?  New Problems:
 Who is it safe to give a loan to? – Data too big
 What products do consumers purchase in sets? – Algorithms must be simplified and very efficient
(linear in size of data if possible, one scan is best!)
 What is the best pricing strategy for products?
– Reams of output too large for humans to comprehend
 Are there unusual stars/galaxies in this data? – Very messy uncleaned data
 Do patients with gene X respond to treatment Y? – Garbage in, garbage out
 What job posting best matches this employee? – Heterogeneous data sources
 How do proteins fold? – Ill-posed questions
– Privacy

Statistics, Machine Learning, Change in Scientific Methodology

and Data Mining
Traditional:
Traditional: New:
New:
 Historic revolution and refocusing of statistics  Formulate hypothesis  Design large experiment
 Statistics, Machine Learning, and Data Mining  Design experiment  Collect large data
merging into a new multi-faceted field  Collect data  Put data in large database
 Old lessons and methods still apply, but are used  Analyze results  Formulate hypothesis
in new ways to do new things  Review hypothesis  Evaluate hyp on database
 Repeat/Publish  Run limited experiments
 Those who don’
don’t learn the past will be forced to
to drive nail in coffin
reinvent it
 Review hypothesis
 => Computational Statistics, ML, DM, …  Repeat/Publish

10
 ML/DM Here to Stay

 Will infiltrate all areas of science, engineering,

public policy, marketing, economics, …
 Adaptive methods as part of engineering process
– Engineering from simulation
– Wright brothers on steroids!
 But we can’
can’t manually verify models are right!
 Can we trust results of automatic learning/mining?

11