Illusions of Confidence

The importance of mathematical thinking

Neil Moakley

Math in the (pseudo) real world

Ms. Smith has just started her new job as the principal of Philadelphia High School. She's concerned that some of the school's 500 students may be cheating on their exams, and so she decides to use a new piece of computer software to find out how bad this problem is. This software will analyze the work of each student and try to determine who cheated and who didn't.

When it's run at Philadelphia High, this software will give an accurate result for 90% of the studentsthat is, it will correctly identify 90% of the cheaters as cheaters, and it will correctly identify 90% of honest students as honest students.

Math in the (pseudo) real world

The software identifies 66 students as cheaters. Angrily, Ms. Smith calls these students to the auditorium to "have a talk with them". As she's walking towards the auditorium, though, Ms. Smith begins to worry, I hope this software did a good job; after all, I don't want to yell at a room full of honest students! But then she remembers that the software is 90% accurate, and breathes a sigh of relief. That means almost 60 of those 66 students are cheaters...close enough!

Questions to Consider
Do you agree with Ms. Smiths reasoning? How did she calculate that almost 60 cheaters were caught? If you disagree with her conclusion, what percentage of the students in the auditorium do you think are cheaters? Try to use your intuition here, without doing any math. Once youve made an educated guess, try to build a mathematical answer to this question.

A mathematical solution
66 = + ( ) = 90% = 10%

90% + 10% = 66
+ = 500 9 + = 660 + = 500 8 = 160 = 20 There are only 20 cheaters at Philadelphia High School; 18 of them were caught, and were sent to the auditorium. = = . %

Visualizing the solution

Philadelphia High School (500 students)

X X X X X X X X
Cheating students (20) Non-cheating students (480)

X X X X X X X X X X X X X X X X X X X X

X X X X X X X X X X X X X X X X X X X X

X X X X X X X X X X X X X X X X X X

X X

Cheaters who were caught (18) Non-cheaters who were "caught" (48)

The Auditorium (66 students)

X X X X

X X X X

X X X X

X X X X

X X X X

X X X X

X X X X

X X X X

X X X

X X X

X X X

X X X

X X X

X X X

X X X

X X X

X X X

X X X

X X

X X

A more complicated question

There are approximately 75.2 million students in the United States The national version of our Cheater Detector can correctly identify 85% of cheaters and 90% of honest students When we run the test on all students, 7,670,000 cheaters are identified. How many of them are actually guilty?
A) 0 to 30% B) 30-60% C) 60-90%

Heres the math

= ; = + = 75,200,000 85% + 10% = 7,670,000
85 + 10 = 767,000,000 10 + 10 = 752,000,000 75 = 15,000,000 = 15,000,000 75 = 200,000 So there are 200,000 cheaters and 75,000,000 honest students in America. What percentage of those who were caught were actually guilty? 85% 170,000 = 2.22% 85% + 10% 7,670,000

And heres why it matters

= ; = * + = 75,200,000 85% + 10% = 7,670,000 85 + 10 = 767,000,000 10 + 10 = 752,000,000 75 = 15,000,000 = 15,000,000 75 = 200,000 85% 170,000 = 2.22% 85% + 10% 7,670,000 These numbers are similar to the actual statistics for mammogram accuracy and breast cancer detection in the United States
(Current estimates suggest that ~86% of positive mammograms are false positives)

almost everyone gets this wrong!

In an informal study that presented similar data, subjects were asked the same intuition question I asked you 95% of physicians surveyed picked (C), and indicated that their estimate of the tests accuracy was between 70-80%

Some ideas to take away

Our intuitions about math will sometimes lie to us This is especially true when dealing with statistics This particular resultfalse positives outweighing true positiveswill occur any time the incidence of the tested behavior is low. Think
Cancer, AIDS, and other diseases Drug testing Security screenings

Further reading
This phenomenon is called the False Positive Paradox or the Base Rate Fallacy
http://en.wikipedia.org/wiki/False_positive_paradox http://en.wikipedia.org/wiki/Base_rate_fallacy

Look for the Monty Hall Problem or the Simpsons Paradox for more examples of counterintuitive statistics
http://en.wikipedia.org/wiki/Monty_Hall_problem http://en.wikipedia.org/wiki/Simpson%27s_paradox

Sources
http://www.census.gov/newsroom/releases/archives/fa cts_for_features_special_editions/cb11-ff15.html http://www.census.gov/prod/cen2010/briefs/c2010br03.pdf http://breastscreening.cancer.gov/data/benchmarks/scr eening/2009/table7.html http://www.cdc.gov/cancer/breast/statistics/index.htm http://www.youtube.com/watch?v=D8VZqxcu0I0 http://www.theatlantic.com/technology/archive/2012/0 4/homeland-securitys-pre-crime-screening-will-neverwork/255971/
(* cancer statistics are rounded estimates for women over aged 40)