1)NewPathstoNewMachineLearningScience

2)HowanUnrulyMobAlmostStole
) l b l l
theGrandPrizeattheLastMoment

Jeff Howbert
February 6,
6 2012
NetflixViewingRecommendations
 RecommenderSystems
DOMAIN:somefieldofactivitywhereusers buy,view,
consume,orotherwiseexperienceitems

PROCESS:
1. use
userss p
provideratings
o de at gs o onitems
te s ttheyhaveexperienced
ey a e e pe e ced
2. Takeall<user,item,rating >dataandbuildapredictive
model
3. Forauser whohasntexperiencedaparticularitem,use
modeltopredict howwelltheywilllikeit(i.e.predict
rating)
g)
 RolesofRecommenderSystems
y Helpusersdealwithparadoxofchoice

y Allowonlinesitesto:
y Increaselikelihoodofsales
y Retaincustomersbyprovidingpositivesearchexperience

yC
Consideredessentialinoperationof:
id d ti l i ti f
y Onlineretailing,e.g.Amazon,Netflix,etc.
y Socialnetworkingsites
Social networking sites
Amazon.comProductRecommendations
 SocialNetworkRecommendations
y Recommendationsonessentiallyeverycategoryof
R d ti ti ll t f
interestknowntomankind
y Friends
y Groups
y Activities
y Media(TVshows,movies,music,books)
y Newsstories
y Adplacements
y Allbasedonconnectionsinunderlyingsocialnetwork
graphandyourexpressedlikesanddislikes
h d d lik d di lik
 TypesofRecommenderSystems
Basepredictionsoneither:

y contentbased approach
y explicit characteristicsofusersanditems
characteristics of users and items

y collaborative
collaborativefiltering
filtering approach
y implicit characteristicsbasedonsimilarityofusers
preferencestothoseofotherusers
 TheNetflixPrizeContest
y GOAL:usetrainingdata tobuildarecommendersystem,
which,whenappliedtoqualifyingdata,improveserrorrateby
which when applied to qualifying data improves error rate by
10%relativetoNetflixsexistingsystem

y PRIZE:firstteamto10%wins$1,000,000
y AnnualProgressPrizesof$50,000alsopossible
 TheNetflixPrizeContest
y CONDITIONS:
y Opentopublic
y Competeasindividualorgroup
y Submitpredictionsnomorethanonceaday
y Prizewinnersmustpublishresultsandlicensecodeto
Pi i t bli h lt d li d t
Netflix(nonexclusive)

y SCHEDULE:
y StartedOct.2,2006
y Toendafter5years
 TheNetflixPrizeContest

y PARTICIPATION:
y 51051contestantson41305teamsfrom186different
countries
y 44014validsubmissionsfrom5169differentteams
44014 valid submissions from 5169 different teams
 TheNetflixPrizeData
y Netflixreleasedthreedatasets
y 480,189users (anonymous)
y 17,770movies
y ratings onintegerscale1to5

y Trainingset:99,072,112<user,movie >pairswithratings
y Probeset:1,408,395<user,movie
Probe set: 1 408 395 < user movie >pairswithratings
> pairs with ratings
y Qualifyingset of2,817,131<user,movie >pairswithno
ratings
g
 ModelBuildingandSubmissionProcess
trainingset probeset
ratings
99,072,112 1,408,395 known
tuning

MODEL validate

makepredictions
RMSEon RMSEkept
public 1,408,342 1,408,789 secretfor
leaderboard quizset testset finalscoring

qualifyingset
(ratingsunknown)
 WhytheNetflixPrizeWasHard

movie17770
movie10
movie1
movie2
movie3
movie4
movie5
movie6
movie7
movie8
movie9
y Massivedataset
y Verysparse
Very sparse matrix
matrix

m
m
m
m
m
m
m
m
m
m

m

user1 1 2 3
only1.2%occupied user2 2 3 3 4
y Extremevariationin user3 5 3 4
numberofratings
b f user4 2 3 2 2
user5 4 5 3 4
peruser user6 2
y Statisticalproperties
p p user7 2 4 2 3
user8 3 4 4
ofqualifyingand user9 3
probesetsdifferent user10 1 2 2
g
fromtrainingset
user480189 4 3 3
 DealingwithSizeoftheData
y MEMORY:
y 2GBbareminimumforcommonalgorithms
y 4+GBrequiredforsomealgorithms
y need64bitmachinewith4+GBRAMifserious
y SPEED:
SPEED
y Programinlanguagesthatcompiletofastmachinecode
y 64bitprocessor
64bit processor
y Exploitlowlevelparallelismincode(SIMDonIntelx86/x64)
CommonTypesofAlgorithms
y Globaleffects
y Nearestneighbors
y Matrixfactorization
y RestrictedBoltzmannmachine
y Clustering
y Etc.
 NearestNeighborsinAction

movie17770
movie10
movie1
movie2
movie3
movie4
movie5
movie6
movie7
movie8
movie9
m
m
m
m
m
m
m
m
m
m

m

user1 1 2 3
user2 2 3 3 4 ? Identicalpreferences
user3 5 3 strongweight
g g
user4 2 3 2 2
user5 2 3 5 4 2 4
Similarpreferences
user6 2
moderateweight
user7 2 4 2
user8 3 1 3 4 5 4
user9 3
user10 1 2 2

user480189 4 3 3
 MatrixFactorizationinAction

70
movie1777
movie10
movie1
movie2
movie3
movie4
movie5
movie6
movie7
movie8
movie9
ovie17770

ovie10
feature1
ovie1
ovie2
ovie3
ovie4
ovie5
ovie6
ovie7
ovie8
ovie9
feature2
< a bunch of numbers >
<abunchofnumbers>
mo
mo
mo
mo
mo
mo
mo
mo
mo
mo

mo
feature3

user1 1 2 3 feature4
user2 2 3 3 4 feature5
user3 5 3 4
user4 2 3 2 2 +
user5 4 5 3 4 reducedrank

feature1
feature2
feature3
feature4
feature5
user6 2 singular
user7 2 4 2 3
value
user8 3 4 4
user9 3
decomposition
user1
user10 1 2 2 (sortof) user2
user3
3

<abunchoff
user480189 4 3 3

numbers>
user4
user5
user6
user7
user8

n
user9
user10

user480189
 MatrixFactorizationinAction

70
movie1777
movie10
movie1
movie2
movie3
movie4
movie5
movie6
movie7
movie8
movie9

movvie17770

feature1

movvie10
movvie1
movvie2
movvie3
movvie4
movvie5
movvie6
movvie7
movvie8
movvie9
feature2
feature3

feature4 user1 1 2 3
feature5
user2 2 3 3 4
+ user3
user4 2 3
5 3
2
4
2
user 5
user5 4 5 3 4
feature1
feature2
feature3
feature4
feature5

multiplyandadd user6 2
features user7 2 4 2 3
(dotproduct) user8 3 4 4 ?
user1 user9 3
user2 fordesired user10 1 2 2
user3
3 < user movie >
<user,movie
user4 prediction user480189 4 3 3
user5
user6
user7
user8
user9
user10

user480189
 ThePowerofBlending
y Errorfunction(RMSE)isconvex,solinearcombinationsof
modelsshouldhavelowererror
y Findblendingcoefficientswithsimpleleastsquaresfitof
ff f f
modelpredictionstotruevaluesofprobeset
y Examplefrommyexperience:
Example from my experience:
y blended89diversemodels
y RMSErange=0.8859
g 0.9959
y blendedmodelhadRMSE=0.8736
y Improvementof0.0123overbestsinglemodel
y 13%ofprogressneededtowin
Algorithms:OtherThingsThatMattered
y Overfitting
y Modelstypicallyhadmillionsorevenbillionsofparameters
y Controlwithaggressiveregularization

y Ti
Timerelatedeffects
l d ff
y Netflixdataincludeddateofmovierelease,datesofratings
y Mostofprogressinfinaltwoyearsofcontestwasfrom
Most of progress in final two years of contest was from
incorporatingtemporalinformation
TheNetflixPrize:SocialPhenomena
y Competitionintense,butsharingandcollaborationwere
equallyso
y Lotsofpublicationsandpresentationsatmeetingswhile
f bli i d i i hil
conteststillactive
y Lotsofsharingoncontestforumsofideasand
Lots of sharing on contest forums of ideas and
implementationdetails
y Vastmajorityofteams:
y Notmachinelearningprofessionals
y Notcompetingtowin(untilveryend)
y Mostlyusedalgorithmspublishedbyothers
One Algorithm from Winning Team
OneAlgorithmfromWinningTeam
(timedependentmatrixfactorization)

YehudaKoren,Comm.ACM,53,89(2010)
Netflix Prize Progress: Major Milestones
NetflixPrizeProgress:MajorMilestones
1.05
QuizSet

1.00
me, starting
June, 2008
MSErroronQ

0.95
RM

0.90

8.43%
9.44%
10.09%
10.00%
0.85
trivialalgorithm Cinematch 2007Progress 2008Progress GrandPrize
Prize Prize

DATE
DATE: Oct 2007
Oct.2007 Oct 2008
Oct.2008 J l 2009
July2009
BellKor in
WINNER: BellKor ???
BigChaos
June25,200920:28GMT
June26,18:42GMT BPCTeamBreaks10%
 G i f Th E bl
GenesisofTheEnsemble
Vandelay me Dinosaur
Industries Planet
5indiv. Opera 3indiv.
Solutions Gravity
5indiv. 4 indiv
4indiv. 7other
7 other
9other
individuals individuals

Operaand
p
G dP i T
GrandPrizeTeam
VandelayUnited 14individuals
19individuals

TheEnsemble
33individuals

www.theensemble.com
June30,16:44GMT
July8,14:22GMT
July17,16:01GMT
July25,18:32GMT TheEnsembleFirstAppears!

24hours,10min
beforecontestends

#1and#2teams
#1 and #2 teams
eachhaveone
moresubmission!
 July26,18:18GMT
BPCMakesTheirFinalSubmission

24minutesbeforecontestends

TheEnsemblecanmakeonemoresubmission
window opens 10 minutes before contest ends
windowopens10minutesbeforecontestends
July26,18:43GMT ContestOver!
Final Test Scores
FinalTestScores
FinalTestScores
NetflixPrize:WhatDidWeLearn?
y Significantlyadvancedscienceofrecommendersystems
y Properlytunedandregularizedmatrixfactorizationisa
powerfulapproachtocollaborativefiltering
f l ht ll b ti filt i
y Ensemblemethods(blending)canmarkedlyenhance
p
predictivepowerofrecommendersystems
p y

y Crowdsourcingg viaacontestcanunleashamazing
g
amountsofsustainedeffortandcreativity
y Netflixmadeoutlikeabandit
y Butprobablywouldnotbesuccessfulinmostproblems
 NetflixPrize:WhatDidILearn?
y Severalnewmachinelearningalgorithms
y Alotaboutoptimizingpredictivemodels
y Stochasticgradientdescent
y Regularization
y Alotaboutoptimizingcodeforspeedandmemoryusage
p g p y g
y SomelinearalgebraandalittlePDQ
y Enoughtocomeupwithoneoriginalapproachthatactually
worked

y Moneymakespeoplecrazy,inbothgoodwaysandbad
y p p y, g y

COST:about1000hoursofmyfreetimeover13months