You are on page 1of 7

CVChess:

 Computer  Vision  Chess  Analytics  


 
Jay  Hack  and  Prithvi  Ramakrishnan  
 
 
 
Abstract   higher-­‐level   analysis   might   immediately   take  
place.  
 
We   present   a   computer   vision   application   and   This   project   is   motivated   by   several  
a  set  of  associated  algorithms  capable  of  recording   established   use-­‐cases   for   similar   systems,   as   well  
chess   game   moves   fully   autonomously   from   the   as   problems   with   current   systems   that   we   can  
vantage   point   of   a   consumer   laptop   webcam.   This   empathize   with.   Perhaps   most   significantly,   the  
consists   of   two   main   algorithms,   (1)   a   hough   process   of   hand-­‐recording   games   and   inputting  
transform-­‐based   algorithm   for   finding   a   them   into   computers   for   algorithmic   analysis   is  
homography   relating   board   coordinates   to   image   extremely   tedious.   Furthermore,   real-­‐time   aids  
coordinates,   and   (2)   a   model   of   chessboard   colors   such  as  score-­‐keeping,  time-­‐keeping,  live  analysis,  
and   occlusions   that   allows   us   to   account   for   and   and   even   automated   coaching   can   be   easily  
infer   piece   movement   in   real   time.   We   provide   a   achieved   given   a   utility   for   identify   moves   in   real  
video  demonstration  of  the  application  applied  to  a   time.   Given   the   prevalence   of   open   research   and  
real  chess  game  and  describe  experiments  in  which   tools   for   chess   analysis,   we   suggest   that   our  
our   developed   algorithms   significantly   outperform   project   will   allow   chess   players   the   convenience  
a   naive   baseline.   All   code   is   open   sourced   and   and  pleasure  of  playing  chess  on  a  physical  board  
available  on  GitHub.  (See  Below)   while   maintaining   the   ability   to   leverage   analytic  
advancements  in  computer  chess.
 
Code:  github.com/jayhack/CVChess  
2.  Previous  work  
Video  Demo:  youtube.com/watch?v=iZOA1ew-­‐zYc  
Previous   work   with   determining   chess  
1.  Introduction positions   from   images   is   very   limited.   The  
majority   of   existing   work   has   been   conducted   by  
While  the  game  of  chess  has  greatly  benefitted   independent   hobbyists,   independent   of  
from   the   information   age,   including   analytical   established   research   institutions,   and   without  
algorithms,   online   playing   environments   and   published   experimentation   or   results.   Most  
game   databases,   there   have   been   relatively   few   research   on   the   problem   of   identifying    
advances  that  improve  the  experience  of  over-­‐the-­‐ chessboard   pieces   has   made   one   or   more   of   the  
board  chess.  One  such  problem  is  the  tediousness   following  simplifications:  
of  recording  the  sequence  of  moves  performed  in  
• The  camera  is  mounted  directly  above  the  
a  game.  
board,  leading  to  minimal  perspective  
We  aim  to  improve  this  aspect  of  the  game  by   distortion  
creating   a   robust,   automated   system   that   • Plain  background,  making  it  easy  to  detect  the  
leverages   computer   vision   in   order   to   provide   chessboard  
insights   into   chess   games   played   on   physical   • Multiple  cameras,  leading  to  minimal  total  
boards.  In  particular,  we  would  like  to  implement   occlusion.  
a   set   of   CV   algorithms   that   allow   us   to   determine  
We  present  examples  of  such  research  below:  
the   full   state   of   the   board   at   all   times   using   widely  
available   hardware   (namely   a   Macbook   Pro   next   Research   involving   finding   chessboard  
to   the   board.)   We   will   then   provide   users   with   gridlines   from   a   top-­‐down   view   is   a   popular  
recordings  of  their  games  that  are  easily  portable   computer   vision   problem   [1],   and   research   has  
to  common  chess  databases  and  engines,  so  that  a   been   made   into   exclusively   detecting   chessboard  

  1      
corners.   One   report   [2]   leverages   the   4. The  program  recognizes  when  the  game  is  
checkerboard  black-­‐and-­‐white  colors  surrounding   finished  (i.e.  an  end  state  is  reached),  and  
chessbaord   corners   as   features   to   detect   terminates  
chessboard  corners,  achieving  comparable  results  
to  other  combined  edge-­‐corner  detectors.     4.  Technical  Details  
Another   report   [3]   uses   a   setup   of   two  
cameras   with   perpendicular   views   of   the   Terminology
chessboard  at  relatively  low  perspective  angles  to   • Board  coordinates:  (2x1)  vectors  that  
reduce   the   probability   of   simultaneous   occlusion   represent  points  on  the  chessboard;  (0,  0)  
of   a   piece   or   square.   To   aid   in   determining   the   describes  the  top  left  corner  of  a8  and  (8,  8)  
orientation  of  the  chessboard  the  setup  also  places   describes  the  bottom  right  corner  of  h1.  
a  marker  on  the  board  at  the  corner  of  a  particular   • Image  coordinates:  (2x1)  vectors  that  
setup.   However,   by   maintaining   the   initial   setup,   represent  points  in  the  image.  
the   researchers   achieve   live   gameplay   with   a   • Board-­‐Image  Homography:  A  (3x3)  projective  
computer   tracking   the   positions   of   pieces   on   the   matrix  that  maps  homogenized  board  
board.   coordinates  onto  homogenized  image  
It   is   also   notable   that   the   OpenCV   function   coordinates.  
findChessboardCorners   does   not   perform   well   at   The   first   task   is   to   find   the   homography   from  
acute   angles   such   as   that   found   by   a   Macbook   board   coordinates   to   image   coordinates   from   an  
Pro’s  camera.     empty   board   taken   in   the   first   frame,   at   the  
perspective  angle  from  a  Macbook  Pro.  
3.  Approach  
4.1  Initialization:  Finding  the  Board-­‐
Our  approach  attempts  to  do  the  following,  in  
Image  Homography  
an  effort  to  be  pragmatic  and  useful  as  a  general-­‐
use  program:   In   order   to   find   the   homography   from   the  
board  to  the  image,  it  is  necessary  to  find  a  set  (at  
• Require  minimal  setup  overhead  
least   four,   but   preferably   many   more)   of   point  
• Leverage  established,  open  source  tools  
correspondences   from   board   points   to   image  
• Maximize  robustness   points.  The  following  algorithm  was  performed  to  
In  an  effort  to  do  this,  we  solved  the  problem   achieve  this:    
with   a   very   simple   setup:   A   Macbook   Pro   (or   any   1. Find  definite  corners  by  narrowing  the  output  
computer  with  a  camera)  on  the  side  of  the  board   of  a  Harris  Corner  Detector  with  a  SIFT  
on   white’s   left,   with   the   screen   roughly   Descriptor  Classifier  and  filtering  on  points  
perpendicular   to   the   table’s   surface.   This   allows   that  closely  snap  to  high-­‐threshold  vertical  
for  easy  setup  and  quick  initialization.  A  full  cycle   and  horizontal  lines  obtained  by  a  Hough  
of  our  program  works  as  follows:   Transform.    
1. The  computer  is  set  up  next  to  an  empty   2. Use  a  RANSAC-­‐like  algorithm  with  linear  and  
chessboard  and  the  program  takes  a  snapshot   geometric  regressions  to  find  indices  of  
of  the  board,  allowing  it  to  determine  a   vertical  and  horizontal  lines  from  the  Hough  
mapping  between  board  squares  and  the   transform  and  assign  each  remaining  image  
image  regions  they  correspond  to.   point  to  an  integer  pair  representing  a  board  
2. The  players  set  up  pieces  on  the  board  and   point.  
gameplay  commences.  
3. After  each  player  makes  a  move,  they  hit  any   Finding  a  subset  of  chessboard  corners  
key  on  the  laptop,  as  they  would  hit  a  chess   First,   we   ran   a   Harris   Corner   Detector   on   the  
clock  in  tournament  play.  The  program  keeps   image   to   get   a   large   set   of   possible   corners   from  
track  of  the  position  after  each  move,   the  raw  image.  
capturing  an  image  of  the  current  state  of  the  
board  every  time  a  key  is  pressed.  

  2      
Figure  1:  On  the  top  is  an  image  with  corners  detected  with  a  Harris  Corner  Detector  overlaid.  On  the  bottom  is  
the  same  image,  after  the  corners  have  b een  filtered  with  a  SIFT  Descriptor  Classifier  and  clustered  using  mean-­‐
shift  clustering.  

The  results  of  a  Harris  corner  detector  to  one   In   order   to   assign   each   of   these   points   board  
of   our   preliminary   images   is   shown   in   (fig.   1).   coordinates,   we   need   to   first   determine   which  
Clearly,  this  has  a  high  recall  in  finding  chessboard   index   (an   integer   in   1, 2, 3, … , 8 )   each   vertical  
corners,   as   every   corner   on   the   board   is   saliently   line   and   each   horizontal   line   correspond   to.   We  
marked.   This   provides   a   good   population   of   perform   the   following   algorithms   to   determine  
candidates   for   more   complicated   classification   these   line   correspondences,   given   four   or   five  
procedures.   lines,  either  all  vertical  or  all  horizontal:  
To   alleviate   this   issue,   we   ran   a   SIFT   For  vertical  lines:  
Descriptor   Classifier   on   the   Harris   Corners,   1. For  each  line,  get  its  x-­‐intercept  with  the  
namely   a   logistic   regression   classifier   trained   on   bottom  of  the  image  of  each  line  
roughly   2000   negative   and   1000   positive   2. Generate  all  possible  line  assignments  that  
examples.   This   both   significantly   reduced   the   satisfy  that  if  line  m  is  to  the  left  of  line  n,  m  <  
number   of   false   positives.   In   order   to   ensure   that   n  
corners   were   not   double   counted,   we   applied   a   3. For  each  assignment,  fit  a  linear  regression  on  
clustering   procedure   very   similar   to   mean-­‐shift   the  (assignment,  x-­‐intercept)  pair  for  each  
clustering.   However,   it   is   worth   noting   that   some   line  and  get  the  correlation  coefficient  
corners  are  missing  from  the  classifier,  but  indeed,   4. Return  the  assignment  with  the  maximum  
the   algorithm   only   needs   a   small   set   of   corners   correlation  coefficient.  
with  assigned  board  coordinates.  
For  horizontal  lines  
Assigning  board  indices  to  each  corner   1. For  each  line,  get  the  logarithm  of  the  average  
height  of  the  line  above  the  bottom  of  the  
To  determine  which  board  corners  each  of  the  
image  
corners  returned  by  the  SIFT  Descriptor  Classifier  
2. Generate  all  possible  line  assignments  that  
is,  we  fitted  the  corners  to  horizontal  lines  using  a  
satisfy  that  if  line  m  is  above  line  n,  m  <  n  
Hough   Transform.   In   order   to   only   capture   lines  
3. For  each  assignment,  fit  a  linear  regression  on  
that  are  actually  lines  on  the  board,  we  use  a  high  
the  (assignment,  x-­‐intercept)  pair  for  each  
threshold,  at  the  expense  of  possibly  missing  lines.  
line  and  get  the  correlation  coefficient  
We   only   consider   points   very   close   to   both   a   4. Return  the  assignment  with  the  maximum  
vertical  line  and  a  horizontal  line.   correlation  coefficient.  

  3      
Figure  2:  On  the  top  are  the  lines  detected  b y  a  Hough  Transform  on  the  points  given  by  the  SIFT  Descriptor  
Classifier,  filtered  to  only  horizontal  lines.  On  the  bottom  are  the  lines,  filtered  to  the  vertical  lines.  

We  take  advantage  of  the  fact  that  the  correct   a   small   set   of   known   point   correspondences  
assignments   will   lead   to   an   approximately   linear   between   the   image   and   board.   This   homography  
sequence   for   vertical   lines,   and   an   approximately   allows   one   to   find   the   image   coordinates   of   any  
geometric  sequence  for  horizontal  lines.   point   on   the   board,   therefore   enabling   one   to  
Finally,   we   notice   that   this   will   only   give   us   select   image   regions   corresponding   to   certain  
accurate  results  up  to  a  shift,  since  results  up  to  a   squares.   Here   we   outline   our   approach   to   finding  
shift  will  all  give  equal  correlation  coefficients.  To   this  homography.  
determine   which   of   the   possible   shifts   is   correct,   Let   𝑃! ,   𝑃!!  be   corresponding   points   in  
we   simply   determine   which   of   the   possible   shifts   board/image   coordinates,   respectively,   for  
is   correct   by   seeing   which   of   the   homography   𝑖 ∈ 1, … , 𝑛 .   Then   ∃𝐻 ∈ ℝ!×!  such   that   ∀𝑖, 𝑃!! =
correctly   matches   the   most   Harris   Corners   (after   𝐻𝑃! .    
filtering   with   the   SIFT   Descriptor   Classifier)   found   𝐻  can   be   determined   by   the   following  
earlier.   overdetermined   system   of   equations   𝑃ℎ = 0  
where  𝑃  is  given  by:  
Computing  the  Board  Image  Homography  
Only   a   subset   of   board   corners   would   be  
detected   by   the   above   algorithm.   We   are  
guaranteed,  however,  that  the  chessboard  consists  
of   an   evenly-­‐spaced   grid   lying   on   a   plane.   Hence,   and  ℎ  is  a  columnized  representation  of  𝐻 .  
there   is   a   projective   transformation   relating  
In  order  to  solve  this  system  of  equations,  we  
points   on   the   board   (in   board   coordinates)   with  
apply   SVD   to   P   and   construct   a   matrix   from   the  
points  in  the  image,  which  can  be  computed  from  
last  row  of  the  the  third  return  matrix  from  SVD.    

  4      
Figure  3:  Example  heatmaps  generated  by  the  program.  The  left  heatmap  displays  squares  with  a  large  
increase  in  piece  color  and  the  right  heatmap  displays  squares  with  a  large  decrease  in  piece  color.  The  correct  
move,  which  the  program  finds  correctly,  is  Nf3,  or  g1-­‐f3.  

b. Find  the  cosine  similarity  between  the  


4.2  Live  Gameplay:  Determining  Moves   expected  and  actual  heatmaps  
Once   the   homography   has   been   determined,   4. Return  the  move  with  the  lowest  cosine  
the  players  set  up  the  pieces  and  the  game  begins.   similarity  
At   the   beginning   of   the   game,   when   the   pieces   Several  approaches  were  examined  to  
have  just  been  set,  the  program  takes  a  snapshot.   generate  expected  heatmaps.    
Using   the   centroids   from   a   k-­‐means   clustering   A   naive   solution  that   was   initially  
algorithm  with  k  =  4,  we  determine  the  four  most   implemented  was  to  fill  the  expected  heatmap  for  
common   colors   on   the   board.   Those   four   colors   increase   in   piece   color   with   zeros,   except   for   the  
will  be: square  into  of  which  a  piece  was  moved,  in  which  
1. The  color  of  the  white  squares   the   value   is   1.   Conversely,   the   expected   heatmap  
2. The  color  of  the  black  squares   for   decrease   in   piece   color   would   be   filled   with  
3. The  color  of  the  white  pieces   zeros,   except   for   the   square   out   of   which   a   piece  
4. The  color  of  the  black  pieces   was  moved,  in  which  the  value  is  1.  
In  live  gameplay,  after  each  move  is  played,  a   This   algorithm   initially   yielded   promising  
new  image  is  taken.  The  image  of  the  board  before   results   for   the   first   few   moves   of   a   game,   but  
the   move   was   played   (taken   after   the   previous   generated   incorrect   predictions   when   the   square  
move  was  taken)  and  an  image  of  the  board  after   that   were   most   occluded   by   the   introduction   of   a  
the  move  was  played  are  compared  as  follows:   piece   was   not   the   square   into   which   the   piece   was  
1. Match  each  pixel  of  each  square  area  to  one  of   moved,   but   instead   the   square   immediately  
the  four  categories  colors  generated  via  k-­‐ behind   or   two   squares   behind   it.   This   was   a  
means  clustering.     common   occurrence,   so   we   modified   our  
2. Find  two  heatmaps  for  increase  and  decrease   algorithm  to  reflect  that.  
of  the  color  of  the  piece  being  moved  in  the   The   solution   that   we   selected   was   to   fill   the  
actual  moves  in  each  square,  and  divide  each   expected  heatmap  for  increase  in  piece  color  with  
of  the  heatmaps  by  the  the  sum  of  its  values.     zeros,   except   for   the   square   into   of   which   a   piece  
3. For  each  move   was   moved   and   the   three   squares   behind   it,   in  
a. Generate  two  expected  heatmaps  for   which   the   values   are   1.   Conversely,   the   expected  
increase  and  decrease  of  the  color  of  the   heatmap   for   decrease   in   piece   color   would   be  
piece  being  moved  in  each  square,  and   filled   with   zeros,   except   for   the   square   out   of  
divide  each  of  the  heatmaps  by  the  sum  of   which   a   piece   was   moved   and   the   three   squares  
its  values.   behind  it,  in  which  the  value  is  1.  

  5      
5.  Experiments   6.  Applications  
5.1  Homography  testing   Our   intention   with   this   application   was   to  
provide   an   API   on   top   of   which   other   developers  
We   tested   the   program’s   ability   to   generate  
could   create   useful   and   insightful   applications.  
the   correct   homography,   and   found   that   while  
Here   we   discuss   potential   future   directions   for  
certain   conditions   affect   the   accuracy   of   the  
development.    
generated   homography,   under   conditions   that  
would   be   optimal   for   human   chess,   the   While   our   application   provides   one   with   the  
homography  is  generated  with  good  accuracy.     ability  to  receive  a  standard  notation  for  any  given  
chess   game   played   on   a   standard   board,   we   have  
Most   significantly,   varying   lighting   conditions  
not   yet   made   efforts   to   integrate   this   with   existing  
significantly,  especially  when  there  is  severe  glare  
chess   engines.   We   believe   that   the   real-­‐time  
on  top  of  several  of  the  points  in  a  row  or  column  
application  of  existing  analytical  algorithms  would  
will  frequently  lead  to  an  incorrect  homography.    
offer   a   great   deal   of   utility   to   players.   In   particular  
However,   in   conditions   when   the   image   is   existing   algorithms   for   determining   a   player's  
taken   by   a   Macbook   Pro   and   visually   has   strengths   and   strategic   characteristics,   predicting  
appropriate   lighting,  the   program   generates   the   future   moves   and   assessing   their   performance  
correct  homography  in  21  out  of  25  test  cases  that   improvements   (e.g.   assigning   chess   ratings,   etc.)  
we   ran.   In   conditions   with   poor   lighting,   should   be   relatively   easy   to   add   to   our   system.  
significant  obtrusive  glare,  or  other  similar  issues,   Such  algorithms  have  never  been  applied  to  chess  
this   accuracy   was   reduced   to   7   out   of   15   test   on   real   boards   using   commercially   available   and  
cases.   common  hardware,  to  our  knowledge.  
In   addition,   we   anticipate   that   users   would  
5.2  Live  game  testing  
benefit  from  the  availability  of  coaching  resources  
We   also   tested   the   program   in   a   number   of   in   games   on   real   boards.   While   several   chess  
complete   games   played   to   determine   how   many   coaching   utilities   currently   exist   (including   those  
moves   (single   player   moves)   the   program   available   through   Stockfish,   an   open   source   and  
correctly   predicts   until   it   makes   a   mistake.   (We   very   popular   chess   engine),   current  
used   this   metric   instead   of   the   total   count   of   the   implementations  require  that  one  play  online  or  in  
number   of   incorrect   moves   in   a   game   because   a   much   more   controlled   and   resource-­‐intensive  
after   the   first   incorrect   move,   following   moves   are   apparatus.    
less   likely   to   be   correct,   since   the   correct   move  
Finally,   we   believe   that   one   of   the   most  
may  no  longer  be  legal.)  
exciting   directions   that   CVChess   will   progress   in  
the   future   is   its   integration   into   augmented   reality  
systems.   Due   to   the   fact   that   it   can   offer   precise  
Game Total  moves First  incorrect  move coordinates   for   real-­‐world   phenomena,   it   could   be  
used,   with   relatively   little   adaptation,   as   a  
1 48 No  incorrect  moves supplement   to   a   projector   pointing   at   a   board,  
annotating   games   in   real   time.   While   animations  
2 39 18
projected  onto  the  board  may  become  distracting,  
3 75 59 we  maintain  that  simple,  minimalist  aids  projected  
onto   the   board   would   enhance   the   game   playing  
4 28 No  incorrect  moves experience.   This   includes,   but   is   not   limited   to,  
visual   indicators   for   when   pieces   are   in  
5 68 40 check/causing   check,   areas   one   can   move   into  
during   check,   lines   of   attack   from   specific   pieces  
and  more.    
  In   conclusion,   we   have   made   the   first   step   in  
developing   a   low-­‐cost,   robust   platform   on   which  

  6      
we   hope   many   useful   chess-­‐related   applications  
are   developed   in   the   future.   We   are   open   to  
collaboration;  please  contact  the  authors  if  you  are  
interested  in  developing  on  top  of  CVChess.  

7.  References  
[1]  Martin  Martin.  Finding  a  Chessboard.  2009.    
[2]   Chua   Huiyan,   Le   Vinh,   Wong   Lai   Kuan.   Chess  
Vision.  2007.  
[3]   Stuart   Bennett   and   Joan   Lasenby.   ChESS   –  
Quick   and   Robust   Detection   of   Chess-­‐board  
Features.  2012.  

  7      

You might also like