HIRSH1

School of Computing
FACULTY OF ENGINEERING
3D Noughts & Crosses with Baxter, a Humanoid Robot

Benjamin Leonardo Hirsh
Submitted in accordance with the requirements for the degree of
BSc Computer Science (Industry)
2014 - 2015
i
The candidate confirms that the following have been submitted.
Items
Report (Hard Copy)
Report (Digital)
Software Code
Format
Report
Signed forms in envelop
Repository URL
Demonstration
Video URL
Recipient(s) and Date

SSO (03/06/15)
SSO (03/06/15)
Supervisor, Assessor
(02/06/15)
Supervisor, Assessor
(02/06/15)
Type of project: Exploratory Software (ESw)
The candidate confirms that the work submitted is their own and the appropriate credit
has been given where reference has been made to the work of others.
I understand that failure to attribute material which is obtained from another source
may be considered as plagiarism.
(Signature of Student)
c 2014 - 2015 The University of Leeds and Benjamin Leonardo Hirsh
ii
Summary
This report documents the development of a software system run on the Universitys
Baxter Research Robot (named Lucas), enabling him to participate in playing 3D Noughts
& Crosses with another human player. The aim was to create a piece of software to be
run in ROS (Robot Operating System) in order to allow Baxter to do this. The final
product had Baxter successfully playing legal games against human opponents.
iii
Acknowledgements
I would like to thank my supervisors, Professor Tony Cohn and Doctor Eris Chinellato,
for assisting me throughout the project with both the development and the writing of this
report, making sure it complies with standards. I would like to thank the other members
of the STRANDS team, Doctor Ioannis Gatsoulis, Paul Duckworth, Jawad Tayyub and
Muhannad Omari, along with Aryana Tavanai, all of whom gave me advice on solutions
and challenged me on my methods to ensure they were the most suitable. The level at
which I now understand of robotics and computer vision could certainly not have been
reached without them. I would also like to thank my flatmate, Sam Brown, for all his
help and support.
Contents
1 Introduction
1.1 Methodology . .
1.2 Objectives . . . .
1.3 Report Structure
1.4 Schedule . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2 Background Research
2.1 Board Games . . . . . . . . . . . . . . .
2.2 3D Noughts & Crosses . . . . . . . . . .
2.2.1 Physical Properties . . . . . . . .
2.2.2 Rules . . . . . . . . . . . . . . . .
2.2.3 Strategy . . . . . . . . . . . . . .
2.2.4 Computer Representation . . . .
2.3 Adversarial Search . . . . . . . . . . . .
2.3.1 Game Tree . . . . . . . . . . . . .
2.3.2 Minimax . . . . . . . . . . . . . .
2.4 Robotic Paradigms . . . . . . . . . . . .
2.4.1 Hierarchical Paradigm . . . . . .
2.4.2 Reactive Paradigm . . . . . . . .
2.5 Robot Operating System (ROS) . . . . .
2.5.1 Installation . . . . . . . . . . . .
2.5.2 Architecture . . . . . . . . . . . .
2.5.3 Topics & Services . . . . . . . . .
2.6 Baxter Research Robot . . . . . . . . . .
2.6.1 Baxter Software Development Kit
2.7 Existing Projects . . . . . . . . . . . . .
2.7.1 Worked Example Visual Servoing
2.7.2 Connect Four Demo . . . . . . .
2.7.3 Baxter Solves Rubiks Cube . . .
2.8 Software . . . . . . . . . . . . . . . . . .
2.8.1 Programming Languages . . . . .
2.8.2 Libraries . . . . . . . . . . . . . .
2.8.3 Version Control . . . . . . . . . .
iv
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
(SDK)
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
3
4
4
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
5
5
5
6
8
9
10
10
11
12
12
13
13
13
13
13
14
15
16
16
16
16
17
17
17
17
CONTENTS
3 Development & Analysis
3.1 Hardware Setup . . . . . . . . . . . . . . . . . . .
3.1.1 Environment . . . . . . . . . . . . . . . .
3.1.2 Grippers . . . . . . . . . . . . . . . . . . .
3.1.3 Custom Board Game . . . . . . . . . . . .
3.2 Vision . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Blob Detection . . . . . . . . . . . . . . .
3.2.2 Contour Detection . . . . . . . . . . . . .
3.2.3 Background Subtraction . . . . . . . . . .
3.2.4 Hough Circle Detection . . . . . . . . . . .
3.2.5 World Coordinate System Transformations
3.2.6 Camera Calibration . . . . . . . . . . . . .
3.2.7 Pixel Alignment . . . . . . . . . . . . . . .
3.3 Game Logic . . . . . . . . . . . . . . . . . . . . .
3.3.1 2D . . . . . . . . . . . . . . . . . . . . . .
3.3.2 3D . . . . . . . . . . . . . . . . . . . . . .
3.4 Robot Control . . . . . . . . . . . . . . . . . . . .
3.4.1 Basic Movement . . . . . . . . . . . . . . .
3.4.2 Sensors . . . . . . . . . . . . . . . . . . . .
3.4.3 Inverse Kinematics . . . . . . . . . . . . .
3.4.4 Head Display . . . . . . . . . . . . . . . .
3.5 Playing Noughts & Crosses . . . . . . . . . . . . .
3.5.1 How to run programs . . . . . . . . . . . .
3.5.2 Structure . . . . . . . . . . . . . . . . . .
3.5.3 Robot Class . . . . . . . . . . . . . . . . .
3.5.4 Board Class . . . . . . . . . . . . . . . . .
3.5.5 Space Class . . . . . . . . . . . . . . . . .
4 Evaluation
4.1 Board Detection . . . . . . . . . . . . . . . .
4.1.1 Pixel Point Accuracy of Hough Circle
4.1.2 Pixel Point Convergence . . . . . . .
4.2 Object Detection . . . . . . . . . . . . . . .
4.2.1 Pixel Point Accuracy of Contours . .
4.3 Game Play . . . . . . . . . . . . . . . . . . .
4.3.1 Vision Classifications . . . . . . . . .
4.3.2 Game Logic . . . . . . . . . . . . . .
4.4 Code Evaluation . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . . . . .
Detection
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
18
18
19
19
20
21
22
24
25
27
28
29
33
33
34
35
36
36
38
39
39
39
40
40
42
42
.
.
.
.
.
.
.
.
.
44
44
44
45
47
47
49
49
50
50
CONTENTS
5 Conclusion
5.0.1 Time Plan . . . . . . . . .
5.1 Extensions . . . . . . . . . . . . .
5.1.1 Depth Sensing . . . . . . .
5.1.2 Error Handling . . . . . .
5.1.3 Game Variants . . . . . .
5.1.4 Artificial Intelligence . . .
5.1.5 Audio & Voice Commands
5.1.6 Head Display Nod . . . .
5.1.7 Other Games . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
52
52
53
53
54
54
54
55
55
References
56
Appendices
58
A Personal Reflection
59
B Ethical Issues Addressed
61
C Video
62
D Code
63
D.1 Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
D.2 New3DtictacBaxter.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
D.3 BoardClass3D.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
E Board Detection Times Under Varying Light Conditions
91
CONTENTS
Chapter 1
Introduction
The aim of this project is to develop a software system which runs on a humanoid robot,
allowing it to play a game of 3D Noughts & Crosses with a human partner. The robot
should be able to observe and interact with the playing area.
1.1
Methodology
This project has been approached with a rapid prototyping process in mind, reducing the
objectives into small, manageable tasks. The nature of the software development works
with this approach as each iteration adds features and correct flaws from the last. In doing
this, if something were to go wrong, the entire project did not need to be restarted from
scratch or changed drastically. Developing software to enable the robot to perform simple
movements in the Robot Operating System (ROS) environment was the initial iteration,
and further, more complex manoeuvres were developed on top of this, according to the
objectives of Section 1.2.
1.2
Objectives
The following are a set of increasingly ambitious and complex objectives drawn out before
the project began:
1. The robot can pick up a playing piece and place it on an arbitrary space on a 3x3
grid. This is a good starting point to ensure that the robot can actually pick up
and place pieces in specific positions.
2. The robot is able to stack up to three playing pieces on top of one another. Due to
the nature of the game, being able to place one playing piece on top of another in
such a way that the stack will not fall over is essential.
3. The robot is able to recognise the current state of the game by detecting pieces
as they are played and remembering what position they are placed into. This is a
computer vision challenge. Using the cameras on the robot, the state of the game
can be monitored, with a trigger to the robot (visual change to the board) to let
him know that the human players turn is over.
4. The robot understands the game rules and can play the game against a human in
a software environment, using the best possible strategy. This is a software task,
implementing the game logic into the robots functionality.
3
CHAPTER 1. INTRODUCTION
5. The robot develops strategy for playing the game over time. This is an AI problem,
where a learning function will be implemented into the robots code. The best
possible strategy preprogrammed will have to be removed at this point and used as
a reference to measure the robots success.
All but objective 5 have been achieved, and the robot is able to play an entire legal
game.
1.3
Report Structure
The sequence of the chapters in this report follows my development process.

Chapter 2 covers research carried out, prior to development. This included documentation on the hardware and software I would be using as well as aspects surrounding the
Noughts & Crosses game and related work.
Chapter 3 documents the bulk of the development process. An analysis of each of the
systems components is performed alongside its development.
Chapter 4 evaluates the completed system, testing each individual components performance.
Chapter 5 is a conclusion and final summary of the project. It also discusses possible
extensions and improvements.
1.4
Schedule
Figure 1.1 shows a Gantt chart of the estimated time it would have taken to complete
each of the objectives. In week 7 (9th March 2015) there was a student progress meeting
in which I prepared a small demo for my supervisor and assessor. The demo will be
referred to throughout this report when mentioning this student progress meeting.
Figure 1.1: Time plan for each objective.
Chapter 2
Background Research
This chapter documents the various aspects of prerequisite research before development
began; Firstly, the nature of the board game including physical properties, rules, best
strategies and any existing computer representations. Next, potential algorithmic approaches to playing the game via a set of rules, which is followed by robotic paradigms
and their relevance. Finally, research into the robot itself and how one can operate it
programmatically via ROS, including any previously done projects where the robot has
been programmed to pick up and place objects. Further research had to be done during
development in order to create alternative solutions to methods that did not work as
planned. This has also been documented in this chapter.
2.1
Board Games
The board game I have chosen for the robot to play is 3D Noughts & Crosses. I chose
this over a more complex game with a complex set of rules, like chess, as I wanted the
focus of the project to be robotics, rather than AI. Incidentally, the robotics aspect would
be relatively easy to do with board games that feature playing pieces across a 2D plane.
Selecting the three-dimensional version of Noughts & Crosses (or Tic-Tac-Toe) was done
so that more work could be put into developing the robots motor functions and vision
methods.
2.2
3D Noughts & Crosses
Just like regular Noughts & Crosses, the objective is to align 3 of your own playing pieces
in a row, whilst preventing your opponent from doing the same. The exception is that,
rather being played in a 3x3 grid, the game is played in a 3x3x3 cube, where the height
of a piece being played, is determined by previously played pieces in that same position
(similar to Connect 4). Figure 2.1 shows a retail copy of the game. Looking at the
image demonstrates that it is not possible to play a piece in a higher position without
other pieces existing below it.
2.2.1
Physical Properties
The retail version comes with 27 playing pieces. Each piece is either a 3D cross or a sphere
(nought). Every piece features a hole running down the interior length. The board is
essentially a small platform with 9 poles, evenly spaced to form a 3x3 grid (when looked
5
CHAPTER 2. BACKGROUND RESEARCH
Figure 2.1: 3D Noughts & Crosses [1].

at from above). The diameter of the poles are no more than a millimetre smaller than
the diameter of the holes in the playing pieces. This ensures that when impaling a piece
on the poles, it falls to the lowest possible position without any friction. This requires
very finite and precise movement, which Baxter was not be able to perform, resulting in
the need for a custom version of the board game (Section 3.1.3).
2.2.2
Rules
The game is for 2 players only. One player is in possession of the noughts playing pieces,
and the other player possesses the crosses. A starting player is randomly determined and
play begins. On a players turn, that player takes one of their playing pieces and places it
on one of the 9 poles. Once the piece has fallen into position, the turn ends and the next
player takes their turn. This action of placing a piece onto the board I will now refer to
as taking a position and the player using crosses as the first player.
The end condition of the game is when either all 27 pieces have filled the playing area,
or one player has taken three positions with his pieces in a straight line. This line can be
diagonal, vertical or horizontal across all axis.
There are a total of 49 winning lines in a 3x3x3 cube (see Figure 2.2). On his blog
[8], Owen Elton talks about how tic-tac-toe could be played in a 3x3x3x3 hypercube.
Calculating the winning lines on a 2D grid can be done by assigning addresses to each
space in the grid, containing the spaces row and column (such as in Figure 2.3).
address = (row, column)
It is possible to identify whether a set of three distinct addresses form a winning line
by looking at the differences in each element of the addresses. Take the set (a,a,A3),
Figure 2.2: Possible winning lines [8].
Figure 2.3: Addresses of grid spaces [8].

notice how the first element of each address is identical and is therefore a winning line.
For horizontal and vertical winning lines, the alphabetic or numeric properties of the cells
addresses are all the same. For the diagonals, the properties of the cells addresses must
be all different, however, a caveat must be added that if both alphabetic and numeric
properties are all different then the set must contain the central cell (b) to qualify as a
winning line [8].
This article helped me when applying these rules to find the 49 winning lines of a 3dimensional cube, in which each address comprised of 3 elements instead of 2. Section 3.5.4
describes in more detail how I applied this information to generate the winning lines
programmatically.
2.2.3
Strategy
The original game of Noughts & Crosses, has a very simple strategy in which if both
players play optimally, the game will result in a draw. Allen Newell and Herbert Simon
created a Tic-Tac-Toe program in 1972 [7] which followed the following set of rules:
1. Win: If the player has two pieces in a row, they can place a third piece to get three
in a row.
2. Block: If the opponent has two pieces in a row, the player must play the third piece
themselves to block the opponent.
3. Fork: Create an opportunity where the player has two possible lines to win.
4. Option 1: The player should create a Block situation with two pieces in a row
to force the opponent into defending, as long as it doesnt result in them creating a
Fork situation.
5. Option 2: If there is a configuration where the opponent can create a Fork situation, then the player should block that fork.
6. Center: A player marks the centre.
7. Opposite corner: If the opponent plays a piece in the corner, the player plays a
piece in the opposite corner.
8. Empty corner: The player plays in a corner space.
9. Empty side: The player plays in a side space.
This rule set was very useful when developing the 2D version of the game to play. In
Section 3.3 I use a similar, but simplified version of these rules for the robot to follow.
For the opening move, there are 9 places the first player can play into. However, this
can be logically simplified to 3 places, as all the corner spaces are strategically equivalent
and all the side spaces are strategically equivalent. This is only for the first turn. Of these
three moves, the one with the most options to win for the first player is the corner [9].
This is because, out of the 8 remaining spaces for the second player, they can only avoid a
loss by taking the centre space. If the first player had taken the centre, the second player
avoids a loss by taking a corner, of which there are 4. This information was not used
however, for the implementation of the game logic in the robot always had the human
player acting first. This decision was made so that certain states of the board could be
forced for testing purposes.
The 3D variant of the game has a different strategy. When played on a 3x3x3 cube,
the game is easily winnable once a player has a piece in the central position [24]. With
the retail version of the game, this can occur when one player takes the centre position on
the lowest level of the board, but also having another piece next to it, forcing the other
player to block by taking a position on the lowest level. Figure 2.4 shows how the second
player is unable to take the central position without losing to the first player on the lowest
level. It is even possible to force a win from a much earlier point in the game by having
the first player take the central space on the lowest level, and then taking the space above
it on their next turn.
Figure 2.4: Two instances where the second player is forced to block.
The result of this is the game devolves into the winner being the player who takes the
first turn. This discovery made the incorporation of the game logic into the robot very
easy. It shows that the game is solvable, which means there is always an optimum move
to take. As the focus of the project is not to develop a challenging AI but rather have
the robot interact in physical space with accuracy, the game did not need to be changed
in light of this.
2.2.4
Computer Representation
There are many online games of 3D Noughts & Crosses. However, due to the flawed
nature of the 3x3x3 variant, the majority of them are played on a 4x4x4 board, which is
much less predictable. I also discovered that it is possible to play the game without the
gravity factor of the retail version. Unfortunately this allows the first player to take the
central position at the start of the game, further simplifying the games strategy.
Figure 2.5: Sequence of moves until game end in 3D [2].

Using screenshots from one of these games, Figure 2.5 shows the next logical moves
from the example in Figure 2.4, once the first player (orange) takes the central position.
10
The second image shows the second player (blue) is forced to block by taking the top
centre position, allowing the first player to take a move leading them to win the game.
Whatever move the second player makes in this state of the game, they cannot prevent
the first from winning.
Switching to the 4x4x4 variant of the game is a possible extension I discuss in Section 5.1.3. Developing the AI for this will be more challenging though, and like the 3x3x3
variant the first player can force a win [12].
2.3
Adversarial Search
There are a number of algorithms that can be used to solve the best strategy for particular
games. This section explores the algorithms which were appropriate for the implementation of the robots game logic.
Games usually fall under adversarial search problems. Noughts & Crosses is a common
example used in game theory, it can be classed as a deterministic, turn-taking, two-player,
zero-sum game of perfect information [19]. The game is deterministic because there is no
elements of chance, such as dice or decks of cards. Such games are much easier to solve
than games of chance. Perfect information refers to all elements of the game being visible
at all times, whereas imperfect information in games is featured as hidden details, such as
an opponents hand of cards in poker. Each move made by a player (with the aim to win
the game) adversely affects the opponents chances of winning. This is known as a zerosum game, and as the players are each trying to ensure they win whilst simultaneously
ensuring their opponent loses, it makes the situation adversarial.
2.3.1
Game Tree
A game tree is a directed graph which shows all the possible states of a game. A full tree
consists of the initial state of the game as the root node, and branches down to all the
possible variations of terminal states at the leaf nodes. Each level of the tree represents
all the moves which can be made by a player for each turn in the game, and this alternates
between the players every level. In the case of a 3x3 grid, the first player can make one
of nine moves from the initial state therefore nine states branch out from the root node.
The next level contains all the possible states of the game after the second player makes
a move in response to the first, and so on. Figure 2.6 shows a more streamlined game
tree for Noughts & Crosses, showing the 3 possible opening moves (mentioned in the
previous section). There are fewer than 9! = 362,880 terminal nodes [19] in the full tree
of a standard game of Noughts & Crosses. This relatively small number means that it is
entirely plausible to generate a game tree for the 3D variant of Noughts & Crosses, and
implement an algorithmic approach to the robots artificial intelligence when playing it
against a human opponent.
11
Figure 2.6: Game tree, showing states of playing in a corner, side or in the centre [23].
2.3.2
Minimax
The minimax algorithm can be used to search a game tree to calculate the best possible
solution at each given state. As mentioned in Section 2.3.1, each level of the Noughts &
Crosses tree alternates between the players. Minimax goes through these levels maximising the chances of winning for one player and minimising for the other. The algorithm can
be described by naming the players Min and Max. Max will go first and both players
will keep playing until the end of the game. This creates a game tree similar to Figure 2.6.
When the terminal states are reached, scores (known as utility values) are placed at the
leaf nodes, a high value for a terminal state where Max won and a low value for where
Min won. These utility values are calculated at all the child nodes in the tree. Figure 2.7
shows an example of how this is done. Starting at the parents of the leaf nodes, Max
will choose the highest value from each of the child nodes (In the left-most branch, Max
chooses a 3 because it is higher than the 2). The chosen value is labelled at the parent
node, and Min now must choose the lowest value from these nodes to label their parent
nodes (in the left-most branch, Min has also picked the 3 because it is lower than the 9).
Going up through the tree, the initial node is reached, which is the 3 in this example.
This path is how the current player decides which move to make. The move being chosen
will follow the path that benefits them (Max) the most whilst hindering the other player
(Min) the most.
The algorithm performs a complete depth-first exploration of the tree, using a simple
recursive computation of the minimax values for each successor state [19]. At each node
(or state) of the tree, depending on whether the depth of that node is either Maxs or
Mins turn, the algorithm will choose the highest or lowest utility value of its child nodes,
respectively. However, in order to work out the utility value of the child nodes, the minimax computation must be run on their respective child nodes. This recursion continues
to the leaf nodes (or terminal states) of the tree, and then the minimax values are passed
up through the tree where they were required, resulting in a tree like Figure 2.7. Once
12
Figure 2.7: Minimax Example [3].

all the child nodes have utility values assigned to them, the decision aspect mentioned in
the previous paragraph can be carried out.
2.4
Robotic Paradigms
A paradigm describes the approach to a problem whilst implying a possible structure for
its potential solution [11]. A robot paradigm makes use of three primitives which can be
used to categorize functions within the robot.
Sense - Handle raw information from sensing/detecting aspects of the robots environment. Examples include camera feeds and range sensors.
Plan - Take information from either sensory input or innate knowledge, then compute appropriate actions to reach a goal state.
Act - Perform actions which result in the manipulation of the environment. This
includes a change in location of the robot itself via movement.
A given paradigm describes the relationship between these three primitives [11].
2.4.1
Hierarchical Paradigm
This paradigm features a top-down approach. The robot first takes sensory data of the
surrounding environment until it has enough data to move onto the next stage. From
the sense stage, it takes the data and formulates a best possible action to perform in the
plan stage. Finally, this action is physically performed by the robots motors in the act
stage. This continually loops until a goal state has been reached. The downside of this
paradigm is that a lot of time is lost during the plan stage. In a dynamic environment
this will come across as an unresponsive robot, since the computations of the plan stage
will not take into account that aspects of the environment can change during this time.
13
However, due to the environment featured in my project remaining static for the duration
of the game, this is the paradigm that I adopted.
2.4.2
Reactive Paradigm
The reactive paradigm comprises of multiple SENSE-ACT couplings, eliminating the need
for a plan stage. This is the solution to unresponsive robots in dynamic environments
for each type of sensory input directly translates to a particular action. The downside
of this paradigm is that for complicated sensory inputs, there must be an increase in
the number of SENSE-ACT couplings for each possible environmental reaction. As the
number of these environmental factors grow (sense), so do the reaction functions (act),
and the difficulty arises in managing them.
2.5
Robot Operating System (ROS)
ROS is a free, open-source framework for writing robotics software. Its flexibility comes
from a wide selection of tools, libraries and conventions. The Baxter research robot used
in this project runs on ROS, so becoming familiar with it was a prerequisite step.
2.5.1
Installation
Each distribution of ROS is targeted at running in specific releases of the Ubuntu distribution of Linux. ROS Indigo is the distribution that the robot is currently running, which is
primarily targeted at the Ubuntu 14.04 LTS release [18]. After installing VirtualBox VM
software on my machine, I installed Ubuntu 14.04 with ROS Indigo in order to familiarize
myself with the ROS framework and follow the tutorials found on the wiki.
2.5.2
Architecture
ROS works very much like a file system. Most of the functionality comes in the form
of terminal commands; roscd to change directory, rosrun to run a program. Every
project created in this file system is stored in the form of a package. The package contains
all software and data, as well as header files to import necessary dependencies. Once a
package is ready to be built, the rosmake command is entered to compile all the packages
in the file system alongside all the required dependencies.
2.5.3
Topics & Services
A robot is usually a series of connected nodes and sensors. The nodes tend to be motors
which allow the robot to interact with the physical environment and the sensors are a
form of input, containing information about the robots surroundings. ROS is able to
14
interact with these aspects via the medium of Topics and Services. rostopic list will
list all the available topics. Topics are the buses in which the nodes in a robot exchange
messages. An example of a topic could be the rotational state of a particular joint in the
arm of a robot.
Services are subscribed to in order to utilize information which is feeding into the
robot. For example, to output the image feed from a camera on the robot, the software
subscribes to that cameras service. rosservice list lists all the available services.
2.6
Baxter Research Robot
Figure 2.8: Baxter Research Robot by Rethink Robotics [4].

Baxter is an industrial robot built by Rethink Robotics and introduced in September
2012 [21]. Baxter is a 3-foot tall (without pedestal; 510" - 63" with pedestal), twoarmed robot with a screen that can be used as an animated face. It weighs 165 lbs
without the pedestal and 306 lbs with the pedestal [16]. Throughout the project I worked
with the robot whilst it was mounted on its pedestal. Every joint in each arm features a
servo motor, whose angles are adjusted in order to achieve movement. Holding the cuff
15
off an arm puts the robot into Zero Force Gravity Compensation mode, allowing free
manipulation of the limb by physical guidance. The robot has 3 cameras, one in the wrist
of each arm and one built into the face display. There are also infra-red sensors on the
wrists which calculate distance. The robot comes with several interchangeable grippers for
the arms each varying in width and length. There is a selection of holes that the grippers
can be screwed into in order to adjust the overall distance between them (see Figure 2.9).
The grippers are opened and closed by a servo motor as well. I will be referring to the
Baxter Research Robot as Baxter for the rest of this report.
Figure 2.9: Camera, infra-red sensor and gripper adjustment [15].
2.6.1
Baxter Software Development Kit (SDK)
The Baxter SDK provides an interface to allow custom applications to be run on the
Baxter. The main interface of the Baxter SDK is via ROS Topics and Services, which is
why learning ROS was a necessity. The SDK comes with libraries to control Baxter from
programs. The Hello Baxter! tutorial on Rethink Robotics wiki shows how to store the
angles of the motors in Baxters arm, manipulate them, and then update the arm with
the new angles. The SDK also includes example scripts to run on Baxter, including the
important tuck_arms.py which I used at the start and end of each development session
to store the robot. There is another example which maps every motor in the robot to
the keyboard. This is a very impractical way of controlling Baxter for the movements are
not precise. However, I found the control of the grippers in this script useful in testing
different gripper sizes on varying objects.
2.7
16
Existing Projects
It was important to find out whether my project had been done before. Finding existing
projects prove useful in seeing where others have succeeded or failed in the research you
are planning to do. In searching for projects identical to my own, I discovered that this
project is entirely unique. However, projects with similar goals and solutions were found.
2.7.1
Worked Example Visual Servoing
This example [14] is targeted at 1st-2nd year undergraduates on courses similar to my own.
The aim of the project is to have Baxter pick up golf balls, and place them into slots in an
egg tray. The example uses Visual Servoing for workspace relationship and calibration,
Visual Object Recognition and target selection. It also makes use of inverse kinematics.
This project helped me greatly during the development phase of my project. Section 3.4.3,
Section 3.2.5, and Section 3.2.7 detail development work which was influenced by this
example, specifics can been seen in each section. I will be referring to this project as the
Golf Example from now on.
2.7.2
Connect Four Demo
The Connect Four Demo [17] features Baxter playing the original game of Connect 4,
either against a human opponent or itself. There is a requirement for a custom game piece
feeder which allows Baxter to pick up the game pieces with ease. The board detection
aspect has the user click and drag over the image coming from the camera feed, to highlight
the board. An HSV blob detector (like the one I use in Section 3.2.1) is used to identify
which colour game pieces have been played.
2.7.3
Baxter Solves Rubiks Cube
This project [6] has Baxter solving the Rubiks cube puzzle. The stages of runtime are
separated into detection, verification, solving and manipulation. Detecting the current
state of the cube is done with the use of a flatbed scanner. Baxter does not utilise any
of its detection hardware for this project. The location of the scanner is taught by using
Baxters Zero Force Gravity Compensation mode. It then scans each of the six sides
of the cube and verifies that it has scanned them correctly. The Kociemba algorithm is
then run to solve the cube and return a list of the required manipulations. Baxter then
performs these manipulations, in a manner similar to that of a human. This project did
not feature any aspects that crossed over with my objectives, therefore I did not refer to
it further during development.
2.8
2.8.1
17
Software
Programming Languages
The available programming languages for ROS are Python and C++. I selected to use
Python, due to its simplicity and the fact that the Baxter SDK examples were also in
Python. The performance advantage gained from using C++ is not necessary for the
purposes of this project.
2.8.2
Libraries
Apart from the Baxter SDK, another library I made use of was OpenCV. As the name
implies, it is an open source set of programming functions aimed at doing computer vision
in both images and video frames. All of the computer vision aspects of the project made
use of this library.
2.8.3
Version Control
A repository was set up on my Bitbucket account for all the source code (including this
report) developed during the project. I used git commands to pull and update code from
the repository when changes were made. This method allowed me to work on the project
from multiple places as well as provided a backup in case of any hard disk failures. The
link to the repository has been included in Appendix D.
Chapter 3
Development & Analysis
This chapter investigates the aims and objectives set in the first chapter, and the possible
solutions to go about achieving them.
The structure of this chapter mimics the hierarchical paradigm of Sense, Plan, Act
(from Section 2.4) used in the robots software. The first section covers the setup of
the hardware used in the project. This is then followed by the Vision (Sense), Game
Logic (Plan) and Robot Control (Act) sections. The final section summarises all the prior
sections and how the components developed in each of them fit together. This includes a
more in-depth look at the code.
Each iteration of development builds upon the previous, with additional features being
added. All the code has been stored in separate files for each iteration and are referred
to as a .py program (e.g examplescript.py). Details on how to run these programs
can be found in Section 3.5.1.
3.1
3.1.1
Hardware Setup
Environment
All development done on the robot had been in a constant environment, with sufficient
lighting.
Figure 3.1: Baxters physical workspace environment
18
CHAPTER 3. DEVELOPMENT & ANALYSIS
19
Figure 3.1 shows the table in which Baxter is stood in front of, which was kept at the
same height throughout the project.
3.1.2
Grippers
The Baxter research robot comes with a number of grippers of varying size which can be
attached to the arm.
Size
Each pair has a minimum width, in Figure 3.2 the two pairs of grippers in the centre were
narrow enough to pick up the custom made game pieces. I settled on the right-most one
as I did not require the longer length. The shorter piece meant that more tension would
be applied to the picked up object, lessening the chances of it being dropped by accident.
Figure 3.2: Electric Parallel Gripper Kit [5].
Shape
The kit comes with several pairs of rubber tips which slot onto the ends of the grippers.
The ends of these tips are either rounded or square. Initially I did not take into consideration the importance of deciding which tip to use. When it came to implementing
Hough Circle detection (See Section 3.2.4) I noticed that the rounded gripper tips were
being detected as circles by the program. You can see in the screenshots that the ends
of the grippers are always in view. Figure 3.8 shows the rounded tips in the image feed.
Changing the tips was a simple matter. The difference the square tips make can be seen
in Figure 3.9.
3.1.3
Custom Board Game
The pieces of the retail version of 3D Noughts & Crosses were too small for Baxters
grippers. Therefore, part of the development process during the project was to create
a custom version of the board game, playable by Baxter. To do this, I cut a 2 metre
20
long, 62mm x 37mm plank of wood into 62mm x 62mm x 37mm blocks. These were then
coloured green (crosses) and blue (noughts). Figure 3.3 shows the playing pieces.
Figure 3.3: Custom board game pieces (originally red).

Figure 3.4 shows the design of the final board. I created a Python script, genBoard.py
to generate this image using OpenCV. This was to ensure symmetry and positioning of
grid spaces and the circles, and matching the image resolution to the pixel dimensions of
A3 paper. OpenCV has functions for drawing perfect squares and circles which were all
anchored off the images central pixel coordinates. The circles were used to calculate the
centre points of the grid positions. More details of this are in Section 3.2.4.
Figure 3.4: New board design, generated in OpenCV.

White circles were drawn onto one side of the human players pieces (Figure 3.5). This
was due to the limitations of background subtraction (described in Section 3.2.3) when
detecting two pieces of the same colour on top of one another. The implementation of
this can be found in Section 3.2.4.
3.2
Vision
To achieve the first objective (Section 1.2), I had to implement the ability of enabling
Baxter to pick up an object from the table, regardless of the objects position. In order
to do this, the robot would first have to correctly identify objects. This section is divided
21
Figure 3.5: Updated custom game pieces.

into the different methods Baxter uses to detect and locate objects in the environment,
along with their analyses.
3.2.1
Blob Detection
To simplify this task, I took advantage of two programs developed by members of the
Activity Analysis group working with the robot Baxter. Both programs subscribe to the
limb camera and work on the output image feed. They are designed to work in conjunction
with one another and are included with the rest of the software developed for the project.
The first program Object_finder.py is used to isolate objects based on their colour.
It features RGB sliders for both the upper and lower ends of the colour spectrum (totalling
in 6 sliders). Adjusting the sliders alters a black and white filter on the image feed from
Baxters limb camera. The purpose of the program is to isolate each object to be tracked.
This is done by adjusting the sliders in such a way that the object appears white, with
the rest of the image feed entirely black (an example of this can be seen in the right image
of Figure 3.6). The program generates a text file containing the saved RGB values for
each object. The Object_tracker.py program uses the generated text file as an input
configuration file.
Blob detection is a mathematical method aimed at detecting regions in a digital image
that differ in properties, such as brightness or colour, compared to areas surrounding those
regions [22]. The Object_tracker.py program uses a blob detector in OpenCV to locate
the objects in the camera feed. By analysing all the pixels of every frame, a cross can
be drawn at the centre point (or centroid) of the similarly coloured groups of pixels (or
blobs), which match the RGB values of the configuration file. This tracking method
was used to differentiate between the different coloured pieces of the custom board game.
Using something like an edge detector would not have been a useful solution as all the
pieces are of similar shape. If using different shaped pieces, such as in the retail version,
using an edge detector would be a viable option. However, the aspects of physically
picking up various shapes of different pieces would add to the complexity of the project,
22
which isnt the projects focus.

Colour Calibration
When working with Baxter in the evenings, the change in lighting would greatly affect
the colour levels coming in from the camera feed. The colours of the game objects I
was working with at the time were red, green and yellow. With the Object_finder.py
program, it was very hard to isolate the original red blocks from the green blocks in low
light conditions. It was possible, but the small amount of pixels detected would result
in a fluctuating target; Baxter would target one block, and as his arm moved over it,
the small amount of light would be blocked out. The robot would then target another
block and move towards it, blocking out the light again. This process of moving back
and forth would repeat to no end. The program for isolating colours uses an HSV (Hue,
Saturation, Value) input from the camera feed. Changing this to RGB and isolating each
channel separately may have fixed this problem. However, due to time constraints I did
not employ this method. Instead I wrapped each of the red painted pieces in a matt blue
insulation tape. This blue was much easier to isolate from the other colours and vice
versa.
Area Threshold
The blob detector uses image moments, a weighted average of the image pixels intensities,
from the resulting mask of a filtered frame (like the right most image of Figure 3.6) to
work out properties such as the area and the centroid of a blob. The target object often
has multiple blobs across its surface. This can be seen on the green blocks of the left
image of Figure 3.6. Here, the green contours drawn around the blobs are showing the
detected areas. A reason for these divisions could be a reflection of light across the surface
of the object, which will split up the blob. To avoid this problem, I set a minimum area
threshold. Using the moments from the image I can draw contours around the blobs which
meet this threshold. Working out this threshold was a trial-and-error process. Having the
threshold too high lead to the object not being detected, whilst having it too low resulted
in one object being detected as multiple objects.
3.2.2
Contour Detection
In a later iteration of the project, a flaw arose in using solely blob detection. When
wanting to detect two pieces of the same colour, the centroid of the blob becomes the
centre of the two objects. In order to differentiate between objects, contours had to be
implemented. A contour is the outline or bounding shape of an object. In OpenCV, it is
possible to draw contours around blobs of pixels. By combining the output mask from the
blob detector with the contour detection features of OpenCV, I was able to draw crosses
23
at the centroids of the contours rather than the blobs. Figure 3.6 shows how this resulted
in being able to accurately differentiate between objects of the same colour.
Figure 3.6: Screenshots of Contour tracking (left) over blob detection output (right).
Detecting the board and working out where all the spaces are, was first attempted
with this contour/blob method. This was when a yellow piece of A3 paper was used for
the board. Using this method, working out the centre point of the paper was trivial for
it was just a very large blob. I then had to calculate the 8 remaining points to make up
the playing grid. Initially, I wanted to only use the output mask from the blob detection.
Testing showed that Baxter converged to the centroid of a blob quicker than the centroid
of a contour based on that blob. This was because the contours constantly adjust based
on the blob, so the centroid is less stable. Unfortunately, OpenCV has no functions to
calculate the dimensions of a blob, which is what was required to establish the grid spaces
on the board. Switching to contours meant I could draw a bounding rectangle around
the contour of the board, which would give me the dimensions I needed. Figure 3.7
shows a screenshot of the tracked board with calculated playing spaces. There was a
Figure 3.7: Blob and contour detection of the board with contours (green), bounding
rectangle (red) and grid spaces (purple).
problem with this implementation, however. The grid centres positions are all based on
the centroid of the contour surrounding the yellow board. If the arm was to move to one
of the outer grid points, part of the board will fall off the edge of the cameras field of
view. When this occurs, the size of the contour gets smaller, which in turn changes where
Baxter believes the centre point to be, because the calculated centroid will shift position.
24
In order to remedy this, I hard coded the dimensions of the A3 paper. After finding a
pose above the centre point of the board (using contours) at runtime, the positions in
Cartesian coordinates of the other points are calculated using these dimensions. This
method changed with the implementation of the new board design. Details of how the
positions above the other grid points are calculated with pixel-to-world transformations,
can be seen in Section 3.2.5.
Analysis
Contour detection proved to not be the most accurate way of calculating the positions
on the original board design, seen in Figure 3.7. Seeing as the board did not feature
an actual grid (only a superimposed grid which was calculated by Baxter) there was
no middle ground for where each player should be placing their pieces. This became a
problem in the 3D implementation of the game. A grid point for the human player was
not exactly the same as where Baxter was calculating it, which resulted in stacks of blocks
not being stable. This was mainly due to the board being a big yellow square where the
human had to guess the locations of the grid lines. For these reasons, the new board
design (Figure 3.4) was produced.
3.2.3
Background Subtraction
The question arose of how the robot would be able to detect and recognise moves made by
the human player. Using the position stored as the centre point above the board object, I
could compare frames from the camera feed of before and after the human player has made
a move. By taking the newer frame and subtracting the old one, the resulting image would
reveal any new changes to the board. This method is known as background subtraction.
The final implementation of this in the project was as follows; after Baxter has calculated
all the board positions from its centre point above the board, it stores a frame from the
camera feed. Then prior to each of Baxters turn in the game, the arm returns to this
centre point, stores a second frame and compares the two. Figure 3.8 shows an example
of how the software processes this information. After using OpenCV to subtract the
background, the new objects are revealed. In order to identify the correct object, I used
the blob detector with contours on the resulting image to calculate the centroid of the new
target object. The distance between this point and each of the nine grid points (shown
in Figure 3.7) are compared, and the shortest distance is used to determine which of the
nine grid spaces the human player has played a piece. This iteration can be seen in the
detectBoardPositions.py program.
Analysis
For the 3D implementation of the game, background subtraction had its limits. The
method worked for when the human player played a piece straight onto the playing grid
25
Figure 3.8: Left to right, top to bottom:

Sequence of background subtraction functionality.
or on top of one of the robots pieces. It did not work if the human player wanted to
play a piece of top of his own piece, because the subtracted image would not show a big
enough difference for the blob detector to recognise another green piece being played. At
an attempt to fix this, the infra-red sensor was reintroduced to scan the heights of each
stack but due to the sensors limitations (described in Section 3.4.2), this solution was not
viable. The most accurate way to detect same-coloured pieces on top of one another was
done via the use of Hough Circle detection and drawing white circles onto one side of the
custom playing pieces (Figure 3.5).
3.2.4
Hough Circle Detection
Hough Transform is a technique used to extract features from an image in computer

vision. The circle Hough Transform is a specialized version of this used to detect circles.
OpenCV has a HoughCircles function which extracts circles from an image and places
them into an iterable object.
The new board was designed to utilize Hough Circle detection. detect_object2.py
was the code I used to experiment ideas using this type of detection. The program uses
OpenCV to draw circles around the points where circles are detected. The centre points
of these circles are stored in an array. The point closest to each corner of the screen is
identified by using Pythagoras theorem to calculate the shortest distance. These four
corners correspond to the four circles of the new board design (see Figure 3.4). The next
step was to calculate the centres of each grid space from the coordinates of the circles
centres. The result of this can be seen in Figure 3.9.
This method works because the generation of the board was completely symmetrical.
The distances between the grid spaces centre points were equal, and the distances between
26
Figure 3.9: Calculating the grid spaces of the new board design, using Hough Circle
detection.
the circles centre points and the nearest grid spaces centre point to that circle were also
equal. The top and bottom rows of the grid were simple to calculate. For the top row,
the change in x and y pixel coordinates was divided by four. Then adding these dx and
dy values to the top-left circle four times and drawing dots each time, results in the top
row of grid centres being drawn. The same calculations were used for the bottom row.
The middle row had some added complexity. The point between the left pair of circles
and the point between the right pair of circles had to first be calculated. Then, using the
same calculations for the other rows, the middle set of grid centres could be drawn.
Background subtraction was not an entirely viable method for detecting moves made
by the human player, the case of a piece being played on top of a same coloured piece, in
particular. The solution to this was to implement the Hough Circle method I used for the
detection of the new board design. Figure 3.5 shows the update of painting equal sized,
white circles onto one side of all the human players game pieces. A new game rule had
to be introduced to utilize this method; when the human player makes the move to play
a piece on top of their own, they must place the new playing piece with the circle side
up. If there is already a circle on the piece being played on, the new piece must be played
with the circle side down. For all other cases such as playing onto an empty grid space or
on top of one of the robots playing pieces, the new piece must be played with the circle
side down.
If the background subtraction method fails to detect any new pieces when Baxter
checks for the human players move, another frame is stored. The Hough Circle method
is applied to this frame to extract the centre points of any circles. The function has
parameters to specify a maximum and minimum diameter for which circles are detected.
This allowed me to isolate the circles on the game pieces from the circles on the corners
of the new board design. Any circles in this new frame have the their distances to
each grid spaces centre point compared. If the distance is less than a certain threshold
27
number of pixels, then that circle is determined to be inside that grid space (similar to
the background subtraction method). The program stores which grid spaces have already
have circles detected inside them. This allows Baxter to differentiate between a circle that
was revealed in a previous turn, a new circle or whether a circle has been occluded by
another piece. The latter of these cases occurs when the human player vertically stacks
three of his pieces in a row.
These Hough Circle methods were implemented into the final iteration of the project,
New3DtictacBaxter.py.
Analysis
Using Hough Circle detection did not only provide the solution to detecting same-colour
stacked pieces, but also provided accurate piece positioning for Baxter via a more reliable
board detection. The method allowed for the board to be oriented at off-angles. As long
as the board was of landscape orientation, then Baxters poses for placing pieces would
be calculated accurately. This did not work with the contour detection, for the bounding
box used to calculate the grid spaces (via the old method) would align itself with the
camera instead of the board.
An initial advantage to using Hough Circles to detect the board, was that the playing
pieces did not obscure the corner circles, allowing the board to be detected mid-play. This
meant that if the board was knocked or shifted out of place for whatever reason, Baxter
would be able to readjust to the new position and correctly update its poses for each grid
space. Having circles painted on the backs of the playing pieces lead me to believe that
this feature would be not be possible due to disturbance of multiple circles in the play
area, affecting detection. In hindsight, I believe the implementation of this feature was
entirely possible due to the parameters on the HoughCircles OpenCV function filtering
the sizes of radii detected, preventing disturbance.
3.2.5
World Coordinate System Transformations
In order to improve the accuracy of the stored poses above each of the grid space centres,
a world coordinate system transformation was implemented.
World Coordinate System (WCS) transformations is a way of converting pixel coordinates to real world coordinates. In the project, the WCS transformation performed was
converting the pixel coordinate differences of the camera feed centre point to each of the
grid space centre points, into world coordinates. These would then be converted with
inverse kinematics into joint angles, allowing Baxter to position its limb above the correct
grid space. This equation [14] shows the conversion from the Golf Example [14]:
B = (Pp Cp ) Cc Dt + Bp + Go
28
where:
B = Baxter coordinates
Pp = target pixel coordinates
Cp = centre pixel coordinates
Bp = Baxter pose
Go = gripper offset
Cc = camera calibration factor
Dt = distance from table
When implementing this into the project, a simplified version was used:
B = (Pp Cp ) Cc Dt + Bp
This was because the code for picking up an object already took into account the gripper
offset. The camera calibration factor had to be calculated by performing a calibration test
on the camera. This test is documented in Section 3.2.6. The method was implemented
into the final iteration, New3DtictacBaxter.py. After the central grid space is aligned
(using the pixel alignment method in Section 3.2.7) with the limbs camera, this equation
is used to calculate the eight surrounding points of the grid. Originally, this was done by
calculating the offsets to the central pose using hard-coded dimensions of the game board.
3.2.6
Camera Calibration
The World Coordinate System transformations in Section 3.2.5, required a constant value
(Cc ) to correctly convert pixel distances into world distances. This constant is referred to
as the camera calibration factor and this section documents the test performed in order
to find it.
Using a method similar to generating the board design, gen_test.py is a script which
generates a symmetrical image as shown in Figure 3.10. All the shapes were drawn using
OpenCV functions. The purple cross in the centre was drawn using the images pixel
dimensions in order to get the exact positioning. The centre points of the circles are also
perfectly represented by smaller purple crosses.
Using the following equation based on the WCS transformation equation from Section 3.2.5:
( DDCt )
Cc =
Pd
where:
29
Figure 3.10: Camera calibration test image.

Dc = world distance between the image centre and circle centre
Dt = distance from table
Pd = pixel distance between the image centre and circle centre
The pixel_calib.py program calculates the camera calibration factor over the
course of runtime. Using the Hough Circle technique from Section 3.2.4, the distances
between all the circles and the centre point is calculated. The cross at the centre of the
printed image must be aligned with the cross drawn on the image feed by the program,
which represents the centre pixel point of the camera. The Dc value of the equation is
the distance measured with a ruler between the image centre point and the circles of the
printed image. The results of this after 1000 iterations can be seen in Figure 3.11. The
camera calibration factor for each circle is calculated and then a final c average is taken
from these. It was this value that was used for any WCS transformations in the project.
As an experiment, I wanted to see if the camera calibration factor stayed the same
for all heights. This would mean that the lens in the camera was curved to allow WCS
transformations to work at any height with the same constant. Dropping the height by
10mm yielded a result that differed by 0.0001, as shown in Figure 3.12. This meant that
the lens did not compensate for different heights and that the calibration factor was a
function of the distance to the table.
3.2.7
Pixel Alignment
There was an issue with picking up objects directly from a pose using World Coordinate
System transformations. In the 3D implementation of the game, a stack of playing pieces
30
Figure 3.11: Camera calibration test results.

can obstruct the line of motion from above the board, to a particular grid space. To solve
this issue, a crane-like motion was adopted to ensure that the limb would only move down
to the plane of the board when it was directly above the desired location. The advantage
of this came in the simplicity of the limb only needing to move through the X and Y axis,
keeping Z axis constant. Once the limb reached a position above the object, the arm could
then descend along the Z axis, using the functionality of the pickupDropwithRange.py
program, and pick up the object. This technique was influenced by the Golf Example
[14].
pickupColour.py is the program that was developed for the project demo. It correctly aligned the limb with the object, and then descended to pick it up once it was
accurately positioned above it. It even followed the object around if it was moved before
the descent. In order to correctly align the arm with the object, the centre pixel of the
image feed had to be aligned with the centre pixel of the object.
The Golf Example [14] had a relatively simple way of doing this. By calculating the X
and Y offsets of the object from the cameras centre, the overall position of the arm could
be updated (using inverse kinematics, Section 3.4.3) along the X and Y axis, keeping the
z position the same. This was done by the following equation for each axis:
O = Dp Cc Do
where:
O = offset
31
Figure 3.12: Camera calibration test results at a lower height.

Dp = displacement of target pixel from the images centre
Do = distance to the top of the object
Notice how similar this equation is to the equation used for WCS transformations in
Section 3.2.5.
An error value is also calculated and the whole process is repeated until the error
becomes small enough to meet a set threshold. At this point, the arms pose will be
updated, but this time the X and Y values were kept constant and the Z value was
changed to the distance to the top of the object.
I chose not to make use of the IR sensor for the descent of the arm for the purposes of
the demo. The reason for was that iterating through the heights until meeting a threshold
was very slow, especially in combination with the iterations for finding the correct X
and Y positions. The Golf Example comes with a setup script, golf_setup.py, which
calculates the distance from the limbs start position to the table, using the IR sensor.
It then outputs this to a file which I then use for my program. It proved much quicker
to calculate all the heights prior to the robots runtime, the only height that had to be
hard-coded was the height of the object, any further heights being a multiple of this.
This pixel alignment method was used to track the centre pixel of the board. Section 3.2.4 describes how this centre point is calculated. Just like when targeting an object,
an error value is updated until a predefined threshold is met, resulting in a pose located
32
directly above the centre of the board. This is when the poses above the other grid spaces
are calculated.
Offsets
After Baxter targets an object, it pick it up by updating the Z value of the arms endpoint
until the distance to the object has been reached. As you can see in Figure 2.9, the camera
is a fair distance away from the gripping point of the grippers. In order to ensure the
grippers enclose around the object, and offset has to be added to the pose when picking
it up.
The implementation of this offset featured a trial-and-error approach in finding the
best values. First, I physically measured the height and width offset from the centre of
the camera lens to the middle of the grippers. These values were added (or subtracted)
to any pose that went into the inverse kinematics function when Baxter lowered its arm
to pick up an object. By stopping the arm right before the grippers would close, I could
measure how far off the grippers were from the centre of the object, and increment or
decrement the offsets accordingly. The height at which the gripper picked up the object
was another offset that had to be set. During the robots setup, it works out the distance
from its arm to the table using the IR sensor. The offset to be set is the distance between
the centre of the IR sensor and the end point of the grippers. The current calculation to
pick up an object of predefined height is a change in the Z axis. The table distance, minus
the objects height, plus the offset gives this value. It was important to ensure that the
gripper did not close around the object at too low a height. This could lead in collisions
with the table and damage to Baxter. If the grippers closed around too near the top of
the object, there is a chance that on occasion the grippers could fail to pick up the object
entirely.
These offsets are also applied when Baxter places pieces onto the game board.
Stacking
As an add-on to the project demo, I implemented a stacking feature and a class-based
structure for the current code. The new structure is described in more detail in Section 3.5.2 and is viewable in the program pickupStack.py. The stacking feature remembers the pose for the location above the first object, locates and picks up the second
object, placing it on the first. The height to release the second object from is calculated
using the vertical offsets. By doubling the stored height of the object and using that
value in the pose for placing down the object, the new height is equal to the height of two
blocks. Due to the new class based structure, this implementation was trivial. It can be
easily modified to stack a higher number of objects (tripling the stored object height to
stack of three blocks, for example).
This add-on was used as the prime functionality of playing the board game. Baxters
33
turn to play a piece into a space on the board, uses the stacking feature to stack the
object onto the space. The level at which the object was stacked on was dependent on
how many objects were currently in that space (if there were zero, then the object was
placed directly onto the board).
3.3
3.3.1
Game Logic
2D
Baxter needed to know how to play the game. According to the objectives, this would
first have to be achieved in a software environment before applying Baxters physical
capabilities.
The first step was to essentially create a simple tic-tac-toe game in Python. The
tictac.py program is a game of the 2D version of Noughts & Crosses, playable in
the Terminal window. Implementing the 2D version of the game is a prerequisite step
before the full 3D version. I set up the code so that it could be easily adapted to higher
dimensions.
The program features play between a human and the computer. On the turn of the
computer, the AI has been set up to follow this sequence of possible moves in order:
1. A winning move
2. A move to stop the opponent winning
3. A free random corner space, if there is one
4. The centre space, if its free
5. A free random side space
The method to check whether a winning move can be made or whether the player
needs to be blocked to prevent them from winning, is done by making a copy of the board
(function of the board class in Section 3.5.4) for every free space and filling each space
with either playing piece. Then each copy of the board is checked against the possible
winning lines (checkWinner() function, also from the board class) to see if that potential
move will result in a win for either side.
This is not an unbeatable AI, it is possible to win by taking two opposite corner spaces.
Now that the AI will try to block the player, a third corner can be taken on the players
next turn, putting them in a position of winning the game. This strategy is stoppable by
taking an opposing side space, as shown in Figure 3.13 and mentioned in Martin Gardners
book [9]. For the sake of testing, I have not programmed in this defence. This is so that
when Baxter comes to playing the game, I now have strategies to force a Win, Loss and
Tie.
34
Figure 3.13: The second player taking the opposing side space of the taken corner.
3.3.2
3D
The 3D implementation of the game took the games current data model, which was a 2D
matrix represented by an array of arrays, and stretched it into another dimension. The
3D matrix of the new implementation featured three of the 2D matrices from the previous
iteration, one for each level of the cube.
Along with a new set of winning lines, which Section 3.5.4 describes the method of
generation, the implementation required a new set of conditions for the AI to play the
game. I kept the code for making a move into the grid on a 2D interface the same; the
(row, col) format of where the move was made now places a piece in the lowest possible
level of that position. This meant the rules followed in Section 3.3.1 did not need to be
altered much. The new sequence of possible moves the AI follows are:
1. A winning move (unchanged)
2. A move to stop the opponent winning (unchanged)
3. The centre space on the middle level, if there is a robot piece below it
4. The centre space on the bottom level, if its free (unchanged)
5. A free random corner space on the bottom level, if there is one
6. The centre space on the middle level, if its free
7. A free random corner space, if there is one (unchanged)
8. A free random side space (unchanged)
This is by no means a perfect AI, but still presents a decent enough challenge whilst
allowing me to force certain board states for testing purposes.
35
Generating Winning Lines

The 2D version of the game had all the possible winning lines hard coded into the script.
This was plausible as there were only 8 possible lines. The 3D version, however, has a
total of 49 winning lines, detailed in Section 2.2.2. With the help of Owen Eltons blog
[8], I devised a way to generate these lines at runtime. The BoardClass3D.py class does
this in its constructor. The generator function generateSolutions(), iterates through
every group of three distinct spaces in a 3x3x3 cube. Certain conditions are then checked
to determine which of these groups form a winning line. Each space in the cube has an
address of three values ranging from 0 to 2. The address is read:
space = (level, row, column)
For example, a bottom corner would be (0,2,2) and the central piece (1,1,1).
Every group of three addresses has two conditions checked on each element of all the
addresses, totalling in six checks (two conditions on all the levels, two conditions on
all the rows, etc.). The first condition is whether an element of the address is the same
across all the addresses. If one element across all the addresses is the same, then we have
a winning line in the 2D game [8]. If two elements across all the three addresses are equal,
then we have a winning line in the 3D game. This works for all the non-diagonal lines in
the cube.
The second condition is whether an element of all the addresses is distinct but also
in an ascending or descending order (all the lines are sorted before these conditions are
checked). Golomb and Hales study the analogue of tic-tac-toe played on a k-dimensional
hypercube of side n [10]. The idea for the second condition was inspired by the paper. Now
if all the elements across the three addresses were either in an ascending or descending
order, then the group would represent one of the four diagonal lines in the cube, moving
through the central space (e.g. [(0,0,0), (1,1,1), (2,2,2)] and [(0,2,2) ,(1,1,1), (2,0,0)]).
Finding the rest of the winning lines of the cube was done using a combination of the
two conditions on each group of three addresses. If two elements across the addresses were
in either an ascending or descending order (as per condition two) and if the third element
across the addresses are equal (condition one), then this group represents a winning line.
3.4
Robot Control
The first successes of controlling Baxter was via the SDK example programs and the
Hello Baxter! online tutorial. The first program I developed was based on this tutorial
and was an important stepping stone in understanding how Baxter sees the environment.
3.4.1
36
Basic Movement
The first objective in Section 1.2 required Baxter to pick up and place a playing piece
into an arbitrary space. The first task I set myself in order to complete this objective was
to program Baxter to pick up and place objects at pre-set locations. Though seemingly
simple and not autonomous, the task became an essential learning experience of how
Baxter follows commands.
The position of the arm is determined by an array of angles, one for each of the
servo motors in that arm. The function joint_angles() returns the array for the
current orientation of the motors. The pickupDrop.py program prompts the user to
move the limb into several positions, storing the joint angles of each position. It then
iterates between them picking up an object from one position and dropping at another.
At first, the program only stored two positions; one for the target object and one for
the destination. The problem with this was that there are multiple solutions for the
motors to move from one state of joint angles to another. This risks the limb getting
into unwanted collisions in the environment, such as with the table. The solution was to
include intermediate positions between the origin and destination to ensure that the path
of the limb was free of any obstacles. The final number of stored positions in this was
four.
3.4.2
Sensors
Baxter has several methods of sensing the surrounding environment. The purpose of the
next iteration was to utilize the infra-red sensor. This would calculate the distance the
limb is from the table in order to pick up an object of predefined height.
The pickupDropwithRange.py program lowers the limb until a threshold height is
met. At this point, the grippers close around the object and the arm picks it up. This
movement is shown in Figure 3.14.
The output of the infra-red sensor is accessed via a ROS Service. In the Python code,
the method I used was to create a Subscriber object to the service /robot/range/right_hand_range/state and a callback function to handle the incoming data from it. Once
the program correctly output readings from the IR sensor, the next step was to implement
a descending crane-like motion to the limb.
Infra-Red Limitations
For reasons described in Section 3.2.3, the method was not sufficient to detect the human
player playing a piece on top of his own. Using the IR sensor on Baxters arm was the
first attempt at solving this problem.
If the background detector did not reveal a new piece being played, Baxter would
move his arm to the positions (set in Section 3.2.5) over where each of the opponents
37
Figure 3.14: Left to right, top to bottom:

Example functionality of the pickupDropwithRange.py program.
pieces were located. Distance readings from the IR sensor would be checked against the
pre-programmed height of the playing pieces in order to determine if a stack of one, two
or three pieces were there. In theory, this solution seemed to be the most efficient and
showed good use of Baxters capabilities. However, in practice the readings from the
sensor were not at all accurate.
I set up a test iteration of the program (now titled broken3DtictacBaxterRange.py)
to take multiple readings from every grid space and repeat this for a number of times.
When the entire grid was empty, the distances for each space were not all equal, but
between repetitions of the test the distance values for each space did not change, regardless
of whether it was correct or not. I then put blocks in each of the spaces, so that the entire
playing grid was one level higher. Upon restarting the test, I noticed that some readings
came back correct but others returned as if there was nothing populating the space. Again,
the readings between repetitions remained consistent.
I considered that the poses for Baxters arm above each position were not accurate
enough, so I modified the test program to take several readings from different locations
within each grid space. The results for this test were much more accurate, though there
were still a couple of spaces which gave readings as if no object was detected there. I
decided to replace the playing pieces on the board so that they were all of one colour. At
this point in the development the available playing pieces were blocks either wrapped in
blue insulation tape or painted green. Interestingly, using only the blue blocks provided
much more accurate results over the green blocks. I put this down to the material being
more absorbing and having a more consistent surface than the naked wood.
38
Lastly, I ran the test again using only the blue blocks and having stacks of them up
to three levels high. The results got much less accurate with this test. The stacks of
one level in height were detected as being two or three levels high, when next to a taller
stack. I deemed this solution not viable, especially due to the possibility that IR sensor
was faulty.
Another solution was to take distance readings as the arm moved the IR sensor across
each row of the grid, constructing a curve of heights. This seemed like a viable option
at the time but again, due to the possibility of the IR sensor being faulty, I thought
it more wise to spend my time finding an alternative solution; using the Hough Circle
method to detect drawn on circles on the backs of the human players playing pieces (see
Section 3.2.4).
3.4.3
Inverse Kinematics
Baxter updates the pose of its limbs by an array of angles in radians. This is very hard to
work with when wanting to perform simple manoeuvres that update the position of the
limbs endpoint. It is possible to output the limbs endpoint in Cartesian coordinates by
subscribing to another service, /robot/limb/right/endpoint_state. There is no topic
or service, however, that enables updating the endpoint by manipulating its coordinates.
Inverse kinematics refers to the use of the kinematics equations of a robot to determine
the joint parameters that provide a desired position of the end-effector [13]. In other
words, it transforms the Cartesian coordinates of the endpoint into joint angles for the
limb. The Golf Example has a function (baxter_ik_move()), which I included in
the code to enable manipulation of the limbs endpoint via Cartesian coordinates. The
pickupDropwithRange.py program contained an initial test done by taking the output
of the endpoint service, decrementing the Z value of the coordinates and then updating
the limbs joint angles using the inverse kinematics function. This decrementing of the Z
value continues inside a loop until the output of the infra-red sensor meets a threshold
equal to the height of the object being picked up. This is how the motion shown in
Figure 3.14 occurs.
Inverse kinematics is used for all movements made by Baxter after its initial start
poses are taken.
Joint Limitations
All the movements in 3D space made with inverse kinematics were done with the hand
(out-most joint) locked facing downwards. The function baxter_ik_move() takes six
parameters. The first three are the Cartesian coordinates (XYZ) of the target point in
space. The latter three are used to determine the orientation of Baxters gripper at this
point. Seeing as I kept this orientation constant throughout the project, it meant that
there were certain points Baxter could not reach with inverse kinematics, even though they
39
were within his range. The problem stems from the concept that a particular configuration
of joint angles will give only one point in 3D space. However, a point in 3D space can be
converted into many joint angles for Baxters arm. If the orientation of the arm is fixed,
then the number of possible joint configurations is severely reduced, sometimes to zero.
The work around to this problem, was to incrementally increase the angle of the pitch
of the arm (the second of the latter three parameters which make up the orientation of
a pose) until a joint configuration is found for the target point. This worked relatively
well, objects which used to be out of Baxters reach were now being picked up. When
they were placed, there was a bit of turbulence for the change in orientation meant that
the object was being dropped at an angle which may cause the piece to not be positioned
correctly. Due to time constraints I was unable to fully test this solution.
3.4.4
Head Display
Baxters head display has a related topic /robot/xdisplay of which images can be
published to. To give the user a sense of where in the code Baxter is, I set different faces
to appear on Baxters head display. There is a default smiling face which will always
display when the robot is between functions. This almost works as a confirmation that
some detection feature has been completed successfully. Whenever Baxter is performing
any form of vision, the camera feed is published live (frame by frame) to the head display
to give the human player a better view of how Baxter is perceiving the board. This also
greatly assisted testing. During the human players turn, a Your Turn image is displayed
with a countdown, to prompt the player to make their move. Finally, either a happy or
angry face is displayed at the end of the game depending on of Baxter won or not.
3.5
Playing Noughts & Crosses
This section concludes the development done on the project. It details how the individual
components work together from a code perspective. It also provides an explanation of
how to run programs I developed.
3.5.1
How to run programs
As mentioned in Section 2.5, Baxter requires a specific software environment to be controlled from. Once ROS environment and Baxter SDK has been installed on a linux
machine, the project folder needs to be added into the src folder of the ROS workspace
(e.g. ros_ws/src). Running catkin_make inside the workspace folder will make sure
dependencies for all projects are installed. After this, the terminal must be connected to
Baxter by running the shell script (./baxter.sh, as mentioned in the Baxter SDK tutorial). Each program can then be run as: rosrun <project folder name> <program
40
name>.py. All code developed for the project can be accessed from the repository, which
can be found in Appendix D.
3.5.2
Structure
The class-based structure facilitated the management of the growing amount code across
iterations. It allowed me to store global variables as class properties, which were useful
when needing to adjust values such as offsets. The final iteration of the program can be
seen in new3DtictacBaxter.py. The main() function is kept outside of the class and
is used to call the class and its various higher level functions. The game loop created in
tictac3d.py has been used in this function, and the rest of the game logic has been
put into the class function take_turn(). Figure 3.15 shows an overview of the main()
function.
I created separate classes to handle and store the state of the board. BoardClass3D.py
features two classes; one for the board (Section 3.5.4) and one for each space (Section 3.5.5)
in the board. Most of the functions are getters and setters used in the robot class. These
are good coding practice to utilize as it prevents any accidental changes to class properties.
3.5.3
Robot Class
The robot class contains all the functions used by Baxter to interact with the game
environment. The class properties contain a number of variables; object dimensions, start
positions, camera offsets and speeds. In this constructor, cameras are configured to pixel
dimensions and subscribed to. A function to move the unused arm out of the way is called
and the actuators of the robot are also enabled.
The class functions are separated into sections; vision, game logic and movement.
Vision
Before the start of the game, the locate_board() function is called to calculate and
store positions of the board. This is done with a combination of Hough Circle detection
(Section 3.2.4) and pixel alignment (Section 3.2.7) to centre the arm over the board.
World coordinate system transformations (Section 3.2.5) are then performed to calculate
the poses for each grid space.
The detect_and_draw() function is the main computer vision aspect of the program.
The function serves as a callback to the subscribed camera. The message passed to the
callback is an image message which is converted to a Cv2 (OpenCV) image using the
CvBridge (an object from the ROS SDK). Functions for detecting the robots pieces and
the board extract features from this image using OpenCV, the workings of which are
described in Section 3.2.
41
The function which handles the detection of moves made by the human player, detect_pieces(),
contains two inner functions that are based on detect_and_draw(). detect_pieces()
features a while loop that notifies the human player of their turn by countdown. At the
end of this countdown the first inner function is called to check if any changes to the
board have been detected. check_circles() detects if a piece with a white circle has
been played by the human player, using Hough Circle detection. If no move is detected,
the subtract_background() function checks a single frame against a frame taken from
an earlier state in the game to detect if any other kind of move has been made, and where
(Using the background subtraction method in Section 3.2.3).
Game Logic
The game logic function take_turn() is called from the game loop when Baxter is to
take its turn. Section 3.3 describes how the robot decides which move to make based on
the current state of the board. This function mainly uses the class functions of the Board
class (Section 3.5.4), such as checkWinner() which determines if a winning move can be
made. The same function is used to check for a winner between every turn, as well as a
function used for checking for ties.
Movement
After updating the board model with the desired move, the place_piece() function is
called to instruct Baxter to show this move in the physical playing area. This function
comprises other functions, which break down the movements as follows:
1. start_position() - returns the arm to the position between the board and the
pieces.
2. toggleDisplayFeed() - sets Baxters head display to output feed from the arm
camera.
3. find_object() - searches for playing pieces using contour detection (Section 3.2.2)
and pixel alignment.
4. pickup_object() - lowers and raises the arm to and from the object with inverse
kinematics, closing the grippers around the object at the low point.
5. drop_on() - Places the picked up piece in a position on the board. The parameters
of this function take a pose stored for the grid space and an integer to determine at
what height the piece should be placed.
As mentioned previously, all movement is made by the robot with inverse kinematics
inside the baxter_ik_move() function. More details of this function can be found in
Section 3.4.3.
3.5.4
42
Board Class
The functions of this class are focussed on the manipulation and observance of the board
state during the game. The constructor of the class generates an empty cube of 3x3x3
spaces and a set of possible winning lines which can be referred to when checking a win.
checkWinner()
The function takes the current state of the board and a playing piece reference as a
parameter. It then returns true if one of the generated winning lines exists in the state
of the board, made up of that playing piece.
isFull()
This function iterates through all the spaces of the top level of the cube and returns true
if none of the spaces are free of content. This function is used to check for ties.
getCopy()
This function uses the copy library from Python to create a deep copy of the board
state. This is so that the copy can be manipulated free from affecting the original. The
use of this function can be seen in the Game Logic section of Section 3.5.3.
3.5.5
Space Class
All the functions of this class are getters and setters and store information about a
particular space on the board. The properties of this class are the contents of the space,
the pose for Baxters arm to position above the space and the pixel coordinates of the
space for use in the background subtraction method (Section 3.2.3). The class also stores
whether a circle has been detected in that space, which is used by the Hough Circle
method (Section 3.2.4).
Figure 3.15: Overview of Baxters runtime.
43
Chapter 4
Evaluation
This chapter evaluates the performance of the robot. The most important aspects of
functionality that required evaluation were the computer vision features. Such features
will be covered using both qualitative and quantitative measures. I deemed aspects of the
robots movements, such as the inverse kinematics solver and other Baxter SDK functions,
unnecessary to evaluate. This was due to the fact that results from such evaluations would
only give indication to the robots performance as a unit, rather than of how well the code
I have written to run on it executes.
Light Conditions
All tests were performed with varying light conditions; minimal, artificial and natural
light.
Minimal light featured all the lights in Baxters workspace turned off and the blinds
on the nearby windows closed.
Artificial light had the lights turned on, and with the blinds kept closed.
With natural light, all the lights were turned off so that the only light coming in
was through the open blinds.
4.1
Board Detection
The final product detected the board via the use of Hough Circle detection (Section 3.2.4).
The circles on the board were evenly spaced so that the detector could calculate the mid
points of the grid spaces.
4.1.1
Pixel Point Accuracy of Hough Circle Detection
This first test was to determine how accurate the calculated centre point of the board
space was. The program hough_eval.py was created to take the calculated centre point
average of the board over the course of its runtime. It also provided a function to print
the pixel coordinates of a mouse click on the image feed. By measuring a central point on
the physical board with a ruler and placing a dot visible enough to click on in the image
feed, I was able to print the exact pixel coordinates boards centre point. This value was
used as a ground truth.
44
CHAPTER 4. EVALUATION
Lighting
Minimal
Artificial
Natural
45
Average Pixel (x,y) Ground Truth (x,y) Difference (x,y)

(467,293)
(470,244)
(3,-49)
(467,293)
(470,244)
(3,-49)
(467,293)
(470,244)
(3,-49)
Table 4.1: Hough Circle pixel point accuracy test results.
Results
The results of the test (Table 4.1) shows a consistent average pixel across all light conditions. These averages were recorded over 1000 frames. Hough Circle detector uses a
grayscale image as an input so the consistency in the calculated averages was expected.
As long as the circles contrast to the background they are drawn on, brightness has little
effect on their detection. The ground truth value was recorded once and used for every
light condition (where a new average was calculated each time). In this case, we can
assume that the ground truth was slightly off centre and due to human error on my part.
A repeat of the test with a better fixed ground truth returned a difference of (2,-1)
across all lighting conditions. Such a small pixel difference does not affect the overall
performance of Baxter.
4.1.2
Pixel Point Convergence
The previous test featured the detection of the board from a stationary arm. This test was
done to evaluate the speed of which Baxter could centre his arm above the boards central
point from its start position, under varying light conditions. The test was performed
by running the final version of the project deliverable and using a stop watch to time
from when the Locating Board... image was displayed until the default.png face was
displayed, which is an indication to the user that Baxter has successfully located the
board.
Results
Appendix E.1 shows the recorded times of 10 runs of the program under three separate
lighting conditions. Table 4.2 shows the mean and standard deviation of the recorded
times. All the times have been plotted onto bar charts for each lighting condition and
feature a horizontal black line, representing the mean time taken in that condition. This
data can be seen together in Figure 4.1 or individually in Appendix E. Several conclusions
can be drawn from the results of this experiment.
According to the mean times (represented by horizontal lines), Baxter located the
board fastest in minimal lighting conditions, however the standard deviation of the recorded
times in this test was much greater than in other conditions. The high deviation infers
that the consistency of Baxters performance suffers in low lighting conditions. It is pos-
46
Lighting
Minimal
Artificial
Natural
Mean Standard Deviation

13.201
34.4
24.616
7.2
15.466
3.4
Table 4.2: Mean and standard deviation of the recorded times.
100
Time (Seconds)
80
60
40
20
Minimal
Artificial
Natural
Figure 4.1: Experiments arranged in ascending order by time taken. Means drawn as
horizontal lines in corresponding colour.
47
sible to see several recorded times straying far from the mean time line. It is unclear
whether the two longest time results are either anomalies or signs of Baxters possible
unreliability in these conditions.
The slowest mean time was recorded in artificial light. From Figure 4.1 and Table 4.2,
it is possible to see that this average is a reliable representation of Baxters performance.
The relatively low standard deviation shows consistency between the times.
Natural lighting conditions showed the most consistent out of all the others. The chart
shows the even spread around the mean time. Having performed this experiment, it was
concluded that natural light was the most suitable condition for Baxter when locating
the game board. This is due to consistency in the recorded times and the low mean time,
which was not far off from the fastest average. However, Baxter still performed adequately
across all light conditions, indicating that there is no specific lighting requirement for the
robot to run in.
4.2
Object Detection
Baxter is able to differentiate between the coloured pieces by using a blob detector. The
output of this detection has contours extracted from it, which is then used to locate
individual objects of the desired colour (Section 3.2.2).
4.2.1
Pixel Point Accuracy of Contours
A test similar to the one carried out in Section 4.1.1 was performed to evaluate the accuracy of the contour detector. In the main program, the robot aligns itself with the centre
point of a bounding box drawn around the extracted contour of an object. The program
eval_cont.py takes an average of the calculated centre points of each object from a stationary point, with the initial setup show in Figure 4.2. Like the eval_hough.py test,
a ground truth value was set for each pieces centre point, based on the returned pixel
coordinates of mouse clicks. The aim was to find out which lighting conditions proved
most suitable for detecting playing pieces.
Results
The results of the test (Table 4.3) under natural lighting conditions were both accurate
and reliable. The offsets from the ground truth values were minimal and the reliability of
the detection can be seen in Figure 4.3 with a stable set of contours.
The results for the test under minimal and artificial lighting conditions were not reliable enough to include in this evaluation. The number of detected contours fluctuated
between frames, resulting in disruption amongst the averages of the centre points, for the
program was unable to determine which centre points were tied to each block. From this
test we are unable to make comparisons on the accuracy of the contour detector under
48
Figure 4.2: Contour accuracy test setup. Blocks 1 to 10 (left to right, top to bottom).
Block Number Average Pixel (x,y) Ground Truth (x,y) Difference (x,y)
1
(459,205)
(458,203)
(1,2)
2
(598,190)
(602,188)
(-4,2)
3
(461,314)
(463,315)
(-2,-1)
4
(594,301)
(594,302)
(0,-1)
5
(466,426)
(467,426)
(-1,0)
6
(596,422)
(599,424)
(-3,-2)
7
(468,552)
(471,553)
(-3,-1)
8
(602,534)
(606,539)
(-4,-5)
Table 4.3: Contour accuracy test results after 1000 frames under natural lighting conditions.
varying light conditions, however by comparing the images of the detected contours, it
still gives indication of its reliability. The artificial light condition image in Figure 4.4
reveals that not all the blocks were detected, and those that were had imperfect contours
drawn around them. Though the number of detected contours was relatively stable, it
was less than the number required by the test so the centre points were not recorded. The
number of detected contours under minimal light had the greatest instability. Figure 4.5
shows a time lapse of the detected blocks and how the number of contours changed. As
with the artificial light results, the centre points of the contours in this test were also not
recorded to to this instability.
In conclusion to this test, the order at which the contour detector had the greatest
stability under varying light conditions is as follows:
1. Natural Light
2. Artificial Light
3. Minimal Light
49
Figure 4.3: Detected contours under natural light.
Figure 4.4: Detected contours under artificial light.

Though natural light is the optimal condition, Baxter will still function correctly in
all other tested conditions.
4.3
Game Play
The final iteration of the 3D Noughts & Crosses program was run on Baxter 10 times under
each lighting condition. It was at this point when the timed results of the experiment in
Section 4.1.2 were recorded. The purpose of evaluating full length run-throughs was to
test how well the individual components synchronised together and to fully test the game
AI.
4.3.1
Vision Classifications
One observation made during the run-throughs was how well the robot classified pieces
played by its opponent. The aspects of the robot evaluated in Section 4.2 were present
50
Figure 4.5: Detected contours under minimal light.

during this but not observed. Of these tests Baxter correctly classified all objects in 100%
of run-throughs across all three varying light conditions.
4.3.2
Game Logic
The test from Section 4.3.1 involved a lot of observance of the robots game logic. Due
to the rule based system, the outcome of each game was predictable. The more I played
the game, the quicker I was able to beat the robot. The number of turns to win a
game converged to five, leaving nine pieces on the board. The results of this evaluation
corresponds to the prior researched carried out and that it does not work as a game. This
is assuming that a game must present some level challenge and a chance for either side to
win. However in Section 2.2.3, it mentions that the game is already broken as the first
player can always force a win.
The robot is able to play the game to a degree which is challenging enough for new
players. Baxter beat all of the three other people I had test the system in their first
game against it. In some cases, Baxter managed to win further games against them. The
human players eventually worked out a strategy to win (not necessarily the quickest).
It is possible to change the code slightly to allow the robot to play first, decreasing the
chances of the human players to win.
4.4
Code Evaluation
The program features a class-based structure (Section 3.5.2) which provides an easy to
understand set of functions and properties. The functions after the constructor are split
into sections; utility/setup (such as subscribers and boolean switches), game logic, track-
51
ing and movement. The data models for the board and individual spaces are stored in
separate classes. The code was written in a style to allow modification of the game.
Chapter 5
Conclusion
The final system meets all but one of the objectives from Section 1.2. This outstanding
objective was to implement a level of intelligence into the robot so that it develops a
strategy for playing the game over time. The goal proved to be too ambitious given the
scope of the project. Therefore, I spent more time ensuring the other objectives were met
and the resulting functionality worked robustly.
Baxter successfully locates and picks up playing pieces and places them into specific
areas of the game board with the ability of stacking them up to three levels.
Baxter is able to recognise moves made by the human player and store their corresponding locations.
Baxter can respond to the human players moves strategically and by following the
game rules. The game has also been implemented into a software-only environment.
5.0.1
Time Plan
Figure 5.1 shows a revised version of the Gantt chart from Section 1.4 with the actual
times it took to complete each objective. The overall time of the project was extended
by three weeks due to a change in the deadline. This change was at a request from me
to the university regarding family related problems at the time, incurring a set back in
progress. The obvious change in the time plan is that the first objective took more time
than expected. This is because the objective had many prerequisite steps, such as locating
the board and calculating the grid positions. The second objective was easy to implement
once a certain number of these steps were completed. Objective four (implementing game
logic in a software environment) took a bit longer than planned because it was worked
on at separate points; the first few days of the segment a 2D version of the game was
developed, and the last few days the code was upgraded to the 3D version. A similar
occurrence happened with objective three; the background subtraction was implemented
in the first five days of the segment, and development on the Hough Circle method started
when the 3D version of the game logic was implemented. The time it took to complete
this objective still managed to stay true to the estimate.
5.1
Extensions
This section discuses the ways in which the project can be extended for increased functionality and different uses. The number of possible extensions is fairly limited by Baxter
52
CHAPTER 5. CONCLUSION
53
Figure 5.1: Actual time plan for each objective.

not being easily modifiable as a unit.
5.1.1
Depth Sensing
One way the robot can have its hardware modified is by the inclusion of a Kinect sensor
(by Microsoft, originally built for the Xbox 360). The Kinect features an RGB camera
and a depth sensor [20], allowing it to perceive environments in 3D. Such a devise can be
used to upgrade the tracking aspects of the project.
Attaching a Kinect to the front of the robot will negate the need for pixel alignment (Section 3.2.7) and World Coordinate System transformations (Section 3.2.5), for
the world coordinates of the block centre points can be calculated from two different
perspectives also giving a much greater accuracy.
By attaching the sensor to the cuff of the robot (facing down towards the board), the
heights of objects played on the board can be detected with the depth sensor. Currently
the background subtraction (Section 3.2.3) and Hough Circles (Section 3.2.4) methods
recognise moves made by the player. Having the Kinect sensor implemented would negate
the requirement to have circles drawn on one side of each of the human players pieces.
5.1.2
Error Handling
For the purposes of time, I did not put much focus into handling any errors that the user
may cause in running the demo.
If the human player played a piece in between two spaces, Baxter will register it as
being played in the first space which has the smallest distance from the pieces centroid.
If Baxter were to play a piece in the empty of those two spaces, it would collide with the
piece played by the human player for it covers an unexpected area. A fix for this would
be to calculate the offset of the humans piece from the centre of space it is registered as
being played in. Baxter could then shift all the playing positions of the other grid spaces
by this offset, so as not to collide with any pieces. An alternative fix would be to have
Baxter pick up the humans piece and place it correctly in the registered space. On the
assumption that the human player will always place their pieces in the designated squares
54
of the board, I did not implement either of these.
5.1.3
Game Variants
There are many variants of Noughts & Crosses. Qubic features a 4x4x4 board and
presents a much bigger challenge to both the player and someone developing an AI to
play it (Though also solvable [12], like the 3x3x3 variant). The code in this project has
been developed to allow modifications to work with larger game boards. Playing the
regular 2D version of the game is also very possible, as it was a prior iteration before the
final. Other variants such as Connect Four are worth a mention but are not as easily
adapted to as official variants of Noughts & Crosses.
5.1.4
Artificial Intelligence
Due to the nature of the board game, there came a small amount of challenge when playing
it over time. The focus of its behaviour was based on the 49 generated winning lines and
strategic value of particular spaces (such as the central space). The system can be made
more intelligent by a number of methods. I learnt from Newell and Simons Tic-Tac-Toe
program[7], that Baxter could be a more formidable opponent if it was aware of the fork
moves, which the entire strategy of Noughts & Crosses is based on. A possible way of
doing this would be to generate an array of all the possible fork moves, like the winning
lines.
Adaptive strategy based on game state could be implemented with Minimax. Section 2.3.2 discusses Minimax and how the algorithm works out a best possible move at
each state. Implementing this into the robots game logic would provide even more challenging game play.
It is possible to program an unbeatable strategy for the robot if it were to play first.
This would not provide a fair game to the human player. Implementing an AI level
(such as easy, medium and hard, where hard utilizes the unbeatable) could simulate
more realistic gameplay, with a further add-on randomizing the starting player.
5.1.5
Audio & Voice Commands
The current method of Baxter recognising a move made by a player, is to continually

loop through detection methods between timers until a change is detected. As this is a
turn-based game, this real-time aspect may hinder a users strategy. A simple solution to
this is to implement a simple voice command, where the human player can say a phrase
(such as turn!) to indicate to the robot that it should start its detection methods to find
where the human has made their move.
The acknowledgement method of a move being successfully detected is currently to
publish the resulting image to Baxters head display. Using a text-to-speech synthesiser,
55
this method can be replaced with the robot speaking to indicate to the human player
that their turn has been detected.
5.1.6
Head Display Nod
Another way of acknowledging a move made by the human player is to manipulate the
position of the head display to nod. Though equally trivial as the speaking method of
the previous section, it provides further exploration of Baxters available functions.
5.1.7
Other Games
Aside from the Noughts & Crosses variants, there are other games this project can be
extended to work for. The functions of the robot Class in the code (described in Section 3.5.3), provide movements which are universal to most board games. Along with the
detection functions, most of the code can be applied to play different games, the only
requirement being a change in the game logic and data models. One possible game could
be Draughts (or Checkers).
References
[1] http://www.jaqueslondon.co.uk/noughts-crosses-3d.html. [Online; last accessed 01-June-2015].
[2] http://logicgame.com.ua/game.php?id=tic_tac_toe_3d&l=en1. [Online; last
accessed 01-June-2015].
[3] http://kylim.tistory.com/91. [Online; last accessed 01-June-2015].
[4] http : / / www . hizook . com / blog / 2012 / 09 / 18 /
baxter-robot-rethink-robotics-finally-unveiled.
[Online; last accessed
01-June-2015].
[5] http://onexia.com/baxter/Baxter-Electric-Parallel-Gripper.html. [Online;
last accessed 01-June-2015].
[6] H. Coles-Abell and V. Pugh. Connect Four Demo. http://sdk.rethinkrobotics.
com/wiki/Connect_Four_Demo. [Online; last accessed 30-May-2015].
[7] K. Crowley and R. S. Siegler. Flexible strategy use in young childrens tic-tac-toe.
Cognitive Science, 17:531561, 1993.
[8] O. Elton. 4d noughts and crosses. http://matheminutes.blogspot.co.uk/2012/
07/4d-noughts-and-crosses.html, 2012.
[9] M. Gardner. Hexaflexagons and Other Mathematical Diversions.
Chicago Press, 1988.
University of
[10] S. W. Golomb and A. W. Hales. Hypercube tic-tac-toe. In More games of no chance

(Berkeley, CA, 2000), volume 42 of Math. Sci. Res. Inst. Publ., pages 167182.
Cambridge Univ. Press, Cambridge, 2002.
[11] R. R. Murphy. Introduction to AI Robotics. MIT Press, Cambridge, 2001.
[12] O. Patashnik. Qubic: 4 x 4 x 4 Tic-Tac-Toe. Mathematical Magazine 53, pages
202216, 1980.
[13] R. P. Paul. Robot manipulators: mathematics, programming, and control : the computer control of robot manipulators. MIT Press, Cambridge, 1981.
[14] V. Pugh. Worked Example Visual Servoing. http://sdk.rethinkrobotics.com/
wiki/Worked_Example_Visual_Servoing.
56
57
REFERENCES
[15] Rethink Robotics. Arms. http://sdk.rethinkrobotics.com/wiki/Arms. [Online;

last accessed 01-June-2015].
[16] Rethink Robotics.
Baxter Research Robot Tech Specs.
rethinkrobotics.com/baxter-research-robot/tech-specs/.
http : / / www .
[17] Rethink Robotics. Connect Four Demo. http://sdk.rethinkrobotics.com/wiki/

Connect_Four_Demo. [Online; last accessed 30-May-2015].
[18] ROS.org. Ros Indigo Igloo. http://wiki.ros.org/indigo/, July 2014.
[19] S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Pearson
Education Limited, 2013.
[20] S.
Totilo.
Natal
Recognizes
31
Body
Parts,
Uses
Tenth
Of Xbox 360 Kotaku.
http : / / kotaku . com / 5442775 /
natal-recognizes-31-body-parts-uses-tenth-of-xbox-360-computing-resources,
January 2010. [Online; accessed 28-May-2015].
[21] Wikipedia. Baxter (robot) Wikipedia, The Free Encyclopedia. http://en.
wikipedia.org/wiki/Baxter_%28robot%29. [Online; accessed 21-April-2015].
[22] Wikipedia. Blob detection Wikipedia, The Free Encyclopedia. http://en.
wikipedia.org/wiki/Blob_detection. [Online; last accessed 01-June-2015].
[23] Wikipedia. Game tree Wikipedia, The Free Encyclopedia. http://en.
wikipedia.org/wiki/Game_tree. [Online; last accessed 01-June-2015].
[24] Wikipedia. Tic-tac-toe Wikipedia, The Free Encyclopedia. http://en.
wikipedia.org/wiki/Tic-tac-toe. [Online; accessed 20-April-2015].
Appendices
58
Appendix A
Personal Reflection
I found the entire project a very engaging experience. Up until this point, my education
featured timetabled lectures, which were assessed via coursework and exams. The move
from this style to one where I could essentially choose the times I wanted to work was a
great change. Responsibility and discipline play a big part when it comes to working by
your own timetable. I found learning this to be enjoyable and even tempts me to continue
research as a career option.
Being given access to the robot at all times proved essential, along with my own
workstation located by it. The near-solidarity of the project was unusual. I was used
to completing coursework in time for deadlines alongside my peers in previous years.
This semester I found myself rather alone in the work I had to do, as everyone else
were off completing their own projects. That being said, I found the PhD and postdoctorate students in the area I had been allocated to work more than helpful and very
accommodating.
The project started slowly. There were problems setting up workstations to work with
Baxter both in the lab and in virtual machines on my laptop. This was time consuming,
and the virtual machines were not even used. Another problem which took several days
lay with initializing the OpenCV library which resulted in a full reinstallation of the
operating system. After these initial setbacks the project gained momentum. Working
with Baxter was very rewarding. Unlike a piece of code that works and outputs data
correctly, Baxter shows a physical representation of successful code. The choice to use
Python was something I thanked myself for. The ease of debugging and the familiarity
of the syntax made the entire coding aspect comfortable. On top of this, the majority
of previous projects and online tutorials relating to Baxter use Python code in their
examples.
A large proportion of the subject matter was new to me. I had never worked with
robots, nor done particularly well in modules involving artificial intelligence. I had not
even taken the computer vision module for the first semester of this year. Nevertheless,
with the online tutorials and help from my colleagues, I gained useful, highly interesting
knowledge which I was able to apply. Computer vision, in the end, turned out to be the
most enjoyable aspect of the project. It shames me to think what I could of learnt if I
had taken it as a module. Having a better knowledge of detection techniques prior to
the project may have resulted in less time being spent on vision, with more time spent
attempting to complete the final objective.
There were two major setbacks over the course of the project. The first was family
related, and incurred in a loss of three weeks where I could have been working. Thankfully,
59
APPENDIX A. PERSONAL REFLECTION
60
the university granted me an extension. The second came when attempting to utilize the
infra-red sensor on Baxters arm to detect heights of stacked playing pieces. This was by
far the most frustrating point of the project and incurred several consecutive late nights
working with the robot. On the assumption that the hardware was malfunctioning, a new,
more robust solution was used. I wish I had not wasted so much time trying to make
the infra-red method work, however, I was very happy to see that there was a possible
alternative solution and that the robot was eventually able to function as desired.
Overall, the project was a great success. Its rather entertaining to watch people
play against the robot and feel shamed by loosing to it on their first try. There are also
talks of having the software used on open days for prospective students, where they can
demo the robot themselves. The amount of invaluable knowledge I gained in doing the
project has started to show. When looking back over the demo projects which I read
over for background reading, I now have a solid understanding of how they work whereas
previously, I found the content confusing and rather daunting. Continuing a career in
robotics was something I never had considered until now.
My advice to future students attempting a project similar to this is to be generous
with time estimates. Robots do not have the second natures and common sense that us
humans do. A simple action that we can perform, such as picking up an orange, needs to
be broken down to many levels to enable a robot to mimic us. Think about how we see an
orange, its colour, its shape. A robot needs to be programmed to know what a circle is
and what orange is. When we pick it up, our sense of touch in our hands give us a large
description of how much force to apply and from what angle to grasp it. Robots need to
be programmed all this information, and many of them do not have an obvious sense of
touch. Baxter was no exception, everything I had to work with came down to what was
seen through a camera, and making movements based on this. This and the fact that
working with hardware in general provides many opportunities for failure, including the
configuration of work environments, is why I suggest leaving a project more time than
originally assumed.
Appendix B
Ethical Issues Addressed
There is no way my project, the nature of which revolves around playing a game, has any
ethical impact. There is an aspect of robots replacing the jobs of humans (in supermarkets,
for example), but my project does not incur such problems.
61
Appendix C
Video
This YouTube link shows a recorded demonstration of the code running on Baxter:
https://youtu.be/DMy8C9eJ4LE
62
Appendix D
Code
D.1
Repository
The following link is a public repository containing all code developed for this project.
The repository contains a readme detailing how to set up the work environment and run
the code. The rest of this appendix contains the code of the final version of the project.
https://bitbucket.org/ben2000677/fyp/
D.2
New3DtictacBaxter.py
#!/usr/bin/env python
import roslib
import sys
import inspect, os
import rospy
import string
from sensor_msgs.msg import Image
from sensor_msgs.msg import Range
from std_msgs.msg import Header
from moveit_commander import conversions
from baxter_core_msgs.srv import SolvePositionIK, SolvePositionIKRequest
import baxter_interface
from cv_bridge import CvBridge
import cv
import cv2
import numpy as np
import math
import random
from BoardClass3D import Board
# init rospy node

rospy.init_node(pickup_and_stack)
# file directory
63
64
APPENDIX D. CODE
directory =
os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
def main():
# get setup parameters
limb, distance = get_distance_data()
# read colour data from file
colours = get_object_data()
# create board class
theBoard = Board()
# create stack class instance
baxter = robot(limb, distance, colours, theBoard)
baxter.send_image(default)
print "limb
= ", limb
print "distance = ", distance
# open the gripper

baxter.gripper.open()
# move to start position
baxter.start_position()
# send image to baxters head display
baxter.send_image(locateboard)
# sleep to wait for camera feedback
rospy.sleep(1)
# displays image feed from camera on head display
baxter.toggleDisplayFeed(1)
# initialises board detection using Hough Circles
baxter.locate_board()
baxter.start_position()
game = True
while game:
theBoard.printBoard()
baxter.detect_pieces()
# prints game state to terminal

# check for players move
65
APPENDIX D. CODE
if theBoard.checkWinner(baxter.playerPiece): # check if human has won
baxter.send_image(anger2)
print(Hooray! You have won the game!)
break
elif theBoard.isFull(): # check if board is full
print(The game is a tie!)
break
baxter.take_turn()
# initialises Baxters game logic
if theBoard.checkWinner(baxter.robotPiece):
baxter.send_image(happy2)
print(The computer has beaten you! You lose.)
game = False
elif theBoard.isFull():
print(The game is a tie!)
break
sys.exit()
rospy.spin()
class robot():
def __init__(self, arm, distance, colours, board):
# piece data
self.playerPiece = X
self.playerPieceIndex = 1 # ( 1 or 2)
self.robotPiece = R
self.robotPieceIndex = 2
# space detection
self.board = board
self.storePointPix = True
self.detectedAllCorners = False
66
APPENDIX D. CODE
self.pixelPoints = []
self.board_tolerance = 0.00375
# board xyz
spaces = self.board.getSpaces()
self.boardZ = len(spaces)
# levels
self.boardY = len(spaces[0]) # rows

self.boardX = len(spaces[0][0]) # columns
# Background subtraction
self.bgsubImgmsg = None
self.imgmsg = None
# Circle piece detection
self.numCircleFrames = 1
self.spacePixelDist = 50.0
# arm ("left" or "right")
self.limb
= arm
self.limb_interface = baxter_interface.Limb(self.limb)
if arm == "left":
self.other_limb = "right"
else:
self.other_limb = "left"
self.other_limb_interface = baxter_interface.Limb(self.other_limb)
# gripper ("left" or "right")
self.gripper = baxter_interface.Gripper(arm)
# object color dictionary data
self.objects = colours
# zeroed lists for pixel coordinates of objects
self.x = np.zeros(shape=(len(colours)), dtype=int)
self.y = np.zeros(shape=(len(colours)), dtype=int)
# speeds
self.normal = 0.8
self.slow = 0.1
67
APPENDIX D. CODE
# start positions
self.start_pos_x = 0.50
# x
= front back
self.start_pos_y = 0.30
# y
= left right
self.start_pos_z = 0.15
# z
= up down
self.roll
= -1.0 * math.pi
# roll = horizontal
self.pitch
= -0.0 * math.pi
# pitch = vertical
self.yaw
= 0.0 * math.pi
# yaw
= rotation
self.pose = [self.start_pos_x, self.start_pos_y, self.start_pos_z, \

self.roll, self.pitch, self.yaw]
# distances
self.distance = distance
self.block_height = 0.04
self.block_pickup_height = self.distance - self.block_height
self.block_grip_height = 0.102
self.block_tolerance = 0.005
# camera parameters (NB. other parameters in open_camera)

self.cam_calib = 0.0022762
self.cam_x_offset = 0.045
# constant for distance = 0.383

# original camera gripper offset
self.cam_y_offset = -0.015
self.width
= 960
self.height
= 600
# Camera resolution
# Enable the actuators

baxter_interface.RobotEnable().enable()
# set speed as a ratio of maximum speed
self.limb_interface.set_joint_position_speed(0.8)
self.other_limb_interface.set_joint_position_speed(0.8)
# calibrate the gripper
self.gripper.calibrate()
# set camera resolution
self.config_camera(self.limb, self.width, self.height)
# subscribe to required camera
self.subscribe_to_camera(self.limb)
APPENDIX D. CODE
# Display camera feed to face display

self.displayCamera = False
self.displayContours = False
self.pub = rospy.Publisher(/robot/xdisplay, Image, latch=True,
queue_size=1)
# move other arm out of harms way
if arm == "left":
self.baxter_ik_move("right", (0.25, -0.50, 0.2, math.pi, 0.0, 0.0))
else:
self.baxter_ik_move("left", (0.25, 0.50, 0.2, math.pi, 0.0, 0.0))
def toggleDisplayFeed(self,on,contours=False):
"""
Toggles booleans to display camera feed on head display.
"""
if contours:
self.displayContours = True
else:
self.displayContours = False
if on:
self.displayCamera = True
else:
self.displayCamera = False
def send_image(self,img_name):
"""
Send the image located at the specified path to the head
display on Baxter.
@param path: path to the image file to load and send
"""
self.toggleDisplayFeed(0) # Turn off feed to head display
global directory
path = directory+/images/+img_name+.png
img = cv2.imread(path)
self.publish_img(img,False)
68
69
APPENDIX D. CODE
rospy.sleep(1)
# Sleep to allow for image to be published.
def publish_img(self,img,resize=True):
"""
Publisher to head display.
"""
if resize:
img = cv2.resize(img, (1024, 600))
msg = CvBridge().cv2_to_imgmsg(img, encoding="passthrough")
self.pub.publish(msg)
# Game Logic
#--------------------------------------------------------------------------------------#
def take_turn(self):
"""
Baxters turn, following game logic.
"""
def random_move(movesList,level):
# Returns a valid move from the passed list on the passed board.
# Returns None if there is no valid move.
possibleMoves = []
for i in movesList:
if not self.board.getPiece(level,i[0],i[1]):
possibleMoves.append(i)
if len(possibleMoves) != 0:
return random.choice(possibleMoves)
else:
return None
# check to see if winning move is possible

for row in range(self.boardY):
for space in range(self.boardX):
copy = self.board.getCopy()
if not self.board.getPiece(self.boardZ-1,row,space): # if space
is free
copy.makeMove(row,space, self.robotPiece)
70
APPENDIX D. CODE
if copy.checkWinner(self.robotPiece):
z = self.board.makeMove(row,space, self.robotPiece)
self.place_piece(row, space, z)
return
# Check if the human player could win on his next move, and block them.
for space in range(self.boardX):
copy = self.board.getCopy()
if not self.board.getPiece(self.boardZ-1,row,space): # if space
is free
copy.makeMove(row,space, self.playerPiece)
if copy.checkWinner(self.playerPiece):
z = self.board.makeMove(row,space, self.robotPiece)
self.place_piece(row, space, z)
return
# Try to take the middle center, if its free and has a robot piece
below it.
if not self.board.getPiece(1,1,1) and self.board.getPiece(0,1,1) ==
self.robotPiece:
level = self.board.makeMove(1,1,self.robotPiece)
self.place_piece(1,1,level)
return
# Try to take the bottom center, if its free.
if not self.board.getPiece(0,1,1):
return
# Try to take one of the corners of the bottom level, if they are free.
move = random_move([(0,0),(2,0),(0,2),(2,2)], 0)
if move != None:
level = self.board.makeMove(move[0],move[1],self.robotPiece)
self.place_piece(move[0], move[1], level)
return
# Try to take the middle center, if its free.

if not self.board.getPiece(1,1,1):
71
APPENDIX D. CODE
return
# Try to take any corner, if its free.

move = random_move([(0,0),(2,0),(0,2),(2,2)], 2)
if move != None:
return
# Move on one of the sides.
move = random_move([(0,1), (1,0), (2,1), (1,2)], 2)
# Tracking Stuff
#--------------------------------------------------------------------------------------#
def get_contours(self,i,hsv):
"""
Gets contours from image.
"""
mask = cv2.inRange(hsv, np.array(self.objects[i][0:3]),
np.array(self.objects[i][3:6]))
# filter and fill the mask
kernel =
cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(self.objects[i][6],self.objects[i][6])
mask2 = cv2.morphologyEx(mask,cv2.MORPH_OPEN,kernel)
ret,thresh = cv2.threshold(mask2,127,255,0)
contours, hierarchy =
cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
return contours
def detect_and_draw(self,imgmsg):
"""
Main detection function, constantly loops through.
"""
APPENDIX D. CODE
72
# converts image message to cv image

img = CvBridge().imgmsg_to_cv2(imgmsg, desired_encoding=passthrough)
# stores img message for background subtraction
self.imgmsg = imgmsg
#-----------------------------------HOUGH_CIRCLES--------------------------------------#
# for detecting the board
# get grayscale version of image
cimg = img[:,:,0:3]
cimg = cv2.cvtColor(cimg,cv2.COLOR_BGR2GRAY)
cimg = cv2.medianBlur(cimg,5)
# extract circles from the image
circles =
cv2.HoughCircles(cimg,cv.CV_HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadius=10,ma
# ensures all 4 corners are visible and that point storing is enabled
if circles != None and self.storePointPix:
circles = np.uint16(np.around(circles))
centers = []
for i in circles[0,:]:
centers.append([i[0],i[1],i[2]])
dis_TR = 100000
dis_TL = 100000
dis_BR = 100000
dis_BL = 100000
# calculate which corners the detected circles are (closest distance)
for i in centers:
dis = np.sqrt((i[0]-1000)**2 + (i[1])**2)
if dis < dis_TR:
dis_TR = dis
TR = i
dis = np.sqrt((i[0])**2 + (i[1])**2)
if dis < dis_TL:
dis_TL = dis
TL = i
APPENDIX D. CODE
dis = np.sqrt((i[0]-1000)**2 + (i[1]-1000)**2)
if dis < dis_BR:
dis_BR = dis
BR = i
dis = np.sqrt((i[0])**2 + (i[1]-1000)**2)
if dis < dis_BL:
dis_BL = dis
BL = i
TR = map(int,TR)
TL = map(int,TL)
BR = map(int,BR)
BL = map(int,BL)
# draw circles around detected circles
cv2.circle(img,(TR[0],TR[1]),TR[2],(0,255,0),2)
cv2.circle(img,(TR[0],TR[1]),2,(0,0,255),3)
cv2.circle(img,(TL[0],TL[1]),TL[2],(0,0,255),2)
cv2.circle(img,(TL[0],TL[1]),2,(0,0,255),3)
cv2.circle(img,(BR[0],BR[1]),BR[2],(255,0,0),2)
cv2.circle(img,(BR[0],BR[1]),2,(0,0,255),3)
cv2.circle(img,(BL[0],BL[1]),BL[2],(150,30,200),2)
cv2.circle(img,(BL[0],BL[1]),2,(0,0,255),3)
# distances between the grid space centres
dx = (float(TR[0])-float(TL[0]))/4
dy = (float(TR[1])-float(TL[1]))/4
dx2 = (float(BR[0])-float(BL[0]))/4
dy2 = (float(BR[1])-float(BL[1]))/4
# middle row beginning and end points
MLx = TL[0]+int(np.round((float(BL[0])-float(TL[0]))/2))
MLy = TL[1]+int(np.round((float(BL[1])-float(TL[1]))/2))
MRx = TR[0]+int(np.round((float(BR[0])-float(TR[0]))/2))
MRy = TR[1]+int(np.round((float(BR[1])-float(TR[1]))/2))
# distances between the grid space centres of the middle row
dx3 = (MRx - MLx)/4
dy3 = (MRy - MLy)/4
self.pixelPoints = range(self.boardX **2)
for i in range(0,self.boardX):
73
74
APPENDIX D. CODE
top =
(TL[0]+int(np.round(dx*(i+1))),TL[1]+int(np.round(dy*(i+1))))
# top row
mid = (MLx+int(np.round(dx3*(i+1))),MLy+int(np.round(dy3*(i+1))))
# middle row
bot =
(BL[0]+int(np.round(dx2*(i+1))),BL[1]+int(np.round(dy2*(i+1))))
# bottom row
self.pixelPoints[i] = top
self.pixelPoints[i+3] = mid
self.pixelPoints[i+6] = bot
cv2.circle(img,top,2,(80*(i+1),0,0),3)
cv2.circle(img,mid,2,(0,80*(i+1),0),3)
cv2.circle(img,bot,2,(0,0,80*(i+1)),3)
# if all the corners are detected
if len(centers) == 4:
self.detectedAllCorners = True
#----------------------------BLOB_DETECTION---------------------------------------#
# for detecting robot blocks
# Blur each frame
simg = cv2.GaussianBlur(img,(5,5),0)
# Convert BGR to HSV
hsv = cv2.cvtColor(simg, cv2.COLOR_BGR2HSV)
i = self.robotPieceIndex
contours = self.get_contours(i, hsv)
my_contours = []
minDist = 10000
for cnt in contours:
if cv2.contourArea(cnt)>1400:
# check contours above a certain area
my_contours.append(cnt)
x,y,w,h = cv2.boundingRect(cnt)
# get dimensions of the
contour
if self.displayContours:
cv2.rectangle(img,(x,y),(x+w,y+h),(0,0,255),2)
cx = x + (w/2)
cy = y + (h/2)
# centre points
75
APPENDIX D. CODE
dist = self.distance_to((cx,cy),(self.width,self.height)) #
target the most-right-bottom blob
if dist < minDist:
minDist = dist
self.x[i-1] = cx
self.y[i-1] = cy
if self.displayContours:
cv2.drawContours(img,my_contours,-1,(0,255,0),3)
if self.displayCamera:
self.publish_img(img)
cv2.imshow(RGB,img)
k = cv2.waitKey(5) & 0xFF
#--------------------------------------------------------------------------------------#
# create subscriber to the required camera
def subscribe_to_camera(self, camera):
if camera == "left":
#callback = self.left_camera_callback
camera_str = "/cameras/left_hand_camera/image"
elif camera == "right":
#callback = self.right_camera_callback
camera_str = "/cameras/right_hand_camera/image"
else:
sys.exit("ERROR - subscribe_to_camera - Invalid camera")
image_topic = rospy.resolve_name(camera_str)
# subscribe to detection topic
rospy.Subscriber(image_topic, Image, self.detect_and_draw)
# used to place camera over target object

def block_iterate(self, iteration, block_centre):
# find displacement of target from centre of image
pixel_dx
= (self.width / 2) - block_centre[0]
pixel_dy
= (self.height / 2) - block_centre[1]
pixel_error = math.sqrt((pixel_dx * pixel_dx) + (pixel_dy * pixel_dy))
76
APPENDIX D. CODE
error
= float(pixel_error * self.cam_calib *
self.block_pickup_height)
x_offset = - pixel_dy * self.cam_calib * self.block_pickup_height
y_offset = - pixel_dx * self.cam_calib * self.block_pickup_height
# update pose and find new block location data
self.update_pose(x_offset, y_offset)
# find displacement of target from centre of image
pixel_dx
= (self.width / 2) - block_centre[0]
pixel_dy
= (self.height / 2) - block_centre[1]
pixel_error = math.sqrt((pixel_dx * pixel_dx) + (pixel_dy * pixel_dy))

error
= float(pixel_error * self.cam_calib *
self.block_pickup_height)
return block_centre, error
def config_camera(self, camera, x_res, y_res):
if camera == "left":
cam = baxter_interface.camera.CameraController("left_hand_camera")
elif camera == "right":
cam = baxter_interface.camera.CameraController("right_hand_camera")
else:
sys.exit("ERROR - open_camera - Invalid camera")
# set camera parameters
cam.resolution
= (int(x_res), int(y_res))
cam.exposure
= -1
# range, 0-100 auto = -1
cam.gain
= -1
# range, 0-79 auto = -1
cam.white_balance_blue = -1
# range 0-4095, auto = -1
cam.white_balance_green = -1
# range 0-4095, auto = -1
cam.white_balance_red = -1
# range 0-4095, auto = -1
def detect_pieces(self):
"""
Detect moves made by the opponent.
"""
def subtract_background():
"""
77
APPENDIX D. CODE
Apply backround subtraction.
"""
# take old and new image
oldImg = CvBridge().imgmsg_to_cv2(self.bgsubImgmsg,
desired_encoding=passthrough)
img = CvBridge().imgmsg_to_cv2(self.imgmsg,
fgbg = cv2.BackgroundSubtractorMOG()
oldImg = oldImg[:,:,0:3]
fgmask = fgbg.apply(oldImg)
img = img[:,:,0:3]
fgmask = fgbg.apply(img)
# subtract images
img1_bg = cv2.bitwise_and(img,img,mask = fgmask)
# Blur image
simg = cv2.GaussianBlur(img1_bg,(5,5),0)
# Convert BGR to HSV
hsv = cv2.cvtColor(simg, cv2.COLOR_BGR2HSV)
# check for contours of the players piece
contours = self.get_contours(self.playerPieceIndex, hsv)
areaMax = 0.0
cx = 0
cy = 0
for cnt in contours:
if cv2.contourArea(cnt)>1400 and cv2.contourArea(cnt)>areaMax:
areaMax = cv2.contourArea(cnt)
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img1_bg,(x,y),(x+w,y+h),(0,0,255),2)
cx = x + (w/2) # Centre points
cy = y + (h/2)
pos = (cx,cy)
if pos != (0,0):
# if a contour was
detected
self.bgsubImgmsg = self.imgmsg
self.publish_img(img1_bg)
# update stored image

# send to face
78
APPENDIX D. CODE
minDist = float(inf)
x = 10
y = 10
# find the closest grid space to the contour
for col in range(self.boardX):
dist = self.distance_to(pos,self.board.getPixel(row,col))
if dist < minDist:
minDist = dist
x = row
y = col
# update the board data model
self.board.makeMove(x,y,self.playerPiece)
return True
else:
return False
def check_circles():
"""
Checks for pieces with circles drawn on them.
"""
centers = []
for frames in range(self.numCircleFrames): # checks a number of
frames
img = CvBridge().imgmsg_to_cv2(self.imgmsg,
# for circle pieces
cimg = img[:,:,0:3]
cimg = cv2.cvtColor(cimg,cv2.COLOR_BGR2GRAY)
cimg = cv2.medianBlur(cimg,5)
# extract circles
circles =
cv2.HoughCircles(cimg,cv.CV_HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadiu
if circles != None:
circles = np.uint16(np.around(circles))
for i in circles[0,:]:
center = (i[0],i[1])
79
APPENDIX D. CODE
if center not in centers:
centers.append(center)
cv2.circle(img,(i[0],i[1]),i[2],(0,255,255),3)
cv2.circle(img,(i[0],i[1]),2,(0,0,255),3)
# show detected circle on head display
self.publish_img(img)
if frames > 0:
rospy.sleep(1)

# check places with a player piece in them
if self.board.getHighPiece(row,col) == self.playerPiece:
dists = []
# checks if the highest piece of the stack already has a
circle
hasCircle = self.board.getCircle(row,col)
for center in centers:
# finds closest grid space
dist =
self.distance_to(center,self.board.getPixel(row,col))
dists.append(dist)
if not dists:
if hasCircle:
# updates board data model
self.board.makeMove(row,col,self.playerPiece,False)
return True
else:
# check to see if a circle piece has been played
if min(dists) < self.spacePixelDist and not hasCircle:
self.board.makeMove(row,col,self.playerPiece,True)
return True
# check to see if a circle piece was removed
elif min(dists) > self.spacePixelDist and hasCircle:
self.board.makeMove(row,col,self.playerPiece,False)
80
APPENDIX D. CODE
return True
return False
# positions arm above boards central point

self.pose = self.board.getPosition(1,1)
self.baxter_ik_move(self.limb, self.pose)
nomove = True
while nomove:
self.send_image(yourturn)
self.send_image(3)
# countdown prompt
self.send_image(2)
self.send_image(1)
self.send_image(go)
rospy.sleep(2)
if check_circles() or subtract_background():
# then there is a change
rospy.sleep(3)
return
def distance_to(self,i,j):
a = (i[0]-j[0])
b = (i[1]-j[1])
c2 = (a*a) + (b*b)
return math.sqrt(c2)
# Movement Functions
#--------------------------------------------------------------------------------------#
def place_piece(self, x, y, level):
self.start_position()
self.toggleDisplayFeed(1,True)
self.find_object()
self.send_image(default)
self.pickup_object()
self.drop_on(self.board.getPosition(x,y),level+1)
def start_position(self):
self.setSpeed(self.normal)
# move back to start position
81
APPENDIX D. CODE
self.pose = [self.start_pos_x, self.start_pos_y, self.start_pos_z, \
self.roll, self.pitch, self.yaw]
def find_object(self,BoardMode=False):
# iterates to find closest pose above object and returns it
if not BoardMode:
tolerance = self.block_tolerance
i = self.robotPieceIndex-1
else:
tolerance = self.board_tolerance
# middle pixel point index of the pixelpoint array
i = int(len(self.pixelPoints)/2)
error
= 2 * tolerance
iteration = 1
# iterate until arm over centre of target
while error > tolerance or not self.detectedAllCorners:
if not BoardMode:
block_centre = (self.x[i],self.y[i])
else:
if self.pixelPoints[i] == i:
# if a middle pixel point has
not been generated

continue
block_centre = (self.pixelPoints[i][0],self.pixelPoints[i][1])
block_centre, error = self.block_iterate(iteration, \
block_centre)
iteration
+= 1
return self.pose
def pickup_object(self):
self.setSpeed(self.slow)
# save return pose
origin = self.pose
# move down to pick up object
self.pose = (self.pose[0] + self.cam_x_offset,
self.pose[1] + self.cam_y_offset,
self.pose[2] + (self.block_grip_height self.block_pickup_height),
APPENDIX D. CODE
self.pose[3],
self.pose[4],
self.pose[5])
self.gripper.close()
# return to origin
self.baxter_ik_move(self.limb, origin)
def drop_on(self,target,level):
# drop object onto target below obj pose param
self.pose = target
self.pose = (self.pose[0] + self.cam_x_offset,
self.pose[1] + self.cam_y_offset,
self.pose[2] + (self.block_grip_height - (self.distance (self.block_height * level))),
self.pose[3],
self.pose[4],
self.pose[5])
self.gripper.open()
self.pose = target
def locate_board(self):
centre = self.find_object(True)
# stop storing pixel points at centre position and update Board class
model
self.storePointPix = False
counter = 0
centrePix = self.pixelPoints[4]
self.board.setPixel(row,col,self.pixelPoints[counter])
self.board.setPosition(row,col,(centre[0] +
((self.pixelPoints[counter][1] - centrePix[1]) *
self.cam_calib * self.distance),
82
83
APPENDIX D. CODE
centre[1] +
((self.pixelPoints[counter][0] centrePix[0]) * self.cam_calib *

self.distance),
centre[2],
centre[3],
centre[4],
centre[5]))
counter += 1
# update pose in x and y direction

def update_pose(self,dx, dy):
x = self.pose[0] + dx
y = self.pose[1] + dy
self.pose = [x, y, self.pose[2], self.roll, self.pitch, self.yaw]
def baxter_ik_move(self, limb, rpy_pose):
"""
Performs all inverse kinematics conversions from cartesian coordinates.
Upgraded to change orientation if valid configuration is not found.
"""
quaternion_pose = conversions.list_to_pose_stamped(rpy_pose, "base")
limb_interface = baxter_interface.Limb(limb)
node = "ExternalTools/" + limb + "/PositionKinematicsNode/IKService"
ik_service = rospy.ServiceProxy(node, SolvePositionIK)
ik_request = SolvePositionIKRequest()
hdr = Header(stamp=rospy.Time.now(), frame_id="base")
ik_request.pose_stamp.append(quaternion_pose)
try:
rospy.wait_for_service(node, 5.0)
ik_response = ik_service(ik_request)
except (rospy.ServiceException, rospy.ROSException), error_message:
rospy.logerr("Service request failed: %r" % (error_message,))
sys.exit("ERROR - baxter_ik_move - Failed to append pose")
if ik_response.isValid[0]:
84
APPENDIX D. CODE
# convert response to joint position control dictionary
limb_joints = dict(zip(ik_response.joints[0].name,
ik_response.joints[0].position))
# move limb
if self.limb == limb:
self.limb_interface.move_to_joint_positions(limb_joints)
else:
self.other_limb_interface.move_to_joint_positions(limb_joints)
else:
# display invalid move message on head display
self.send_image(errorjoint)
self.pose = [self.pose[0], self.pose[1], self.pose[2], self.pose[3],

self.pose[4]-(0.01 * math.pi), self.pose[5]]
print adjusting
self.baxter_ik_move(limb,self.pose)
self.send_image(default)
if self.limb == limb:
# if working arm
quaternion_pose = self.limb_interface.endpoint_pose()
position
= quaternion_pose[position]
# if working arm remember actual (x,y) position achieved

self.pose = [position[0], position[1],
self.pose[2], self.pose[3], self.pose[4], self.pose[5]]

def setSpeed(self,speed):
self.limb_interface.set_joint_position_speed(speed)
# Setup Functions
#--------------------------------------------------------------------------------------#
def get_distance_data():
# read the setup parameters from setup.dat
file_name = directory + "/setup.dat"
try:
f = open(file_name, "r")
except ValueError:
APPENDIX D. CODE
sys.exit("ERROR: golf_setup must be run before golf")
#find limb
s = string.split(f.readline())
if len(s) >= 3:
if s[2] == "left" or s[2] == "right":
limb = s[2]
else:
sys.exit("ERROR: invalid limb in %s" % file_name)
else:
sys.exit("ERROR: missing limb in %s" % file_name)
#find distance to table
s = string.split(f.readline())
if len(s) >= 3:
try:
distance = float(s[2])
except ValueError:
sys.exit("ERROR: invalid distance in %s" % file_name)
else:
sys.exit("ERROR: missing distance in %s" % file_name)
return limb, distance
def get_object_data():
Objects_pointer = open(directory+/objects.txt, r)
OBJ = {}
for line in Objects_pointer:
line = line.strip(,\n)
if line == END:
break
fields = line.split(,)
fields = map(int, fields)
OBJ[fields[0]] = fields[1:len(fields)]
return OBJ
85
APPENDIX D. CODE
if __name__ == "__main__":
main()
D.3
BoardClass3D.py
#!/usr/bin/env python
import copy
class Board():
def __init__(self,x=3,y=3,z=3):
self.cube = []
self.spaces = []
# create a 3D matrix of empty spaces
for i in range(0,z):
level = []
for j in range(0,y):
row = []
for k in range(0,x):
space = Space()
row.append(space)
self.spaces.append((i,j,k))
level.append(row)
self.cube.append(level)
self.solutions = self.generateSolutions()
def generateSolutions(self):
"""
Generates winning lines for 3D Noughts & Crosses
"""
def allEqual(x,y,z):
return (x==y) and (y==z)
def isSorted(x,y,z):
return (x > y and y > z) or (z > y and y > x)
solutions = []
for a in self.spaces:
for b in self.spaces:
86
87
APPENDIX D. CODE
for c in self.spaces:
# check every combination of 3 distinct spaces
if a != b and b != c and a != c:
line = [a,b,c]
line.sort()
if not line in solutions:
level_eq = allEqual(a[0],b[0],c[0])
row_eq = allEqual(a[1],b[1],c[1])
col_eq = allEqual(a[2],b[2],c[2])
level_sort = isSorted(a[0],b[0],c[0])
row_sort = isSorted(a[1],b[1],c[1])
col_sort = isSorted(a[2],b[2],c[2])
# logic statements to check if line is a winning line
if ((level_eq and row_eq) or (level_eq and col_eq) or
(col_eq and row_eq)) or \
(level_sort and row_sort and col_sort) or \
((level_eq and row_sort and col_sort) or (level_sort
and row_eq and col_sort) or (level_sort and
row_sort and col_eq)):
solutions.append(line)
return solutions
def getCopy(self):
return copy.deepcopy(self)
def checkWinner(self,letter):
board = self.cube
for solution in self.solutions:
if
board[solution[0][0]][solution[0][1]][solution[0][2]].getContent()
== letter \
and
== letter \
and
== letter:
return True
def makeMove(self, row, col, letter):
"""
Updates the data model with a new piece
88
APPENDIX D. CODE
"""
for level in range(len(self.cube)):
if not self.getPiece(level,row,col):
self.setPiece(level,row,col,letter)
return level
def isFull(self):
# Return True if every space on the top of the board has been taken.
Otherwise return False.
for row in self.cube[len(self.cube)-1]:
for space in row:
if not space.getContent(): # if space is free
return False
return True
def setPosition(self, row, col, pose):
self.cube[0][row][col].setPose(pose)
def getPosition(self, row, col):
return self.cube[0][row][col].getPose()
def getSpaces(self):
return self.cube
#used to be self.spaces
def setPiece(self, lvl, row, col, piece):

self.cube[lvl][row][col].setContent(piece)
def getPiece(self, lvl, row, col):
return self.cube[lvl][row][col].getContent()
def getHighPiece(self, row, col):
"""
Returns the highest piece in a grid space
"""
cube = self.getSpaces()
if self.getPiece(0,row,col) and not self.getPiece(len(cube)-1,row,col):
#make sure stack is not full or empty
for level in range(len(cube)-2,-1,-1):
piece = self.getPiece(level,row,col)
if piece:
return piece
89
APPENDIX D. CODE
else:
return 0
def setPixel(self, row, col, pixel):
self.cube[0][row][col].setPix(pixel)
def getPixel(self,row, col):
return self.cube[0][row][col].getPix()
def printBoard(self):
"""
Prints the board data model to terminal
"""
for level in range(len(self.cube)-1,-1,-1):
counter = 0
print " 0 1 2"
for row in self.cube[level]:
print " -------"
string = str(counter)
for space in row:
string += "|"
if not space.getContent():
string += " "
else:
string += space.getContent()
print string + "|"
counter += 1
print " -------"
class Space():
def __init__(self):
self.piece = 0
self.pose = (0,0,0,0,0,0)
self.pixel = (0,0)
def setPose(self,pose):
self.pose = pose
def getPose(self):
return self.pose
APPENDIX D. CODE
def setContent(self,piece):
self.piece = piece
def getContent(self):
return self.piece
def setPix(self,pixel):
self.pixel = pixel
def getPix(self):
return self.pixel
90
Appendix E
Board Detection Times Under Varying
Light Conditions
Attempt Minimal Light (secs) Artificial Light (secs)

1
18.10
18.10
2
13.67
39.78
3
13.76
28.16
4
13.27
32.32
5
29.17
24.62
6
09.65
27.10
7
81.34
13.40
8
13.91
20.32
9
95.95
19.05
10
23.19
23.31
Natural Light (secs)

10.00
21.19
13.85
16.15
18.32
12.93
20.78
14.25
13.59
13.60
Figure E.1: Times taken to locate the game board under varying light conditions.
91
APPENDIX E. BOARD DETECTION TIMES UNDER VARYING LIGHT CONDITIONS92

100
Time (Seconds)
80
60
40
20
Figure E.2: Recorded times in Minimal lighting conditions, arranged in ascending order.
40
Time (Seconds)
35
30
25
20
15
Figure E.3: Recorded times in Artificial lighting conditions, arranged in ascending order.
22
Time (Seconds)
20
18
16
14
12
10
Figure E.4: Recorded times in Natural lighting conditions, arranged in ascending order.

HIRSH1

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HIRSH1

Uploaded by

Copyright:

Available Formats

School of Computing

3D Noughts & Crosses with Baxter, a Humanoid Robot

Recipient(s) and Date

Type of project: Exploratory Software (ESw)

c 2014 - 2015 The University of Leeds and Benjamin Leonardo Hirsh

B Ethical Issues Addressed

The sequence of the chapters in this report follows my development process.

Figure 1.1: Time plan for each objective.

3D Noughts & Crosses

CHAPTER 2. BACKGROUND RESEARCH

Figure 2.1: 3D Noughts & Crosses [1].

CHAPTER 2. BACKGROUND RESEARCH

Figure 2.2: Possible winning lines [8].

Figure 2.3: Addresses of grid spaces [8].

CHAPTER 2. BACKGROUND RESEARCH

CHAPTER 2. BACKGROUND RESEARCH

Figure 2.5: Sequence of moves until game end in 3D [2].

CHAPTER 2. BACKGROUND RESEARCH

CHAPTER 2. BACKGROUND RESEARCH

CHAPTER 2. BACKGROUND RESEARCH

Figure 2.7: Minimax Example [3].

CHAPTER 2. BACKGROUND RESEARCH

Robot Operating System (ROS)

Topics & Services

CHAPTER 2. BACKGROUND RESEARCH

Baxter Research Robot

Figure 2.8: Baxter Research Robot by Rethink Robotics [4].

CHAPTER 2. BACKGROUND RESEARCH

Figure 2.9: Camera, infra-red sensor and gripper adjustment [15].

Baxter Software Development Kit (SDK)

CHAPTER 2. BACKGROUND RESEARCH

Worked Example Visual Servoing

Connect Four Demo

Baxter Solves Rubiks Cube

CHAPTER 2. BACKGROUND RESEARCH

Figure 3.1: Baxters physical workspace environment

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.2: Electric Parallel Gripper Kit [5].

Custom Board Game

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.3: Custom board game pieces (originally red).

Figure 3.4: New board design, generated in OpenCV.

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.5: Updated custom game pieces.

CHAPTER 3. DEVELOPMENT & ANALYSIS

which isnt the projects focus.

CHAPTER 3. DEVELOPMENT & ANALYSIS

CHAPTER 3. DEVELOPMENT & ANALYSIS

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.8: Left to right, top to bottom:

Hough Circle Detection

Hough Transform is a technique used to extract features from an image in computer

CHAPTER 3. DEVELOPMENT & ANALYSIS

CHAPTER 3. DEVELOPMENT & ANALYSIS

World Coordinate System Transformations

CHAPTER 3. DEVELOPMENT & ANALYSIS

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.10: Camera calibration test image.

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.11: Camera calibration test results.

CHAPTER 3. DEVELOPMENT & ANALYSIS

Figure 3.12: Camera calibration test results at a lower height.

CHAPTER 3. DEVELOPMENT & ANALYSIS

CHAPTER 3. DEVELOPMENT & ANALYSIS