9 Robot Vision

Intro to Robots
Processing Pictures:

Without the fluke board we cant process pictures taken

by the robot.
However we can process regular .jpg files.
Caution: Processing can take a long time so keep your
pictures to 600 x 600 pixels. If you are just experimenting
try to keep your pictures even smaller 200x200 pixels.

Intro to Robots
Picture Functions:

from myro import *

picture = makePicture(somefile.jpg) # read from a .jpg file

show(picture)

newPic = pictureCopy(picture)

picWidth = getWidth(picture); picHeight = getHeight(picture)

<list of pixels> = getPixels(picture)

<some Pixel> = getPixel(picture,x,y)

setRed(pix, <n>) # 0 <= n <= 255, red == 255

setGreen(pix, <n>) # 0 <= n <= 255, red == 0
setBlue(pix, <n>) # 0 <= n <= 255, red == 0

redValue = getRed(pix); greenValue = getGreen(pix); blueValue = getBlue(pix)

x = getX(pix); y = getY(pix)

savePicture(picture,anotherfile.jpg) # save to a .jpg file

Intro to Robots
Robot Vision:

Without the Fluke board, Scribbler can not take pictures.

However we can explore how we would process
photographs taken by Scribbler and to use results of that
analysis.

Intro to Robots
Making your own Picture:

newPic = makePicture(100,100)
show(newPic)

black by default

onePixel = getPixel(newPic,10,5)
print getRed(onePixel) #initial red value
setRed(onePixel,255)
.
print getRed(onePixel) #new red value
show(newPic)

Intro to Robots
Making your own Picture: height of picture

for yValue in range(0, getHeight(newPic)):

aPixel = getPixel(newPic, 10, yValue)
setRed(aPixel,255)
setGreen(aPixel,0)
setBlue(aPixel,0)
show(newPic)

# or make it into a function

def drawVerticalRedLine( picture, xPos ):

for yPos in range(0, getHeight(picture) ):
aPixel = getPixel( picture, xPos, yPos)
setRed(aPixel,255)
setGreen(aPixel,0)
setBlue(aPixel,0)

drawVerticalRedLine(newPic,10)
show(newPic)

Intro to Robots
Real Photograph

Photo taken by Scribbler

applePic = takePicture()
OR
applePic = loadPicture(apple.jpg)

drawVerticalRedLine(applePic,10)
show(applePic)

Intro to Robots
Scan Real Photograph

If we draw the vertical red line continuously across the

photo we will cover up the photo.
We would rather have the following behaviour. run

What we are doing is:

drawing a red line,
waiting a fraction of a second,
putting back the original pixels and
drawing the line again further on.

Intro to Robots
Robot/Program Vision:

The robot and our program dont see purple, they each
see a combination of red, green and blue.
r,g,b = getColors(pixel)

Colors with low values of red, green and blue are

generally dark and with high values, generally light.
width=height=500
myCanvas = GraphWin("Circles", width, height)
myCanvas.setBackground("white")
for i in range(1,25):
c = Circle(Point(randrange(0,width),
randrange(0,height)),20)
run red = 10*i
c.setFill(color_rgb(red, red/2, red/4))
c.draw(myCanvas)
wait(1)

wait(5)
myCanvas.close()

Intro to Robots
Filtering Photographs:

Have you ever heard the expression, The world is not

black and white, but various shades of gray.
Some times seeing things as black and white helps
filter out other irrelevant or confusing messages.
Go back to our original picture
and ask, What parts of the
photo are brightest?

Intro to Robots
Brightness Filter:

We know that higher red, green and blue values mean

brighter object.
Brightness: average color value > 175
myPicture = takePicture()
show(myPicture)
for pixel in getPixels(myPicture):
rValue = getRed(pixel)
before
gValue = getGreen(pixel)
bValue = getBlue(pixel)
avgValue = ( rValue + gValue + bValue) / 3.0
if avgValue > 175 :
#Turn the pixel white
setRed(pixel,255);setGreen(pixel,255)
setBlue(pixel,255)
else: after
#Otherwise, turn it black.
setRed(pixel,0);setGreen(pixel,0)
setBlue(pixel,0)
show(myPicture)

Intro to Robots
Redness Filter:

Suppose you want to find the reddest thing in a photo

(and if the photo was taken by the robot, have the robot
turn towards that thing).
In the previous example most of the filtered apple was
black, why?
Redness:
high red value (> 175),
low green and blue values (< 175).

Intro to Robots
Redness Filter:

Program for highlighting red

myPicture = takePicture()
show(myPicture)
def filterRed(picture):
for pixel in getPixels(picture):
rValue = getRed(pixel)
if rValue > 175:
#Turn the pixel white
setRed(pixel,255)
setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0)
setGreen(pixel,0)
setBlue(pixel,0)

filterRed(myPicture)
show(myPicture)
an awful lot of red

Intro to Robots
Greenness Filter:

Same program for green

myPicture = takePicture()
show(myPicture)
for pixel in getPixels(myPicture):
gValue = getGreen(pixel)
if gValue > 175:
#Turn the pixel white
setRed(pixel,255)
setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0)
setGreen(pixel,0)
setBlue(pixel,0)

show(myPicture)

funny how a green filter makes a red apple stand out

Intro to Robots
Primarily Red Filter:

myPicture = takePicture()
show(myPicture)
Def findRedAreas(picture):
for pixel in getPixels(myPicture):
rValue = getRed(pixel)
gValue = getGreen(pixel)
bValue = getBlue(pixel)
if rValue > 175 and
gValue < 175 and bValue < 175:
#Turn the pixel white
setRed(pixel,255)
setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0)
setGreen(pixel,0)
setBlue(pixel,0)
more like it; just two red
show(myPicture) hot spots
Intro to Robots
Find the Center of Redness:

Simple Technique: Find the average x-coordinate of all

white pixels in the filtered picture.
This will point to the center of redness, which may
unfortunately be somewhere in between two major red
(now white) areas

average x-coordinate

Intro to Robots
Center of Redness of our Photography

def AverageXofWhitePixels(picture):
sumX = 0.0 # We use floating point values.
counter = 0.0 # We use floating point values.
for xPos in range(0, getWidth(picture) ):
for yPos in range(0, getHeight(picture) ): Vertical line to one side because
pixel = getPixel(picture,xPos,yPos) redness of the apple shadow is
value = getGreen(pixel) pulling the red center to the right.
if value > 0 :
# add x-coord of every white pixel
sumX = sumX + xPos
counter = counter + 1
averageX = sumX / counter
return int(averageX) #Return an Integer

myPicture = takePicture()
picture = copyPicture(myPicture)
picture = findRedAreas(picture)
XposAvg = AverageXofWhitePixels(picture)
drawVerticalRedLine(myPicture,XposAvg)
show(myPicture)

Intro to Robots
Processing Red Areas Individually:

Call each red area a blob.

Our problem is:
find a blob
count its pixels
calculate its x-coordinate center
Easiest way to traverse the picture is
for pix in getPixels(picCopy):
...

two blobs one blob

Intro to Robots
Red Blobs one-at-a-time:

Problem: Processing the pixels left-to-right, top-to-

bottom we wont know when we process the line
indicated by the red arrow if the blobs are two or one.
Solution: Keep a list of partially-built blobs and as we
find that two blobs are really one, combine them into
one.
no
processing
continue
longer processing
processing on
aanew
same
newblob;
blob
blob
or is it the same blob?

two blobs one blob

Intro to Robots
BlobList Data Structure:

A BlobList is a data structure that contains a list of blobs

and has many methods for accessing that list.
from blob import * def dropBlob(self, b):
i = self.getIndex(b)
class BlobList: del self.contents[i]
def __init__(self):
self.contents = [ ] def writeOut(self):
for b in self.contents:
def add(self,b): b.writeOut()
self.contents.append(b) print ""

def findBlob(self,c): def getBiggestBlob(self):

for b in self.contents: maxNdx = -1
if b.inBlob(c): maxSz = 0
return b for i in range(len(self.contents)):
return None b = self.contents[i]
size = b.getSize()
def getIndex(self,b): if size > maxSz:
for i in range(len(self.contents)): maxSz = size
if b == self.contents[i]: maxNdx = i
return i return self.contents[maxNdx]
return -1
Intro to Robots
Blob Data Structure:

A Blob is a contiguous red section of a filtered photo. All

pixels are connected either vertically or horizontally.
A blob is built by finding a red point, not already part of
another blob and adding it and all the connected red
points on the same vertical line to the same blob.

As the point, x, is added

to a new blob, we continue
x . to add all the red pixels
on the same vertical line as
the point, x, and connected
to x

Intro to Robots
Complicated Blobs

x . these points are part of the

A initial blob containing x

these points are not part of

once the line labeled A is part of the
same blob as x, the extension of the
vertical line containing x will also
become part of the same blob
the initial blob containing x
because although on the same
vertical line, the two lines are
not connected by red pixels
eventually the entire donut will be
a single blob

Intro to Robots
 total blob
How to Build a Blob: in picture

pixels added
to program
blob
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . 2.nd . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
1st
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
Which is the last pixel to be added to the program blob?

Intro to Robots
Blob Strategy

Each time we add a pixel to a blob we give it a color that

is not red and not black and not used in any other blob.
combined blob
previous blob new blob
select a red pixel
create a new empty blob with new color
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
. . . . .
combine the two blobs
if pixel to the right is already in another blob:
. . . . .
combine the two blobs
get the pixel immediately above the previous pixel . . . . .
select the pixel immediately below the original red pixel
while pixel is red: . . . . .
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
. . . . .
if pixel to the right is already in another blob:
combine the two blobs
. . . . .
get the pixel immediately below the previous pixel

Intro to Robots
 Blob Code: .....
.....
from myro import * .....
class Blob:
.....
xCoordinates 12 13 14

vertcialCounts = {12:2, 13:3, 14:1}

def __init__(self, c=None,x=None):
self.colors = [ ]; self.blobPixelSum = 0 ### blob consists of a list of colors in use
self.verticalCounts = { }; self.midPoint = -1 ### and a dictionary of vertical counts
if c != None: ### key = xCoord; value = number yPixels
self.colors.append(c)
if x != None:
self.verticalCounts[x] = 0

def incPixelCount(self,xCoordinate,cnt):
self.blobPixelSum = self.blobPixelSum + cnt
if self.verticalCounts.has_key(xCoordinate): ### xCoord already belongs to blob
self.verticalCounts[xCoordinate] = ### handles the donut case
self.verticalCounts[xCoordinate] + cnt
else:
self.verticalCounts[xCoordinate] = cnt

def inBlob(self,c):
return c in self.colors ### tests if a color is already in the blob; if the same color
### is found in two blobs, they are the same blob

Intro to Robots
Blob Code:

def mergeDictionaries(self,b):
newD = { }
d1 = self.verticalCounts ### {x1:c1, x2:c2, , xn:cn}
d2 = b.verticalCounts
for d in d1.keys():
newD[d] = d1[d]
if d2.has_key(d):
newD[d] = newD[d] + d2[d] ### add vertical counts if xCoord in both blobs
del d2[d]
for d in d2.keys(): ### get the remaining xCoords not gotten in first loop
newD[d] = d2[d]
return newD

def combine(self,b):
newB = Blob()
newB.colors = self.colors + b.colors ### combine disjoint color lists
newB.blobPixelSum = self.blobPixelSum +
b.blobPixelSum
newB.verticalCounts = self.mergeDictionaries(b) ### combine vertical counts
return newB

Intro to Robots
Blob Code:

def calculateMidPoint(self):
sum = 0
for x in self.verticalCounts.keys(): ### same calculation as the single blob problem
sum = sum + x*self.verticalCounts[x]
self.midPoint = sum/self.blobPixelSum

def getMidPoint(self):
if self.midPoint == -1:
self.calculateMidPoint()
return self.midPoint

def getSize(self):
return self.blobPixelSum

def writeOut(self):
print self.colors
print "sum of pixels: ", self.blobPixelSum
self.calculateMidPoint()
print 'midPoint ' , self.midPoint
print 'color list', self.colors

Intro to Robots
Generate New Colors:

Our filtered picture is all black (0,0,0) or red (255,0,0).

We have 224 2 other colors to choose from.
More than enough for the pictures we will process.
from myro import *

class AvailableColors:
generates blue-green colors
def __init__(self):
on the fly in the ranges:
self.g = 100; self.b = 0
100 <= green <= 255
0 <= blue <= 255
def getNext(self):
approximately 37500 different
g = self.g; b = self.b
colors
self.b = self.b + 1
if self.b > 255:
self.g = self.g+1
self.b = 0
return Color(0,g,b)

Intro to Robots
Alternate AvailableColors Implementation:

from myro import *

class AlternativeAvailableColors:
def __init__(self): creates a complete lost of colors
self.contents = [ ] up front and then returns these
self.index = 0 colors one at a time as requested
for i in range (100,256):
for j in range(0,256):
self.contents.append(Color(0,i,j))

def getNext(self):
c = self.contents[self.index]
index = index + 1
return c

Intro to Robots
Data/Code Comparisons:

Both AvailableColors and AlternativeAvailableColors

have exactly the same interface:
ac.getNext()

But their implementations are very different.

AvailableColors has almost no data (variables g and b)
and a complicated algorithm for getNext().
AlternativeAvailableColors has a complicated data
structure (self.contents, a list of all available colors) but a
very simple implementation of getNext().
Lesson to Learn: There is a trade-off between data
structure and algorithm; if one is simple the other is
usually complex.

Intro to Robots
 Main Program:
leftPixel
rightPixel
rightPixel = ==
ac= getPixel(pictureCopy,x-1,yNdx)
getPixel(pictureCopy,x-1,yNdx)
getPixel(pictureCopy,x+1,yNdx)
getPixel(pictureCopy,x+1,yNdx)
AvailableColors()
clr
yNdx y+1= ac.getNext()
=yCount = yCount
ififwhile
bl getRed(leftPixel)
=getRed(rightPixel)
if getRed(rightPixel)
BlobList()
x =yNdx
getX(pix)
>= 0 and 0+
==== yNdx
==and
00and
and- y -1
picHeight
yNdx
yNdx
while =
==yNdx getHeight(pictureCopy)
yNdx
yNdx +<- 1picHeight
1
b.incPixelCount(x,yCount) and> >0:>0:0:
getGreen(leftPixel)
getGreen(leftPixel)
getGreen(rightPixel)
getGreen(rightPixel)
y<=picHeight:
getY(pix)
getRed(pixel) > maxRed: > 0:
if pixel
yNdx
pixel = =getPixel(pictureCopy,x,yNdx)
getPixel(pictureCopy,x,yNdx)
getRed(pixel)
bl.add(b) > maxRed:
for pixleftColor
leftColor
rightColor
rightColor
bin == getColor(leftPixel)
getColor(leftPixel)
==getColor(rightPixel)
= Blob(clr,x) getColor(rightPixel)
getPixels(pictureCopy):
setColor(pixel,clr)
pixel
if
ifsetColor(pixel,clr)
not=
not getPixel(pictureCopy,x,yNdx)
ifpixelb.inBlob(leftColor):
b.inBlob(leftColor):
notb.inBlob(rightColor):
b.inBlob(rightColor):
redValue ==pix
getRed(pix)
oldB
if redValueoldB=>=bl.findBlob(leftColor)
bl.findBlob(rightColor)
bl.findBlob(rightColor)
maxRed:
for each pixel
bb==b.combine(oldB)
b.combine(oldB)
if pixel is red:
bl.dropBlob(oldB)
bl.dropBlob(oldB)
create a new empty blob and new color
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
if pixel to the right is already in another blob:
combine the two blobs
get the pixel immediately above the previous pixel
select the pixel immediately below the original red pixel
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
if pixel to the right is already in another blob:
combine the two blobs
get the pixel immediately below the previous pixel
add blob to blob list

Intro to Robots