USING PYTHON TO DEVELOP YOUR

VISION ALGORITHM

A simple line follower

Note: you must have seen previous webinar

about naoqi and choregraphe before seeing
this one

Alexandre Mazel
Explorer at Nao Labs
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
AGENDA FOR TODAY

NAO at a glance

Image Processing

Integration in a choregraphe behavior

Digression: porting to c++ and performance
comparison
NAO AT A GLANCE
NAO FOR EDUCATION

ALDEBARAN
400+ employees in 5 offices worldwide
Worldwide leader in B2A humanoid robotics
5000+ NAO sold over 40 countries
NAO
Platform for education and research purposes
Many contents for all school sectors & levels

5-part
curriculum
with 10 modules
comprehensive introducing
examples Programmin 15 activities for
g with NAO Secondary Education
&
18 activities for
Higher Education
NAO AT A GLANCE
WHAT CAN I DO?

I am the most advanced interactive system for

Academics
MOVE SENSE
25 Degrees of 2 HD camera
Freedom 4 microphones
Smooth and precise 8 Force Sensing
coreless motors Resistors
controlled by Inertial Center
software 2 Bumpers, 2 Sonars
Complex movement
capabilities
INTERACT THINK
2 loudspeakers
Intel Atom 1,6 GHz
Multiples LEDs
CPU
Tactile Sensors
1Gb Mb RAM
Prehensile
8 Gb Flash Memory
Hands
Software Suite
Infrared
Sensors
WIFI Connexion
NAO AT A GLANCE
SOFTWARE SUITE

Choregraphe Webots for Monitor

NAO

Graphical
Physical
Ergonomic Interface
Development of Simulation Engine to monitor actuators
Behaviors
Behaviors and sensors data

Ergonomic and Simulation and
user-friendly validation
Interface
SDK

Compilation and debugging tools

Access NAO with Matlab, Java, Python, C++, .NET, MS
Robotics Studio
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
AGENDA FOR TODAY -
oldies
A. Image Processing
1. Retrieve an image in the python world from NAO camera
2. Process the image
B. Integration in the complete behavior in choregraphe
3. Box and sub boxes
4. Tuning the right head position using video monitor
5. Omnidirectional Walk
6. Installing a time out
7. Various way to finish the behavior
C. Disgression
8. Efficiency compared to c++
9. How to port it to c++ (but why?)
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
AGENDA FOR TODAY

NAO at a glance

Image Processing

Integration in a choregraphe behavior

Digression: porting to c++ and performance
comparison
IMAGE PROCESSING IN PYTHON
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
NAOQI Vision Architecture
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
STANDARD C++ DEVELOPMENT

Write C++ module and method

Cross compile the module

Send it to your NAO

Relaunch Naoqi
See results

Debug
Cross compile the module
Send it to your NAO
Relaunch Naoqi
See results

Test
Cross compile the module
Send it to your NAO
Relaunch Naoqi
See results

Tune
Cross compile the module

Send it to your NAO

Relaunch Naoqi

See results
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
DEVELOPMENT IN CHOREGRAPHE

Write python
Run the code (press F5)

Debug
Run the code (press F5)

Test
Run the code (press F5)

Tune
Run the code (press F5)
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
RETRIEVING AN IMAGE: SUBS

def connectToCamera( self ):

try:
self.avd = ALProxy( "ALVideoDevice" )
strMyClientName = self.getName()
nCameraNum = 1
nResolution = 1
nColorspace = 0
nFps = 5
self.strMyClientName =
self.avd.subscribeCamera( strMyClientName,
nCameraNum, nResolution, nColorspace, nFps )
except BaseException, err:
self.log( "ERR: connectToCamera: catching error: %s!" % err )

def disconnectFromCamera( self ):

try:
self.avd.unsubscribe( self.strMyClientName )
except BaseException, err:
self.log( "ERR: disconnectFromCamera: catching error: %s!" %
err )
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
RETRIEVING AN IMAGE

def getImageFromCamera( self ):

"""
return the image from camera or None on error
"""
try:
dataImage = self.avd.getImageRemote( self.strMyClientName )

if( dataImage != None ):

Image = (
numpy.reshape(
numpy.frombuffer(dataImage[6], dtype='%iuint8' %
dataImage[2]),
(dataImage[1], dataImage[0], dataImage[2])
)
)
return image

except BaseException, err:

self.log( "ERR: getImageFromCamera: catching error: %s!" % err )
return None;
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING

img = self.getImageFromCamera();

img contains now a numpy buffer, you can use all numpy and OpenCV2
functions on it!

As a quick glimpse:

avg: give the average luminosity of an image

argmax: the most luminosity of an image
Slicing: reduce to a portion of the image
More details on: http://docs.scipy.org/doc/numpy/reference/
or
http://scipy-lectures.github.io/advanced/image_processing/

threshold: focus on some points from the image

filter2D: apply filter to detect stuffs
And also erode, region, feature detection...
More details on http://docs.opencv.org/trunk/doc/py_tutorials/py_tutorials.html
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
AGENDA FOR TODAY

NAO at a glance

Image Processing

Integration in a choregraphe behavior

Digression about performance
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING A LINE FOLLOWER

a simple vision extractor:

NAO will see the white line position and
orientation and then walk along it.
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING A LINE FOLLOWER

Step 1: Apply a filter to detect vertical line: OpenCV2.filter2D using a [-1,2,-1]

kernel.
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING A LINE FOLLOWER

Step 2: Threshold the picture using OpenCV2.threshold, the image becomes a

B&W image.
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING A LINE FOLLOWER

Step 3: Find the abscissa of the first white pixel using numpy.argmax
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING A LINE FOLLOWER

Step 4: Compute some rough approximation of an offset and orientation of

the line, using numpy.where and numpy.average.
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
PROCESSING THE METHOD

def detectLine( img, bVerbose = False ):

"""detect a line in an image, Return [rOffset, rOrientation] or [None,None], if no line detected
- rOffset: rough position of the line on screen [-1, +1] (-1: on the extrem left, 1: on the extrem right, 0: centered)
- rOrientation: its orientation [-pi/2,pi/2]"""

nWidth = img.shape[1]; nHeight = img.shape[0];

kernel = -numpy.ones((1,3), dtype=numpy.float);

kernel[0,1] = 2;

img = cv2.filter2D(img, -1, kernel);

# thresholding to remove low differential

retval, img = cv2.threshold( img, 45, 255, cv2.THRESH_TOZERO );

aMaxL = numpy.argmax(img, axis=1 );

aMaxLWithoutZeros = aMaxL[aMaxL>0];

if len( aMaxLWithoutZeros ) < 4:

print( "WRN: abcdk.image.detectLine: detected line is very short: %s" % aMaxLWithoutZeros );
return [None, None];

aNonZeroIdx = numpy.where(aMaxL != 0)[0]; # here we retravelling thru the list, it's not optimal (TODO: optimise!)
nFirstNonZero = aNonZeroIdx[0];
nLastNonZero = aNonZeroIdx[-1];
nHeightSampling = nLastNonZero - nFirstNonZero;

aLine = aMaxLWithoutZeros;

nSamplingSize = max( min(len(aLine) / 40, 8), 1 );

rTop = numpy.average(aLine[:nSamplingSize]); # first points
rMed = numpy.average(aLine[len(aLine)/2:len(aLine)/2+nSamplingSize]);
rBase = numpy.average(aLine[-nSamplingSize:]); # last points

rOrientation = ((rTop-rBase))/nHeightSampling; # WRN: here it could be wrong as the aLine has zero removed, so perhaps the top and bottom are not at top
or bottom !

return [(rMed/nWidth)*2-1, rOrientation];

INTEGRATION IN THE COMPLETE
BEHAVIOR IN CHOREGRAPHE
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
LINE FOLLOWER

Let's jump to choregraphe!
DIGRESSION
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
DIGRESSION port to c+
+
Python:
kernel = -numpy.ones((1,3), dtype=numpy.float)
kernel[0,1] = 2
img = cv2.filter2D(img, -1, kernel)

C++:
int nKernelRows = 1;
int nKernelCols = 3;
kernel = cv::Mat::ones( nKernelRows, nKernelCols, CV_32F );
cv::filter2D(img_GreyScale, img_GreyScale, -1 , kernel,

cv::Point( -1, -1 ), 0,
cv::BORDER_DEFAULT );
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
DIGRESSION port to c+
+

Python:
retval, img = cv2.threshold( img, 45, 255, cv2.THRESH_TOZERO )

C++:
// thresholding to remove low differential
cv::threshold( img_GreyScale, img_GreyScale, 45, 255,
cv::THRESH_TOZERO );
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
DIGRESSION port to c+
+

Python:
aMaxLWithoutZeros = aMaxL[aMaxL>0];

C++:
std::vector<int> aMaxLWithoutZeros;
aMaxLWithoutZeros.reserve( nHeight );
for( unsigned int i =0; i < nHeight; ++i)
{
val = aMaxL[i];
if (val > 0)
{
aMaxLWithoutZeros.push_back( val );
}
}
USING PYTHON TO DEVELOP YOUR VISION
ALGORITHM
DIGRESSION port to c+
+

Computation time on NAO V4 per images in

QVGA (320x240)

Python:
buffer conversion to numpy: 5ms
detectLines: 14 ms

C++:
buffer conversion: 0ms (none)
detectLines: 5 ms
 THANK YOU FOR YOUR ATTENTION !
Contact us for more info:

education@aldebaran-robotics.com

NAOAcademics

@NAOAcademics