You are on page 1of 54

Vision for Automation

Ranga Rodrigo
Department of Electronic and Telecommunication Engineering
University of Moratuwa
Sri Lanka
ICIAfS 2008 Vision for Automation Workshop
December 10, 2008
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 1/43
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 2/43
Introduction
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 3/43
Introduction
What Is Computer Vision?
The goal is the emulation of the visual capability
of human beings using computers.
In other words, computer vision is making the
machine see as we do!
It is challenging.
Steps:
1
Image acquisition
2
Image manipulation
3
Image understanding
4
Decision making
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 4/43
Introduction
What Is Computer Vision?
The goal is the emulation of the visual capability
of human beings using computers.
In other words, computer vision is making the
machine see as we do!
It is challenging.
Steps:
1
Image acquisition
2
Image manipulation
3
Image understanding
4
Decision making
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 4/43
Introduction
What Is Computer Vision?
The goal is the emulation of the visual capability
of human beings using computers.
In other words, computer vision is making the
machine see as we do!
It is challenging.
Steps:
1
Image acquisition
2
Image manipulation
3
Image understanding
4
Decision making
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 4/43
Introduction
What Is Computer Vision?
The goal is the emulation of the visual capability
of human beings using computers.
In other words, computer vision is making the
machine see as we do!
It is challenging.
Steps:
1
Image acquisition
2
Image manipulation
3
Image understanding
4
Decision making
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 4/43
Introduction
Main Driving Technologies
Signal processing.
Multiple view geometry [2].
Optimization.
Machine learning.
Hardware and algorithms.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 5/43
Applications
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 6/43
Applications
Applications
1
Automotive:
Lane departure warning systems.
Head tracking systems for drowsiness detection.
Driver assistance systems.
Reading automobile license plates, and trafc
management.
Photography:
In camera face detection [6], red eye removal, and
other functions.
Automatic panorama stitching [1].
1
(From http://www.cs.ubc.ca/spider/lowe/vision.html)
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 7/43
Applications
Applications
Movie and video (a very big industry):
Augmented reality.
Tracking objects in video or lm and solving for 3-D
motion to allow for precise augmentation with 3-D
computer graphics.
Multiple cameras to precisely track tennis and cricket
balls.
Human expression recognition.
Software for 3-D visualization for sports
broadcasting and analysis.
Tracking consistent regions in video and insert
virtual advertising.
Tracking for character animation.
Motion capture, camera tracking, panorama
stitching, and building 3D models for movies.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 8/43
Applications
Camera Tracking
Source: http://www.2d3.com/capability
Show 2d3 video.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 9/43
Applications
Applications
Games:
Tracking human gestures for playing games or
interacting with computers.
Tracking the hand and body motions of players (to
control the Sony Playstation).
Image-based rendering, vision for graphics.
General purpose:
Inspection and localization tasks, people counting,
biomedical, and security. etc.
Object recognition and navigation for mobile
robotics, grocery retail, and recognition from cell
phone cameras.
Laser-based 3D vision systems for use on the space
shuttles and other applications.
Image retrieval based on content.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 10/43
Applications
Applications
Industrial automation (a very big industry):
Vision-guided robotics in the automotive industry.
Electronics inspection systems for component
assembly.
Medical and biomedical (maturing):
Vision to detect and track the pose of markers for
surgical applications, needle insertion, and seed
planting.
Teleoperations.
Quantitative analysis of medical imaging, including
diagnosis such as cancer.
Security and biometrics (thriving):
Intelligent video surveillance.
Biometric face, ngerprint, and iris recognition.
Behavior detection.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 11/43
Applications
Minimal Invasive Surgery
Source: http://www.davincisurgery.com/surgery/system/index.aspx
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 12/43
Applications
Areas of Advancement
Hardware.
Image segmentation.
3-D reconstruction.
Object detection.
Navigation.
Scene understanding.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 13/43
Applications
Areas of Advancement
Hardware.
Image segmentation.
3-D reconstruction.
Object detection.
Navigation.
Scene understanding.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 13/43
Vision in Automation
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 14/43
Vision in Automation
Whats
needed?
cameras
software
actuators
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 15/43
Vision in Automation
Cameras
Camera, and a frame grabber.
IEEE 1394 or USB cameras.
Ethernet cameras.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 16/43
Vision in Automation
Source: http://www.ptgrey.com/products/chameleon/index.asp
Source: http://www.matrox.com/imaging/products/vio/home.cfm
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 17/43
Vision in Automation
Source: http://www3.elphel.com/sites/default/les/images/nc353 io geo.jpeg
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 18/43
Vision in Automation
Inspection
Color
Barcode scanning
Character recognition
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 19/43
Vision in Automation
Inspection: Examples
Defects in parts, measurement of size.
Robotic bin picking.
If each slot is lled in a carton of pills.
Character recognition.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 20/43
Vision in Automation
Visual Servoing
Uses vision in the servo loop [3].
Dynamic look and move needs the accuracy of the
vision sensor and robot end-effector.
Having visual feedback in the control loop increases
the overall accuracy of the control loop.
Visual Servoing
Machine vision can provide closed-loop position
control for a robot end-effectorthis is referred to as
visual servoing.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 21/43
Vision in Automation
Visual Servoing
Uses vision in the servo loop [3].
Dynamic look and move needs the accuracy of the
vision sensor and robot end-effector.
Having visual feedback in the control loop increases
the overall accuracy of the control loop.
Visual Servoing
Machine vision can provide closed-loop position
control for a robot end-effectorthis is referred to as
visual servoing.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 21/43
Vision in Automation
Visual ServoingCamera Conguration
End-effector mounted
Fixed conguration
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 22/43
Vision in Automation
Servoing Architectures
Is the control structure hierarchical, with the
vision system providing set-points as input to the
robots joint level controller, or does the visual
controller directly compute the joint-level inputs?
Is the error signal dened in 3-D (task space)
coordinates, or directly in terms of image
features?
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 23/43
Vision in Automation
Servoing Architectures
Dynamic position-based look-and-move
structure.
Dynamic image-based look-and-move structure.
Position-based (direct) visual servo structure.
Image-based (direct) visual servo structure.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 24/43
Vision in Automation
Dynamic Position-Based
Look-and-Move Structure
c
x
d
Cartesian
control law

+
Joint controllers and power amps
Image feature
extraction
Pose
estimation
video f
c

x
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 25/43
Vision in Automation
Dynamic Image-Based
Look-and-Move Structure
f
d
Feature space
control law

+
Joint controllers and power amps
Image feature
extraction
video f
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 26/43
Vision in Automation
Position-Based (Direct) Visual Servo
Structure.
c
x
d
Cartesian
control law

+
Power amps
Image feature
extraction
Pose
estimation
video f
c

x
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 27/43
Vision in Automation
Image-Based (Direct) Visual Servo
Structure.
f
d
Feature space
control law

+
Power amps
Image feature
extraction
video f
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 28/43
Software Tools
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 29/43
Software Tools
Software Tools
Octave or Matlab.
C or C++ with a library such as OpenCV.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 30/43
Software Tools
Image Processing using Octave or
Matlab
Simple and quick.
A lot of library functions.
Interpreted.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 31/43
Software Tools
Octave Examples
Image reading and writing.
Histograms.
Filtering.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 32/43
Software Tools
Image Processing using OpenCV
Power of C++.
Well coded.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 33/43
Software Tools
OpenCV Examples
1
Image reading and writing.
2
Edge detection.
3
Template matching.
4
Capturing video.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 34/43
Examples of State-of-the-Art
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 35/43
Examples of State-of-the-Art
Segmentation Using Graph Cuts [5]
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 36/43
Examples of State-of-the-Art
Segmentation Using Graph Cuts [5]
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 36/43
Examples of State-of-the-Art
Segmentation Using Graph Cuts [5]
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 36/43
Examples of State-of-the-Art
Segmentation Using Graph Cuts [5]
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 36/43
Examples of State-of-the-Art
3-D Reconstruction
Can we obtain a 3-D view of a scene, given only
a set of (2-D) images?
Yes. Using multiple view geometry, we can
reconstruct a scene.
Show Leibe et al. video [4].
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 37/43
Examples of State-of-the-Art
3-D Reconstruction
Can we obtain a 3-D view of a scene, given only
a set of (2-D) images?
Yes. Using multiple view geometry, we can
reconstruct a scene.
Show Leibe et al. video [4].
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 37/43
Examples of State-of-the-Art
Object Detection: Face Detection
Show OpenCV sample.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 38/43
Examples of State-of-the-Art
Navigation: Sanfords Robot Stanley
Show video.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 39/43
Summary
Outline
1
Introduction
2
Applications
3
Vision in Automation
4
Software Tools
5
Examples of State-of-the-Art
6
Summary
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 40/43
Summary
Conclusion
Vision-based automation is promising.
Solutions are simple in a controlled
environment.
State-of-the-art is very interesting.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 41/43
Summary
Conclusion
Vision-based automation is promising.
Solutions are simple in a controlled
environment.
State-of-the-art is very interesting.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 41/43
Summary
Conclusion
Vision-based automation is promising.
Solutions are simple in a controlled
environment.
State-of-the-art is very interesting.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 41/43
Summary
Thank you.
OpenCV examples, and Octave examples are here:
http://www.ent.mrt.ac.lk/ ranga/publications.html
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 42/43
Summary
Matthew Brown and David Lowe.
Recognising panoramas.
In Proceedings of the 9th International Conference on Computer Vision, pages
12181225, Nice, France, October 2003.
Richard Hartley and Andrew Zisserman.
Multiple View Geometry in Computer Vision.
Cambridge University Press, 2nd edition, 2003.
Seth Hutchinson, Gregory D. Hager, and Peter I. Corke.
A tutorial on visual servo control.
IEEE Transactions Robotics and Automation, 12 No. 5:651670, 1996.
Bastian Leibe, Nico Cornelis, Kurt Cornelis, and Luc Van Gool.
Dynamic 3D scene analysis from a moving vehicle.
In Proceedings of the IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, pages 18, Minneapolis, MN, June 2007.
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake.
GrabCut: Interactive foreground extraction using iterated graph cuts.
ACM Transactions on Graphics: Proceedings of the 2004 SIGGRAPH Conference,
23(3):309314, August 2004.
Paul Viola and Michael Jones.
Rapid object selection using a boosted cascade of simple features.
In Proceedings of the IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, pages 511518, Hawaii, December 2001.
ICIAfS 2008 Robotics and Automation Workshop Vision for Automation 43/43

You might also like