QuickTime™ and a

decompressor
are needed to see this picture.

Vision for robotics in LabVIEW:

from base concepts to application
examples

Ignazio E M Piacentini
ignazio.piacentini@imaginglab.it
www.imaginglab.it

Kuka Workshop, Rivoli, 20 May 2009

 QuickTime™ and a
decompressor
are needed to see this picture.

Agenda

The NI product offer

Basic concepts and requirements

A 'platform' for vision-based robotics applications

Generalising the application sw

Few application examples

Questions and answers

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 2

 QuickTime™ and a
decompressor
are needed to see this picture.

A long story: back in the early nineties …

Concept VI: a vision

library for LabVIEW in
the very early nineties
(France)

Acquired by NI
Corporate in 1996

Development of a full
line of Vision hw and
sw products in NI

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 3

 QuickTime™ and a
decompressor
are needed to see this picture.

NI LabVIEW
Embedded

Graphical development environment (FPGA)

PC-based systems to embedded Sensor
real-time targets
Handheld

Integrated measurements,
Wireless
analysis, connectivity
and control Networked I/O

Vision System

Reconfigurable Control I/O

Industrial Computer (PXI)

PC

Workstation

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 4

 QuickTime™ and a
decompressor
are needed to see this picture.

NI Vision Capabilities

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 5

 QuickTime™ and a
decompressor
are needed to see this picture.

NI Vision Platform

NI Vision Develoment NI Vision Builder for

Module Automated Inspection
Programming tools for LabVIEW, Configure, benchmark, and deploy
LabWindows/CVI, and MS Visual Studio without programming

NI Vision Acquisition Software

Acquire, save, and display images from 1000s of cameras

NI Compact PXI-based PC-based

NI Smart Cameras
Vision Systems Vision Systems Vision Systems

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 6

 QuickTime™ and a
decompressor
are needed to see this picture.

NI Vision Development Module

Programming libraries for
LabVIEW, (LabWindows/CVI,
Microsoft Visual Studio)

Hundreds of image
processing and machine
vision functions

Includes Vision Assistant
 Prototypes and benchmarks your
application
 Generates code for LabVIEW, Visual
Basic, and C

Think, solve, prototype

and then program!!
@Kuka, 20th May 2009 Vision for robotics in LabVIEW 7
 QuickTime™ and a
decompressor
are needed to see this picture.

Vision Builder for Automated Inspection

Ease of Use
 Interactive, menu-driven
vision application software
 Connect, configure & deploy
to CVS and Smart Cameras

Fast Development
 40+ powerful machine vision
tools
 Advanced decision-making
 Branching and looping
capabilities

Industrial Inspection
 Isolated digital and serial I/O
 Built-in industrial protocols
 Complete pass/fail
management

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 8

 QuickTime™ and a
decompressor
are needed to see this picture.

NI Frame Grabbers

PCI, PCIe, and PXI boards

Analog
 Standard and non-standard

Camera Link
 Base, medium and full config

IEEE 1394 control adaptors

 IEEE 1394a and 1394b

GigE Vision interfaces

 Optimized for NI software

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 9

 QuickTime™ and a
decompressor
are needed to see this picture.

NI Compact Vision System

Real-time, rugged & reliable

Multiple 1394 cameras (up to 16)

Industrial connectivity
 29 Digital I/O lines (TTL & Isolated)
 ENET, RS232, VGA

Built-in 256MB flash, 1394 HD

support

User-defined I/O w/ LabVIEW FPGA

Configure with Vision Builder AI, or

Program with NI LabVIEW

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 10

 QuickTime™ and a
decompressor
are needed to see this picture.

The NI Smart Camera Family

Multiple sensors, starting with
• Monochrome VGA and SXGA CCDs
Partial scan (½, ¼) and Binning

Direct Drive Lighting Control

• Strobe generation (5V TTL, 24V)
Powerful processing
• Current lighting controller • PowerPC 400/533 MHz
• DSP 720MHz coprocessor

Dual Gigabit Ethernet

• I/O extension
• Debugging
Industrial I/O
•Industrial communication • Opto isolated 2-in & 2-out
• RS232
• Encoder support

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 11

 QuickTime™ and a
decompressor
are needed to see this picture.

The vision platform: from Smart Cameras to

multicore PC

The hardware selection is

dictated by the application
needs:
 Conventional PC
 Industrial Panel PC
 PXI-based industrial PC (Win-based or
RT OS)
 Compact Vision System (Real Time
OS)
 Smart Camera (Real Time OS)

The software platform remains
the same!!

See Smart Cam videoclip

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 12

 QuickTime™ and a
decompressor
are needed to see this picture.

The NI Vision Difference

High-performance and ease-of-use
• High-level tools and low-level functions
• Single API for all NI Vision hardware
• LabVIEW and C code generation with Vision Assistant

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 13

 QuickTime™ and a
decompressor
are needed to see this picture.

The NI Vision Difference

Open platform for interoperability and connectivity
• Camera Link • GigE Vision
– NI PCIe-1429 (Full) – NI PCIe-8231
– NI PCIe-1430 (Dual Base) • Analog
– NI PCIe-1427 (Base) – NI PCI-1410 (4-ch mono.)
– NI PCI/PXI-1428 (Base/Medium) – NI PXI-1409 (4-ch mono.)
– NI PCI-1426 (Base) – NI PCI/PXI-1411 (1-ch color/mono.)
• IEEE 1394 – NI PCI-1405 (1-ch color/mono.)
– NI PCI/PXI-8252 (1394.a) • Parallel Digital
– NI PCI-8254R (1394.a) – NI PCI-1422 (RS422, LVDS)
– NI PCIe-8255R (1394.a & 1394.b) – NI PCI-1424 (RS422, LVDS, TTL)

NI Vision Acquisition Software

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 14

 QuickTime™ and a
decompressor
are needed to see this picture.

The NI Vision Difference: scalability

Scalable software / scalable hardware

From Smart Camera to PC (industrial or standard)

Configurable or programmable sw

Adapting hw and sw to the complexity of the solution PC-Based Vision
Systems
b VI EW
AI or La PXI Vision
de r
n Buil Systems
Visio Compact Vision
Systems

Smart Cameras

Size
Embedded PC-Based
Smart Cameras Vision Systems Vision Systems
@Kuka, 20th May 2009 Vision for robotics in LabVIEW 15
 QuickTime™ and a
decompressor
are needed to see this picture.

A quick demo in VBAI (Vision Builder for

Automated Inspection) and just a glance
at the Vision library in LabVIEW ….

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 16

 QuickTime™ and a
decompressor
are needed to see this picture.

The essence of vision-robotics integration:

the equivalent of LabVIEW 3-icon demo

Acquire the
image

Extract the
information

Download
info to
robot &
execute

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 17

 QuickTime™ and a
decompressor
are needed to see this picture.

Machine vision, GPM, and ….
We will not enter into

the details of machine
vision but ..

Geometric and
conventional pattern
matching need to be
complemented by other
checks

Parts may have different
orientations / views

It is necessary to
discriminate
recognizable / pickable /
conformal (QC) parts

Some inherent
complexity can be
removed with a ‘smart’
user interface

The end user must be
able to set pass / fail
criteria, vision and
robotics origin and
zeros

Auto-calibration tools
are a must
@Kuka, 20th May 2009 Vision for robotics in LabVIEW 18
 QuickTime™ and a
decompressor
are needed to see this picture.

Generalising the application

Putting all the 'tools'

together: Chameleon
& Vision Inspector

A 'general' application
for p&p in different
area

In-process inspection

Targeted to the end
user / operator (the
'customers' of the
machine
manufacturers

Remote support /
configuration service

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 19

 QuickTime™ and a
decompressor
are needed to see this picture.

A quick word on 3D imaging

3D imaging has a great potential in robotics applications

Available technologies: laser scanning, stereo-imaging, time of flight

ImagingLab is currently working on 3D and has recently released a
LabVIEW wrapper for the Ranger series of SICK-IVP cameras

Well known example: random bin-picking (as opposed to palletizing)

IL has developed a LabVIEW interface (under contract with SICK) for the
Ranger series cameras hand has a close collaboration with SICK-IVP

Acq e processing speed are an important issue

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 20

 QuickTime™ and a
decompressor
are needed to see this picture.

Few application
examples …

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 21

 QuickTime™ and a
decompressor
are needed to see this picture.

A complex application: testing DIMM memory

modules @ EEMS (ex Texas Instruments) (1’ 35’’
movie)

Fully automated functional testing

of DIMM memory modules

64 interchangeable motherboards/
256 memory slots

2 robots with common working
space (collision avoidance)

Full auto-guide / auto-learn / on-
board camera

Total testing cycle time
compatible with production line

Repetitive project

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 22

 QuickTime™ and a
decompressor
are needed to see this picture.

EST Projet (Educare per la Scienza e la Tecnologia):

teaching vision & robotics at the ‘Leonardo da Vinci’
Science Museum in Milan

Un progetto, anzi due, per il

Museo della Scienza di
Milano

Two ‘machines’ sponsored by NI,

Mitsubishi, IL

Recognising shape and colour,
assembling and disassembling

Interactive

Taking care of an audience of
variable age / designing for zero
or little maintenance

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 23

 QuickTime™ and a
decompressor
are needed to see this picture.

Flexible manufacturing / flexible feeding:

terminal blocks assembly

Components feeding / loading

Assembly

Highly modular

Flexible / riconfigurable feeding
station

Flex feeder: computer-controlled
vibration amplitude, direction and
frequency

One single software platform

Advanced tools for auto-calibration

Advanced tools for system
configuration

Quality control of the selected
parts

CAD img courtesy of Brazzale Automazione

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 24

 QuickTime™ and a
decompressor
are needed to see this picture.

A rapid feasibility study and prototyping

for spray painting of 'antique looking'
furniture components

Automatic part detection

Generation of trajectory
generation

Direct downloading to robot
controllers

Camera mounted on robot
arm: detection & guidance

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 25

 QuickTime™ and a
decompressor
are needed to see this picture.

The OPERA detector and 'bricks'

In total there are
206336 bricks, ~ 1.8 kton

spectrometer

2 x 31 Walls
(each containing
3328 bricks)

SM2

210.000 bricks: 125 SM1

x 100 x 74 mm

Alternate layers of Pb and photographic film

Special Pb with low residual radiation emission

Very high sensitivity film produced by Fuji
@Kuka, 20th May 2009 Vision for robotics in LabVIEW 26
 QuickTime™ and a
decompressor
are needed to see this picture.

Conveyor tracking in pharmaceutical

packaging

Dual line with 2 vision
systems / 4 DENSO
SCARA + 2 FANUC

Large selection of
different items

Curved transparent /
translucid plastic

Multiple pattern
matching / multiple
object views

……

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 27

 QuickTime™ and a
decompressor
are needed to see this picture.

Building an ‘abstraction layer’: how to handle

different brands of robots

An ‘action’ or command is essentially identical for different brands of
robots

A high level command (or program) common to ‘all’ robots is
translated via a robot-specific plug-in

Command High level LabVIEW robot protocol

Kuka Denso Mitsubishi Staubli XYZ

Protocol Protocol Protocol Protocol Protocol
…

Robot Controller

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 28

 QuickTime™ and a
decompressor
are needed to see this picture.

Questions &
answers …

@Kuka, 20th May 2009 Vision for robotics in LabVIEW 29