Image and Video Processing Using MATLAB

April 12, 2011 Rochester, NY

2011 The MathWorks, Inc. 1

Agenda

9:00 a.m. 9:15 a.m. 10:30 a.m. 10:45 a.m. 11:30 a.m. 12:00 p.m.

Welcome and MathWorks Overview Image Processing with MATLAB Break Video Processing with MATLAB System Design and Hardware Targeting Summary and Wrap-up

From MathWorks

Sr. Account Manager

Ed Donovan ed.donovan@mathworks.com 508-647-4251

Application Engineer
Stephanie Kwan

Agenda

9:00 a.m. 9:15 a.m. 10:30 a.m. 10:45 a.m. 11:30 a.m. 12:00 p.m.

Welcome and MathWorks Overview Image Processing with MATLAB Break Video Processing with MATLAB System Design and Hardware Targeting Summary and Wrap-up

Where is image processing used?

Common Image Processing Tasks

Perform segmentation Deblur and remove noise Measure image features Perform edge detection Apply morphological operations Enhance image contrast Perform image registration Apply transforms

Common Image Processing Challenges

Reading and writing to various file formats Visualizing images and intermediate results Testing algorithms with what-if scenarios Identifying causes of algorithm failure Processing large images with limited memory Executing algorithms faster

Technical Computing Tasks

Access
Files Data Analysis and Modeling

Explore and Discover

Share
Reporting and Documentation

Software

Algorithm Development

Outputs for Design

Code and Applications

Hardware

Application Development

Deployment

Automate
8

MATLAB A Platform for Technical Computing

Access
Files Data Analysis and Modeling

Explore and Discover

Share
Reporting and Documentation

Software

Algorithm Development

Outputs for Design

Code and Applications

Hardware

Application Development

Deployment

Automate
9

Core MathWorks Products

The leading environment for technical computing

The industry-standard, high-level programming language for algorithm development Numeric computation Data analysis and visualization Toolboxes for signal and image processing, statistics, optimization, symbolic math, and other areas Foundation of MathWorks products

10

Image Processing Toolbox

Perform image processing, analysis, visualization, and algorithm development

Image analysis Image enhancement Spatial transformation Image registration Morphological operations ROI-based processing Image display and exploration

11

Demo: Camera Pipeline

From Sensor Data to Image

Scene

Image

12

Demo: Camera Pipeline

1. Noise Reduction 2. Demosaic

3. Tone Mapping
4. White Balance 5. Gamma Correction

13

Image Acquisition Toolbox

Acquire images and video directly into MATLAB and Simulink

Configure device properties Perform background acquisition

Synchronize multimodal devices

Configure, acquire, and preview live video data using a graphical interface

14

Image Acquisition Toolbox Hardware Support

Industry standard support:

Frame grabbers

Analog Camera Link

DCAM compatible FireWire (IIDC 1394) GigE Vision Common OS webcam interfaces

Operating system support:

Windows Linux Macintosh
15

Image Acquisition Toolbox Hardware Support

Manufacturers include:
Allied Vision Technologies

Basler
Baumer DALSA FLIR

Hamamatsu
Lumenera Matrox Imaging National Instruments

PixeLINK
Point Grey Qimaging Sony

See Supported Hardware Pages for more information

16

Where is Large Imagery Used?

Satellite imagery Aerial surveys Super-resolution Image sequences and stacks Volumetric data Multispectral and hyperspectral Mosaics and panoramic imagery

17

Common Large Image Challenges

Out of memory errors Slow processing

18

Some Steps to Handle Large Images

Try memory command to see available space

Preallocate space for variables Minimize numerical data types Load small portions of an image Visualize only what is necessary Incorporate more hardware resources

Can lead to complex code

19

blockproc and rsetwrite

Block processing
Automatically divides an image into blocks for processing Reduces memory usage Processes arbitrarily large images

Reduced resolution data set

Avoids memory demands in visualizing large images

20

Demo: Cell Segmentation with blockproc

21

Improving Performance

MATLAB
Preallocate space for variables Identify bottlenecks with Profiler Vectorize code

Image Processing Toolbox

Many toolbox functions are faster as of R2010a

Parallel Computing Toolbox

Utilize multicore processors, clusters, and GPUs

22

Parallel Computing Toolbox

Going Beyond Serial MATLAB Applications

Worker

Worker Worker

TOOLBOXES

Worker Worker

BLOCKSETS

Worker
Worker

Worker

23

Parallel Computing enables you to

Larger Compute Pool

Larger Memory Pool

Speed up Computations

Work with Large Data

11 12 13 14 15 16 17 17 19 20 21 22 26 27 28 29 30 31 32 33 34 35 36 37 41 42 43 44 45 46 47 48 49 50 51 52

24

Distributing Tasks (Task Parallel)

Processes

Time

Time
25

Demo: Camera Pipeline and Parallel Computing

26

Parallel Computing Toolbox Support for GPUs

Automatically use GPUs with supported built-in MATLAB functions

conv2, filter2

Run supported MATLAB code on the GPU

arrayfun

Create kernals from existing CUDA code and PTX files Run kernals on the GPU from MATLAB

27

Why would you want to use a GPU?

Speed up execution of computationally intensive simulations For example:

Performance: A\b with Double Precision

28

Why use MATLAB for Image Processing?

Read and write many image file formats Visualize and explore images interactively Connect directly to cameras and frame grabbers Use a large library of built-in functions Quickly build custom image processing algorithms Block-process large images to avoid memory issues Process images faster with multiple cores and clusters

29

Agenda

9:00 a.m. 9:15 a.m. 10:30 a.m. 10:45 a.m. 11:30 a.m. 12:00 p.m.

Welcome and MathWorks Overview Image Processing with MATLAB Break Video Processing with MATLAB System Design and Hardware Targeting Summary and Wrap-up

30

Agenda

9:00 a.m. 9:15 a.m. 10:30 a.m. 10:45 a.m. 11:30 a.m. 12:00 p.m.

Welcome and MathWorks Overview Image Processing with MATLAB Break Video Processing with MATLAB System Design and Hardware Targeting Summary and Wrap-up

31

Common Video Processing Tasks

Convert between color spaces Enhance video contrast Perform segmentation De-interlace video frames Calculate running statistics Overlay text and graphics

32

Common Video Processing Challenges

Reading and writing to various file formats Managing buffers and intermediate storage Testing video algorithms with test cases Accessing standard video algorithms Visualizing results and intermediate steps Displaying text and graphics

33

Demo: Working with Video in MATLAB

Read in and write out data using VideoReader and VideoWriter

What happens if the video file is very long or the frame size is large?

34

System Objects

MATLAB objects that represent algorithms and I/O capabilities that process data frame by frame Instantiate and configure

reader = vision.MultimediaFileReader(viptraffic.avi)
step(reader)

Execute within a loop

Standard API

reset isDone close

35

Video Processing in MATLAB

myVid = VideoReader(myvideofile.avi); numFrames = myVid.NumberOfFrames; numIter = 10; opticalFlowIn = zeros([size(currentFrame) 5]); opticalFlowOutput = zeros([size(currentFrame) numFrames]); i = 1; Explicit state management while i <= numFrames opticalFlowIn(:,:,2:end) = opticalFlowIn(:,:,1:end-1); opticalFlowIn(:,:,1) = read(myVid,i); flow = opticalFlow(opticalFlowIn(:,:,1),opticalFlowIn(:,:,5), horn-schunck,numIter,magitude-squared);

opticalFlowOutput(:,:,i) = flow;
i = i+1; end implay(opticalFlowOutput,30)

Explicit indexing Need to maintain buffer

36

Video Processing with System Objects

reader = vision.VideoFileReader reader.Filename = myvideofile.avi; viewer = vision.DeployableVideoPlayer(framerate,30); optical = vision.OpticalFlow optical.Method = horn-schunck; optical.OutputValue = Magitude-squared; optical.ReferenceFrameDelay = 3; optical.MaximumIterationCount = 10;

Initialize objects

while ~isDone(reader) currentFrame = step(reader); OF = step(optical, currentFrame); step(viewer, OF); end

In-the-loop code is much simpler

Implicit states, buffering, and indexing

Video player runs in-the-loop

37

Demo: Car Detection Using Optical Flow Use optical flow to detect and count moving vehicles on a road

38

Motion Estimation

Motion estimation applications:

Object tracking Segmentation Interpolation Compression

Many methods include:

Optical flow Block matching Template matching

39

Demo: Interpolation with Block Matching Use block matching to perform motion estimation and interpolate video

40

Computer Vision System Toolbox

Design and simulate computer vision and video processing systems
Read and write multimedia files Display videos Overlay text and graphics Perform pre- and post-processing Perform motion-based processing Detect and track objects Recognize features Design computer vision systems

41

What is Computer Vision?

Automating the process of extracting information from images and video to understand a real world scene
Image Processing Remove noise Adjust contrast Measure Computer Vision Detect Identify Classify Recognize Track Interpretation

Pedestrian Bicyclist Truck Car Traffic violation Accident

42

Common Computer Vision Applications

Detect and track objects Automatic target recognition Count objects in a scene Classify or cluster objects Automatically register images Create mosaics and panoramas Perform stereo vision

43

Statistics Toolbox
Perform statistical analysis, modeling, and algorithm development

Principle components analysis K-means clustering Gaussian mixture models Nave Bayes classification K-nearest neighbor search Support vector machines Boosted decision trees
44

Why use MATLAB for video processing and computer vision?

Read and write to many video file formats Manage buffers, indexing, and states with System objects Integrate live data from cameras and frame grabbers Access algorithms for video processing and computer vision Overlay text and graphics annotations on video data Provides tools for designing computer vision systems

45

Agenda

9:00 a.m. 9:15 a.m. 10:30 a.m. 10:45 a.m. 11:30 a.m. 12:00 p.m.

Welcome and MathWorks Overview Image Processing with MATLAB Break Video Processing with MATLAB System Design and Implementation Summary and Wrap-up

46

Technical Computing Tasks

Access
Files Data Analysis and Modeling

Explore and Discover

Share
Reporting and Documentation

Software

Algorithm Development

Outputs for Design

Code and Applications

Hardware

Application Development

Deployment

Automate
47

From MATLAB to Implementation

Simulink & Stateflow

MATLAB Coder

MATLAB Compiler MATLAB Compiler

Simulink

Coder

Simulink HDL Coder

MATLAB Builder NE

MATLAB Builder JA

MATLAB Builder EX

.exe
.dll .exe .exe .lib CodeCOM.mex .NET C .dll

.dll .lib

COM

.NET

C Code

HDL Code

www www

Java Java

ExcelExcel Add-in Add-in

48

MATLAB Compiler

Automatic conversion of your MATLAB programs into standalone applications and shared libraries Support for the full MATLAB language and most toolboxes Royalty-free deployment

49

MATLAB Coder
Generate C and C++ Code from MATLAB Code

Maintain one design in MATLAB Design faster and get to C/C++ quickly Verify behavior of generated code Accelerate computationally intensive portions of MATLAB code Test systematically and frequently Spend more time improving algorithms in MATLAB Compatible with Computer Vision System Toolbox

50

Common MATLAB to C Tasks

Deploy MATLAB algorithms on Windows/Linux desktop PC Integrate MATLAB algorithms with existing C code Hand-off code to software engineers for embedded processor implementation Accelerate user-written MATLAB algorithms

51

Broad support for Features and Functions

Broad set of language features and functions/system objects supported for code generation.
Matrices and Arrays Data Types Programming Constructs Functions
MATLAB functions and subfunctions Variable length argument lists Function handles Supported algorithms > 400 MATLAB operators and functions > 200 System objects for Signal processing Communications Computer vision

Matrix operations N-dimensional arrays Subscripting Frames Persistent variables Global variables

Complex numbers Arithmetic, Integer math relational, and Double/singlelogical operators precision Program control Fixed-point (if, for, while, arithmetic switch ) Characters Structures Numeric classes Variable-sized data System objects

52

Demo: Camera Pipeline Revisited

Generate C code from MATLAB code

53

Targeting DSPs and FPGAs

MATLAB and Simulink
Algorithm development Debugging and profiling System design
Fixed-Point Modeling Generate code

Verify design Link to Embedded Software

Integrated Development Environment

Compiler Build automation tools Debugger

DSP/ FPGA
54

Demo: Camera Pipeline System Design

From Sensor Data to Image

Scene

Image

55

Why use MATLAB and Simulink for embedded hardware design?

Automatically convert MATLAB to C code Reduce the number of versions of code Simplify the design process Reuse MATLAB and legacy code in models Convert floating-point algorithms to fixed point Optimize algorithms for DSPs and FPGAs Test and validate an algorithms implementation

56

Agenda

9:00 a.m. 9:15 a.m. 10:30 a.m. 10:45 a.m. 11:30 a.m. 12:00 p.m.

Welcome and MathWorks Overview Image Processing with MATLAB Break Video Processing with MATLAB System Design and Hardware Targeting Summary and Wrap-up

57

What Youve Seen Today

MATLAB for image and video processing Image Processing Toolbox
Visualize, analyze, and process images

Image Acquisition Toolbox

Test image processing algorithms with live data

Computer Vision System Toolbox

Visualize, analyze, and process video Motion estimation, tracking, and stereo vision

Parallel Computing Toolbox

Process images faster using multiple core and clusters

MATLAB Coder
Convert MATLAB code to C/C++ code
58

Doheny Eye Institute Develops Next-Generation of Retinal Prosthesis with MathWorks Tools

Challenge
Develop next-generation, higher-resolution retinal prostheses
Illustration of a retinal prosthesis prototype.

Solution
Use MathWorks tools to develop, simulate, and automatically generate code for real-time image processing algorithms

With Video and Image Processing Blockset and Target Support Package, we rapidly prototype our image and video processing algorithms on the DM642 board. This

Results
Development time reduced from months to weeks DSP deployment streamlined Patient testing improved

can save me days or weeks of time.

Neha Parikh Doheny Eye Institute

59

Given Imaging Develops Camera-in-aCapsule Using MATLAB to Improve the Diagnosis of Gastrointestinal Disorders

Challenge
Create an alternative to endoscopy and other invasive gastrointestinal imaging procedures

Solution
Use MATLAB and companion toolboxes to develop and implement a swallowable video capsule

The PillCam SB 2 video capsule.

Results
Fast, efficient development Easy access to precise diagnostic information Improved patient care

With MATLAB, we simulated the intended system and fine-tuned it at the early stages of implementation, enabling us to develop critical engineering programs that met requirements on the first iteration.
Rafi Nave Given Imaging

Link to user story

60

Additional Articles and User Stories

Newsletter: Robot Soccer

Newsletter: JPL Mars rover

User Story: Dutch Epilepsy Clinics Foundation

Newsletter: Parking assistant at BMW

61

From a Leading Textbook Author ...

"I have used a number of commercial image processing packages over the years, and prefer the MathWorks Image Processing Toolbox for several reasons: the wide variety of functions it provides, the users ability to write additional functions with minimal effort, the quality of the software, and the high level of support."
Rafael C. Gonzalez Professor Emeritus University of Tennessee
62

Support and Community

63

Consulting Services
Accelerating return on investment
A global team of experts supporting every stage of tool and process integration

Process and Technology Standardization Full Application Deployment Component Deployment Advisory Services

Process Assessment

Jumpstart
Migration Planning

Research

Advanced Engineering

Product Engineering Teams

Supplier Involvement
64

Continuous Improvement

Process and Technology Automation

Training Services
Exploit the full potential of MathWorks products

Flexible delivery options:

Public training available worldwide Onsite training with standard or customized courses Web-based training with live, interactive instructor-led courses Self-paced interactive online training

More than 30 course offerings:

Introductory and intermediate training on MATLAB, Simulink, Stateflow, code generation, and Polyspace products Specialized courses in control design, signal processing, parallel computing, code generation, communications, financial analysis, and other areas
65

MATLAB Central

Community for MATLAB and Simulink users Over 1 million visits per month File Exchange
Upload/download access to free files including MATLAB code, Simulink models, and documents Ability to rate files, comment, and ask questions More than 12,500 contributed files, 300 submissions per month, 50,000 downloads per month

Newsgroup

Web forum for technical discussions about MathWorks products More than 300 posts per day
Commentary from engineers who design, build, and support MathWorks products Open conversation at blogs.mathworks.com
66

Blogs

Based on February 2011

Connections Program
More than 400 add-on products and services that complement and extend MathWorks products:

Specialized third-party toolboxes for MATLAB Interfaces to third-party software and hardware products Specialized training courses and consulting services System integrators and suppliers that incorporate MathWorks products

67

Book Program
More than 1200 books for educational and professional use, in 26 languages

Controls Signal processing Image processing Biosciences Communications Mechanical engineering

Mathematics Aerospace engineering Environmental sciences Chemistry Finance Electronics

68

Technical Support

Resources

Over 100 support engineers All with MS degrees (EE, ME, CS) Local support in North America, Europe, and Asia Comprehensive, product-specific Web support resources

High customer satisfaction

95% of calls answered within three minutes 70% of issues resolved within 24 hours 80% of customers surveyed rate satisfaction at 80100%

69

For More Information

Experiment with product by downloading a trial Peruse videos, webinars, user stories, and demos online Contact us
Talk to a sales representative to get answers to your questions Discuss your projects with MathWorks applications engineers

70