Workshop On Mobile Visual Search

R&D Centre Europe Visual & Sensing Division
www.uk.mitsubishielectric-rce.eu
WORKSHOP ON MOBILE VISUAL SEARCH Fingerprinting the Web with Image Signatures
Miroslaw Bober Paul Brasnett Mitsubishi Electric R&D Centre Europe December 2009
2009 R&D Centre Europe B.V. Visual & Sensing Division
Export Control : NLR
VIS09-PPT168
Fingerprinting the Web Why?
Who are her friends? Who holds the copyright? Who is she? What web-pages does she appear at? show me all! Is that a part of a larger image? Does she know Leonardo?
In which museum can I see this painting? Do I already have a copy of that image on my laptop?
I want to see other images of her!
What is her telephone number? Somebody is taking this painting across the border is that OK?
I want a full resolution picture of her!
VIS09-PPT168
Overview
Need for standardised Visual Signature Tools MPEG7 Image and Video signature tools Image Signature Requirements Evaluation & Development Process Design of Image Signature Performance Use Scenarios Conclusions
VIS09-PPT168
Need for standardised Visual Signature Tools
There is always a need to identify items

e.g. ISBN, barcodes, car license plates etc
There is a need to identify multimedia content Identifier standards exist e.g. International Standard Audio-visual Number However:
99.999% digital content exists without any identifier even if image content is registered, no automated search/identification is possible
Content Signatures can either

1. provide a link between content and its identifier number or, 2. form an identifier itself.
Image and Video signatures are based on analysis of content

2009 R&D Centre Europe B.V. Visual & Sensing Division Export Control : NLR VIS09-PPT168
MPEG7 Image and Video signature tools
Work started in January 2006, a part of ISO/IEC 15938-part 3 (MPEG-7 Visual) Tough requirements formulated July 2007 Call for proposals issued April 2009 Image Signature Standard completed Work on Video Signature close to completion
Region Shape
Colour Layout
Camera Motion
Boom up Track right Dolly forward Boom down Dolly backward Track left
Face Detection and Recognition
Dominant Colour
Edge Histogram
Homogeneous Texture
Colour Structure
Contour Shape
VIS09-PPT168
Image Signature Requirements
How many images on the Web? 100 billion?+ What does this mean for the requirements? False alarm rate 1 in 20,000,000 Matching speed 100 million+ per second on a typical PC (global)
General Requirements: Robustness Independence Fast Matching Fast Extraction Compactness Non-alteration of content Self-contained Coding Independence
Image Signature Requirements Partial matching (cropping) Heavy compression Scaling & aspect ratio change
VIS09-PPT168
Performance Evaluation
Two Datasets
Colour/grayscale photographs, logos & graphics of varying sizes Independence: 135,000 images, ~10 billion pairs Robustness: 10,000 images x 44 modifications
VIS09-PPT168
MPEG Evaluation & Development Process
Requirements and Evaluation Conditions
Call for Proposals Issued
Technologies Received
Standard Published
Improve Technology & Prepare Standard
Evaluations Performed & Technology Selected
VIS09-PPT168
Trace Transform
The trace transform is a generalization of the Radon transform The image is traced by straight lines along which a functional of the image is computed. Each line is characterised by two parameters; Trace transform is effectively the map of the values of the functional as a function of these parameters By choosing the functionals , one can construct features that are invariant to various image distortions, like affine transforms or illumination changes. A second functional (also known as diametrical functional) is applied to the columns of the trace transform producing 1D function called a circus function Problems of occlusion can be tackled with the help of local image kernels, a localized version of the Trace transform
VIS09-PPT168
How do we get rotational invariance?
3.85
x 10
x 10
3.8
2.5
3.75
2
3.7
3.65
3.6
3.55
FFT magnitude
1.5
3.5
0.5
3.45
0 0 10 20 30 40 50 60 70 80 90 100
3.4
100
200
300
400
500
600
x 10
0.5
-0.5
-1
-1.5
10
20
30
40
50
60
70
80
90
100
VIS09-PPT168
10
Example of rotational invariance
3.85 3.9 3.85 3.85 3.8 3.8 3.8 3.75 3.75 3.75 3.7 3.7 150 150 3.7 3.65 3.65 3.6 3.6 250 250 3.55 3.55 3.55 3.5 3.5 3.5 3.45 3.45 3.45
x 104 x 10
4 4
50 50
100 100
200 200
300 300
350 350
0 0
100 100
200 200
300 300
400 400
500 500
600 600
3.4 3.4 0 0
100 100
200 200
300 300
400 400
500 500
600 600
x 10
2.5
1.5
0.5
10
20
30
40
50
60
70
80
90
100
VIS09-PPT168
11
Multiresolution Trace Transform
We need to increase robustness Idea: create a multiresolution image representation by building a pyramid of trace transforms Fast can be done directly in trace domain Linear and angular dimension
VIS09-PPT168
12
Trace Cube
A three-dimensional (3D) representation of the image is generated in terms of the parameters d, and t This new representation, which is a 3D volume, is referred to as the trace-cube. The trace-cube carries significantly more information then the conventional trace transform, including information related to non-linear regions in the image, such as circles or ellipses.
VIS09-PPT168
13
Combining binary descriptors
Component binary strings are combined to create the global image signature 512 bits
+ FFT
512 bits
+ FFT
VIS09-PPT168
14
Global Image Signature
VIS09-PPT168
15
Local Image Signature
A scale-space representation of the image is constructed by low-pass filtering with Gaussian kernel. Features are detected by finding maxima in the location-scale space Two detectors are used to provide good detection of the location and scale of features modified scale corrected Laplacian of Gaussian (filter) scale-adapted Harris operator Image intensity gradients are calculated using the Scharr operator
The Laplacian of Gaussianis is applied over 12 levels in scale space The locations and scales with the highest value are chosen as potential features. Candidate features corresponding to a weak response from Harris operator are removed
16
Local Image Signature II

Between 32 and 80 feature points are selected Their location and quantised gradient direction are stored The global trace transform method is used to extract a binary feature signature from circular regions centered at feature points. Regions are extracted to sub-pixel level accuracy and have scaledependent radius, normalised to 32 pixels
VIS09-PPT168
17
MPEG-7 Image Signature Syntax
VIS09-PPT168
18
Local
levels Modified Items Brightness change Colour to monochrome conversion JPEG compression with varying Quality factors Colour reduction Gaussian Noise Histogram equalization Auto-levels Blur Simple Rotation Geometric transformation Scaling Flip AVERAGE
2009 R&D Centre Europe B.V. Visual & Sensing Division Export Control : NLR
Heavy 99.81
Medium 99.91 99.99
Light 100.00
Mean 99.91 99.99
99.69
99.95 99.34
100.00 99.86 99.91
99.88 99.60 99.51 95.41 99.95
99.04
99.58 95.41 99.95
99.98 100.00 100.00
99.99 100.00 100.00 100.00
99.98 100.00 100.00
99.98 100.00 100.00 100.00 99.48

VIS09-PPT168
19
Local + Global
levels Modified Items Brightness change Colour to monochrome JPEG compression with varying Quality factors Colour reduction Gaussian Noise Histogram equalization Auto-levels Blur
Heavy 99.97
Medium 99.98 99.99
Light 100.0 0
Mean 99.98 99.99
99.97
99.99 99.84
100.0 0 99.99 99.98
99.99 99.92 99.88 93.15 100.00
99.76
99.88 93.15 100.00
100.00
100.0
100.0
100.00
VIS09-PPT168
20
Local + Global
Rotation Simple Rotation Scaling Translation Geometric transformation Flip Aspect Ratio Change Crop Skew Perspective Combined (C,T,S) AVERAGE
99.99 100.00 100.00 75.64
100.0 100.0 100.0 91.85 100.00 100.0
100.00 100.00 100.00 99.42
100.00 100.00 100.00 88.97 100.00
100.00 99.48 100.00 99.85 99.85
100.00 82.11 98.27 98.36 99.26 97.77
54.95 94.86 96.15 98.33
91.89 99.96 99.08 99.60
VIS09-PPT168
21
Usage Scenarios - The missing Link?

Media Usage Monitoring
Tracking & recording multimedia usage Is somebody using my logo illegally?
Personal library management

Managing alternative versions of the same content Have I copied this MemoryStick to my PC?
Content linking
Inferring links between content based on analysis of media content
Metadata Association
Quickly and accurately obtain content metadata information
Copyright management
Royalty collection, brand management
22
Traditional Image Search keyword based
Wojciech Machowski
Index
VIS09-PPT168
23
Web Search
Approximately 1.5 million images from Flickr Several hundred random seed keywords used e.g.
oxide, rank, perplexity, mirrors, throne, walk, hens, seat etc.
Image Signature extracted and stored in database

24
Example Query
VIS09-PPT168
25
VIS09-PPT168
26
VIS09-PPT168
27
MPEG-7 Image Signature Tools

Global Signature
512 Bits per image False alarm rate of 0.05 parts per million 99.29% correct detection Up to 100 million images per second searching
Local Signature
Between 3,584 and 7,424 bits per image False alarm rate of 10ppm 98.57% correct detection on cropping, translation Over 100,000 images per second searching
ISO/IEC 15938-3:2002/Amd. 3 Image Signature Tools

Published: April 2009
VIS09-PPT168
28
Conclusions
MPEG-7 Visual Signature Tools
Standardised Multimedia Content Identification Robust, Low False Alarm, Fast Searching
Suitable for usage monitoring, copyright management, content linking, content metadata association MPEG-7 Image Signature Tools
Published April 2009
MPEG-7 Video Signature Tools

Expected To be Published July 2010
Maybe this is the missing link that you need for your killer application!
VIS09-PPT168
29
R&D Centre Europe Visual & Sensing Division
MERCE R&D
with an eye to the future!

thank you for your attention (m.bober@uk.merce.mee.com)
VIS09-PPT168
30

Workshop On Mobile Visual Search

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Workshop On Mobile Visual Search

Uploaded by

Copyright:

Available Formats

R&D Centre Europe Visual & Sensing Division

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Fingerprinting the Web Why?

I want to see other images of her!

I want a full resolution picture of her!

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Need for standardised Visual Signature Tools

There is always a need to identify items

Content Signatures can either

Image and Video signatures are based on analysis of content

MPEG7 Image and Video signature tools

Face Detection and Recognition

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Image Signature Requirements

Export Control : NLR

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

MPEG Evaluation & Development Process

Requirements and Evaluation Conditions

Call for Proposals Issued

Improve Technology & Prepare Standard

Evaluations Performed & Technology Selected

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

How do we get rotational invariance?

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Example of rotational invariance

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Multiresolution Trace Transform

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Combining binary descriptors

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Global Image Signature

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Local Image Signature

Local Image Signature II

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

MPEG-7 Image Signature Syntax

2009 R&D Centre Europe B.V. Visual & Sensing Division

Export Control : NLR

Medium 99.91 99.99

Mean 99.91 99.99

100.00 99.86 99.91

99.88 99.60 99.51 95.41 99.95

99.58 95.41 99.95

99.98 100.00 100.00

99.99 100.00 100.00 100.00