Image Indexing and Retrieval: Multimedia Information Access

Image Indexing and
Retrieval
Lecture 8
Multimedia Information Access
Dr Crawford Revie
(with thanks to Prof Fabio Crestani)
Outline of lecture
z
z
z
Why do we require image retrieval?

Overview of techniques
Examples of systems
NB if you print these slides out then you will probably

want to use colour (or at least for some of them, 18-28 in
particular don't make a lot of sense in b/w!)
MIA Lecture 8
Crawford Revie (2006)
Motivation and Application Areas

z
Management of Image Archives

z
z
Art Galleries & Museums

WWW Image Indexing
Science Databases (Medicine, Astronomy,

Geography)
Industry specific
z
z
z
z
Trademark Databases
Textiles & Fabrics
Advertising
Architecture & Design
MIA Lecture 8
Approaches
There are 3 main approaches used in practice:
1. Keyword based
z
manual / semi-automatic / automatic
2. Based on visual properties

z automatic
3. Concept based
z mostly manual (still in 'research' mode)
MIA Lecture 8
Keyword approach: indexing

z
Images are annotated using keywords
But:
z
manual annotation is very expensive (as it is

exceedingly time consuming)
low level visual properties are almost impossible to
index consistently using manual mark-up
even for 'high level' properties manual annotation is
prone to subjectivity
Take a look at the Google Image Labeler 'game'

http://images.google.com/imagelabeler
MIA Lecture 8
Keyword approach: retrieval

z
Since image description is textual, we use an

almost straight-forward application of standard
IR techniques (stop-word removal, stemming,
indexing, etc.)
z
hypertext links have proved to be useful for retrieving

images ("retrieval by browsing")
thesauri and vocabularies can be more necessary
here than in standard IR (e.g. AAT) see also later
discussion of structured keywords and concept-based
retrieval
MIA Lecture 8
From manual to automatic

keyword assignment
z
Keywords can be assigned to an image by

analysing the text associated with the image
z
z
z
z
this includes the alt and caption attributes of the

<img> and <table> HTML tags
text elsewhere on a web page containing the image
text of the link pointing to the image
even the name of the file containing the image
MIA Lecture 8
Automatic keywords
Why are
these
pictures
retrieved?
MIA Lecture 8
Automatic keywords
Check the
text on
the web
page; the
caption;
and the
filename
MIA Lecture 8
Structured keywords: using a

database
Some systems
use a DBMS
to handle
keywords
and searches
MIA Lecture 8
10
Visually based approaches

z
Often referred to as Content Based Image

Retrieval (CBIR)
Similarity between query and documents is

calculated based on visual features:
z
z
z
colour
texture
shape
Visual features may be detected automatically

or semi-automatically
MIA Lecture 8
11
Feature vectors
z
Images are represented using a set of feature

vectors
If i = ( If i1 , K, If in )
Queries are represented with the same set of

feature vectors
Qf i = (Qf i1 , K , Qf in )
MIA Lecture 8
12
Vector features
z
Each feature has its own representation

z
range of values, variability, etc.
Feature vectors may provide a synthetic view

of a certain feature
In IR each word is represented by one feature

exactly, but here one image characteristic may
be represented by many features
z
similar to audio retrieval
MIA Lecture 8
13
Similarity functions
z
Need to choose similarity functions carefully:

z
should be a good approximation to human perception

of similarity between images
should have properties that help speed up
computations
Different types of similarity evaluations may

need to be combined to compare overall
similarity (e.g. shape, colour, texture, etc.)
MIA Lecture 8
14
Colour based retrieval

z
z
Arguably easiest; earliest to be used

Process is as follows:
z
represent image as a rectangular pixel raster (e.g.

1024 columns and 768 rows)
represent each pixel as a quantified colour (e.g. 256
colours ranging from red through violet)
count the number of pixels in each colour bin (this will
produce a vector representation)
compute vector similarity (e.g. using the normalised
inner product)
MIA Lecture 8
15
Colour based matching
Let's
compare
some
images
retrieved
using
keyword:
Godzilla
MIA Lecture 8
16
Colour histograms (for two samples)
MIA Lecture 8
17
Texture matching
z
Texture characterizes small-scale regularity

z
Described by several types of features

z
smoothness, periodicity, directionality, etc.
Match region size with image characteristics

z
colour describes pixels, texture describes regions
computed using filter banks, Gabor wavelets
Perform weighted vector space matching

z
usually in combination with a colour histogram
MIA Lecture 8
18
Texture matching
MIA Lecture 8
19
Image segmentation (shape)

z
Global techniques alone yield low precision

z
Segment at colour and texture discontinuities

z
e.g. in Berkeleys Blobworld we use ellipses
Represent relative position of objects

z
like flood fill in Photoshop
Represent size shape & orientation of objects

z
colour & texture are better at characterising objects, not full

images
e.g. angles between lines joining the centers
Segmentation allows us to perform object rotation and

scale-invariant matching
MIA Lecture 8
20
"Flood fill" in Photoshop
More sophisticated techniques are needed

MIA Lecture 8
21
CBIR systems: Examples

z
Commercial systems
z
z
Virage
QBIC
Academic
z
z
z
z
Blobworld
VisualSeek
Chabot
Viper
MIA Lecture 8
22
QBIC
You can sketch
an example of
what you are
looking for
MIA Lecture 8
23
QBIC
Can you spot

the similarity
to your 'query
definition'?
MIA Lecture 8
24
Berkeley Blobworld
Can you
spot the
similarity
to your
'query
definition'?
MIA Lecture 8
25
Berkeley Blobworld
MIA Lecture 8
26
Blobworld Segmentation (1)
MIA Lecture 8
27
Blobworld Segmentation (2)
MIA Lecture 8
28
Viper: query by example
Provide
examples
by indicating
relevant images
MIA Lecture 8
29
Viper (QBE)
Retrieves
similar
images
MIA Lecture 8
30
Concept based approach

z
Knowledge of the application domain is required

z
System assigns concepts (index terms) to part of the

image:
z
e.g. indexing of medical images requires knowledge of

medicine(!)
vocabulary + domain specific
automatic concept assignment: very imprecise and

ambiguous process
manual concept assignment: time-consuming and
highly subjective process
Few experiments, semi-automatic seems best so far
MIA Lecture 8
31
The future?
z
Simple image retrieval is commercially available

z
colour histograms, texture, limited shape information
Segmentation-based retrieval is still in the lab
Some way off:

z
automatic identification and recognition of objects in

images and videos
conceptual image retrieval
MIA Lecture 8
32

Image Indexing and Retrieval: Multimedia Information Access

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Image Indexing and Retrieval: Multimedia Information Access

Uploaded by

Copyright:

Available Formats

Image Indexing and

Why do we require image retrieval?

NB if you print these slides out then you will probably

Crawford Revie (2006)

Motivation and Application Areas

Management of Image Archives

Art Galleries & Museums

Science Databases (Medicine, Astronomy,

Crawford Revie (2006)

manual / semi-automatic / automatic

2. Based on visual properties

Crawford Revie (2006)

Keyword approach: indexing

Images are annotated using keywords

manual annotation is very expensive (as it is

Take a look at the Google Image Labeler 'game'

Crawford Revie (2006)

Keyword approach: retrieval

Since image description is textual, we use an

hypertext links have proved to be useful for retrieving

Crawford Revie (2006)

From manual to automatic

Keywords can be assigned to an image by

this includes the alt and caption attributes of the

Crawford Revie (2006)

Crawford Revie (2006)

Crawford Revie (2006)

Structured keywords: using a

Crawford Revie (2006)

Visually based approaches

Often referred to as Content Based Image

Similarity between query and documents is

Visual features may be detected automatically

Crawford Revie (2006)

Images are represented using a set of feature

Queries are represented with the same set of

Crawford Revie (2006)

Each feature has its own representation

range of values, variability, etc.

Feature vectors may provide a synthetic view

In IR each word is represented by one feature

similar to audio retrieval

Crawford Revie (2006)

Need to choose similarity functions carefully:

should be a good approximation to human perception

Different types of similarity evaluations may

Crawford Revie (2006)

Colour based retrieval

Arguably easiest; earliest to be used

represent image as a rectangular pixel raster (e.g.

Crawford Revie (2006)

Colour based matching

Crawford Revie (2006)

Colour histograms (for two samples)

Crawford Revie (2006)

Texture characterizes small-scale regularity

Described by several types of features

smoothness, periodicity, directionality, etc.

Match region size with image characteristics

colour describes pixels, texture describes regions

computed using filter banks, Gabor wavelets

Perform weighted vector space matching

usually in combination with a colour histogram