Digital In-Line Holographic

Microscope Algorithms for

Micro-Organisms Detection and
Three Dimension Tracking
Ahmed AYOUB, PhD
Szabolcs TOKES, PhD
ICENCO 2007
Computer and Automation Research Institute
Hungarian Academy of Science
Benha High Institute of Technology
Benha University
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Contents
Motivation
Principles of Digital Holographic Microscopy
Recent Results
Three-D Object Tracking
Future Work
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Motivation
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Holography
Recording
Reconstruction
Development
of photo plate
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Principle
Record the interference pattern
(hologram) by an optical sensor, e.g.
CCDor CMOS
Reconstruct using fast digital
algorithms
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Goal
Create a lensless microscope
Record 3D interference pattern
information
Several images of several depths can be
reconstructed digitally
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Digital Holographic Microscopy
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
The Optical Setup
Laser wavelength: 635m
CCD camera resolution: 10.6m (10241024)
Objective: several 3D objects
Pinhole size: 1m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
The Optical Setup
CCD camera
p=10.6m
N=1024
3D Object
Pinhole
Microlens
Laser source
(Red) 635nm
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
DHM Project
Experimental Setup
Reconstruction
Algorithm
Diffractive Optics
Recognition
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Lszl Orz, PhD
Digital Holography Project
Ahmed Ayoub, PhD
Pter Divs
Sndor Tth
Szabolcs Tks, PhD
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
System Calibration
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Fundamental Formulas
, )
2
1
2
2
1
L L
L
d
Np Np

| |
> + ~
|
\ .
, )
2
1
1 1
r
L L L
L
L
L L

= ~
2
2
min
1 1
1
2 2
r
p Np
L Np

| |
= ~
|
\ .
Resolution calculated from Numerical apprature
Relation between object-distance and
reconstruction distance
Nyquist criteria
laser wavelength
p CCD camera pixel pitch
N Number of Pixel in one row
a =N*p CCD row width
L CCD distance from Pinhole
L
1
Object distance from Pinhole
L
1max
Maximum allowed object distance (limited
by Nyquist criteria) from Pinhole
L
r
Reconstruction distance
L
rmin
Minimum allowed reconstruction distance
(limited by Nyquist criteria) from Pinhole
d reciprocal of the Resolution
M Magnification at the L1max plane
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Derived Formulas
2
1max
2
2L
L
Np
~
2
1max
1
2
L Np
M
L L
= ~
Allowed L
1max
as function of L
Magnification at L
1max
plane
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Recording Dimensions
L
L
1max
L
1
a
=
N
*
p
CCD plane
Maximum distance to
place an object!
Pinhole plane
Object plane
a side length of the CCD
N number of pixels
p pixel pitch
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Reconstruction Dimensions
Reconstruction zone
Recording zone
L
1
L
1max
L
L
r
L
2
3D Object
CCD plane
Pinhole plane
Reconstructed
image
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Fundamental Formulas
L1mi n as a functi on of L
p=10.6um, N=1024, =635nm
0
5
10
15
20
25
30
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
L1min [mm]
L

[
m
m
]
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Fundamental Formulas
Resolution as a function of L
p=10.6m, N=1024, =635nm
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
0 10 20 30 40 50 60
L [mm]
d

[
u
m
]
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Focus and Magnification Maps
The present in-line digital holographic microscope (DHM) shows a linear behavior
for the focus (small deviation because of human observation error) but an
exponential one for the magnification.
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Magnification Map
The present in-line holographic microscope (DHM) experimental magnification
map behaves similarly to its computed one.
2
1max
1
2
L Np
M
L L
= ~
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Recent Results
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Reconstruction Algorithm
Loading hologram image
matrix (H)
Loading reference beam
image matrix (R)
+=H-R
Convolve + with G
G Kernel function of reconstructing
plane wave Fourier transform
Focus Reached?
Save
reconstructed
image
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Reconstruction Program
Load
hologram
Hardware
settings
Reconstructio
n settings
status
Progress
image
Output
settings
Navigatio
n settings
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Result 1 - Test Pattern
Minimum Resolution
(Group 0, Element 1)
Maximum Resolution
Finest line thickness
2.2m
(Group 7, Element 6)
Group Number
Element Number
Original test pattern
1951 USAF resolution chart
www.edmundoptics.com
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Digital Focusing for Group 7, Element 6
Reconstruction
Resolution: 1024x1024
J une 13, 2006
Original test pattern
1951 USAF resolution chart
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Reconstruction
Resolution: 1024x1024
Microscope focus: 3.75mm
Time of reconstruction: 2sec/layer
Group 7, Element 6
(2.2m)
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Result 2 - Detecting Cat Cortex Neurons C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Detecting Cat Cortex Neurons
By our present DHM
Resolution: 1024x1024
J une 13, 2006
Cat brain neurons with optical
600x Magnification
Neurons
Synapses?
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
CNN-Like Processing
Blur
Diffuse Edges Contour
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Result 6 Eurytemora
Detected Eurytemora
Resolution: 1024x1024
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Video Reconstruction
1. Record a video of holograms series (B)
2. Decompose the holograms into a series of single ones (C)
3. Reconstruct every hologram at several depths (A)
4. Combine reconstructed images of the same reconstruction
depth in one output video (A)
i. enhance the reconstructed frames of video (C)
ii. Label the video date, resolution, wavelength,
reconstruction depth (C)
iii. Object detection step will be applied later to follow an
object moving in depth.
Tools:
A. our present DHM
reconstruction
algorithm
B. record.exe
C. VideoMach.exe
Decompose into
time frames
Time
Depth
Recorded video of
holograms
(4D information)
Series of Single
holograms
(3D contents)
Each set is a series of
reconstructed 2D images from
a single hologramat different
depths. Focusing in depth.
Pick up time series of
holograms
Reconstruct each
frame in depth
Combine frames at
certain depth
Reconstructed video at
certain depth
PAST
Future
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Video Reconstruction
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Three-D object Tracking
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Objectives
Short term objective
Track micro organisms moving in depth
Long term objective
Construct 3D model of micro organisms
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Frame#3 Frame#N Frame#2
Objectives
Frame#1
+
+
+
+
Time, frame
3D Object Tracking
4.5
5
5.5
6
6.5
7
7.5
8
1 3 5 7 9
1
1
1
3
1
5
1
7
1
9
2
1
2
3
2
5
2
7
2
9
3
1
3
3
Time, Frame
D
e
p
t
h
,

u
m
In-depth
reconstructions
Detected
object
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Algorithm
No
Yes
Load frame#1 as
previous frame(PF)
Start a counter, i,
withi=2
Load frame#i as
current frame(CF)
DF=CF-CP
Extract templateTF
fromCF
CG
j
=TFNF
j
ThresholdCG
j
s
FindNF
jmax
AddNF
jmax
to output
file
Set PF=NF
jmax
Set i=j
max
Last
frame?
Close output file
A
A
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Extract Largest Object Template
Template
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Extract Largest Object Template
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Correlate Template with Neighborhood
Correlate the extracted template with 9
neighboring frames (3 spatial x 3
temporal)
Spatial frames are those of the next 3 depth
reconstructions at certain time
Temporal frames are those of the next 3
frames in time at the same depth
The highest correlation peak shows where
the object had moved to
The detection process repeats
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Daphnia are members of a collection of
animals that are broadly termed as
"water fleas".
Get their name from their jerky
movement through the water.
Very sensitive to poor water conditions:
halide (e.g. chloride or fluoride)
metal ions (e.g. sodium, potassium,
magnesium and calcium)
dissolved toxins (e.g. copper and zinc)
Increased concentrations can cause
immobility and death.
Daphnia is often used to monitor water
quality
Biological sensor
Daphnia
Image of Daphnia using transmission diode
(courtesy: The Acadia Centre for Microstructural Analysis)
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Recent Results

Reconstructed and labeled image

of Daphnia using the present
algorithm.

Anatomy of Daphnia
(courtesy: Minnesota
department of natural
resources)
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Recent Results C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Challenges and Open Problems
Quality
Higher magnification factor (presently~500x)
Higher resolution (presently~2m)
Speed
Faster digital reconstruction algorithm
Dedicated hardware for real time applications
(FPGA is recommended)
Recognition and 3D modeling
Autofocus algorithm
CNN UM operations
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
General Applications
Protein Crystal Growth Monitoring
Charged Particle Trace Detection
Neurons Activity
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Other Workgroups Results
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Free-Drifting Digital Holographic Imaging System
Thin Layers of Harmful Algal Blooms
www.me.jhu.edu/~lefd/shc/HoloSub_site/Digital_Holosub.htm
Laboratory for Experimental Fluid Dynamics
Mechanical Engineering Department
J ohns Hopkins University, USA
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Free-Drifting Digital Holographic Imaging System
www.me.jhu.edu/~lefd/shc/HoloSub_site/Digital_Holosub.htm
Laboratory for
Experimental Fluid
Dynamics
Mechanical Engineering
Department
J ohns Hopkins University,
USA
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Copepod Nauplius goes after a dinoflagellate but the dino
escapes!
www.me.jhu.edu/~lefd/shc/HoloSub_site/Digital_Holosub.htm
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Dalhousie University
Dalhousie University, Canada
Department of Physics and Atmopsheric Science
Figure A a contrast hologram of a test pattern plate.
Figure B the reconstructed image of the plate
Figure C the image has been zoomed in on the central portion
The smallest 3 bars of the test pattern (near the right of figure C) are 11 microns across.
fizz.phys.dal.ca/~kreuzer/DIH.html
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
Woods Hole Oceanographic Institute
REMUS (Remote Environmental Monitoring Units) 100 is a compact, light-
weight, autonomous underwater vehicle designed for operation in coastal
environments up to 100 meters in depth. The REMUS 100 can be configured
to include a wide variety of standard and/or customer specified sensors and
systemoptions to meet your unique autonomous missionrequirements
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m
C
lic
k
to
b
u
y
N
O
W
!
P
D
F-XChang
e
w
w
w
.d
oc u-t r ac k
.c
o
m