Stereovision to Calculate Depth and 3D Map

College: Cummins College of Engineering for women

Guide: Mrs. Shubhangi V. Tikhe svtikhe@rediffmail.com

Team Members

Anagha Kulkarni anagha.kulkarni03@gmail.com

Amruta Mohod amrutamohod@gmail.com

Priya Rokade priya.rokade12@gmail.com

Dhanashree Shetty shetty.dhanashree@gmail.com

Abstract --- An important research topic in the field of image
processing is stereo vision. For acquisition of 3-Dimensional
information in real space, stereo vision system is suitable. Stereo
vision is the process of constructing a 3D model of a scene through
processing two 2D images of this scene. The main problem is the
construction of a disparity map. This is a map that describes which
points in the two images corresponds to the same point in the 3D
scene.
In the stereo system, 3D real world position is derived from
translation of coordinates between cameras and world. Thus, to use
stereo vision, it is needed to construct a precise system which
provides kinematically precise translation between camera and
world coordinate, in spite of intricacy and hardness. In our paper,
we propose an approach to develop a system which can easily
obtain 3D information by direct computation of position with
translation of coordinates, disparity in stereo pair of image which is
used to find the reference depth of objects. The algorithm is based
on correlation and uses the epipolar constraint.
Keywords -– Stereo Vision, Epipolar Geometry, Camera
Calibration, Rectification, Disparity
I. INTRODUCTION
Making a machine to see objects is one of the most
important tasks in artificial intelligence, manufacturing
automation, and robotics.
Computer Vision is one of the fastest growing areas
within Computer Science. Aided by rapid recent progress in
hardware and software design, computer vision projects are
making use of vast increases in processing and memory
capacities to enhance their performance. In order for computers
to effectively processes, segment and analyze visual input of
their environments it is often a requirement that the system is
able to obtain data of the surrounding world in a format that can
be easily equated to the actual environment in which the system
finds itself. In the case of many vision systems this could be a 3
dimensional representation of the real world. For humans this is
a task that we achieve quite naturally from an early age and it
soon becomes second nature for us to accurately judge distance,
perspective and space, however, when human vision systems are
analyzed it becomes apparent that the brain uses a multitude of
techniques to give us a sense of the three dimensional world in
which we live.
In order for a computer vision system to obtain depth
data from a scene it is possible to use a number of different
techniques. Three dimensional scene data can be obtained from
sources including object shading, motion parallax data,
structured light or laser range finders. However, perhaps the
most obvious technique is that of stereo vision. In a system
analogous to a pair of human eyes, the input to two cameras
observing the same scene can be analyzed and the differences
between the two images used to compute object depth and hence
a model of the scene that the system is viewing. The utilities of a
robust implementation of such a system are many and
potentially include applications in areas such as space flight,
face recognition, immersive video conferencing and industrial
inspection to name just a few.[7]
Stereo vision, which is inspired by human visual
process, computes the disparity between correspondence points
in images captured by multiple cameras for distance
measurement. In Robots, Visual interpretation is an extremely
challenging problem. Robot vision technology is needed for the
stable walking, object recognition and the movement to the
target spot.
Stereo vision of three dimensional spaces would make
robot have powerful artificial intelligence. Stereo Vision is an
important research topic in the field of image processing. The
problem is to compute a 3D model of a scene from two (or
possibly more) 2D images. Each pixel in these images is
represented by a number, corresponding to a grey level. In order
to construct a 3D model of a scene a disparity map is needed.
Such a map describes for each point in the source image where
to find the best corresponding point in the target image. From a
disparity map and the calibration information of the cameras
that generated the two images a 3D model of the scene can be
constructed.
Computational stereo vision involves inferring the 3-
dimensional depth of a scene from the small differences
between multiple views of the same scene. The most general
definition of computational stereo vision involves only two
views of the same scene. This is also the most difficult problem
in stereo vision, since the amount of available data is small
relative to multi-view stereo vision and hence the ambiguity of
the problem is high. With stereo vision, the disparity or image
distance d between matching points XL and XR in the left and
right images is inversely proportional to the depth z, with the
baseline distance b and focal length f the constant factors. The
concept of stereo vision itself is easily grasped, but the stereo
matching problem of finding the correct match is much more
ambiguous.
Real-time stereo vision has been widely used in
intelligent robot navigation, smart human-computer interaction,
intelligent surveillance, etc. The estimation of the disparity
between two images of the same scene is a long-standing issue
for the machine vision community [1]. Stereoscopic vision is
based on the principal, first utilized by nature, that two spatially
differentiated views of the same scene provide enough
information so as to perceive the depth of the portrayed objects.
Thus, the importance of stereo correspondence is apparent in the
fields of machine vision, virtual reality, robot navigation,
simultaneous localization and mapping [2], [3], depth
measurements and 3-D environment reconstruction. The two
alternatives for estimating disparity are either to precisely align
the stereo camera rig and then perform the demanded
rectification (leading to simple scan line searches),or to have
arbitrary stereo cameras setup and avoid any rectification
(performing searching throughout blocks). Accurately aligned
stereo devices are very expensive.
II. CAMERA CALIBRATION
Camera calibration is the process of relating the ideal model
of the camera to the actual physical device and of determining
the position and orientation of the camera with respect to a
world reference system. In order for it to be possible to
reconstruct a scene from a stereo image pair it is necessary that
several important properties about each of the cameras be
known. Obtaining values for these properties is known as
calibration. Depending on the model used, there are different
parameters to be determined. Techniques for camera calibration
loosely fall into three categories linear, non-linear and two-step.
Linear techniques assume a simple pinhole camera model and
do not account for lens distortion effects which turn out to be
“significant in most off-the-shelf charge coupled devices” .In
non-linear methods a relationship between parameters is
established and then an iterative solution is calculated through
minimizations. The parameters to be calibrated are classified in
two groups:
1) Internal (or intrinsic) parameters: Internal geometric and
optical characteristics of the lenses and the imaging device.
2) External (or extrinsic) parameters: Position and orientation
of the camera in a world reference system [7].
Calibration is important for accuracy in 3D reconstruction.
In particular it is a critical task for stereovision analysis.
Calibrating stereo cameras is usually dealt with by calibrating
each camera independently and then applying geometric
transformation of the external parameters to find out the
geometry of the stereo setting [6].
Methods for estimating camera parameters rely typically on
targeted test-fields and correspondences between targets and
their images on one or more frames. For multi-image
configurations, precise 3D test-fields can be replaced by simple
2D patterns, typically of a chess-board type. With unknown
exterior orientation (Fiala and Shu, 2005), a further advantage of
such patterns is the fact that their high contrast and regularity.
Several freely available algorithms exist for estimating interior
and exterior orientation parameters based on chess-board
patterns imaged from different points of view. Among
functional tools presented in this context, Bouquet’s Camera
Calibration implemented in C++ and included in the Open
Source Computer Vision library distributed by Intel is probably
best known. The initialization step includes manual pointing of
the four chessboard corners on all images and knowledge of the
number of nodes per row and column. Node locations can thus
be first approximated and then identified with sub-pixel
accuracy by a point operator [6].
If lens distortion is strong, one may need to supply
approximations for its coefficients. Initial values are provided
by the plane-based calibration algorithm, while an iterative
adjustment provides a final solution for camera and pose
parameters. This is achieved using image sets of typical chess-
board patterns (alternating light and dark squares of equal size),
which is the sole a priori information needed. Only those
extracted image points are kept which may be ordered in two
groups of lines referring to the main orthogonal directions on the
object plane. Due to the regularity of the pattern, establishing
point correspondences among views is then a trivial task,
although possibly involving object systems which differ in-
plane rotation and translations [6].
A. Camera calibration algorithm
1) Corner extraction: With parameters chosen after some tests,
the Harris corner operator with sub-pixel accuracy, made
available by Bouguet, is applied to grayscale images with
equalized histograms.
2) Point ordering: After extraction of feature points on an
image, the medians of their x, y coordinates are calculated.
These will normally indicate a point close to the centre of the
chess-board pattern. The median is preferred against the mean
value due to its lower sensitivity to the presence of ‘noisy
points’ outside the chess-board. The point selection and ordering
algorithm is initialized by choosing the closest feature point as
‘base point’ B. All extracted points around B in a window of
size equal to ⅓ of the image size are calculated and sorted
according to distance from it. Assuming that B is indeed a valid
node, the principal directions of the pattern must now be
identified by avoiding points not corresponding to pattern nodes
but also points on chess-board diagonals. The linear segment s
from ‘base point’ B to the nearest extracted point is formed.
Identification of the main directions succeeds by comparing the
grey values of the pixels on either side of s [6].
3) Point correspondences: The final outcome of this step is a set
of points characterized by the number of the respective chess-
board row and column with which they are associated. The
lower row appearing on an image is row 1 and is arbitrarily
associated with the object X-axis. The column to the far left is
column 1 and associated with the object Y-axis. Thus, the point
belonging to these two lines is point (1, 1) of this image and is
associated with the origin (point 1, 1) of the chess-board XY
system. If this point does not actually appear on an image or has
not been detected, the adjacent node detected on this image is
numbered accordingly, e.g. (2, 1) or (1, 2) etc. The process is
repeated for all images. Hence, thanks to the symmetric nature
of the pattern, it may be assumed that point correspondences
among frames as well as correspondences with the chess-board
nodes have been established. This provides an answer to the
problem of correspondences, which is seen as the most difficult
part in automatic camera calibration and is often solved
manually (Fiala and Shu, 2005) [6].
4) Initial value: Instead of the linear solution of plane-based
calibration, the approach adopted here relies on the two
principal vanishing points (VP) of the images. These are found
by line-fitting adjustment to nodes ordered in pencils of
converging lines. In each direction, initial estimates of VP
locations are obtained from the two lines with >3 points forming
the largest angle. If the distance of a VP from the image centre
exceeds by more than 40 times the size of the image format
(equivalent to a rotation angle ~1.5° for a moderately wide-
angle lens), this VP is considered at infinity. If both VPs are
finite, their locations are refined in a single adjustment, in which
coefficients of radial lens distortion are also included in the
unknowns. Using diagonals with >3 points, the VP of the
diagonal direction which falls between the two principal VPs is
also included as an unknown to enforce the vanishing line
constraint. In case one VP is finite, it is estimated from all
participating lines along with the radial distortion coefficients
[6].
B. Epipolar geometry
It used to the geometry of stereo vision. When two
cameras view a 3D scene from two distinct positions, there are a
number of geometric relations between the 3D points and their
projections onto the 2D images that lead to constraints between
the image points. To find the corresponding pixel, we only have
to search along the epipolar line (1D instead of 2D search). This
search space restriction is known as epipolar constraint [8]
Fig:1: Epipolar geometry
Each camera captures a 2D image of the 3D world.
This conversion from 3D to 2D is referred to as a perspective
projection. The camera is therefore modelled by its perspective
projection matrix (PPM)) Since the two focal points of the
cameras are distinct, each focal point projects onto a distinct
point into the other camera's image plane. These two image
points are denoted by eL and eR and are
called epipoles or epipolar points. The line OL–X is seen by the
left camera as a point because it is directly in line with that
camera's focal point. However, the right camera sees this line as
a line in its image plane. That line (eR–xR) in the right camera is
called an epipolar line. Symmetrically, the line OR–X seen by
the right camera as a point is seen as epipolar line eL–xLby the
left camera. An epipolar line is a function of the 3D point X, i.e.
there is a set of epipolar lines in both images if we allow X to
vary over all 3D points. Since the 3D line OL–X passes through
camera focal point OL, the corresponding epipolar line in the
right image must pass through the epipole eR (and
correspondingly for epipolar lines in the left image). This means
that all epipolar lines in one image must intersect the epipolar
point of that image. In fact, any line which intersects with the
epipolar point is an epipolar line since it can be derived from
some 3D point X [9].

The process of camera calibration gives us both a

model of the camera’s geometry and a distortion model of the
lens. In practice, no lens is perfect

III. IMPLEMENTATION
Fig: 2

A. Undistortion Implementations of correlation algorithms required in

Distortion is explained in detail in section.Using the
parameters extracted from calibration images are undistorted.
B.Rectification
Image rectification is an important step in the three
dimensional analysis of scenes. For stereo vision, image
rectification can increase both the reliability and the speed of the
disparity estimation process. This is because in the rectified
images, the relative rotation among the original images has been
removed and the disparity search happens along the image
horizontal or vertical scan lines. The rectification process
requires certain camera calibration parameters or weakly
calibrated (uncelebrated) epipolar geometries of the image pair
[10].
To reduce disparity estimation complexity, the used
cameras are usually arranged in a parallel-axis configuration or
equivalently, the stereo image pairs are carefully rectified using
the camera geometry. The main function of a stereo
correspondence algorithm is to match each (in the case of dense
stereo) or some (in the case of sparse stereo) pixels of the first
image to their corresponding ones in the second image. The
outcome of this process is a depth image, i.e. a disparity map.
This matching can be done as a 1-D search if the stereo pairs are
accurately rectified. In rectified pairs horizontal scan lines reside
on the same epipolar lines, as shown in Fig. 2
A point P1 in one image plane may have arisen from
any of points in the line C1P1, and may appear in the alternate
image plane at any point on the so-called epipolar line
E2(Eipipolar Constrain).Thus, the search is theoretically
reduced within a scan line, since corresponding pair points
reside on the same epipolar line. The difference on the 2
horizontal coordinates of these points is the disparity. The
disparity map results from assigning to each matched pixel the
disparity value of its correspondent. Afterwards, the depth of the
scenery can be derived from the disparity map, since the more
distant an object is the larger disparity value it is expected to
have [4].
the next stage of the reconstruction process can be greatly
simplified if the input images can be rectified. The process of
rectification involves a 2D image transformation of the input
images such that corresponding image points are located on
equivalent image scan lines. Utilizing geometric properties
inherent to epipolar geometry (see Figure 4), given a point and
its projected location on one image plane, it is possible to
calculate on which epipolar line in the other image plane the
point will appear. This epipolar constraint allows us to calculate
and perform the rectification 2D transformation of the original
input images [7].
Fig 3
The epipolar constraint expresses the relation between
two images of the same scene. The effect of the rectification is
such that the correspondence problem is reduced to one
dimension since we only have to search for matching points
across a single horizontal line of the matching input image.
Figure 5 shows the results of rectifying some input images after
calibration of a stereo rig and the capture of a stereo image pair.
Analysis of the rectified image pairs shows that matching points
are indeed positioned on matching scan lines showing this to be
a useful rectification. With the rectification of the input images
complete it then becomes possible to begin attempting
additional stages in the reconstruction process [7].

C. Stereo correspondence problem

 The dense correspondence problem consists of finding
a unique mapping between the points belonging to two images
of the same scene. If the camera geometry is known, the images
can be rectified, and the problem reduces to the stereo
correspondence problem, where points in one image can
correspond only to points along the same scan line in the other
image. If the geometry is unknown, then we have the optical
flow estimation problem. In both cases, regions in one image
which have no counterparts in the other image are referred to as
occlusions [11].
Scharstein and Szeliski have provided an exhaustive
comparison of dense stereo correspondence algorithms. Most
algorithms generally utilize local measurements such as image
intensity (or color) and phase, and aggregate information from
multiple pixels using smoothness constraints. The simplest
method of aggregation is to minimize the matching error within
rectangular windows of fixed size. Better approaches utilize
multiple windows adaptive windows which change their size in
order to minimize the error, shift able windows or redicted
windows, all of which give performance improvements at
discontinuities. [4]

D. Disparity

A disparity map is a depth map where the depth Fig 4

information is derived from offset images of the same scene.
Depth maps can be generated using various other methods, such
as time-of-flight (sonic, infrared, laser). Although these active
methods can often produce far more accurate maps at short
distances, the passive method we use has its benefits, including
applicability at long distances [14].
As mentioned above, differences between two images
gives depth information. These differences are known as
disparities. The key step to obtaining accurate depth information
is therefore finding a detailed and accurate disparity map.
Disparity maps can be visualized in grayscale. Close objects
result in a large disparity value. This is translated into light
grayscale values. Objects further away will appear darker.
Fig. 4 and Fig. 4 illustrate the disparity estimation
algorithm. As described in Fig. 1, the object of interest
registered on different x-position in an image. However, both
images share similar features. By repeating shifting one of the
images in x-axis, in this case the algorithm shift the right image,
at a certain point, both images will exhibit the maximum
number of overlapping pixels. As a result, the shift value that
produced this maximum overlapping area is used to represent
the disparity value for this object of interest. [13]
Obtaining depth information is achieved through a
process of four steps. Firstly the cameras need to be calibrated.
After calibrating the cameras the assumption is made that the
differences in the images are on the same horizontal or epipolar
line [4].
The secondly step is the decision as to which method is
going to be used to find the differences between the two images.
Once this decision is made, an algorithm to obtain the disparity
map needs to be designed or decided on. The third step is to
implement the algorithm to obtain the disparity information. The
final step is to use the disparity information, along with the
camera calibration set in step one, to obtain a detailed three
dimensional view of the world [12].

Fig 5
1) Feature Based Disparity: This method looks at features in
one image and tries to find the corresponding feature in the
other. The features can be edges, lines, circles and curves.
Nasrabadi applies a curve segment based matching algorithm.
Curve segments are used as the building block in the matching
process. Curve segments are extracted from the edge points
detected. The centre of each extracted curve is used as the
feature in the matching process. [12]
2) Area Based Disparity: There are two techniques that are
used in this algorithm. In both these methods a window is placed
on one image. The other image is scanned using the same size
window. The pixels in each window are compared and operated
on. These are then summed to give a coefficient for the centre
pixel. These techniques have been developed by Okutomi and
Kanade [12]
D. Triangulation
The same method that is used in navigation and
surveying is used to calculate depth. Basic triangulation uses the
known distance of two separated points looking at the same
scene point. From these parameters the distance to the scene
point can be calculated. This same basic idea is used in stereo
vision to find depth information from two images. Figure 6
below graphically shows the geometry.[4]
Fig 6
In the above arrangement, two cameras (C, C’) see the same
feature point (S). The location of the point in the two image
planes is denoted by A and A’. When the cameras are separated
by a distance T, the location of A and A’ from the cameras
normal axis will differ (denoted by U, U’). Using these
differences, the distance (Z) to the point can be calculated from
the following formula:
Z=f*T/U-U’
In order to calculate depth however, the difference of U and U’
need to be established.[4]
This formula allows us to calculate the real world distance of a
point. If what we are interested in is relative distance of points
rather than exact distance we can do this with even less
information. The base offset and focal length of the camera are
the same for both images. Hence the distance of different points
in the images will vary solely based on this disparity
component. Therefore we can gauge relative distance of points
in images without having the base offset and focal length [14].
IV CONCLUSION
In this paper, we propose an approach for stereo vision to
calculate depth and 3D map. In our system, we have used
cameras so it gives more accuracy in distance calculation than
sensors. The algorithm which we have used gives better
disparity map which leads to 3D visual perception. Our system
works well under following conditions:1)good image quality
2)proper camera calibration 3)object distance up to 5 feet.
ACKNOWLEDGMENT
This paper is outcome of the help, co-operation and guidance
from various corners. We take this opportunity to express our
deep sense of gratitude to our internal guide Mrs.S.V.Tikhe and
project incharge Mr. Saurabh Mengale. They took keen and
personal interest in giving us constant encouragement and
timely suggestions.
REFERENCES
[1] D. Marr and T. Poggio, “Cooperative computation of stereo
disparity,”Science, vol. 194, no. 4262, p. 283, 1976.
[2] D. Murray and J. little, “Using real-time stereo vision for mobile robot
navigation,” Autonomous Robots, vol. 8, no. 2, pp. 161–171, 2000.
[3] D. Murray and C. Jennings, “Stereo vision based mapping and navigation
for mobile robots,” in Proc. IEEE Int. Conf. on Robotics and Automation, vol. 2,
1997, pp. 1694–1699.
[4] L. Nalpantidis, A. Amanatiadis, G. Sirakoulis, N. Kyriakoulis, and A.
Gasteratos,” Dense Disparity Estimation Using a Hierarchical Matching
Technique from Uncalibrated Stereo Vision”, 2009 IEEE
[5]. Valsamis DOUSKOS, Ilias KALISPERAKIS, George karras“AutomaticC
Calibration of Digital Cameras usingPlanar chess board pattern”Department of
Surveying, National Technical University of Athens
[6]. http://Calibration of Stereo Cameras for Mobile Robots.html
[7]. Daniel Bardsley “A Correlation Based Stereo Vision System For Face
Recognition Applications”2004, University of Nottingham
[8].http://en.wikipedia.org/wiki/Image_rectification
[9].http://en.wikipedia.org/wiki/Epipolar_geometry
[10 Changming Sun, “Uncelebrated Three-View Image Rectification”,. IMAGE
AND VISION COMPUTING, VOL.21, NO.3, PP.259-269, MARCH 2003
[11]. http://www.cs.umd.edu/~ogale/papers/ogaleIJCV05shape.pdf
[12]. Paul Munro, Antony P. Gerdelan “Stereo Vision Computer Depth
Perception”
[13]. Yen San Yong and Hock Woon Hon.”Disparity Estimation for Objects of
Interest”
[14].http://www.Wikidot.com