2014 ICCE-Taiwan

,QWHJUDWLRQRI/L'$5DQG&DPHUD'DWDIRU'UHFRQVWUXFWLRQ
1

Tsung-I Chen1, Yu-Xiang Zhang1, Chia-Yen Chen1, Member, IEEE and Chia-Hung Yeh2, Member, IEEE
1
Dept. of Computer Science and Information Engineering, National University of Kaohsiung, Kaohsiung, Taiwan
2
Dept. of Electrical Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan

(SfM) problem, which recovers the shape of objects based on

Abstract-- In this work, we propose an integration of the
LiDAR device and a pair of binocular cameras for egomotion
estimation and 3D reconstruction. The proposed system
simultaneously acquires both range data and image data as input
and calibrates the LiDAR and the cameras to determine the
correspondences between them. Egomotion is determined as the
system moves within the environment and finally the egomotion
is used to register and integrate the data from the LiDAR device
and the cameras to construct a 3D model of the environment. 3D
reconstructions for both indoor and outdoor environments are
performed to demonstrate the effectiveness of the proposed
method.
I. INTRODUCTION
In recent years, 3D related technologies have received
increased attention. 3D applications which were not possible
previously have become more and more popular. For example,
the self-driving car developed by Google [1] uses a light
detection and ranging (LiDAR) device to provide information
of the vehicle's 3D surroundings in real-time. The particular
device gained recognition for its usage in mobile sensing since
several of the winning teams in the 2007 US DARPA Urban
Challenge incorporated it on their vehicles [2].
In this work, our ultimate goal is to perform large scale 3D
reconstruction of the environment, using a 3D reconstruction
system that we have constructed. The system consists of a
device and two cameras, all of which mounted rigidly upon a
mobile platform. To achieve this goal, we first perform
calibrations for the LiDAR device and the cameras to
determine their correspondences; then we need to perform
spatial feature detection and matching on the images to obtain
a sparse point cloud. As the system moves with time, temporal
feature tracking is also performed on the images sequence to
calculate intra-frame motion. We use an auto-calibration
method to calculate the extrinsic parameters of the LiDAR and
the cameras, which are used to align the point clouds scanned
by the LiDAR along the path. Experimental results have
shown that the proposed system and method is able to achieve
integrated dense reconstruction for large scale scene.
II. RELATED WORK
Early research on egomotion started in the 1980's when the
Stanford AI Lab wanted a cart equipped with cameras to be
able to deduce its own motion [3]. Egomotion is often defined
as the calculation of sensor motion in 3D space from 2D
images [4], and has since become increasingly important in
robotics, autonomous navigation, 3D reconstruction, and so on.
Egomotion estimation is similar to the structure from motion
images of the object taken at different viewpoints. Similar
calculations are also seen in visual odometry (VO), which
estimates the movement of a body based on consecutive
images [5]. In simultaneous localization and mapping (SLAM)
or visual SLAM (V-SLAM), the task is to have a robot or a
vehicle build a map of an environment using sensory
feedbacks, while keeping track of its own location within the
environment [6].
III. PROPOSED METHOD
The proposed method works as follows. In the first stage,
the laser scan of a calibration target is obtained, and the IR
images of the target are recorded simultaneously. The pixels
belonging to the laser trails of the scanned object are identified
from the captures images. Each of these trails is then mapped
to a laser rangefinder according to its position in the image.
Once mapped, we associate each laser point to a pixel in the
image which denotes the expected 2-D location where the 3-D
point is supposed to be projected onto. Finally, the calibration
data is constructed. Fig. 1 shows the 3D reconstruction device
used in our work.
Fig. 1. The LiDAR and stereo camera system for 3D reconstruction.
A. Device calibrations
Conventional sensor calibration approaches utilize a set of
correspondences to adjust parameters, so that the deviation
between the expected results and the observations is
minimized. We use automatic calibration method proposed in
[7], the method is able to correctly detect laser trails from the
infrared images and match them to the depth data acquired by
the multi-layer laser. Fig. 2 shows the results after calibration,
where the coloured points represent range data acquired by the
LiDAR device. The points are transformed based on the
calibrated correspondences between the LiDAR device and the
cameras and overlaid on one of the binocular stereo images.

978-1-4799-4851-2/14/$31.00 2014 IEEE 93

 2014 ICCE-Taiwan
Fig. 2. Projection of calibrated depth data onto images
A. Egomotion estimation
We perform egomotion estimation using images obtained
from stereo cameras. The method proposed in [8] is used to
determine the path of the 3D reconstruction system as it
moves along a path. Fig. 3 shows the path obtained using the
selected egomotion estimation method. To demonstrate the
performance of egomotion estimation, we moved the system
forward and back along the same straight path. As Fig. 3
shows, the estimated paths overlap almost perfectly.
Fig. 3. Paths showing egomotion estimated using the method in [8], the units
are in mm.
B. 3D reconstruction
In the reconstruction step, the range and image data at each
point of acquisition are registered using the correspondences
calculated from the calibration process, and the egomotion
estimated along the path of reconstruction is used to align the
data acquired at different points on the path, such that over the
entire course, a 3D model of the environment can be
reconstructed. We selected two locations for reconstruction, an
indoor location as shown in Fig. 4(a) and an outdoor location
spanning over 10m in range as shown in Fig. 4(b).
(a) (b)
Fig. 4. 3D reconstruction locations (a) indoor and (b) outdoor.
IV. EXPERIMENT RESULT
The indoor and outdoor 3D reconstructions are shown in
Fig. 5(a) and (b), respectively. As can be seen from the results,
the 3D reconstructions at different points along the paths are
integrated quite well to produce very dense 3D models of the
tested environments.
94
(a) (b)
Fig. 5. 3D reconstruction for (a) indoor and (b) outdoor.
V. CONCLUSIONS
In this work, we integrated a multi-beam LiDAR device and
a pair of binocular cameras to achieve egomotion estimation
and large scale 3D reconstruction. The system simultaneously
acquires both range data and image data, calibrates the LiDAR
device and the cameras to determine the correspondences, and
calculates egomotion as the system moves through the
environment. The calibrated device parameters and egomotion
are used to register and integrate the range data from the
LiDAR device and the image data from the cameras to
construct a 3D model of the environment. Both indoor and
outdoor experiments have been performed to demonstrate the
practicality of the proposed 3D reconstruction system. In
future work, we will continue to improve the system with
respect to its reconstruction accuracy and efficiency.
ACKNOWLEDGMENT
The authors would like to thank the National Science
Council of Taiwan for sponsoring this work under grant
numbers NSC 102-2221-E-390-021, NSC 103-2218-E-110-
006, NSC102-2221-E-110-032-MY3 and NSC101-2221-E-
110-093-MY2.
REFERENCES
[1] G. Erico, "How Google's Self-Driving Car Works," [online] Available
from: http://spectrum.ieee.org/automaton/robotics/artificial-intelligence
/how-google-self-driving-car-works [Accessed 18 July 2013].
[2] P. Stone, P. Beeson, T. Merili and R. Madigan, "DARPA Urban
Challenge Technical Report," technical report, Austin Robot
Technology, 2007.
[3] H. Moravec, "Obstacle Avoidance and Navigation in the Real World by
a Seeing Robot Rover," Ph.D. thesis, Univ. of Stanford, CA, 1980.
[4] M. Irani, B. Rousso and S. Peleg, "Recovery of Ego-Motion using Image
Stabilization," in Proceedings of the IEEE Computer Society Conference
on Computer Vision and Pattern Recognition 1994, pp. 21-23, 1994.
[5] D. Nister, O. Naroditsky and J. Bergen, "Visual Odometry," in
Proceedings of the IEEE Computer Society Conference on Computer
Vision and Pattern Recognition 2004, Vol.1, pp. 652-659, 2004.
[6] B. Williams and I. Reid, "On Combining Visual SLAM and Visual
Odometry," in Proceedings of the IEEE International Conference on
Robotics and Automation 2010, pp. 3494-3500, 2010.
[7] C.-Y. Chen, H.-J. Chien, "Geometric Calibration of a Multi-layer
LiDAR System and Image Sensors using Plane-based Implicit Laser
Parameters for Textured 3-D Depth Reconstruction", J. Visual
Communication and Image Representation,
DOI:10.1016/j.jvcir.2013.08.005, Aug. 2013.
[8] C.-Y. Chen, J.-H. Zhang, T.-I Chen and C.-F. Chen, "3D Egomotion
from Stereo Cameras Using Constrained Search Window and Bundle
Adjustment," In Proc. IVCNZ '13, Wellington, New Zealand, 2013.