2011 Second International Conference on Intelligent Systems, Modelling and Simulation

Implementation of In-vehicle Multi-sensor Information Fusion Gateway for Cooperative Driving

Ting-Ying Wei, Zhi-Liang Qiu, Chung-Ping Young
Department of Computer Science and Information Engineering National Cheng Kung University Tainan, Taiwan {p78981192, p76984542, cpyoung}@mail.ncku.edu.tw

Yu-Tsung Lo
Institute for Information Industry Tainan, Taiwan yutsung@iii.org.tw

AbstractActive safety is an important feature of a modern vehicle to provide precaution warning or compensatory control before the pre-crash stage of vehicle safety. All vehicle signals and information are acquired by several in-vehicle sensors on ECUs or surrounding vehicles, and integrated in vehicle gateway through in-vehicle or vehicle-to-vehicle communications. The information exchanged among the host and surrounding vehicles provides comprehensive vehicle and driving status of each vehicle, so the driver can drive more safely with the cooperative driving mechanism. The demonstration system consists of a vehicle gateway, which is based on a heterogeneous multi-core processor, consisting of one ARM core for I/O control and system management and two DSP cores for intensive computation of information fusion. After reaching within the communication limit, the vehicles set up time synchronization and then exchange vehicle information. The acquired sensor data and received vehicle information are time aligned and fed into PAC DSPs for information fusion, which consists of four phases, signal processing, location mapping, trajectory prediction and risk assessment. The risk assessment evaluates the probability of car crash and broadcast the warning message to its surrounding vehicles, if the risk level is raised beyond a threshold. The cooperative driving is fulfilled by sharing the vehicle information and emergency warning through wireless communication, so a driver can be aware of the dangerous situation and also be suggested an adequate response earlier. Keywords- Multi-sensor; information fusion; cooperative driving; vehicle gateway; in-vehicle network; vehicle-to-vehicle

I.

INTRODUCTION

To well protect drivers and passengers from car collision the driving status and vehicle data are acquired and processed by a variety of car electronics before the pre-crash stage of vehicle safety to determine any potential risk condition. Some accidents happen because the driver cannot predict or observe the operation of other drivers. Increasing the response time will largely decrease the probability of car accident. The traffic crashes caused by the human factor in United States are reported about 8090% of all incidents by National Highway Traffic Safety Administration (NHTSA) [1], so active vehicle safety is one of the major research topics for intelligent transportation system (ITS) [2]. The safety concern is the most important issue for vehicle operation, while cooperative driving improves the vehicle
978-0-7695-4336-9/11 $26.00 2011 IEEE DOI 10.1109/ISMS.2011.13 19

safety by exchanging the vehicle information to enhance the driving information and the response time [3], [4]. Electronic Control Unit (ECU), which is an embedded system dedicated to a designated vehicle function, connects with other versatile ECUs via in-vehicle communications to form a distributed sensor network. A vehicle links with its surrounding vehicles via vehicle-to-vehicle communications to become a larger scale distributed computing system, which is a vehicular ad-hoc network [5]. A vehicle gateway obtains the in-vehicle driving status and vehicle data, and broadcasts them to the neighboring vehicles, while it also receives the broadcast data from the opposite parties [6], [7]. In order to realize a vehicle gateway, a heterogeneous multi-core based platform is employed, where it consists of an ARM processor as a general purpose processor (GPP) and a digital signal processor (DSP) as a special purpose processor (SPP). An embedded Linux kernel running on the GPP as a central console is responsible for graphical user interface (GUI), task management, I/O and communication protocol stack, and the SPP functions as a computation engine for processing vehicle data. All tasks are categorized into real-time and non-real-time, while the task manager handles the non-real-time tasks on GPP, and dispatches the real-time tasks to the SPP [8]. The heterogeneous multi-core platform interconnects with other microcontroller-based data acquisition ECUs via a Controller Area Network (CAN) bus. Cooperative driving improves vehicle safety by exchanging vehicle data, so the driving status of neighboring vehicle can be observed in advance and thus the response time will be shortened. The vehicle gateway implements the information fusion of in-vehicle sensor data and wireless communication packets. The vehicle data is exchanged through vehicle-to-vehicle communications and the GPP of vehicle gateway informs the SPP that the latest vehicle data were received. The information fusion engine in SPP will integrate local vehicle sensor data and the acquired neighboring vehicle information, and implement information fusion to generate several safety indexes and produce warning messages. At first, multi-sensor information fusion procedure processes the acquired vehicle signals, retrieves vehicle information of surrounding vehicles, and integrates them in a time sequential linked list. The relative distance, angle and velocity of each surrounding vehicle to the host are recalculated by cross-evaluating the information from versatile

sensors of each car, so all vehicles are represented by a tree structure rooted from the host and mapped onto a highway map. The trajectory of each surrounding vehicle is tracked by continuously recording its location information, and the upcoming path in next few seconds is predicted. The risk assessment will determine the risk level by estimating the possible duration before a car will collide with the host. If a potential collision will occur, an alarm signal and collision avoidance manipulation will warn and prompt to the driver of the host car and broadcast it to the surrounding vehicles. II. MULTI-SENSOR INFORMATION FUSION Each sensor type has its advantages and limitation to measure environmental parameters, while the same type of sensors in different cars may have different degrees of precision or fault. In order to provide full coverage to the detection of surrounding objects coming close to a host vehicle to prevent from collision, a variety of sensors are employed to acquire supportive, complementary or redundant information. The raw or preprocessed data dont provide comprehensive knowledge to risk assessment, but they have to be integrated and processed to generate a dynamic information set, which includes historical track and upcoming event prediction. The procedure of fusing multisensor information includes the following stages: signal processing, location mapping, trajectory prediction and risk assessment. Figure 1 depicts the data flow of multi-sensor information fusion.
Sensor Input

established and a negotiation session is applied for determining a clock master to initiate time synchronization. Though every car may have a global positioning system (GPS) receiver to obtain the global time, the differences of software layers and hardware architecture among all vehicle gateways will cause the deviation of local time when GPS time is acquired in application of each car. The implementation of time synchronization is to minimize the clock drift between two local clocks, so the data sampling of sensors will be activated simultaneously. B. Signal Processing The input signals from accelerometer, gyroscope, radar and GPS receiver are processed or received by the microcontrollers of ECUs or vehicle gateway, and stored in time sequential linked lists. The images captured by cameras run through the image processing in DSP for the detection of lane departure, headway and blind spot. Since the image processing takes more time to accomplish, all other sensor data have to be managed first and proceed to the next stage if necessary. Moreover, the preprocessed information of first data fusion stage from surrounding vehicles will be received sometime after the first stage. The sensor data, image processing results and received information packets arrive in the gateway at different time and they are merged according to time alignment. The sampling rate of the sensor is defined as 200 Hz by considering the car speed is 120 Km/hr. That will be around 16.7 cm movement of the car every 5 ms sampling period. The sampling rate is increased when the speed becomes faster or during a collision warning duration. The frame rate of captured images is normally 30 fps, but, if the headway to tracked target is larger than specified safety distance, the frame rate can be decreased. Each vehicle doesnt transmit the image to its surrounding vehicle, because the images, requiring lots of bandwidth, are not necessary for other vehicles. Each vehicle transmits its vehicle information packet during the designated time-divided duration to prevent from packet collision. Figure 2 shows the in-vehicle data acquisition and vehicle-to-vehicle information exchange.
Location Velocity Acceleration Yaw, Roll Distance Image The sampling rate is proportional to vehicle speed OR Increased because of emergency brake 200 Hz @ 120 KM/hr (16.7 cm/5ms) Frame rate is 30 fps

Signal Processing Communication Communication Input Communication Input Input Data Alignment Location Mapping Trajectory Prediction Risk Assessment

Host Vehicle

Actuator Output

Communication Output
Surrounding Vehicle

Vehicle ID Location Velocity Acceleration Yaw, Roll Distance Emergency

Transmitted once when communication channel is set up

Figure 1. Data flow of multi-sensor information fusion.

The packet length is proportional to vehicle speed Packet is transmitted every 100 ms

A. Time Synchronization When a vehicle is approaching within the defined dataexchanging limit of other vehicle, which is smaller than wireless communication range, a communication channel is

Figure 2. In-vehicle data acquisition and vehicle-to-vehicle information exchange.

20

C. Location Mapping Current GPS system cannot provide high precision position data to civilian usage, so the relative distances among vehicles calculated from the GPS data cannot be counted on for collision avoidance. The GPS data can only be used for global positioning route planning or track recording, while the relative coordinates of the surrounding cars are more crucial for estimating the safety distances. Therefore, the global positioning of the host vehicle on a map is roughly determined by the GPS data and slightly adjusted by other sensor data, while the surrounding vehicles are located into this map according to the relative distance, position and velocity to the host. Merging the signals from radar, accelerometer, wheel speed sensor, and yaw sensor generates the relative distance, position and velocity, while the received information from surrounding vehicles is involved for cross checking. When the reception of GPS signal is interfered, the location aware services can continue by applying composite navigation with information fusion from the sensors mentioned above. D. Trajectory Prediction The trajectory of each surrounding vehicle is plotted by accumulating the time sequential information of vehicle position, velocity and acceleration. According to the current and past vehicle information, the moving path during the next few seconds will be calculated. For normal or drowsy driving condition, the vehicle should move along a straight or curved route, or repetitively depart from lane, while the trajectory prediction can be done by linear estimation. If there are any specific events, like obstacle avoidance, car crash or emergency brake, the moving path of the event car will change abruptly. When this situation happens suddenly, the host vehicle receives the broadcast emergency warning from the event car. The host increases the sampling rate of its own sensors and decreases the cycle time of vehicle information packet reception, so it can obtain more detailed vehicle information of event car during abrupt course change for more accurate trajectory prediction. For example, the gyroscope and accelerometer data are important, because a car sometimes rolls over after the accident and this is not a normal vehicle driving style. The prediction should involve more input parameters, like yaw, roll and lateral sliding, and therefore the side impact, rollover and other emergency scenarios are considered. The road circumstances are also part of the conditions for evaluating future trajectory. For example, the trajectory prediction for a straight or a curved road is different. The angle measured from the yaw sensor is common on a curved road, but may be the indication of lane departure on a straight road when drowsy driving. E. Risk Assessment After the future trajectory of each surrounding vehicle is calculated, each relative trajectory to the host is generated. Different levels of risk will be determined depending on the area crossing of the relative trajectory towards the host and the expected time. The faster the time and the closer the trajectory to the host, the higher the risk level is. The

crossing area located in the front part is more dangerous than the rear part. Figure 3 demonstrates the crossing area and the assigned risk levels, while it is not scaled.
Risk #4 Risk #3

Risk #2 Risk #1

Figure 3. The crossing area of relative trajecgtory and the risk level.

The relative trajectories to each other of the vehicles before the host are determined, too. A vehicle may not have threat to the host vehicle, but it is very likely to collide with another vehicle and thus the traffic accident will immediately threat the vehicle safety of the host. This risk assessment is to pre-warn the possibility of a chain collision. III. VEHICLE GATEWAY The realization of the vehicle gateway consists of two parts: hardware architecture and software implementation. A. Hardware Architecture The demonstration system consists of a vehicle gateway connecting with several signal acquisition ECUs via CAN bus and IEEE 802.15.4 wireless communication. This vehicle gateway is based on a heterogeneous multi-core embedded platform for I/O control, system management and signal processing computation. The vehicle gateway also functions as central console with GUI for human-machine interface and connects to the Internet by 3G communications. Figure 4 demonstrates the block diagram of vehicle gateway connecting with a few ECUs for information fusion.
CCD Camera SD card Touch Panel

ECG
802.15.4 Latitude Longitude Date/time 802.15.4

Heartbeat

PAC Duo
Pressure Gyroscope E-compass CAN CAN
Velocity Engine temperature Engine rotation Tire pressure

GPS DSP1 DSP2 ARM

Roll

Yaw

Revolution G sensor

Velocity

OBD-II

Acceleration

DSRC

3G Coordination Time Velocity Acceleration Heartbeat

Surrounding Vehicle

Service Center

Figure 4. Block diagrams of the demonstration system with multi-sensor information fusion gateway.

21

Figure 5 demonstrates part of the hardware platforms realized for vehicle gateway and ECUs with sensors connected by CAN bus.

Figure 5. Demonstration of system realization of vehicle gateway and ECUs connected through CAN.

1) Vehicle gateway Because of the advance of integrated circuit technology, an ECU not only shrinks the size and reduces the power consumption, but provides more complicated functionality and data processing capability. Moreover, a multi-core processor is feasible for decreasing the amount of chip components, eliminating the cable connection, and lowering down the cost. A heterogeneous multi-core processor PAC Duo SoC, which stands for Parallel Architecture Core, is developed by Industry Technology Research Institute (ITRI), Taiwan and is employed for the core component of the vehicle gateway. It is a chip-level heterogeneous multiprocessor SOC composed of an ARM926EJS and two PAC DSP cores of the same architecture [9]. The ARM926EJS serves as the GPP while two DSPs can be treated as SPP to cooperate with the GPP. Each DSP core in PAC Duo has a 64 KB local memory and resides on the 32-bit AXI bus. Communication with the ARM processor can be achieved through an AXI-AHB bridge since the ARM processor resides on the 32-bit AHB bus. PAC Duo supports inter-processor communication (IPC) at hardware level through hardware mailbox mechanisms and shared memory [10]. The former is interrupt-driven, allowing a processor to send interrupts to another processor for event notification. The latter allows the processors to share data or states. There are four banks of shared memory on the platform. Two banks of shared memory (128 KB SRAM and 128 MB DDR2 DRAM) reside on the AXI bus while the other two (256 KB SRAM and 128 MB SDRAM) reside on the AHB bus [9]. 2) Electronic control unit A few ECUs are implemented by applying the development boards from the processors chip vendors for system demonstration. The ECUs are used for acquiring vehicle sensor data, diagnostic information and human

physiological signals, and transmit them to the vehicle gateway after some pre-processing computation. Microchip PIC18 The Microchip PIC18Fxx8 is an enhanced flash microcontroller with CAN. The PIC18, which includes a RISC CPU running up to 10 MIPS, with 16-bit wide instructions and 8-bit wide data path, is dedicated to the ECUs functioning as low-end I/O control or data acquisition. There are 4 timers, one capture/compare/PWM module and up to 8 channels 10-bit ADC modules and a Master Synchronous Serial Port operating with SPI or I2C mode [11]. Renesas R32C/118 The Renesas R32C/118 is the high-end microcontroller, with 32-bit CISC architecture, multiplier, multiplyaccumulate unit, and floating point unit, applied for vehicle infotainment. The R32C, involving broad selection of onchip peripheral devices provides flexible functions and interfaces for general control and input/output, where there are nine channels of serial interface, one channel of multimaster I2C-bus interface, and two channels of CAN module [12]. TI CC2430 The Texas Instrument CC2430 is a System-on-Chip (SoC) solution, combining the CC2420 RF transceiver with an enhanced 8051 MCU, flash memory, RAM and other features, for IEEE 802.15.4 and ZigBee applications. The CC2430 has low power consumption feature, so it is suited for the applications with long-operated battery-powered remote or wireless devices, for example, tire pressure sensor or drivers physiological signal recorder [13]. 3) Communications In order to obtain the pre-processed sensor data and vehicle information from other ECUs or surrounding vehicles, several wired or wireless communication interfaces are required. CAN The Controller Area Network (CAN) is a serial bus communications protocol developed by Bosch, German in the early 1980s, and it was standardized as ISO standard protocol for in-vehicle networking. The CAN protocol, which has multi-master nodes operated in bus topology using two wires for communication, is based on CarrierSense Multiple Access/Collision Detection (CSMA/CD) [14]. Both Microchip PIC18 and Renesas R32C/118 have onchip CAN modules, which are employed for connecting to PAC Duo to transmit the acquired sensor signals, like gyroscope, e-compass and accelerometer. OBD II On Board Diagnostics (OBD) is a set of diagnostics data variables and also with a relatively standard connector plug, so every car can accommodate the same DLC (Data Link Connector) and generates the same generic DTC (Diagnostic Trouble Codes). The OBD-II is an ISO standard and is regulated as the required equipment in a vehicle to generate the codes. OBD-II provides real-time data acquisition which can be used for not only fault diagnostics, but some useful vehicle information, like the temperature of engine, the rotation rate and velocity of the vehicle [15]. The ELM327

22

series of OBD-II, which has 38400 bps data rate, is used for monitoring the velocity of the car and the temperate and rotation rate of the engine. IEEE 802.15.4 The IEEE 802.15.4 specification defines wireless medium access control (MAC) and physical layer (PHY) for 2.4 GHz low data rate wireless personal area networks (WPANs) operated in either star or peer-to-peer topology within relatively short distances. The communication implements the CSMA/ collision avoidance (CA) mechanism and the transfer rate is up to 250 Kbps [16]. The TI CC2430 is implemented for in-vehicle wireless communication. The PAC Duo, which is originally not capable of IEEE 802.15.4 communication, utilizes UART to connect with a TI SmartRF04 Evaluation Board with a CC2430 Evaluation Module for tire pressure and physiological data reception. Dedicated Short Range Communications (DSRC) The Federal Communication Commission (FCC) of US has allocated 75MHz of spectrum in 5.9 GHz band (5.850 GHz - 5.925 GHz) as Dedicated Short Range Communications (DSRC) in 1999 to support low-latency vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications for improving traffic safety and highway efficiency. DSRC is a multi-channel wireless standard and based on the IEEE 802.11a PHY and the IEEE 802.11 MAC layers. The DSRC standard is developed by IEEE working groups Wireless Access in Vehicular Environment (WAVE). There are seven communication channels, consisting of one control channel and six service channels. Multi-channel coordination, which is an extension to the IEEE 802.11 WAVE [17] mode supporting a multi-channel system with the IEEE 802.11 MAC and PHY via control and service channels, provides mechanisms for prioritized access, channel routing and coordination, and data transmission. The WAVE/DSRC Communication Unit (WCU) developed by ITRI is proposed for vehicle-to-vehicle communication development evaluation. 4) Sensors Different types of sensors are used for acquiring vehicle diagnosis or driving status. The raw data are pre-processed by the ECUs and then merged in the vehicle gateway. Some off-the-shelf sensors, like GPS receivers, are manufactured as full function modules, so the processor in the module processes the signals and output the measuring results with defined format through a standard interface. Image Charge Coupled Device (CCD) sensors are commonly used in video equipment for image capturing. The Swann C510R color camera features a 1/4-inch CCD image sensor with 400 TV Lines or 512 x 492 (NTSC) resolutions. Infrared night vision allows cameras to capture images in total darkness or low-light conditions by using IR LEDs as illumination [18]. The captured image by CCD camera fed into the DSP doing image processing is used for supplementary road condition recognition.

Global Positioning System (GPS) GPS receivers are used for positioning, navigating, and determining the time. The Globalsat BR-355 GPS Receiver incorporates the SiRF Star III GPS chipset and an active patch antenna, so a higher degree of GPS accuracy can be reached [19]. The received lattitude and longitude are received by each car for route tracking only, since its not so accurate for determining the relative distance to its surrounding vehicle. Accelerometer A three-axis micromachined accelerometer (MMA7260QT, Freescale Semiconductor) [20] was mounted on an accelerometer board, and the output signals were fed to the ADC of the PIC18 microcontroller board. B. Software Implementation The software of the whole demonstration system can be separated into several parts. On the vehicle gateway, there is GUI with non-real-time operating system on ARM and the information fusion real-time computation on PAC DSP. On ECUs, the data acquisition and processing are implemented on non-OS environment of each microcontroller board. A heterogeneous multi-core aware embedded software platform was customized and implemented on the ITRI PAC for a variety of automotive applications, while the Linux is on ARM processor and the C/OS-II is on PAC DSP. An IPC mechanism, which takes advantages of hardware features, was designed to fulfill the heterogeneous multi-core interconnection among ARM and DSPs [6]. The real-time process migration between DSPs was realized for load balancing enhancement. To improve the energy consumption, the scheduling and dynamic system configuration for power management was implemented on ARM processor and PAC DSP, respectively. Therefore, the renovation for the multicore OS is characterized with real-time, power management aware and fault tolerant. For the central console, MeeGo, which is a Linux-based open source mobile operating system project supported by Intel and Nokia, provides user interface and subsystem interconnection [21]. This realization for the software platform of automotive applications involves heterogeneous multi-core embedded software interfacing with MeeGo. The software on each microcontroller development board is executed on a non-OS environment with interrupt service routines for servicing A/D conversion, interface communication, DMA and timer/counter. IV. CONCLUSION The vehicle gateway based on a heterogeneous multicore platform is establishing, and several ECUs are interconnected with the gateway through wired or wireless communications. The multi-sensor information fusion is implemented for computing the relative position, velocity and acceleration to the host vehicle, so the potential collision risk can be assessed and the cooperative driving feature is provided by exchanging information with surrounding vehicles through wireless communications. This design proposes the realization of cooperative driving to enhance the active safety.

23

ACKNOWLEDGEMENT This study is conducted under the Open Software Platform Technology for Mobile Device Development Project of the Institute for Information Industry which is subsidized by the Ministry of Economy Affairs of the Republic of China. REFERENCES
[1] National Highway Traffic Safety Administration, Motor vehicle traffic crash fatality counts and estimates of people injured for 2005, 2006. Robert Bosch GmbH, Safety, Comfort and Convenience Systems, Wiley, 2006. S. Weihua, Y. Qingyan and G. Yi, Cooperative Driving based on Inter-vehicle Communications: Experimental Platform and Algorithm, Proc. Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 5073-5078, 2006. G. K. Mitropoulos, et al., Wireless Local Danger Warning: Cooperative Foresighted Driving Using Intervehicle Communication, Intelligent Transportation Systems, IEEE Transactions on, vol. 11, no. 3, pp. 539-553, 2010. H. Hartenstein, K. Laberteaux, Vehicular Applications and InterNetworking Technologies, John Wiley & Sons, 2010. P. Pyungsun, Y. Hongbin, M. Heeseok and J. Jaeil, An OSGi Based In-vehicle Gateway Platform Architecture for Improved Sensor Extensibility and Interoperability, Proc. Computer Software and Applications Conference, 2009. COMPSAC '09. 33rd Annual IEEE International, pp. 140-147, 2009. Y. He, Q. Guihe, T. Zhizhong and D. Jinnan, Network-based invehicle body bus control system, Proc. Control and Decision Conference (CCDC), 2010 Chinese, pp. 3128-3131, 2010.

[8]

[9] [10] [11] [12] [13] [14] [15] [16]

[2] [3]

[4]

[17]

[5] [6]

[18] [19] [20] [21]

J. Chen, et al. Building Multi-kernel Embedded System on PAC Multi-core Platform, Proc. Quality Software (QSIC), 2010 10th International Conference on, pp. 465-472, 2010. Industrial Technology Research Institute, "PACDSP V3 Software Developers Manual-Volume 1: Processor Architecture," 2009. Industrial Technology Research Institute, "PAC DUO Programmings Reference," 2009. Microchip, PIC18F2480/2580/4480/4580 DataSheet, 2009. Renesas Electronics, R32C/118 Group Datasheet, 2009. Texas Instruments, CC2430 Data Sheet, 2010. Robert Bosch GmbH, CAN specification, 1991. Gateway Vehicle Inspection Program, On-Board Diagnostics (OBDII) Fact Sheet, 2007. IEEE Std 802.15.4-2006, IEEE Standard for Information technology- Telecommunications and information exchange between systems-Local and metropolitan area networks-Specific requirements Part 15.4: Wireless Medium Access Control (MAC) and Physical Layer (PHY) Specifications for Low-Rate Wireless Personal Area Networks (WPANs), 2006. IEEE Std 1609.3-2007, IEEE Trial-Use Standard for Wireless Access in Vehicular Environments (WAVE) Networking Services, 2007. Swann, C510R Professional CCTV Camera with Night Vision, 2007. USCG Navigation Center, GPS SPS Signal Specification,1995. Freescale Semiconductor, Three Axis Low-g Micromachined Accelerometer, 2008. MeeGo. [Online] http://meego.com.

[7]

24