Customizable Embedded 3D Surround

View Turn-Key Solution on Apalis iMX6

SoM

- Author: Thomas Hase, TES Electronic Solutions GmbH

Multi-Camera 3D Surround View systems are already established in many passenger vehicles. Demand
for such systems now also grows for special vehicles like construction machines, harvesters, buses or
military vehicles. In comparison to personal cars, these vehicles are produced in a rather moderate
volume and have a high need for customization of the optical system.

2
In this blog post, we present a customizable 3D Surround View Turn-Key solution based on the Toradex Apalis iMX6
SoM based on the NXP i.MX 6 SoC, that can be quickly adapted concerning the specific requirements of the targeted
vehicle.

3
Please watch this video from Embedded World 2017 in Nuremberg, Germany, where this system solution was
presented:

Video Link:
https://www.youtube.com/watc
h?v=Y67DhP6qaUs

4
We also explain how to get familiar with the topic of 3D surround view, evaluating the solution via a step-by-step
approach and finally how to come to a product meeting your specific needs.

TES 3D Surround View Stitching Library the heart of the system solution
TES Electronic Solutions provides a 3D Surround View software library designed for embedded systems, CPU/GPU
independency, and high customizability. Detailed information can be found on the TES 3D Surround View Website.

5
The library takes raw fish-eye pictures from memory and computes a 3D View of the vehicle in its maneuver situation.

TES 3D Surround View Highlights:

Increases safety and comfort through improved situation awareness:
Excellent stitching and 3D scene generation
Watch your maneuver situation from any position outside or inside the vehicle
Any number of predefined viewing positions (Views) possible with animated transitions between different views
Many parameters configurable per View
Highly customizable and portable
Independent from the used type of cameras (the software expects raw fish-eye pictures in memory)

6
3D Surround View Turn-Key Solution

In the following, we present a complete HW/SW system solution based on the Toradex Apalis iMX6Q IT Module allowing
easy hardware customization to meet specific customer demands.

System Setup

7
The System setup consists of the following components:
Processor: Toradex Apalis iMX6 IT Computer On Module with Heatsink
Toradex Ixora Carrier Board

Camera System:
First Sensor Mobility Area View Kit DCAVK including
4x 1MPixel MJPEG Ethernet Cameras (PoE), DC3K-1-EAP
4x 5m Ethernet cables (up to 25m possible for each camera)
Micrel Gigabit Ethernet (PoE) Switch
Metal Plate for Desktop Application, additionally 4 Vacuum cups for vehicle application included

Software:
TES 3D Surround View Software under Linux on Apalis iMX6
8
Resource Usage:
The 3D Surround View algorithm mainly exploits the GPU. The CPU is only used for system control and in the case of
view-changes.
Configuration: 4 MJPEG IP cameras, combined 80Mbit data rate @30 fps and 1Mpx resolution. Low polygon car model,
1280x1024 screen resolution at 60Hz refresh rate.
GPU: 3-6 GFlops (proportional to pixel amount depicted on screen)
RAM usage: 60 MB in total
4* 10 MB for camera data buffers
2 MB for 3D environment and car model
18 MB for camera video textures

9
This turn-key system solution can be rapidly reproduced for evaluation and prototyping purposes. Of course, also the
separate components can be exchanged or adapted to the target application needs, e.g.:

The cameras can be exchanged against other types.

A dedicated application specific main board can be designed taking the Ixora Carrier board as a reference.
The user interface and HMI overlay of the 3D Surround View software can be easily adjusted.
The 3D model of the vehicles can be easily exchanged.

Both Toradex and TES, are open to provide the necessary support.

10
Getting started and familiar with 3D Surround View
3D Surround View is not only a topic of running an algorithm on an embedded platform and be done with it. There
are many aspects of the system setup that need to be considered and require customization for specific use cases,
e.g.:
Selection of the appropriate cameras regarding lens, resolution, physical data transmission (cable length!),
power supply, number of cameras etc.
Positioning of the cameras (usually the higher the better)
Defining the appropriate viewing positions, i.e. virtual position and viewing vector of the observer in the 3D
scene
Optimizing the 3D Surround View parameters, e.g. Environmental model (shape and size of the projection bowl),
stitching method and parameters.
User interface for the 3D Surround View System
3D Model of the vehicle, potentially with transparency.
11
Just consider an excavator compared to a personal car: On an excavator, you need significantly longer cables to the
cameras, the cameras are mounted typically much higher. Latency is not that much of a concern as the driving speed is
lower and maybe you just want to apply only 3 cameras or mount the cameras un-symmetrically because the driver can
see the left front part of his vehicle very well while he cannot see the right front part at all.

So it might be a good idea to first play with a 3D Surround View system to get an idea of
such application aspects. For this purpose or for those who just want to give the TES software
a try there is an offline PC demonstrator available at zero cost. TES provides this
demonstrator on request in a zip archive with no need of installation (just unpack). Any
modern Windows PC should be sufficient to run this software. The program comes with
prerecorded MJPEG Ethernet streams in *.PCAP-format and runs these streams through the
TES 3D Surround View software.

12
The PC keyboard and mouse can be used to change the viewer position: predefined views are available on the number
keys and free-flight is possible via mouse and (Shift-) arrow keys. Also, some of the 3D Surround View system
parameters like the bowl-size and -shape as well as the stitching parameters can be adjusted. Do not hesitate to get in
touch with TES (coordinates below) to get this Software.

First steps in the Lab and on the vehicle

Now that you have played with the software (and assuming it does more or less what you need) you might
want to give it a try with real-life cameras, either in lab conditions or right away on the vehicle.

The fastest way to do that is to get a First Sensor Mobility Area View Kit DCAVK (contents see above
under System Setup) and the TES 3D Surround View Windows PC Software in Online-Mode (Evaluation
Version). This kit allows lab setups as well as on-vehicle installation using a Windows Laptop. Up to 20m
cable length (between cameras and switch) has been tested so far. Also here TES can assist you in purchasing
this setup. 13
And now on the embedded hardware

Having tested the system under lab conditions and on the vehicle that target platform aspects now get focus (again). I.e.
you want to see your setup running on the potential target hardware. At that point, you may want to take your Apalis
iMX6 module with Ixora carrier board (or buy some from Toradex if you do not have those, yet) and get the TES 3D
Surround View software for that target platform (prototype version). I.e. you get rid of the PC and use the Apalis/Ixora
platform instead. The TES prototype software for that target platform has the same functionality as the evaluation
version for PC and you can re-use the configurations that you have setup at the earlier stage now on the target.

14
Getting to a product

For a product, you now have to customize the system to meet your specific application requirements. I.e. you
use the Apalis iMX6 module as core and might re-design the carrier board based on the Ixora design data to
meet your specific requirements regarding, interfaces, power supply, mechanics etc.

In that step or even before you also may want to exchange the cameras with other types.

Again, TES can assist you also in the hardware and system customization, development and (if wanted)
manufacturing process.

15
Summary

We have introduced an embedded 3D Surround View Turn-Key Solution and explained a process to get you
familiar with this topic starting at zero cost. Following this process, your product development risk is
significantly reduced because you can evaluate the system in a step-by-step approach and at the same time
determine your application specific requirements. Finally, we have introduced the steps to come to a product
based on this turn-key solution. Just get started!

16
Thank you