Developing Embedded Automotive Products

Embedded products for automotive applications typically follow a very rigid

development process; the details vary from Original Equipment Manufacturers
(OEMs), but the need for risk mitigation is the same

It's not unusual for processes, tools, and techniques to be similar within a specific industry,
and the automotive industry is a case in point. Automotive development for both heavy
vehicles and cars involves a set of development tools, techniques, and processes designed to
provide a standardized approach to delivering a quality embedded product.

Typical, too, of the automotive as well as many other industries are the various advocacy
groups that promote this expected approach. One such group is the Automotive Industry
Action Group, or AIAG (Chrysler Corp., Ford Motor Company, General Motors Corp.:
1995). The AIAG publishes an overview of processes, from concept through production
phase, in its Advanced Product Quality Planning and Control Plan (APQP) Reference
Manual. (See Table 1.) In general, the phases are:

• Conceptualization
• Product design and development
• Process design and development
• Product and process verification
• Launch.

The APQP approach is usually implemented as a high-level waterfall model. However, it

offers no restrictions to spiral or agile approaches to software development. The manual
specifies a minimum number of deliverable documents and activities. It does not specify a
limit to the amount of documentation, although the formality of the Production Part Approval
Process (PPAP) requires a submission - in full form - of at least 18 documents, often many
more.

The higher-level automotive quality standard is ISO/TS 16949:2002, which is a superset of

the ISO 9001 quality system standard with an automotive spin. The 16949 standard does not
explicitly prescribe APQP, but APQP aptly meets the requirements of the standard.

Detailing the Approach

There are many embedded parts on today's automobile, and managing all of the suppliers is
difficult and time-consuming. Potential suppliers to automotive organizations are rigorously
scrutinized as to business aspects as well as technical capabilities.

The conceptual phase is usually started with a Request for Proposal (RFP) from a customer
attempting to communicate his or her development needs through specifications, interviews,
meetings, and reviews.

Many organizations have a particular set of methods and criteria for this evaluation. Some
approaches are proprietary; others use well-established methodologies. The end result
captures the customer's needs or requirements and identifies any conflicting requirements.
This sets the scope of the project - and failure in this phase translates to probable failure at the
end of the project. The voice of the customer surpasses the voice of the engineer, unless a
safety concern is presented.

APQP documentation suggests that the development team use techniques to benchmark the
capabilities of competing companies. In the event of a novel or new product, this competitive
benchmarking is unlikely. Benchmarking is often difficult, but it is possible to research
product specifications in various trade journals and even to purchase and dismantle similar
systems or subsystems from a competitor to determine performance capabilities and thus
establish realistic targets.

Once the need and performance requirements have been identified, a response to the RFP is
formulated and issued. This makes the most use of the supplier's expertise and generates
multiple concepts for the customer organization. Pre-approved suppliers submit their
proposals to achieve the product objectives, largely addressing fit, key product interfaces, and
end-customer interactions with the product. Their interpretations generally consist of
drawings and physical models, though they can also make use of software simulations or a
digital mockup (DMU).

Proposals are often followed up with presentation materials, additional drawings, and a rough
outline of cost and delivery possibilities. Eventually these activities produce a chosen
supplier, and formal technical documentation of the product begins in earnest, including
detailed hardware and software requirements.

By the conclusion of the concept phase, the development team will have identified the design
direction, the supplier, and the product goals, and enlisted management support for the
project. This gives the development team a good foundation and a clear picture of the scope it
is allowed for subsequent phases of product and process development.

Product and Process Development

Now that the direction for the product is clear, the development team can produce a set of
specifications identifying the product's behavior in terms of software, hardware, and
durability requirements. These specification documents describe not only the desired
"normal" functional behavior, but also the environment in which the product will function, as
well as failure behaviors. There are automotive standards that assist with these definitions
(for example, the Society of Automotive Engineers standard SAE J1455). These documents
go beyond providing development targets; they are also used to confirm that the design has
achieved targets and met performance demands during the test and verification phase.

Time to market is prized in APQP organizations, and they tend to develop products and
processes simultaneously, or nearly simultaneously. Sometimes process design and
development will start after product design and development commences, but otherwise they
run in parallel.

During the product development work, there are multiple releases of the product. These
iterations have an increasing level of capability and functional maturity. Some organizations
assign part numbers for each iteration of the hardware and software; any change to form, fit,
or function for either portion requires new part numbers. Other organizations may treat the
hardware and software as one part by assigning a new part number as if the aggregate were
one entity.
To bridge any gap during these intermediate deliveries and the specifications, release notes
are provided by the designing organization. These notes provide the customer organization
and test and verification personnel with the known or perceived capability of the product. In
the early going, engineers theorize how the design may behave; however, to confirm these
theories, the designers may execute simulation runs, and engineering verification tests will be
conducted whenever possible on the parts. Typical design confirmation activities include:

• Calculations
• Simulation
• Engineering verification activities
• Functional and system integration testing
• Design Failure Modes and Effects Analysis
• Other design reviews.

Design reviews are part of the product development process for automotive applications. The
reviews have a number of goals, some specific to the design (engineering peer reviews), some
from the organizational perspective (manufacturing and service). Experience suggests that
more frequent and diligent reviews will provide a more effective end result. Organizational
policy will dictate the minimum set and type of reviews. The embedded team may opt for
more reviews within the development group.

The automotive world employs a very powerful tool in Design Failure Modes and Effects
Analysis (DFMEA). The DFMEA requires a cross-disciplinary team of development
engineers to analyze the functions of the product using a logical approach.

One such example is the functional analysis system technique (FAST). This technique is used
to develop a table that captures failure modes, causes, effects, and qualitative values related
to these events. The tool works best when teams use it to solve design issues while working
with the design, or in advance of the final design. It is possible to use FAST for embedded
software development as well.

The DFMEA is a design tool for upfront thought on potential problems. This allows the
development team or project manager to focus the test cases upon the areas that present the
most probable and/or highest impact, narrowing the focus of the design effort and verification
efforts.

Production processes also reflect the increasing sophistication and capabilities of the product
under development, and they benefit from the same types of techniques and critiques as the
development work. Production processes are proposed and verified and, where possible,
recycled from existing production methods, such as:

• Production process development and documentation

• Trial production runs
• Production process capability measurements and corrections
• Gauge repeatability and reproducibility
• Process Failure Modes and Effects Analysis
• Other process design reviews.

Like the DFMEA tool used for the design, the Process Failure Mode and Effects Analysis
(PFMEA) tool critiques the production process, generating actions and changes in the
proposed processes or confirmation actions for the processes. Indeed, any business that
requires processes can benefit from a critical review and identification of areas that pose a
risk to success or customer satisfaction. This early critical review, followed by actions and
follow-ups, improves the service even before the service is available.

Product and Process Verification

Any embedded software development team will also need to consider design verification. No
matter the organization, an independent verification and validation (IV&V) team should be
included. The objectivity of the verification team must not be tainted by the development
efforts. After all, those who have developed the software and hardware have already made
interpretations of the specifications and customer requirements.

This independent group will have its own interpretations of the specifications, and will
develop test specifications accordingly. Some organizations outsource parts of the
verification activities. This is especially true where the environmental aspects of the product
are concerned, because this equipment represents a high dollar investment.

Verification activities are done in parallel with the development activities. They provide
feedback to the product development and manufacturing process teams for improvement in
their respective areas.

Software functional and component performance verification is part of the delivery from the
supplier. The results of these tests form part of the release notes for the individual
software/hardware package.

In the automotive world, a summary document called a Design Verification Plan and Report
(DVP&R) is used to capture all of the activities that will verify that the design meets or
exceeds requirements. This document provides a synopsis of the test conducted as well as the
results of the testing. The activities in the DVP&R should be derived from the DFMEA,
which has a detection column specifically for this purpose. A minimal design verification
plan should accomplish the following:

• Fully exercise all inputs (stress to limits)

• Exercise all output similarly
• Use random testing to provide unexpected input/output combinations
• Use as much of the system hardware as is available (other hardware on the data bus)
• Provide some level of extreme testing to push the software/hardware combination to
the limit
• Confirm the requirements are met.

Acquiring material in the early phases is quite costly. Specially tooled prototype parts in
small volumes take time from production lines, and standalone prototype lines that are only
used occasionally carry a high price burden per piece.

In the automotive world, this cost prohibition is overcome in part via simulation. The teams
have access to a multitude of simulators that can simulate the hardware and software
components of the product as well as any data bus. The same hardware in the loop simulator
can often serve as a development fixture to simulate component and feature interactions with
the new proposed component and feature set.
Other possibilities include products such as Matlab/Simulink®, LabView®, and similar tools
capable of communicating on the various vehicle buses and analog-to-digital conversions.
These tools are not just verification tools; they also facilitate creation of requirements
documents or specifications that detail the component as well as functionality.

In a well-equipped software team, in-circuit emulators (ICEs) are available for both
development and early testing of the code. These allow hardware and software interactions to
be observed in real time (actually, pseudo-real time) and provide early verification of the
software in the target system. One of the defects of most ICEs is that they possess their own
power supply, which sometimes will affect the verisimilitude of the emulation. Both
emulation and simulation allow for product feature refinement in advance of the product's
availability, as well as early verification before the prototype stage.

An ICE provides a software/hardware replica of the normal processor (sometimes using the
real processor), whereas a simulator does not have to provide the hardware level of support.
Usually, an embedded developer will use simulators to provide data traffic for the bus.

Manufacturing development is quasi-parallel with the product development phase. The

intention is to shorten the delivery time for the product as well as to ensure that it can be
manufactured. Additionally, the team will develop automated manufacturing testing of the
product's software and hardware.

In cases where sophisticated electronics are available, the software can test itself - for
example, using a boundary scan. Software can also be written to help self-calibrate the
product. In addition, software can be used to provide extreme conditions - for example, data
bus overloading.

The parts created from the rapidly intensifying production processes can be used by the
customer for additional verification activities. These parts, level 2 and level 3, are used for
component level and systems or vehicle integration verification activities. This level of
verification is crucial because it provides real platforms for the embedded product, going
beyond what can be achieved during the early stages using emulators and simulators.

As the design and production processes mature (part level 4 and 5), the parts are used for
increasingly demanding activities, such as vehicle durability confirmation, which often
includes harsh weather and driving conditions. Vehicles equipped with these parts may spend
significant time in severely cold and hot regions, accruing many miles during multi-state,
multi-regional trips and providing critiques of everything from product performance to
sounds that may emanate from the component. These tests represent the extremes of the
product use.

Process validation activities are designed to prove the capability of the production processes.
Even a perfect product is useless if a part cannot be produced at the intended volumes.
Validating processes requires a well-documented production process. All hardware and
software, for the product as well as the manufacturing processes, are frozen. The production
processes are then stressed via run-at-rate, which produces the product at the parts per minute
proposed to meet the customer's needs. These parts are then validated to establish the
"process capability" of the production line - in other words, an evaluation of the line ability.

Launch
Verified product is put onto early-build vehicles, which are used for plant material handling
as well as validation. Product documentation is developed for the manufacturing facility
installing the product. This documentation is based upon the technical specifications and
functional documents. Suppliers offer technical support during the early launch activities.

The developed embedded software can be subject to a number of possibilities after launch
and during manufacturing. One programming option is post-launch product programming,
either during manufacturing or at the customer's site with special tools. Additionally,
maintenance of and upgrades to the software must be considered when developing its
architecture. Product traceability and configuration management demands extend past the
initial production launch.

Closing the Loop

The feedback, evaluation, and corrective actions portion of the development is not really a
phase, but rather activities or systems that run parallel to the project. As in most industries,
rapid feedback of test results is a prerequisite to solving non-conformance and quality
problems quickly.

Key metrics are identified and monitored. Metrics that move outside the expected
performance, indicating a trend toward non-conformance, require corrective actions.
Sometimes these corrective actions extend past the product launch. Automotive product
development organizations use a tool known as the eight disciplines, or "8D" model, whose
steps are as follows:

1. Assemble a cross-functional team with the required expertise.

2. Define the problem.

3. Implement and verify interim containment actions - temporary fixes.

4. Identify and verify root cause.

5. Identify and verify permanent corrective actions; preventive actions are also chosen.

6. Implement and validate permanent corrective actions.

7. Prevent recurrence of the problem/root cause.

8. Recognize the efforts of the team.

Emergency situations require emergency actions, such as stopping the manufacturing line
immediately if, for example, the verification team or the software team determines the
software has a safety issue.

For addressing problems in which it is possible to determine "good" units from "bad" units,
containment actions will be started. (It may be possible to ship the "good" units, provided
there is confidence in the ability to discern, via test, between the two states.) Permanent
corrective actions require determining "root cause," tracing the failure back to the causing
element. Just to be sure, any remedial action taken must be verifiable to confirm that it has
corrected the problem.

While 8D particularly follows production issues, it is possible for the development team to
use this tool to formally address problems found during the development work. It is important
to note that the ultimate goal is to eliminate the element causing the problem. Experience
suggests that frequently the true culprit is not identified; often a symptom is treated, but not
the root cause or causes.

Feedback at this point may sound as if it is coming a bit late in the process. In reality, if
appropriate feedback is to be provided, the metrics that mean something to the product and
project success must be identified early - then monitored vigorously by either technical staff
or project managers, depending upon the metric.

This feedback system communicates identified risks and allows for subsequent mitigation
actions. Incorporating the feedback loop identified in APQP with a comprehensive risk
management plan goes a long way toward reducing the risk of the project and product.

A comprehensive test or verification approach will also reduce product and project risk
exposure. One such technique, used by the Department of Defense, is the Test and Evaluation
Master Plan (TEMP), which provides a comprehensive and systematic approach to
verification activities.

In this approach, the customer and the developers agree ahead of time to deliver a sequence
of software packages. Each new software package is a superset of the previous package. Each
package is fully functional. Each package has verification activities performed. The
verification covers development verification, operational or integration verification, and
deployment verification or live exercises. This approach allows for quick traceability of
discovered problems to a unique software delivery.

Not All About Automotive Apps

The production process for an embedded product can be defined as a set of individual,
incremental steps. Each step typically has a set of processes associated with it. The process
control plan is a perfect tool for such sequences. The steps have key metrics identified to
determine capability, and recurring measurements are taken to assess the state of the
individual process.

However, the use of this tool need not be restricted to the production process. Development
processes and software processes can benefit from the same critical eye - in other words, the
PFMEA. For areas found to be at risk, additional controls can be devised and placed on the
process.

The key to realizing the biggest benefit from DFMEA and PFMEA reviews is to perform
them early in their respective process areas. Early use reduces development waste; it is
cheaper to make changes to the product and process concepts and documentation than it is to
make changes to equipment and processes already in place.

One of the biggest strengths of the APQP model has to be the articulation of an easily
understandable development process that encompasses customer input through to production.
This document provides a framework, not explicit detail that would force particular solutions
from the adopting organization.

Suppliers and customers can easily see how their processes map together, since both sides
likely incorporate some content originating from this document. This mapping is further
facilitated with the introduction of a common language and understanding of terminology. No
matter where you go in the automotive world, these principles are known and used to some
degree.