Environmentally Adaptive Fault Tolerant Computing

(EAFTC)
Jeremy Ramos, Dean W. Brenner, Gary E. Galica, and Chris J. Walter
Honeywell Inc., Defense & Space Electronic Systems
13350 U.S. Highway 19 North
Clearwater, Florida 33764-7290
727-539-4311/ 727-539-2584
jeremy.ramos@honeywell.com
dean.brenner@honeywell.com
galica@psicorp.com
cwalter@wwtechnology.com

Abstract—1,2The application of commercial-off-the-shelf
(COTS) processing components in operational space
missions with optimal performance and efficiency requires a
system-level approach. Of primary concern is the need to
handle the inherent susceptibility of COTS components to
Single Event Upsets (SEUs). Honeywell in conjunction
with Physical Sciences Incorporated, and WW Technology
Group has developed a new paradigm for fault tolerant
COTS based onboard computing. The paradigm is called
“Environmentally Adaptive Fault Tolerant Computing”
(EAFTC.) EAFTC combines a set of innovative
technologies to enable efficient use of high performance
COTS processors, in the harsh space environment, while
maintaining the required system availability.
TABLE OF CONTENTS
INTRODUCTION .........................................................1
TECHNOLOGY ADVANCE ..........................................2
CONCEPT OF OPERATION .........................................2
IMPLEMENTATION METHODOLOGY .........................2
HARDWARE IMPLEMENTATION ...............................3
SOFTWARE FRAMEWORK .........................................5
EAFTC AND RP MIDDLEWARE ...............................6
TECHNOLOGY DEVELOPMENT .................................8
CONCLUSION .............................................................8
REFERENCES .............................................................9
ACKNOWLEDGEMENTS .............................................9
BIOGRAPHIES ............................................................9
INTRODUCTION
Science and defense missions alike have ever-increasing
demands for data returns from their space assets. In recent
time we have seem a significant increase in the capability of
the instruments deployed in space [1,2,3]. The traditional
implementation approach of data gathering, data
compression, and data transmission is no longer sustainable.
The vast amounts of data being generated can not be
transmitted via available downlink channels in reasonable
time. The industry proposed solution is to reduce the
demand on the downlink by moving processing onto the
spacecraft. This approach is hampered by the limited
capabilities of today’s on-board processors and the
prohibitive cost of developing radiation hardened high-
performance electronics [4,5]. This has partly encouraged
the industry to consider the use of COTS components [6].
Furthermore, the recent adoption of silicon-on-insulator
(SOI) technology by COTS integrated circuit foundries is
resulting in devices with moderate space radiation tolerance
[7,8]. Despite all of the progress, COTS components
continue to be highly susceptible to SEUs. Therefore,
technology must be developed that capitalizes on the
strengths of COTS devices while overcoming their
susceptibility to SEUs without negating their benefits.
The popular approach for mitigating SEUs is to employ
fixed component level redundancy [9]. One major
disadvantage of fixed redundancy is low efficiency and
unrealized system capacity. The EAFTC paradigm
facilitates use of COTS components in SEU abundant
environments, while maintaining adequate levels of system
efficiency and capacity. It accomplished this by adaptively
configuring the level of fault tolerance in the system as
mandated by the mission environment and mission
application. The three fundamental elements of EAFTC are:
• Real time environmental sensing
• COTS based computer architecture that supports
adaptable configuration levels of fault tolerance
• System controller to optimize performance and
efficiency while maintaining reliable operation
In this paper we discuss the EAFTC paradigm in detail. We
also discuss ongoing efforts to advance EAFTC as a
technology to adequate levels for use in operational
missions.

1
0-7803-8870-4/05/$20.00© 2005 IEEE
2
IEEEAC paper #10787, Version 4, Updated October 27, 2004
1
 TECHNOLOGY ADVANCE sensor suite, and target computer3. The fundamental
process implemented in the system consists of three steps:
State of the art onboard processing computers consist
mostly of radiation hardened components based on COTS 1. Sense the environment
equivalents. Though COTS compatibility offers many
benefits, including adoption of commercial software, large 2. Assess the environmental threat to the system’s
amounts of NRE are required for the initial silicon availability
implementation. Additionally, radiation hardened
components lag their commercial counterparts in overall 3. Adapt the processing system’s configuration to
performance and capability by at least 1 to 2 orders of effectively mitigate the threat presented by the
magnitude. There are a number of factors that contribute to environment
this deficiency. Most predominantly, radiation-hardening
techniques for microelectronics require the use of fixed SEU Alarm Energy Level
<<sensor>>
transistor or gate level redundancy. The additional logic EAFTC
increases the power required to perform the same unit of controller Process
Environmental Deployment
computation. Measurement
(i.e. temp,
Sensor
Response Health Target Computer
power, etc.) <<onboard
History Deployment
<<sensor>> computer>>
An approach towards improvement is the use of true COTS Plan

microprocessors and Field Programmable Gate Arrays Spacecraft

ephemeris

(FPGAs.) This approach avoids the high cost and long

development time associated with radiation hardened
equivalents. However, COTS devices are highly susceptible
to SEUs. The popular SEU mitigation approach is to use
component level N-module redundancy, which results in
low efficiency and low capacity due to the overhead of 1/3,
or more. Furthermore, the level of redundancy is fixed and
often unnecessary. To overcome the deficiencies of fixed
redundancy there are two characteristics of space missions
that we focused on, the variability of space environment and
task level criticality. Most missions will have a mix of
processes with varying criticality. This characteristic of
mission processing can be exploited to increase the systems
efficiency by applying redundancy at the task level.
Furthermore, the variability of the space environment
provides a temporal and orbital position dependency on the
necessary redundancy. EAFTC exploits both of these
characteristics to mitigate SEUs via redundancy, but unlike
N-module redundancy, it employs replication selectively
and adaptively as mandated by the environment and the
particular task.
Hence, EAFTC as a technology mitigates faults, and in
particular SEUs in COTS devices, while increasing the
system’s overall efficiency and capacity. EAFTC does this
by optimally applying fault tolerance over the life of the
mission as demanded by the task criticality and in-situ
environmental measurements.
Direction of Data Flow
Figure 1-1. EAFTC based system
In general, EAFTC can be implemented to accept a
multitude of environmental inputs that can induce faults in
the target computer4. However, for this particular
discussion, environmental monitoring is focused on
measurements of high-energy particle flux. By monitoring
flux of high-energy particles, it is possible to assess the
systems overall susceptibility to SEUs.
Sensor measurements5 and target computer health are
continuously monitored by the EAFTC controller and
combined with a mission defined application task
Deployment Plan. The Deployment Plan contains task level
criticality requirements as well as other pertinent
information used by the controller. Based on that input, the
controller determines the reliability and availability threat
posed by the environment on the target computer. The
controller then generates the requisite signals to adapt the
target computer, thereby, countering the hostile
environmental threat. The target computer in response
optimally employs the requested fault tolerant mechanism.
This process is performed in real-time and on-line as an
integral part of the overall system operation.

IMPLEMENTATION METHODOLOGY
CONCEPT OF OPERATION
Figure 1-2 shows an implementation methodology that
The goal of an EAFTC based system is to optimally employ incorporates fault tolerant techniques at each level of the
system level fault tolerance based on historical and in-situ system hierarchy.
environmental conditions. Figure 1-1 is a block diagram of
a representative EAFTC based system. The major functional 3
e.g. payload computer system
elements of the system are a controller, environmental 4
For example temperature, available power, etc.
5
i.e. SEU Alarm, temperature, and spacecraft ephemeris
2
 <<processor>> #4
Mission APC:
Data Processor
<<processor>> #3
Mission Faults <<processor>> #2 Benchmark
APC:
RHPPC SBC: Application
Data Processor
System Methods <<spacecraft interface System Controller Linux
<<processor>>
Horizontal#2
Services
EAFTC A/B>> Benchmark
APC:
System
EAFTC Application
Data Processor
Components
Linux
Application Faults Experiment Comm.
Experiment Manager
Configuration
Horizontal Services
BenchmarkControl
Spacecraft System
Software Methods Experiment Monitors
<<processor>>
Experiment #1
Configuration
Configuration
Application
Components
Linux Memory
<<processor>> #1
SIFT Middlewares, Roll-back, Check-Pointing, Algorithm Design, RHPPC SBC:
VxWorks
System Controller
Configuration
System
Horizontal
APC:
Control
Profiling
Services
others System
Data Components
Processor
Configuration
Configuration Control
Memory
Configuration Memory
System Profiling
System Profiling
Module Faults
Hardware Methods <<28V Power>>
Replication, Scrubbing, Heart Beating, Watchdogs, others <<Switch Fabric A>>

<<Switch Fabric B>>

Component Faults

Component Selection
<<device>>
Radiation Hardened, COTS, Military Grade Power Supply A/B
<<device>>
SEU
Alarm:Environmental
Sensors A/B

Figure 1-2 Fault Tolerance Design Methodology

The foundation of this methodology starts with component
selection through testing, screening, and analysis. At the
next layer, fault tolerant hardware design is used to address
component faults. In the proposed system, the use of
hardware redundancy is adapted to mitigate faults in the
critical system subassemblies. The third layer of fault
tolerance uses software-based methods for faults not
covered by the lower layers in the system. The final fault
tolerance layer requires system level methods, such as
EAFTC, to address those faults not covered by the
application. This overall fault tolerance strategy is
employed in the design of our Target Computer and
provides an optimal system solution contributing
significantly to overall system capacity and efficiency.
HARDWARE IMPLEMENTATION
A Target Computer hardware architecture is proposed that
is based on Honeywell’s Integrated Payload System. Figure
1-3 depicts the hardware realization of a COTS-based
Target Computer for EAFTC. The basic hardware elements
of the system are a System Controller, a cluster of Data
Processors, Packet Switched Fabric, and an Environmental
Sensor Suite.
Figure 1-3 Target Computer Block Diagram
System Controller
The System Controller is implemented using redundant,
Radiation Hardened Single Board Computers. A highly
reliable, radiation hardened System Controller provides a
platform for the deployment of critical control software
such as the EAFTC controller. A good candidate for a
system controller is the Honeywell radiation hardened
RHPPC Single Board Computer (SBC.) This radiation
hardened SBC is based on Motorola 603e microprocessor
technology [5]. Key features of the RHPCC SBC are
summarized in Table 1-1.
Table 1-1. RHPPC SBC Key Features
Key Features
3.3V and 5.0 V Power
RHPPC delivering 100 MIPS
Peripheral Enhancement Component support chip
4MB EEPROM with Single Error Correction and Double
Error Detection
512KB EEPROM
128 MB DRAM with SuperEDAC
6U x 220mm Euro Card Form Factor
Max Power Draw 15W
Mass >3lbs
Redundant 1553 (interface to spacecraft computer)
32-bit 33MHz PCI (interface to cluster and MIB
electronics)

Data Processors
The Data Processor cluster uses COTS based processors
with Honeywell’s unique architecture called the Adaptive
Processing Computer (APC.) The APC is a multi-mode
device that combines the use of COTS microprocessors and
FPGAs on a single platform. The APC employs a COTS

3
IBM PowerPC 750FX microprocessor and a Xilinx VirtexII the most flexibility by retaining a programmable
6000 FPGA6. Figure 1-7 is a block diagram of the APC. microprocessor and access to custom hardware. The APC is
capable of dynamic switching between modes, useful in
Front Panel
many applications.
Ethernet
PPC
RAM
The APC’s flexibility allows application designers to adapt
Interrupts

750fx
MPC Bus Control
MPC Bus Address Data

the target processor to a variety of mission level

UART RX/TX (PE) UART RX/TX (CM)
EEPROM
64 MB

requirements. Greater efficiency can be achieved by using

DRAM
MPC Bus Data

Mem Data
64 MB User IO PROM

more custom hardware modules in the FPGA. Similarly,

Mem Address/Cntr
DRAM
Select Map Bus
CM Interrupt
Configuration
greater processing performance can also be realized in
4 MB
EEPROM PE/PC Sleep RAM
VirtexII 6K Health
Manager
Actel
FPGA modules. However, for applications that require ease
128 KB
Mem Address/Cntr CM Reset Mem Data
PROM
EEPROM
Watchdog Interrupt Mem Address/Cntr

of programmability, the microprocessor mode would be a

Local PCI better fit. The APC facilitates these and other
PCI-to-PCI Bridge Ethernet
implementation alternatives not typically available in on-

PWR (3.3,1.5,+/-12V)
+ Arbiter
Actel
Interface APC Interrupt board processor modules. Some key features of the APC are
Timer Synch
APC Health

listed in Table 1-2.

RIO

Thermistor
To Front
PCI

JTAG
Panel

cPCI Connector

Table 1-2. APC Key Features

Figure 1-4 Adaptive Processing Computer Block Diagram Key Features

750 fx @ 650 MHz Delivering 1300 MIPS
The left side of the APC block diagram shows the COTS VirtexII 6000 Processing Element/Processor Controller
computer’s resources and the right side radiation hardened PCI 32-bit 33 MHz
Configuration Manager and supporting functions. The Rapid I/O
Configuration Manager handles APC mode changes, basic 128 MB DRAM with Super EDAC
FPGA configuration, FPGA configuration memory 4MB EEPROM with SECDED EDAC
scrubbing, low-level health monitoring, and power mode
Configuration Manager with support FPGA SEU
control. The APC implements three primary modes of
mitigation
operation: microprocessor, custom processor, and hybrid
PCI-to-PCI bridge facilitating a local PCI bus
processor. Operating mode is determined by the active
Ethernet development interface
configuration of the FPGA labeled Processing
6U x 220 mm Euro Card Form Factor
Element/Processor Controller in the block diagram.
Mass <3 lbs
While in microprocessor mode the APC’s FPGA is Max Power Draw 20W
configured as a Processor Controller and the microprocessor
is enabled. In this mode the APC behaves much like a SBC.
The Processor Controller FPGA hosts all of the support Packet Switched Fabric
functions for the PPC including IO, memory controller, All system modules are interconnected via a Packet
interrupts, timers, etc. When enabled as a custom process, Switched Fabric based on the RapidIO (RIO) commercial
the microprocessor is disabled and can not execute industry standard [11]. RIO is recognized by many as the
software. In this mode the FPGA is configured as a leading edge COTS interconnect. State of the art payload
Processing Element and hosts a full-custom application data processor interconnects are mostly based upon
including all IO and processing logic. The logic in the multidrop configurations such as MODULE BUS, PCI and
Processing Element is defined by the image loaded into the VME. Multidrop systems distribute available bandwidth
FPGA’s configuration memory by the Configuration over each module in the system but also produce points of
Manager who in turn takes its commands from software on contention among participant nodes often resulting in
the System Controller. The third APC capability is hybrid system level bottlenecks. In contrast, RIO implements a
mode operation, in which the FPGA hosts the Processor packet-switched, point-to-point interconnect allowing,
Controller for the microprocessor as well as application multiple full-bandwidth point-to-point links to be
specific modules. This mode can be likened to a co- simultaneously established between end-nodes in a network.
processor system. Application specific modules include RIO reduces contention and delivers more bandwidth to the
Digital Signal Processing (DSP) functions, data application.
compression, vector processors, etc. As with the custom
mode, the use of application specific modules results in high Figure 1-5 shows a RIO system based on two building
efficiency and performance yields [10]. This mode offers blocks: a RIO end-node, and a RIO switch. Each end-node
in the system is outfitted with a RIO network interface
6
The IBM 750fx and Xilinx VirtexII devices have been identified having a point-to-point link to a shared RIO Switch. The
as suitable COTS devices for the flight experiment.
4
switch receives and routes packets to the appropriate module, Figure 1-6 shows a module with three sensors,
destination. The non-blocking nature of RIO allows controller electronics, and network interface.
concurrent routing of multiple packets. For example: sensor
data can be stored in Bulk Memory at the same time as the
processor accesses the General Purpose IO. By using
multiple switches in a system, topologies consisting of
hundreds or thousands of nodes can be achieved. Proton Ion COTS Proton
scintillator and scintillator and scintillator and
Photo Detector Photo Detector Photo Detector

Processor Processor Threshold Output Threshold Output Threshold Output

Alarm Analog/ Alarm Analog/ Alarm Analog/

Digital Electronics Digital Electronics Digital Electronics
Control and Control and
Control and
Data Data
Data
RapidIO Bulk
Sensor Data
Switch Memory
SSM

PWR (3.3,1.5,+/-12V)
Backplane
Controller
FPGA

Thermistor
SSIO
General Non-volatile
Purpose I/O Memory cPCI Connector

Figure 1-6. SEU Alarm Module Block Diagram

Figure 1-5. RapidIO Example System

RIO interfaces are based on LVDS signaling technology SOFTWARE FRAMEWORK

and can achieve bandwidths of up to 60 Gbits/s for each
active link. A 16 bit RIO system with two active point-to-
point links is capable of 120 Gbits/s providing >120x
performance increase over a 33 MHz 32 bit Compact PCI
based system.
A notable benefit of the RIO protocol is its extensive error
detection and recovery mechanism. By combining retry
protocols, cyclic redundancy codes (CRC) and
single/multiple error detection, RIO handles all in network
errors without application intervention. This inherent error
handling and recovery capability proves ideal for space
applications that require a highly reliable interconnect.
Environmental Sensor Suite
EAFTC relies, to a great extent, on the system’s capability
to sense its environment. As part of PSI’s proprietary
The Target Computer’s software framework as shown in
Figure 1-7. The basic components of the framework are the
Operating System/System Software, Fault Tolerant System
Controller/Node, EAFTC controller, Messaging
Middleware, and Reliable Platform Middleware. The
objective of the Target Computer software framework is to
provide the system developer with a dependable yet familiar
software platform. The user’s software consists of mission
specific Payload Control and Communications hosted on
the System Controller, and application processes distributed
across the Data Processor cluster. All software components
may be developed using COTS environments and
associated Application Program Interfaces (APIs).
<<processor>> #4 APC: Data Processor
Direction of Data Flow
<<processor>> #3 APC: Data Processor
COTS
<<processor>> #2 APC: Data Processor

Reconfigurable Environmentally-Adaptive Computing <<processor>> #1 RHPPC <<processor>> #1 APC: Data Processor

Technology (REACT), a miniature embedded radiation SBC:System Controller

monitor, the SEU Alarm has been developed. The SEU EAFTC
Controller
FT System
Controller
FT System
Node
Application

Alarm is based on flight-proven technology originally

developed for PSI’s radiation diagnostic instrumentation Payload
Messg. <<PCI>> Messg. RP
Control and
[12]. An advantage of the SEU Alarm over state of the art Comm.
Middleware Middleware Middleware

sensors, other than its small foot print, is that it is System System

specifically designed to support SEU rate predictions. VxWorks:OS Software

Components
Linux:OS Software
Components

The SEU Alarm provides continuous monitoring of the <<Synch. Ser. IO>>
<<device>> <<RIO>> <<device>>
:RIO Switch
:SEU Sensor
proton and heavy-ion fluxes that cause single event upsets.
The basic components of the SEU Alarm are a small block Figure 1-7 Software Framework
of scintillators coupled to a photodetector. A number of
these simple devices can be consolidated onto a single

5
The proposed Operating Systems are VxWorks [13] for the
System Controller and Linux for the Data Processor cluster.
VxWorks provides the capabilities necessary for the
deployment of real-time control processes such as those
implemented by the EAFTC controller, Fault Tolerant
Controller, and Payload Control and Communications.
VxWorks also provides a familiar platform for developers
of these types of applications. The Data Processor cluster,
unlike the System Controller, is the domain of the science
application developer. In this case Linux is the preferred OS
due to its popularity in the scientific community. To
mitigate the problems associated with the interaction of
heterogeneous operating systems we have introduced the
use of a COTS messaging middleware. The messaging
component of GoAhead’s SelfReliant Middleware [14]
provides a common interface for communication between
Linux and VxWorks software components along with a
variety of practical messaging services such as publish–
subscribe, and replicated databases. Messaging within the
Data Processor cluster is via Reliable Platform (RP)
Middleware, which is also responsible for the Software
Implemented Fault Tolerance (SIFT) [15] in the cluster.
Together, the OS and Middlewares provide the base
platform on which other software is implemented.
EAFTC AND RP MIDDLEWARE
EAFTC relies heavily on the use of two key software
components Reliable Platform Middleware (RP), and the
EAFTC controller as described in this section.
EAFTC Controller
The EAFTC controller is the core control of an EAFTC
based system. Since the integrity and dependability of the
entire system relies on the EAFTC controller its realization
must be highly reliable. Hence, we have selected the
implementation of the EAFTC controller as a software
component hosted on a highly reliable System Controller.
This implementation will give the most flexibility for future
use and adaptations.
Figure 1-8 shows internal functions of the EAFTC
controller in the context of a characteristic system
implementation. The details of each internal component in
the EAFTC controller are described below.
EAFTC Controller
Spacecraft
ephemeris Environmental Deployment
Server
SEU Alarm
<<sensor>> Flux Measurments
Deployment
Alert Level
Generator
History
Direction of Data Flow
Figure 1-8 EAFTC Controller Block Diagram
Environmental Server - Given the variety of possible
sensory input, a function has been defined to collect and
organize sensor signals into abstract representations that
may be shared with other EAFTC components. The
Environmental Server encapsulates the low-level interfaces
to each of the sensors in the system, including the sampling
of each signal.
Health Monitor - The Health Monitor is responsible for
monitoring the state of each target system compute
resource. Signals such as heartbeats, redundant output
consistency mismatches, watchdog time-out, etc are
collected via the Fault Tolerant Controller/Node
components and provided to the Health Monitor. Given
predefined policies, the Health Monitor makes a
determination of the health for each Data Processor in the
APC cluster. This information is then shared with the
Deployment Generator where it is used in determining the
system’s task deployment.
History Database - Although reacting to immediate sensory
input may be adequate for many applications, the ability to
predict near future threats to the system can provide
significant advantages. In particular, adapting fault
tolerance to address anticipated threats reduces the exposure
of the system to faults. A History Database is a key
component of the predictive filter implemented in the Alert
Level Generator. Sensor measurements from prior
spacecraft orbits are maintained in this database and
subsequently retrieved by the Alert Level Generator.
Alert Level Generator - The process of evaluating the
environmental threat to the system is implemented in the
Alert Level Generator. Given the current sensory input,
historical database, and a set of system specific thresholds
the Alert Level Generator outputs a discrete threat level for
the system. The core algorithm of the Alert Level
Generator is an Adaptive Linear Predictive Filter that
generates a particle flux prediction. Based on the prediction,
a series of user defined thresholds are evaluated to
determine the current system alert level to be used by the
Deployment Generator in determining the system’s process
deployment.
Deployment Plan - The on-line behavior of an EAFTC
controller varies based on the target environment, system
level requirements, target application, target system
architecture, and other implementation specific factors.
FPGA
This application specific behavior is captured as a user
FPGA
Configuration
Controller
Configuration/
Refresh defined parameter set. In particular, the Deployment Plan
Plan
describes the desired system dependability for a given
Health
Target
Computer spacecraft position, threat level, and time. The Deployment
<<onboard
Generator Monitor Health computer>>
Plan is fine grained and it is defined by the requirements of
CPU
Process
Deployment
each individual application process.
Configuration
Controller
Deployment Generator - Once the system threat level has
been assessed, the Deployment Generator acts to counter
6
the threat. Given the Deployment Plan, target system
health, and alert level, the Deployment Generator produces
a new system deployment. The process of generating a new
deployment is primarily based on determining the lowest
cost distribution of application processes (including number
of replicas) across the available target resources. The
generated deployment is then sent to each node in the
cluster where local actions implemented by the Fault
Tolerant Node software fulfil the deployment requests.
Specifically, the Fault Tolerant Node collaborates with the
RP Middleware, discussed below, to deploy fault tolerance
as requested.
Configuration Controllers - The Configuration Controllers
are each designed to interface with a particular target
system. Given a new deployment, each Configuration
Controller generates the low-level signals to effect required
changes in the target system. In the proposed system, two
Configuration Controller types are implemented. The first is
responsible for interaction with APC nodes operating in
microprocessor mode. The second interacts with APC
nodes operating in custom processor mode.
Reliable Platform Middleware
The role of WW Technology’s RP in the overall EAFTC
solution is to implement SIFT. The RP manages the fault
tolerance of applications and services distributed across
clusters of processors by establishing a consistent
framework and common context in which the system
operates.
Hosted
Applications
Monitoring Services Reliable Platform API
API
System Capability Frame Data Process
Monitoring
Schedule Integrity Group
Service Service Service
• Application Monitoring
• RP Co mponent Monitoring
Application Services
Cluster Services
Local Resource
Monitoring
(Synchronization, Application
Service Management)
Local Services (Scheduler, Networking, OpSys Services)
Native Hardware, Operating System, and Vendor Device Drivers
Figure 1-9. WWTG Reliable Platform Middleware Block
Diagram
RP consists of a set of services that facilitate the
implementation of reliable systems through the dependable
management of redundant/replicated resources. The RP is
ideally suited for addressing the needs of composing
systems utilizing COTS hardware and software
components, as it offers a software based dependability
solution that provides transparent Fault Detection, Isolation
and Removal (FDIR) services, enabling hosted applications
to provide uninterrupted delivery of service in the presence
of faults.
Figure 1-9 shows a block diagram of the RP and its
relations to other software elements of the system. The main
RP framework components are described as follows.
Local Services - These services are local to each processor
in the distributed system. These services provide local
functionality required for a processor to perform useful
work in the cluster. Examples of these types of services
include networking, local scheduling, timing, and inter
process communications.
Cluster Synchronization - A service that establishes a
dependable distributed time base that is consistent across
the entire system. This service is based on a message
passing technique and uses local physical clocks at each
component to form a logical system clock. The Cluster
Synchronization service is scalable and efficiently
establishes the time base across processors. This time base
is used as a backbone for scheduling distributed operations
across the cluster.
System Configuration Services - These services are used to
establish and control the configuration of the cluster. The
cluster configuration comprises the system physical
resources and logical capabilities. The System
Configuration Service interacts directly with the EAFTC
Fault Tolerant Node component, which in turn
communicates with the Fault Tolerant Controller. The
EAFTC controller sends its generated deployment via the
Fault Tolerant Controller/Node to each processor’s System
System Configuration Service where deployment changes are
Organizati on AP I
finally effected.
••• System Capability
Management
• Element Discovery
System Monitoring Services - These services supply the
system with the ability to dynamically assess the health of
the cluster and localize failed processors and application
Local
Resource
processes. Assessments are made with a cluster wide
Management
perspective using distributed decision-making and
integrated monitoring information from across the cluster.
Failure notifications from this service are forwarded to the
EAFTC Health Monitor via the Fault Tolerant
©WW Technology Group 2004 Controller/Node components.
Process Group Management - The proposed approach for
enhancing the availability and dependability of payload
applications relies on replication. The set of replicated
instances are managed as a “process group,” which is a
peer-to-peer entity in which the support services of each
replica are constantly checking the performance/behavior of
its local replica against that of its remote peers.
Scheduling - A scheduling mechanism is available to the
hosted applications. This mechanism will initially provide
7
simple indications to application processes as to when to
perform its execution cycle and when interaction with other
support services may be performed. This scheduling
mechanism is based on the common time base established
through cluster synchronization. Operations controlled by
this scheduling service can be coordinated in time across all
elements of the cluster.
Data Integrity - A data integrity capability ensures
consistent data sets across replicas. A deviation from this
consistent data by a replica is to be interpreted as an error
by that replica. This capability allows for hosted
applications to expose internal state data facilitating warm
starts of additional resources as they come on-line.
Additional replicas may join an established group by
adopting the internal state of the existing replicas.
The RP offers its services in a highly flexible manner,
supporting a distribution of applications that is not
necessarily tied to the physical realization of the cluster.
The RP utilizes a clustering approach to manage a cluster
processor. Application replicates are hosted on each RP-
Enabled resource via the RP Interface (RPI), rendering the
application “unaware” of the fact that it has been replicated,
or to what extent it has been replicated. The RP works in
the background to monitor application behavior and
recognizing when a fault has resulted in application
divergence. Of particular importance, the RP not only
provides dependability to hosted applications, but the RP is
in-and-of itself dependable, capitalizing internally on the
same techniques and properties conveyed to hosted
applications.
The prototype consisted of a combination of Honeywell and
COTS elements. All components were functionality
integrated to include software to represent a functional
EAFTC system. The software included all of the basic
functions of an EAFTC system implemented at an adequate
level of fidelity for TRL4 validation. The prototype was
demonstrated in a lab environment. The space environment
and SEU Alarm response where simulated with data
captured in an Environmental Scenario file derived from
SPENVIS radiation models [17].
SEU Alarm Prototype
An SEU Alarm prototype was implemented and
demonstrated in the context of a REACT system. PSI
demonstrated, through measurements and modeling, that the
SEU alarm has sufficient sensitivity and count rate
capability to detect SEU-producing particles on orbit7.
Furthermore, PSI demonstrated that the SEU Alarm may be
used to adapt fault tolerance on-line.
CONCLUSION
EAFTC is an advanced fault tolerant system paradigm with
potential use in many NASA and DoD missions. As a
follow on to the study effort the NMP is sponsoring a flight
experiment of select technologies. EAFTC is a candidate
for this flight demonstration. The overall goal of the
proposed experiment is to show that EAFTC is a
competitive and a low-risk solution for missions needing
COTS high-performance on-board payload processing.

TECHNOLOGY DEVELOPMENT

EAFTC has been developed to a great extent under the

sponsorship of the NASA New Millennium Program (NMP)
[16]. Under the NMP Space Technology 8 program, a study
phase contract conducted from November 2003 through
July 2004 advanced the EAFTC technology to a
Technology Readiness Level 4 (TRL4.) To this end the
development team implemented prototypes of an EAFTC
system and an SEU Sensor. Additionally, low-fidelity
predictive performance models where developed for each
prototype.

EAFTC System Prototype

An EAFTC system prototype and an associated predictive
model have been implemented with all of the corresponding
functional elements of the EAFTC technology. The
technology components demonstrated in the prototype
system, are a Target Computer, and EAFTC controller. In
addition to the technology components, instrumentation was
integrated with the prototype for measurement of key 7
For the Xilinx FPGAs, protons with energies greater than a few
system performance factors.
MeV have the potential to produce SEUs.

8
 REFERENCES
[1] “An Overview of Earth Science Enterprise”, NASA
Goddard Space Flight Center, FS-2002-3-040-GSFC,
March 2002
[2] Wallace M. Porter And Harry T. Enmark, “A System
Overview of The Airborne Visble/Infrared Imaging
Spectrometer (AVIRIS)”, JPL Pasadena, California
[3] H.L. Huang, “Data Compression of High-spectral
Resolution Measurements”, Satellite Direct Readout
Conference for the Americas, December 2002
[4] J. Marshall and R. Berger, “A Processor Solution for the
Second Century of Powered Space Flight,” Digital
Avionics Systems Conferences, 2000. Proceedings.
DASC. The 19th ,Volume: 2 , 7-13 Oct. 2000,
Pages:8.A.2_1 - 8.A.2_8
[5] Gary R. Brown, “Radiation Hardened PowerPC 603e ™
Based Single Board Computer,” 20th Digital Avionics
Systems, 2001. Oct 2001
[6] E. R. Prado et al.,“A Standard Approach to Spaceborne
Payload Data Processing,” IEEE Aerospace Conference,
March 2001.
[7] F. Irom et al., “Single-Event Upset in Evolving
Commercial Silicon-on-Insulator Microprocessor
Technologies, Nuclear and Space Radiation Effects
Conference 2003
[8] Xilinx Corporation, “QPro Virtex 2.5V Radiation
Hardened FPGA,” Xilinx Web site
http://www.xilinx.com/, Nov. 2001
[9] Daniel P. Siewiorek and Robert S. Swarz, Reliable
Computer Systems Design and Evaluation 3rd edition,
,MA: AK Peters Ltd., 1998.
[10] J.S. Donaldson, “Push the DSP Performance Envelope”,
Xilinx Xcell Journal, Spring 2003
[11] RapidIO Trade Association Web site
http://www.rapidio.org/
[12] Physical Sciences Inc. Web site
http://www.psicorp.com/index.shtml
[13] Wind River Systems Weeb site
http://www.windriver.com/
[14] GoAhead Web site http://www.goahead.com/
9
[15] P. Ellis and C. J. Walter, “Fault Tolerant Discovery and
Formation Protocols for Autonomous Composition of
Spacecraft Constellations,” IEEE Aerospace Conference,
2003, pp 837-852.
[16] New Millennium Program Web site
http://nmp.jpl.nasa.gov/
[17] Space Environment Information System Web site
http://www.spenvis.oma.be/spenvis/
ACKNOWLEDGEMENTS
We would like to thank the following people and
organizations for their contributions to this work: Gary
Galica and Robin Cox from Physical Sciences Inc.; Chris
Walter, Peter Ellis and Brian LaValley from WW
Technology Group; GoAhead Software; NASA New
Millennium Program; and the entire Honeywell team
including John Samson, Lee Hoffman, Jeff Wolfe, and
Jason Copenhaver.
BIOGRAPHIES
Jeremy Ramos has been a Systems
Engineer with Honeywell Defense and
Space Electronic System since 1999. Mr.
Ramos received a B.S. degree in
Computer Science and Engineering from
the University of South Florida. Prior to
his engineering career Mr. Ramos served
for 7+ years with the United States Army as a Technician in
the Army Ordnance Corps. Most recently Mr. Ramos has
served as the Technical Director of the Honeywell
Reconfigurable Space Computer (HRSC) project, MNP
Space Technology 8 Study Phase project, and a number of
other technology development efforts. Mr Ramos’s expertise
includes computer architecture, system simulation, and
reconfigurable computing.
Dean Brenner is a Program Manager for
the Satellite Enterprise Team of
Honeywell Defense and Space Electronic
Systems. He has 14 years of experience in
the field of digital signal processing,
communications systems, and payload
processing development for military space
operational systems. Prior to becoming a Program
Manager for Honeywell, Mr. Brenner was Principle Staff
Systems Engineer for the Advanced Systems engineering
group and supported several advanced flight development
programs including Space-Based Infrared Systems (SBIRS).
Mr. Brenner served as the Principal Investigator for the
NMP Space Technology 8 Study Phase EAFCT Project.
 Chris J. Walter is the founder of WW
Technology Group and has been actively
involved in the field of fault tolerant and
distributed computing for real-time
control systems for over 20 years. Dr.
Walter coauthored an IEEE Computer
tutorial text on Advances in Ultra-
Dependable Distributed Systems. His
work on a fault tolerant “X-by-Wire” architecture was
transitioned to the Navy’s New Virginia Class Submarine
(VSSN) for the fault tolerant Ship Control System. DARPA,
ONR, Army, Navy, Air Force, JPL and NASA have
supported his research. He is a member of the IFIP WG
10.4 on Dependable Computing and of the IEEE Fault-
Tolerant Technical Committee. His interests include
distributed middleware, reconfigurable computing,
systematic design methods and analysis tools for
dependable systems, on-line diagnosis, and real-time
mission critical computing problems.

Gary E. Galica is currently Group

Leader of Radiation Technologies at
Physical Science Inc. (PSI). He has 15
years of experience in developing high-
reliability instrumentation for space,
aircraft and terrestrial application. Since
joining PSI in 1995, he has been involved
in the development of experiments and
instrumentation for a variety of space science and
environmental applications, including several generations
of Light Particle Detectors that measure the high energy
protons, electrons, and alpha particles in space. At PSI,
Dr. Galica and the Radiation Technologies Group have
developed and delivered 5 flight sensors, 3 engineering
model sensors, 5 breadboard sensor and 1 ground test
facility since 1995. Dr. Galica received his Ph.D. in
Physical Chemistry from the Massachusetts Institute of
Technology in 1988, and his B.S. in Chemistry from Tufts
University in 1983. His doctoral thesis work included the
development of novel experimental and theoretical
techniques for the study of photodissociation dynamics in
small polyatomic molecules.

10