Engineering The Architecture Of

Distributed Control Systems

January 2002
Eric Runnerstrom
MPR Associates
erunnerstrom@mpr.com
703-519-0200

Distributed Systems Architecture

This methodology was developed by MPR to enable the rigorous

design of distributed control systems that will perform as required.
The methodology was developed and implemented as part of the
Navy Research Laboratorys Damage Control Architecture for
Reduced Manning (DC-ARM) Program.
The methodology ensures that the technology demonstrated by the
DC-ARM program can be effectively applied to other platforms.
It also could be be applied to other types of distributed systems,
such as distributed sensor systems.

Jan. 2002 2

Distributed Systems Architecture

Outline

Definition Of Supervisory Control System

Premise
The Process
Firemain Example
Further Development

Jan. 2002 3

Distributed Systems Architecture

Definition Of Supervisory Control System

A system, automated to some extent, that:
Monitors and controls multiple, integrated sub-systems
Enables a human supervisor to interact with the sub-systems
Provides the information necessary so that the responses of
sub-systems and the actions of personnel can complement one
another.

Jan. 2002 4

Distributed Systems Architecture

Premise
Projects that implement complex control systems
often experience:
Cost overruns
Schedule delays
Performance that does not meet expectations, even after
additional delays and costs, and substantial changes to
the system

Performance
Performanceproblems
problemsare
areexacerbated
exacerbatedwhen
when
systems
systemsoperate
operateoff
offdesign
designor
orequipment
equipmentfails.
fails.
Jan. 2002 5

Distributed Systems Architecture

Premise
The principles for designing distributed control systems
are evolving
Industry is able to build distributed control systems, but is still
learning how to engineer their design

Engineering the architecture of distributed control systems

will
Provide a basis for preventing the cost, schedule, and performance
problems often experienced during development
Enable optimizing the architecture of the control system with
respect to acquisition program criteria
Enable effective human-systems integration

AAmethodology
methodologyfor
forengineering
engineeringthe
thearchitecture
architectureof
of
distributed
distributedcontrol
controlsystems
systemsadvances
advancesthe
thestate-of-the-art.
state-of-the-art.
Jan. 2002 6

Distributed Systems Architecture

The Process
Define Control Decisions
Develop Control Decision Logical Architecture
Define Candidate Hardware Architectures
Evaluate Candidate Hardware Architectures &
Select Optimum
Develop Software Architecture

Jan. 2002 7

Distributed Systems Architecture

1. Define Control Decisions

Functional Analysis to define control decisions
Characterize Fire

FIRE
LOCATION

FIRE SEVERITY

No Boundary
Needed

NO FIRE
Fire Severity?

"SMALL" FIRE

Characterize Damage Top Level

Compartments
Containing Damage

"MEDIUM" FIRE
"LARGE" OR
"FULLY DEVELOPED"
FIRE

Compartment
Characteristics
Determine
Boundary
Compartments

Boundary
Needed

Water Mist
Functioning In
Boundary Spaces?

YES

Maintain Fire
Boundary With
Water Mist

Maintain
Boundary With
Water Mist

Calculate Time For

Fire Space To Reach
500 C = T(500)

** TV = Time To
Vertical Spread =
T(500) + 5 Min.

** TH = Time To
Horizontal Spread
= T(500) + 9 Min.

Can Personnel Set

Boundary Before Fire
Spreads?

Define
Operational
Priorities

Control Water
Mist System

NO

NO
Predict
Personnel
Performance

YES

Water Mist
Functioning In
Fire Space?

Characterize
Fire
FIRE PREDICTED
IN COMPARTMENT

Functional Analysis Enables:

System functional requirements driven
by top level program requirements
Fundamental integration of:
Sub-systems with one another
Sub-systems and controls
Humans and sub-systems

Database of attributes for each control

decision

NO

YES

Compartment
Characteristics

Is Boundary Space High

Enough Priority To Apply
Personnel?

YES
Maintain
Boundary With
Personnel

NO

Maintain Fire
Boundary With
Personnel

Jan. 2002 8

Distributed Systems Architecture

2. Develop Control Decision Logical Architecture

Three levels of control logic for systems or compartments:
Device Level Logic

Device Level Control Logic requires only inputs

available at the controlled device itself
Similarly, Compartment Level Control Logic requires
only inputs available at the compartment itself

System Level Logic System Level Control Logic requires inputs from

more than one device in the system

Similarly, Zone Level Control Logic requires inputs
from more than one compartment in the zone

Ship Level Logic

Ship Level Control Logic requires inputs from more

than one system or zone

Other applications could have different boundaries for the control decision logic levels.
Jan. 2002 9

Distributed Systems Architecture

2. Develop Control Decision Logical Architecture

Ship Level Logic requires
input from more than one system.

System Level Logic requires

inputs from more than one
device in the system.

Ship Level
Logic

System
A
Logic

Device Level Logic requires

only inputs available at the
device itself.
Smart
Device #1
Device Logic
Sensors/Input Actuator

Smart
Device #2

Example: Correlation of firemain

status & compartment status is
plant level logic.

Example: Firemain flow

balance is system level logic.

System
B
Logic

Example: DC-ARM
Smart Valve uses
device level logic.

Device Logic

Device #3
Sensors/Input

Sensors/Input Actuator
Jan. 2002 10

Distributed Systems Architecture

2. Develop Control Decision Logical Architecture

Functional requirements for the control system are
defined by:
Definitions and attributes of control decisions
Control decision logical architecture

Consider providing redundant logic for the same

function to enhance reliability and survivability
Recognize trade-off of increased complexity and cost
Control
Controldecision
decisionlogical
logicalarchitecture
architectureprovides
providesaabasis
basisfor
for
the
thesynthesis
synthesisof
ofcandidate
candidatehardware
hardwarearchitectures
architecturesthat
that
meet
meetthe
thesame
samefunctional
functionalrequirements.
requirements.
Jan. 2002 11

Distributed Systems Architecture

3. Define Candidate Hardware Architectures

Hardware Architecture =
Control functions that will be performed on each
processor
Where processors will be located that collect data and
execute control decisions
Redundancy and separation of processors
Redundant processors can improve reliability
Separated, redundant processors can improve
survivability
Consider need for communications among data sources
and processors
Communications load typically increases
significantly during a casualty
Jan. 2002 12

Distributed Systems Architecture

3. Define Candidate Hardware Architectures

Processors can be located at:
Device Level; e.g. valves, pumps,
fans, dampers, A/C plants
Compartment Level
System Level; e.g. controllers for
firemain, ventilation, chilled
water, electrical distribution
Zone Level
Ship Level
Other locations

Decision logic can be

executed in processors at
almost any level; for example,
extremes include:
All decisions executed at device
level processors
All decisions executed at a ship
level processor

Hardware architecture that

mirrors decision logical
architecture may provide best
performance

Communications
Communicationsload,
load,cost,
cost,reliability
reliability&
&survivability
survivabilitycan
canvary
vary
significantly
significantlydepending
dependingupon
uponthe
thehardware
hardwarearchitecture.
architecture.
Jan. 2002 13

Distributed Systems Architecture

4. Evaluate Candidate Hardware Architectures

Criteria are determined by the acquisition program
Typical criteria might include:

Acquisition Cost
Life-Cycle Cost
Reliability
Survivability
Ease of Troubleshooting and Maintenance
Affordability of Likely Upgrades

Select
Selectthe
thehardware
hardwarearchitecture
architecturethat
thatprovides
providesthe
thebest
best
balance
balanceamong
amongacquisition
acquisitionprogram
programcriteria
criteriaand
andpriorities.
priorities.
Jan. 2002 14

Distributed Systems Architecture

5. Develop Software Architecture

Modular software architecture that mirrors control
decision logical architecture probably is a good
baseline
Provides logically consistent basis for:
Development
Troubleshooting
Maintenance
Maximizes plug and play capability for upgrades

Adjust software architecture considering:

Hardware architecture
Communications
Jan. 2002 15

Distributed Systems Architecture

DC-ARM
SHADWELL Firemain Example
Following slides are an example of applying
the architecture engineering process to the
DC-ARM firemain aboard SHADWELL.

Jan. 2002 16

Distributed Systems Architecture

Firemain Example - Functional Analysis

Firemain

Firemain Self Monitor

Monitoring Data**

Pump, Valve, & Pipe Segment Availability

Pump Status & Valve Position
Valve & Pump Response To Commands
Pressures

**Monitoring Data includes data from sensors for component material condition,
component operating data, and system hydraulic data, as needed.
Human Supervisor's Evaluation

Assess Material Condition & Readiness of

Firemain

Commands To Operate
Pumps & Valves

PM & CM Performed

Component Tag-Out
Preventive & Corrective
Maintenance Requirements

Control Firemain &

Provide Information To
Isolate Ruptures &
Maintain Vital Services

Component Material
Condition & Readiness

Perform Firemain Maintenance

Operating Status of Firemain: Pump & Valve Status;

Fluid Pressures, Flows, Temp., Etc., As Needed.

Does Condition
Indicate Probable
Damage?

NO

Fire Plug Availability

Fire Plugs Vital To Assigned Tasks

Readiness Information for Each Firemain Service;

E.G. Fireplugs, Sprinkling, AFFF, Etc.
Continue

YES
Location/Compartment of Damage
Description of Damage

Evaluate Readiness of
Firemain

Typically, data is passed to links without waiting for human

supervisor's assessment. Supervisor's assessment, when
available, updates and may override previous assessments.

Characterize
Damage - Top
Level

Predict
Personnel
Performance

Maintain
Boundary With
Personnel

Attack Small,
Medium, Large, Or
Fully Developed
Fire

Control Firemain

Link To Monitor Ship Systems,

Compartments, & Pre-Damage
Predictions

Link To Attack
Major Fire

Link To Set
Boundaries

Link To
Restore
Firemain

Link To
Firemain
Reflexive
Operation

Link To Respond To
Probable Fire & Extinguish
Minor Fire

Maintain Firemain
Jan. 2002 17

Distributed Systems Architecture

Firemain Example - Functional Analysis

Examples of attributes defined for each action in the
functional analysis
Level in control logical hierarchy
Device, system, or ship
Primary or back-up means of accomplishing the action
Discrete or continuous action
For discrete actions, need an initiating event
Intended effect of the action
Effects if the action is not performed when it should be
Effects if the action is performed when it should not be
Allocated to machine or human
Inputs required and locations of input sources
Outputs and locations of users of the output
Jan. 2002 18

Distributed Systems Architecture

Firemain Example - Decision Logical Architecture

Ship Level Control Decisions
Provide information to the human supervisor to isolate ruptures and maintain vital
services
Provide information to the human supervisor about the operational status of firemain
components
Start other pump if operating pump is in rupture segment

System Level Control Decisions

Isolate ruptures by closing valves closest to the rupture
Redundant with device level rupture isolation
Flow balance logic
Inputs required from multiple valves
Logic is different from device level to minimize common mode failure

Assess material condition and readiness of firemain

Inputs required from multiple devices

Device Level Control Decisions

Isolate ruptures by closing valves closest to the rupture
Rupture path logic
Inputs and actuator are at the valve
Jan. 2002 19

Distributed Systems Architecture

Firemain Example - Candidate HW Architectures

Device Level Processors
Rupture path device level logic - YES
Maximum survivability
Excellent graceful degradation
Simple
Minimum impact on data communications
Acceptable cost
Flow balance system level logic - NO
Survivability not enhanced beyond rupture path device level
logic
Logic to achieve graceful degradation likely to become
unmanageably complex
Increases data communications load
Provide information to the human supervisor to isolate ruptures
and about firemain operability - NO
HCI is not applicable to device level processor
Jan. 2002 20

Distributed Systems Architecture

Firemain Example - Candidate HW Architectures

System Level Processor(s)
Rupture path device level logic - NO
Execute at device level
Flow balance system level logic - CANDIDATE
May enhance survivability if:
Separated from ship level processor
Ship level processor executes redundant logic
Dynamic application reallocation is provided

Some increase in data communications load

Increased cost for: additional computer, associated communications, more
complex software for dynamic application reallocation
Additional redundant computers may increase survivability
Start other pump if operating pump is in rupture segment - CANDIDATE
May be a candidate for system processor if automated logic can be developed

Provide information to the human supervisor to isolate ruptures and about

firemain operability - NO
HCI is not applicable to system level processor
Automated assessment of firemain operability could be done at a system level
processor if applicable logic is developed
Jan. 2002 21

Distributed Systems Architecture

Firemain Example - Candidate HW Architectures

Ship Level Processor
Rupture path device level logic - NO
Execute at device level
Flow balance system level logic - CANDIDATE
May enhance survivability if:
Ship level processor is redundant and separated from system level computer

Some increase in data communications load

Start other pump if operating pump is in rupture segment - CANDIDATE
Without logic for automation, a human decision is needed - HCI is at the ship level
processor

Provide information to the human supervisor to isolate ruptures and about

firemain operability - YES
HCI is at ship level processor

Jan. 2002 22

Distributed Systems Architecture

Firemain Example - HW Architecture Decision

Execute system level logic in
redundant, separated computer

More survivable
Less processor usage
More complex
Higher data communications load
More hardware cost
Higher development cost
More equipment to maintain

OR

Execute system level logic in ship

level computer

Less survivable
Greater processor usage
Less complex
Lower data communications load
Less hardware cost
Lower development cost
Less equipment to maintain

Could eliminate redundant system level logic for rupture isolation

No automatic back-up for device level control
Minimum cost and complexity

Quantitative analyses could be performed to assess options

Communications have a significant influence on survivability
Jan. 2002 23

Distributed Systems Architecture

Firemain Example - Software Architecture

Firemain Device Level Software Modules
Rupture Isolation - Rupture Path Logic

Firemain System Level Software Module

Redundant Rupture Isolation - Flow Balance Logic
Firemain Readiness

Firemain Ship Level Software Module

Display Information To Isolate Ruptures & Maintain Vital Services
Display Information To Enable Assessment Of Firemain Operability
Remote Control Of Valves & Pumps
Jan. 2002 24

Distributed Systems Architecture

Further Development
Extend to compartment-zone structured architecture
Fire detection
Access closure status
Watermist actuation

Investigate integration of device-system structured

architecture and compartment-zone structured architecture

Jan. 2002 25