l-l

TACTICAL DECISION MAKING UNDER STRESS (TADMUS): MAPPING A PROGRAM

OF RESEARCH TO A REAL WORLD INCIDENT-THE
USS VINCENNES.
Joan Hall Johnston, Jams A. Cannon-Bowers, & Eduardo Salas
Naval Air Warfare Center Training Systems Division Code 4961
12350 Research Parkway
Orlando, FL 32826-3275, USA
ABSTRACT
Such events as the one involving the USS Vincennes, where the
decision to initiate countermeasures was the incorrect one, have
focused attention on the human factor in decision making. The
objective of the TADMUS program has been to apply
developments in decision theory, individual and team training,
and information display to the problem of enhancing tactical
decision quality under conditions of stress. Sponsored by the
Offtce of Naval Research, TADMUS is in its 8th year as a
cooperative program in human factors and training involving
SPAWAR Systems Center, San Diego, NAWCTSD, as well as
other Navy, industrial, and academic organizations. The
technology is being demonstrated and evaluated in the context
of surface ship air warfare scenarios. This address will describe
how the TADMUS program was founded and how it has
progressed on a variety of R&D issues having to do with
advanced training and human factors in order to address real
world problems.
INTRODUCTION
On JuIy 3, 1988 an AEGIS cruiser named the USS
VINCENNES accidentally shot down a commercial aircraft,
Flight 655, over the Arabian Gulf killing 290 people. A
number of consequences resulted from this catastrophe that
included the initiation in 1990 of an Office of Naval Research
sponsored research and development program named Tactical
Decision Making Under Stress (TADMUS).
The major
program goal has been to prescribe empirically-based
principles and guidelines to enhance team tactical decision
making performance in complex, knowledge rich
environments. Therefore, program objectives have focused on
applying developments in decision theory, individual and team
training, and information display to the problem of enhancing
tactical decision quality under conditions of stress (Ref 1).
TADMUS is in its eighth year as a cooperative program in
human factors and training involving the Naval Air Warfare
Center Training Systems Division, Space and Naval Warfare
Systems Center, San Diego as well as other Navy, industrial,
and academic organizations. To date, the written products
generated from this program in its first six years number more
than 200 publications in the form ofjoumal articles, technical
reports, book chapters, and symposium proceedings. In
addition, over 100 product transitions have been provided to the

fleet training communities in the form of lectures, workshops,

demonstrations and implementations of training tools, methods,
and strategies. Finally, three large-scale advanced embedded
training research initiatives have resulted (Ref 2; Ref 13). This
paper describes how the strategic planning for the TADMUS
program enabled us to design a roadmap for achieving a
program of research accomplishments for advancing current
and future developments of Navy combat team training.
BACKGROUND

AND APPROACH

Three TADMUS program thrusts were identified for the

purpose of meeting its program goal: definition and
measurement of critical decision tasks; examination of stress
effects on decision making; and development of training and
simulation principles (Ref 1). Program success would be
assessed by the emphasis on empirical research, rapid
transition of research products to the Navy training
community; and research findings that would support the
development of advanced team training technologies. Below
is a description of program requirements that defined our
strategy for meeting program goals.
Technical Advisors Board. A Technical Advisory Board
populated by senior researchers and high level fleet
representatives was appointed to review technical progress
twice per year until program completion. Without a doubt, the
frequent meetings ensured that researchers remained focused
on areas of work that were going to bear fruit; overall, the
Technical Advisory Board helped keep the research on track.
Subiect Matter Expertise. Inclusion of Navy subject matter
expertise, training commands, and ship teams in the design
and development of research experiments and products was an
integral part of the TADMUS plan, To date, virtually
hundreds of active duty officers and enlisted men and women
have played a role in shaping the research products so they
would have a realistic and substantive impact on current and
future Navy training.
Defining and Designing a Realistic Team Task. The
VINCENNES incident was due, in part, to errors resulting
from the Air Warfare (AW) team interactions, therefore, the
research domain was to examine the AW task in a ships
Combat Information Center (CIC). The CIC is the central
information processing and tactical decision making area for a

Paper presented at the RTO HFM Symposium on Collaborative Crew Performance in Complex Operational
Systems, held in Edinburgh, United Kingdom, 20-22 April 1998, and published in RTO MP-4.

1-2
surface combatant (Ref 4). The main focus of research would
be the Air Warfare team on an Aegis capable ship, which is
composed of the Commanding Officer, Tactical Action
Officer, Air Warfare Coordinator, Tactical Information
Coordinator, Identification Supervisor, and the Electronics
Warfare Supervisor. During air warfare, the AW team
performs a series of tasks including detecting, tracking,
identifying radar contacts, taking action on these contacts, and
performing battle damage assessment (Ref 4).
Once the operational task was determined, then choices had to
be made regarding a team-based research testbed. High costs
and lack of access meant that utilizing a ships CIC or a shorebased high-fidelity team trainer was out of the question.
Therefore, a five-person networked PC-based system called
the Decision Making Evaluation Facility for Tactical Teams
(DEFTT) was developed to support research experiments.
Although, as a result of a research tradeoff, DEFTT was low
in functional fidelity, it was determined that the system
presented the tactical problem with enough cognitive fidelity
that we could simulate the actual AW team activities taking
place in the real combat system (Ref 5; Ref 6).
Teams. Without a doubt, conducting team experiments was
the most challenging of all the research tasks. We determined
that most of the experiments would include fve-person teams.
Consequently, as with developing DEFTT, it was not realistic
to expect our research participants to be highly experienced
operators that had worked together as a team for an extended
period of time (i.e., intact). Therefore, we developed a plan so
that we could eventually gain access to such teams. We
organized our research tasks to be conducted at three
shorebased training commands where we could have access to
novice, experienced, and very experienced trainees. In
addition, we spent the first two years of the program at a high
fidelity combat team training facility where we conducted
numerous interviews and developed a database from which
our research scenarios were developed. We were able to
establish DEFTT at one of the school commands, and another
command had already adopted it as a trainer for division
officers, department heads, and prospective commanding
officers. In addition, we established a low fidelity 3-person
team trainer at a school command that allowed novice Navy
recruits to participate in some of the basic research
experiments. To date, over 280 teams of Navy trainees have
participated in the research. Once we had established a
reliable research protocol we were able to have intact and ad
hoc experienced ship teams participate in the research (over
IO teams to date), as well.

Event-Based Scenarios, Stressors. and Measurement. To

ensure that the innovative training could be evaluated, we
developed a strategy-the
Event-Based Approach to Training,
or EBAT-to
support research scenario design, measurement

tool development, and operational stressor implementation

(Ref 7; Ref 8). We structured two pairs of AW Arabian Gulf
scenarios. Each pair was composed of one low and one high
stress scenario, but both shared the same events. Stressors
were defined as workload (e.g., added aircraft and ships) and
information ambiguity (e.g., increased number of difficult
problems to solve). Each scenario was developed with three
significant events whereby individual and team behaviors
could be specified by subject matter experts, and observed by
trained raters (e.g., two hostile aircraft popup at close range to
the teams own ship). The EBAT strategy thus provided a
way to ensure we could assess individual and team
performance with measurement tools that were designed to
capture performance processes and outcomes (Ref 7). Next is
a brief description of the tools as they related to EBAT.
The Behavior Observation Booklet was designed for assessing
individual task processes. At each scenario event, we
identified task requirements at the individual operator level so
that they had a performance score for each event. An outcome
score measured by the Sequenced Actions and Latencies
Index represents the ability to perform the tasks correctly and
on time. The Air Warfare Team Observation Measure
assesses team level performance for information exchange,
initiative, supporting behaviors, and communication. The
team outcome measurement tool (Air Warfare Team
Performance Index) assesses timeliness and accuracy as a
team on the detect-to-engage sequence. As a result of
developing these tools, we were able to assess the
stressfulness of research scenarios, and to guide assessment of
the impact of the TADMUS training (Ref 9).
The Training Research Agenda. In the beginning of
TADMUS, syntheses of the research literature on decision
making, teams, and stress were conducted to identify and
develop innovative training strategies (Ref 1). Consequently,
a research agenda was designed so as to test the individual
effects of such training on enhancing skills in critical thinking
for decision making, teamwork and team self-correction,
handling stress exposure, and leadership (Ref 10). In addition,
training strategies and methods were tested to assess the
impact of part task training, cross-training, and instructor
training to enhance performance feedback strategies (Ref I I ;
Ref 6; Ref 12). The data collection effort started in 1992 and
continues through 1999 in order to establish the impact of an
integrated training program with the decision support system
developed by SPAWAR in San Diego. To date, empirical
data has been collected from over 95 five-person teams
(including experienced intact ship teams) and findings have
shown training imposed significant improvements in
performance, often at levels of 30-40 percent (Ref 10).
Product Transitions: Short-term and Long Term. Although an
applied research program, emphasis was placed on ensuring
the fleet would gain short and long term benefits from our

l-3
findings. In the short-term, numerous such activities as
workshops and demonstrations have and continue to take
place. As an example of building a strategy to transition
training to the shipboard environment, we demonstrated
empirical support for training teamwork skills (Team
Dimensional Training) in the laboratory and then in the
shipboard environment, whereby we eventually gained the
support and endorsement of the afloat training group for
incorporating and implementing Team Dimensional Training
(Ref 12; Ref 13). For the long-term, we have initiated several
advanced research programs to ensure that shipboard
embedded training that includes our TADMUS training
methods, tools, and strategies will be incorporated into new
ship platforms of the 21 Century (Ref 2).

6.

Kirlik, A., Rothrock, L., Walker, N., & Fisk, A. D.,

Simple Strategies or Simple Tasks? Dynamic Decision
Making in Complex Worlds, in the Proceedings of
the Human Factors and Ergonomics Society 40th Annual
Meeting, Human Factors and Ergonomics Society, 1996,
I, pp. 184-188.

7.

Hall, J. K., Dwyer, D. J., Cannon-Bowers, J. A., Salas, E.,

& Volpe, C. E., Toward Assessing Team Tactical
Decision Making Under Stress: The Development of a
Methodology for Structuring Team Training Scenarios.
in the Proceedings of the 15th Annual
InterserviceiIndustry Training Systems Conference,
December 1993, pp. 87-98.

SUMMARY

8.

Oser, R. L., Cannon-Bowers, J. A., Dwyer, D. J., & Salas,

E., Establishing a Learning Environment for JSIMS:
Challenges and Considerations, in the Proceedings of
the 19h annual InterserviceiIndustry Training, Simulation
and Education Conference, December 1997, pp. 144- 153.

9.

Johnston, J. A., Smith-Jentsch, K. A., & Cannon-Bowers,

J. A., Performance Measurement Tools for Enhancing
Team Decision Making, in M. T. Brannick, E. Salas, &
C. Prince (Eds.), Team Performance Assessment and
Measurement: Theory, Methods, and Applications,
Mahwah, NJ: USA, LEA, 1997, pp. 31 l-327.

AND CONCLUSIONS

We have described the strategy we used to guide us on the

roadmap for achieving the TADMUS objectives. The
Technical Advisory Board, subject matter expertise, research
task design and team participants, EBAT, the research agenda,
and short and long-term transitions were crucial to ensuring
the programs ongoing success. In conclusion, the vital
components in all of these tasks were: (1) ensuring that
empirical results were based on a reasonably realistic task that
included team participation, and (2) that fleet participationthe customer-had
input throughout the program.
REFERENCES
1. Cannon-Bowers, J.A., Salas, E., & Grossman, J.D.,
Improving Tactical Decision Making Under Stress:
Research Directions and Applied Implications, in the
International Applied Military Psychology Symposium,
1991.
2.

Naval Air Warfare Center Training Systems Division.

Research and Development Project Summaries.
NAWCTSD, October 1997.

3.

Zachary, W., Bilazarian, P., Bums, J., & Cannon-Bowers,

J. A., Advanced Embedded Training Concepts For
Shipboard Systems, in the Proceedings of the 19h
annual Interservice/Industry Training, Simulation and
Education Conference, December 1997, pp. 670-679.

4.

CG 52 Class Combat System Doctrine, Naval Surface

Force. COMNAVSURFLANT
Instruction C35 16.39.
March 1989.

5.

Holl, R. E. & Cooke, J. R., A

decision (RPD) model of rapid
Proceedings of the I lth annual
Training Systems Conference,

Recognition-primed
decision making, in the
Interservice/Industry
1989, pp. 337-342.

10. Salas, E., Cannon-Bowers, J. A., & Johnston, J. H.,

How can you turn a team of experts into an expert team?
Emerging training strategies, in C. Zsambok & G. Klein
(Eds.), Naturalistic decision making , Hillsdale, NJ,
USA, LEA, 1997, pp. 359-370.
11. Duncan, P. C., Rouse, W. B., Johnston, J. H., CannonBowers, J. A., Salas, E., & Bums, J. J., Training Teams
Working in Complex Systems: A Mental Model-Based
Approach. in W. B. Rouse (Ed.), Human/technology
interaction in complex systems, 8, 1996, pp. 173-23 1.
12. Smith-Jentsch, K.A. Johnston, J.H., Cannon-Bowers,
J.A., & Salas, E., Team Dimensional Training: A
Methodology for Enhanced Shipboard Training, in the
Proceedings of the 1gth annual meeting of the
Interservice/Industry Training, Simulation and Education
Conference, December 1997.
13. Smith-Jentsh, K.A., Payne, S. C, & Johnston, J.H.
Guided Team Self-Correction: A Methodology for
Enhancing Experiential Team Training, in K.A., SmithJentsch (Chair), When, How, and Why Does Practice
Make Perfect?, Paper presented at the 1 llh annual
conference of the Society for Industrial and
Organizational Psychology, 1996.