Applying IRT & CTT in Situation Awareness Decision-

Making Ability Measure for Pilots (SAD-MAM_P):

A Comparative Psychometric Test Analysis

Prof. Jonathan V. Macayan

Department of Psychology
Mapua Institute of Technology
Situation Awareness Decision-Making
Ability Measure for Pilots (SAD-MAM_P)
◦ To respond to the assessment needs of Civil Aviation
Authority of the Philippines in pilot licensing (Student
pilot – Private pilot: Single engine planes: CESSNA
152M, 172M).

◦ To design less obtrusive assessment techniques in

measuring pilot’s situation awareness (SAD-MAM_P vs.
Actual flight testing and Simulation test).

◦ To address issues on causes of air crash due to human

error as attributed to lack of situation awareness.

Rationale
Prof. Jonathan V. Macayan
MIT-Department of Psychology
 Causes of Fatal Accidents by Decade (percentage)
Cause 1950s 1960s 1970s 1980s 1990s 2000s All

 Pilot Error 40 32 24 25 27 26 29

 Pilot Error  (weather related) 11 18 14 17 21 17 16

 Pilot Error (mechanical related) 7 5 4 2 4 3 5

 Total Pilot Error 58 57 42 44 53 46 50

 Other Human Error 0 8 9 6 8 8 6

 Weather 16 10 13 15 9 9 12

 Mechanical Failure 21 20 23 21 21 28 22

 Sabotage 5 5 11 13 10 9 9

 Other Cause 0 2 2 1 0 1 1

The table above is compiled from the PlaneCrashInfo.com accident database and represents
1,300 fatal accidents involving commercial aircraft, world-wide, from 1950 thru 2009 for which
a specific cause is known. Aircraft with 10 or less people aboard, military aircraft , private
aircraft and helicopters are not included.

Prof. Jonathan V. Macayan

MIT-Department of Psychology
Prof. Jonathan V. Macayan MIT-
Department of Psychology
Prof. Jonathan V. Macayan MIT-
Department of Psychology
SITUATION AWARENESS DECISION-
MAKING ABILITY MEASURE for
PILOTS (SAD-MAM_P)

Prof. Jonathan V. Macayan MIT-

Department of Psychology
SAD-MAM_P is designed to contribute to
the assessment of ability for aviation
schooling in the Philippines. The test is
intended to focus on the level of those
who are flying at single engine planes:
CESSNA 152M, 172M.

The Instrument (SAD-MAM_P)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
24 Item ability test
a. 6 Items (Procedural)
b. 10 Items (Application)
c. 8 Items (Comprehension)

Note: The items in the test aim to measure

situation awareness of pilots that is
reflective of their decision-making by
choosing the most appropriate response in
a given flight situation.

The Instrument (SAD-MAM_P)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
 (Procedural) Item No. 2. Capt. Magno and Capt. Bernardo
are flying their way to Manila from Calapan using an
instrument rated aircraft. They took the route which will
pass over Taal and Tagaytay. Upon flying over Tall, they
noticed that they shall be passing through a bulk of clouds
ahead of them in over Tagaytay. What is the most
important thing that they should do upon penetrating the
clouds?
a. Request for instrument approach.
b. Monitor the attitude indicator to see if they are on level flight.
c. Find an alternate route by using GPS so you may be able to go
out of the clouds.

Sample Items (SAD-MAM_P)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
Scenario
RPC1512 : Domestic ramp, 1512, good morning!
DOMESTIC RAMP : 1512, good morning!
RPC1512 : 1512, on Philcox hangar, request to taxi to Fox 1 Bravo.
DOMESTIC RAMP :Cleared to taxi to Fox 1 Bravo 1512.
(Airplane Taxiing)
RPC1512 : 1512 on Fox 1 Bravo.
DOMESTIC RAMP : 1512 switch to 118 for tower.
RPC1512 : 1512, Switching to frequency 118, good day!
TOWER : Good morning 1512!
RPC1512 : Request for take-off, destination Calapan via Taal, 1512
TOWER : 1512, hold short on Fox 1 Bravo.
RPC1512 : Hold short on Fox 1 Bravo, 1512.

Sample Items (SAD-MAM_P)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
 (Comprehension) Item No. 1. In this
confirmation by the tower and RPC1512, what was
initially requested by RPC1512 to Domestic Ramp?

a. 1512 requested to taxi to F1B

b. 1512 requested to taxi to F2B
c. 1512 requested to taxi out of the hangar

Sample Items (SAD-MAM_P)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
(N = 144) Student Pilots from a private
flying school in Manila.

Gender: Male = n130; Female = n14

Note: The participants have completed the

necessary flying theories in their ground
schooling and are preparing for actual flight
training.

Participants
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 SAD-MAM_P was administered to a randomly selected
participants (N=144) in a private flying school in Manila.
Examinees were given a limited time (35 minutes) to
answer the entire test. Time pressure is incorporated,
since decision-making in actual flights especially in
emergency situations require prompt execution. After
giving complete instructions on the test (including the time
limit), the examinees were requested to answer the test to
the best of their knowledge. Debriefing proceeded after the
completion of the test.

Procedures (Test- Administration)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
CTT IRT (1PL)
 Descriptive Analysis  Unidimensionality
◦ Mean & SD Analysis

 Reliability Analysis  FIT Indices

(Internal Consistency)
◦ Cronbach’s alpha  Reliability Indices
◦ Split-half reliability (Person & Item)

 Item Analysis  Item Difficulty

◦ Item Difficulty = (pH + pL)/2
◦ Item Discrimination = pH – pL

Procedures (Data Analysis)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
 Comparative CTT IRT
Indices

Item Reliability Cronbach’s alpha Item Reliability &

RMSE

Item Characteristic
Item Difficulty Item Analysis
Curve (ICC)
(pH + pL)/ 2

Procedures (Data Analysis)

Prof. Jonathan V. Macayan MIT-
Department of Psychology
CTT IRT (1PL)
 Descriptive Analysis  Unidimensionality Analysis
◦ Mean (18.50) & SD (2.70) (Person Separation Reliability; MSE
=.86 : RSE = .85) = 1.01

 Reliability
 FIT Indices
Analysis
INFIT 1.00; OUTFIT was 1.10
(Internal Consistency)
◦ Cronbach’s alpha (α=.457)
 Reliability Indices
◦ Split-half reliability ( =.398) Person = .43; RMSE of .57
Item = .34; RMSE of .21
 Item Analysis
 Item Difficulty
◦ Item Difficulty (24 items = Easy items- item 2, 3, 6, 7, 9,
10,11,15,18,19,20,22,24
Average)
Difficult items- item 1, 4, 5, 8,
◦ Item Discrimination(5, 7,12,
12, 13, 14, 16, 17, 21, 23
13, and 18 are Reasonably
Good )

Results
Prof. Jonathan V. Macayan MIT-
Department of Psychology
Prof. Jonathan V. Macayan MIT-
Department of Psychology
Prof. Jonathan V. Macayan MIT-
Department of Psychology
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 22 out of 24 items passed the criteria of fit in the
1PL-Rasch Model, which indicates that the items
in the test represent the expected ability of the
target examinees (student pilot).

 The data also supports the assumption of

unidimensionality (1.01) of the construct being
measured by the items (N=24) in the test, which
is the situation-awareness decision-making
ability of pilots.

Discussion
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 CTT and IRT both provided a marginal level of
reliability estimate of the instrument when data
was tested for item consistency (CTT =.478; IRT
=.34); such findings can be attributed to the
quantity and quality of observations (scores) in
the test, where all the participants (N=144)
performed well in answering the majority (at
least 50%) of items correctly (lowest score = 12/
highest = 24; total mean score= 18.50/ standard
deviation = 2.70/ sample variance =7.49).

Discussion
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 The results of item difficulty vary between CTT
and IRT, where all items are classified as average
in the traditional method while the Rasch model
delineated easy and difficult items in the test. For
example, IRT was able to detect the easiest
(Item no. 2) and the most difficult (Item no.5)
items in the test, such findings were not detected
in CTT.

Discussion
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 The analysis of data also provided an important
aspect of the instrument’s validity in terms of fit
(INFIT =1.00; OUTFIT = 1.10). This indicates
that the matching of item difficulty to person’s
ability (or vice-versa) was well-defined and
corresponding.

Discussion
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 Other methods of reliability testing aside from internal
consistency can be recommended (i.e alternate form,
inter-rater, test-retest) to further verify the reliability of
the measure.

 Thispsychometric analysis conducted to SAD-MAM_P was

the first attempt to examine the psychometric worthiness
of the test. Further, analyses are recommended since this
test of ability requires justifiable verification of
essentiality in licensing student pilots. Perhaps, thorough
analysis using 2PL and 3PL models of IRT can be helpful
in establishing a well-calibrated licensing test for pilots.

Recommendations
Prof. Jonathan V. Macayan MIT-
Department of Psychology
 THANKS FOR
LISTENING!!!
Prof. Jonathan V. Macayan
Department of Psychology
Mapua Institute of Technology