Validity:

Factor Analysis
Assessment Institute
by
Dena A. Pastor, Ph.D.
James Madison University


Scenario
You have been charged with
assessing the following goal for an
educational program at your
university:
GOAL: As a result of completing this
program, the student will increase in the
extent to which they know and care
about multicultural issues

MASQUE
To assess this goal, you administer the Munroe
Multicultural Attitude Scale Questionnaire
(MASQUE; Munroe & Pearson, 2006) to students
before and after the educational program
MASQUE consists of 18 Likert items
1 = strongly disagree, 6 = strongly agree
The MASQUE is intended to measure three
domains:
(a) a knowledge domain, assessing the extent to which one
knows about multicultural issues,
(b) a care domain, assessing the extent to which one cares
about multicultural issues, and
(c) an act domain, assessing the extent to which one acts to
solve multicultural problems
Domains are hierarchical
Validity of MASQUE
Our concern are the inferences we are
making based on the MASQUEs scores
correct?
Validity refers to the degree to which
evidence supports the inferences made
from the scores
Evidence can be: experts evaluation of the
instrument, relationship between MASQUE and
measures of similar or different constructs,
theoretically expected group differences on
MASQUE


Validity of MASQUE
Acquiring validity evidence for the
MASQUE may also entail a study of the
hypothesized relations among the items
and the quality of the items
One method of acquiring such information is
factor analysis
These studies should be some of the first
studies executed in building the case for
the validity of an instruments scores
(Benson 1998; Benson & Hagtvet, 1996)

Item Number Item Content Subscale
1 I realize that racism exists. Know
2 I know that social barriers exist. Know
3 I understand that religious beliefs differ. Know
4 I understand that sexual preferences may differ. Know
5 I understand that gender-based inequities exist. Know
6 I accept the fact that languages other than English are spoken. Know
7 I do not understand why people of other cultures act differently. Know
8 I am sensitive to respecting religious differences. Care
9 I am sensitive to differing expressions of ethnicity. Care
10 I am emotionally concerned about racial inequality. Care
11 I am sensitive toward people of every financial status. Care
12 I am not sensitive to language uses other than English. Care
13 A person's social status does not affect how I care about people. Care
14 I do not act to stop racism. Act
15 I actively challenge gender inequities. Act
16 I do not actively respond to contest religious prejudice. Act
17 I respectfully help others to offset language barriers that prevent communication. Act
18 I do not take action when witnessing bias based on people's preferred sexual orientation. Act
Sum items for each of the three subscales. Sum all items for an overall score.
See:
Munroe, A., & Pearson, C. (2006). The Munroe multicultural attitude scale questionnaire.
Educational and Psychological Measurement, 66(5), 819-834.
Know Care Act
Table of Specifications for MASQUE (illustrating how to score MASQUE)
Hypothesized Relationships
Latent
Constructs
Items
Know Care Act
Latent
Constructs
Items
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
If this model is correct, we would expect a particular pattern to the
item correlation matrix:
Items tapping
into the same
dimension
should have
stronger
correlations
(purple) than
items tapping
into different
dimensions
(tan)
Information Obtained from
an Exploratory Factor Analysis (EFA)
Based on item correlation matrix, we can use EFA to
explore the:
Number of factors or dimensions underlying item
responses
With MASQUE, we would hope to see 3 factors
Relationship of items with factors
With MASQUE, we would hope to see items 1-7 on factor 1, 8-
13 on factor 2 and 14-18 on factor 3
Relationship among factors
With MASQUE, we would hope to see moderate positive
correlations among factors, with adjacent factors more
strongly correlated

Uses of Factor Analysis
To examine whether test blueprint
holds
Are items hanging together in the
manner intended?
Are there nuisance factors we need to
diminish?
To decide how to score the items
Total score or subscale scores?
Report reliability for total scores or
subscale scores?
Construct
validity
questions
EFA vs. PCA & CFA
We will cover only Exploratory Factor
Analysis (EFA) used with polytomous
items (more than 2 response
categories)
EFA is similar but not the same as
Principal Components Analysis (PCA)
If think there is an underlying construct
driving your responses to items, use EFA
EFA is different from Confirmatory Factor
Analysis (CFA) which is more rigorous
If not sure about # factors, use EFA
Statistics Self-Efficacy Example
Responses to six items are collected from a
large group of college students
X1 through X3 are three different items
written to measure self-efficacy to learn
statistics
X1 X3 a.k.a. SELRN1 SELRN3
X4 through X6 are three different items
written to measure current statistics self-
efficacy (response scale 1-6)
X4 X6 a.k.a. CSE1-CSE3
Do 2 factors underlie the responses to the 6
items?
Responses from 112 college students
Each item responded to on a 1-6 scale
1 = no confidence at all
6 = complete confidence
Commonly used
software to perform
exploratory factor
analyses:
-SAS
-SPSS
Note: item
responses should be
polytomous, with at
least 4 response
categories; if not,
other FA techniques
need to be utilized
Correlation Matrix
0 . 1 81 . 58 . 33 . 42 . 46 . 6
0 . 1 67 . 32 . 43 . 46 . 5
0 . 1 32 . 42 . 36 . 4
0 . 1 87 . 68 . 3
0 . 1 76 . 2
0 . 1 1
6 5 4 3 2 1
X
X
X
X
X
X
X X X X X X
Start off with a correlation matrix
eyeball the matrix for patterns
Items X1, X2, & X3
seem to correlate
positively and strongly
with one another
Items X4, X5, & X6 seem to
correlate positively and
strongly with one another
Note how X1,X2 & X3
and X4, X5, X6 dont
seem to correlate as
strongly with one
another
Two factor solution
looks plausible
Total Variance Explained
3.636 60.594 60.594 3.407 56.781 56.781 2.312 38.536 38.536
1.297 21.620 82.214 1.094 18.233 75.014 2.189 36.479 75.014
.468 7.798 90.013
.307 5.114 95.126
.176 2.940 98.066
.116 1.934 100.000
Factor
1
2
3
4
5
6
Total % of Vari ance Cumulative % Total % of Vari ance Cumulative % Total % of Vari ance Cumulative %
Initial Ei genval ues Extracti on Sums of Squared Loadi ngs Rotati on Sums of Squared Loadings
Extracti on Method: Pri nci pal Axi s Factoring.
What do we get out of Factor Analysis?
Pattern Matrix
a
.686 .167
.975 .004
.936 -.090
.072 .652
-.072 1.004
.016 .836
SELRN1
SELRN2
SELRN3
CSE1
CSE2
CSE3
1 2
Factor
Extraction Method: Principal Axis Factoring.
Rotation Method: Oblimin with Kaiser Normalization.
Rotation converged in 7 iterations.
a.
Factor Correlation Matr ix
1.000 .506
.506 1.000
Factor
1
2
1 2
Extraction Method: Principal Axis Factoring.
Rotation Method: Oblimin with Kaiser Normalization.
Use this
information to
decide on # of
factors to
retain.
If we decide
to retain 2
factors,
well obtain
the
correlation
between
factors as
well as the
relationship
between
each item
and each
factor. We
use the
latter in
naming
the factor.
Factor Analysis Steps
Start with your correlation matrix
Is it appropriate to compute correlations?
Are relationships linear? Are item responses
continuous? Are correlations sizeable?
Extract factors
Among other things, the extraction method
gives rise to factor eigenvalues, used in the
next step
Decide how many factors to retain
Rotate factor pattern matrix to obtain an
interpretable solution
Interpret rotated factor pattern &
structure matrix
Number of Factors to Retain
Several Rules of Thumb you can use to
decide how many factors to retain:
Keep as many factors as have eigenvalues > 1
Keep as many factors as explain some pre-
specified % of variance
Examine Scree Plot
Preferred method over eigenvalues > 1
Explore many solutions and interpret results
Ultimate goal is to have an interpretable
solution! (does the way in which the items relate to factors
in final rotated pattern/structure matrix make sense?)


Use
more
than one
rule in
making
your
decision
Total Variance Explained
3.636 60.594 60.594 3.636 60.594 60.594 2.516 41.926 41.926
1.297 21.620 82.214 1.297 21.620 82.214 2.417 40.289 82.214
.468 7.798 90.013
.307 5.114 95.126
.176 2.940 98.066
.116 1.934 100.000
Component
1
2
3
4
5
6
Total % of Variance Cumulative % Total % of Variance Cumulative % Total % of Variance Cumulative %
Initial Eigenvalues Extraction Sums of Squared Loadings Rotation Sums of Squared Loadings
Extraction Method: Principal Component Analysis.
Number of Factors to Retain
Eigenvalues > 1
rule: We would
retain 2 factors
since 2 have
Eigenvalues > 1

Proportion of variance
explained rule: Might decide
that we will retain as many
factors as explain at least 75%
of the variance here, that
would result in keeping 2
factors

Number of Factors to Retain
Stevens (1996)
The recommendation is to retain all
eigenvalues in the sharp descent
before the first one on the line
where they start to level off.


Scree
plot
indicates
that two
factors
should
be
retained.
Number of Factors to Retain
Found agreement in this example among
all rules of thumb!
Retain two factors
Still recommend examining solutions for 1,
2, 3 and 4 factors and determining which
is most interpretable.
For instructional purposes, lets proceed
by specifying that two factors be retained

Rotation
After asking for a 2 factor solution,
we obtain an unrotated pattern
matrix illustrating how each item
relates to each of our 2 factors
This unrotated pattern matrix is
not interpretable
We need to rotate this unrotated
pattern matrix to simplify the
solution and make it interpretable

-1.00
-0.80
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
-1.00 -0.80 -0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00
Factor 2
F
a
c
t
o
r

1
X1
X4
X5
X6
X3
X2
Plot of Unrotated Solution
X1
X2
X3
X4
X5
X6
Factor Matrix
a
.747 -.238
.862 -.460
.747 -.491
.621 .305
.795 .556
.728 .427
sel rn1
sel rn2
sel rn3
cse1
cse2
cse3
1 2
Factor
Extracti on Method: Pri nci pal Axi s Factori ng.
2 factors extracted. 16 i terati ons requi red.
a.
Unrotated Pattern Matrix
-1.00
-0.80
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
-1.00 -0.80 -0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00
Factor 2
F
a
c
t
o
r

1
X1
X4
X5
X6
X3
X2
Plot of Rotated Solution
(oblique)
Take original axes in black and turn
them slightly to new position in red
We have simple
structure!
Rotated Pattern
Matrix
Pattern Matrix
a
.686 .167
.975 .004
.936 -.090
.072 .652
-.072 1.004
.016 .836
sel rn1
sel rn2
sel rn3
cse1
cse2
cse3
1 2
Factor
Extracti on Method: Pri nci pal Axis Factori ng.
Rotati on Method: Obli mi n wi th Kai ser Normal i zati on.
Rotati on converged in 7 i terati ons.
a.
Oblique vs. Orthogonal Rotation
In prior slide, rotated axes allowing them
to be at some angle other than 90 degrees
This allows factors to be correlated with one
another
Known as oblique rotation
Could rotate axes forcing the angle to
remain at 90 degrees
This forces factors to be uncorrelated
Known as orthogonal rotation
In most situations, oblique rotation more
appropriate
Rotated Pattern Matrix vs.
Rotated Structure Matrix
rotated pattern matrix:
standardized/beta/pattern weight
represent only the direct relationship
between factor and item
rotated structure matrix:
structure weights or correlations
represent direct AND indirect
relationship between factor and item
Rotated pattern matrix will give
you a clearer idea of how each
item uniquely relates to factor
and may be easier to interpret
Report both since they convey
somewhat different information

Both of
these
matrices are
computed
AFTER
rotation,
both
represent
relationships
between
factor and
item

Pattern Matrix
a
.686 .167
.975 .004
.936 -.090
.072 .652
-.072 1.004
.016 .836
sel rn1
sel rn2
sel rn3
cse1
cse2
cse3
1 2
Factor
Extracti on Method: Pri nci pal Axis Factori ng.
Rotati on Method: Obli mi n wi th Kai ser Normal i zati on.
Rotati on converged in 7 i terati ons.
a.
Structure Matrix
.771 .514
.977 .497
.891 .383
.402 .689
.436 .968
.438 .843
sel rn1
sel rn2
sel rn3
cse1
cse2
cse3
1 2
Factor
Extracti on Method: Pri nci pal Axis Factori ng.
Rotati on Method: Obli mi n wi th Kai ser Normal i zati on.
It is common to use at
least .30 rule of thumb
to interpret salient
coefficients
If an item has a loading
greater .30 it it is said to
load on a factor
In some cases
(particularly with small
samples) a cutoff of .40
or greater may be
warranted

Factor Correlation Matr ix
1.000 .506
.506 1.000
Factor
1
2
1 2
Extraction Method: Principal Axis Factoring.
Rotation Method: Oblimin with Kaiser Normalization.
Moderate positive
relationship between factors
Interpretation
It appears that a two factor structure is
plausible
Variables X1 to X3 contribute to the first
factor
Variables X4 and X6 contribute to the second
factor
If this were a real research project, it
would be very important to examine the
items of these two factors for a common
theme (SE to learn stats vs. current
stats SE)
Pursue additional studies!
CFA on a separate sample to see if
factor structure found using EFA
holds and can stand up to a more
rigorous test
External validity studies
Do factors relate (or not relate) to other
variables in ways anticipated by theory?
Do subgroups differ (or not differ) on
factors in ways anticipated by theory?

Sample Size
Requirements for EFA
Some authors suggest a subject to item
ratio of
10 to 1
20 to 1
Others argue that absolute sample size is
the issue (e.g., at least 200, 500, 1000)
Stronger the relationships between items
and factors, the less sample size is an
issue
The more the better!
References
MASQUE
Munroe, A., & Pearson, C. (2006). The Munroe multicultural
attitude scale questionnaire. Educational and Psychological
Measurement, 66(5), 819-834.
FA used for construct validity purposes:
Benson, J. (1998). Developing a strong program of construct
validation: A test anxiety example. Educational Measurement:
Issues and Practice, 17, 10-17.
Benson, J., & Hagtvet, K.A. (1996). The interplay among design,
data analysis, and theory in the measurement of coping. In M.
Zeidner and N.S. Endler (Eds.). Handbook of coping: theory,
research, applications (pp.83-106).
Factor Analysis and related approaches:
Benson, J. & Nasser, F. (1998). On the use of factor analysis as a
research tool. Journal of Vocational Education Research, 23(1),
13-33.
Hair, J,.F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998).
Mutlivariate Data Analysis, 5
th
Ed. Upper Saddle River, NJ:
Prentice Hall. (Chapter 3)
Preacher, K. J., & MacCallum, R. C. (2003). Repairing Tom Swifts
electric factor analysis machine, Understanding Statistics, 2, 13-
43.

Web Resources:
Using SPSS for EFA
http://www.ats.ucla.edu/STAT/spss/
output/factor1.htm


http://www.statisticshell.com/factor.
pdf