Developing a self-reported

scale

Santo Imanuel Tonapa

1
Characteristics of measurement in nursing
 Abstract concept
socioeconomic status (SES)
self-esteem

 Indirectly measure
education, occupation SES
observed variable
latent variable

2
 Rosenberg's Self-Esteem Scale
Strongly Agree Disagree Strongly
Agree Disagree
1. I feel that I am a person of worth, at □ □ □ □
least on an equal plane with others.
2. I feel that I have a number of good □ □ □ □
qualities.
3. All in all, I am inclined to feel that I □ □ □ □
am a failure.
4. I am able to do things as well most □ □ □ □
other people.
5. I feel I do not have much to be proud □ □ □ □
of.
6. I take a positive attitude toward myself. □ □ □ □
7. On the whole, I am satisfied with myself. □ □ □ □
8. I wish I could have more respect for myself. □ □ □ □
3
9. I certainly feel useless at times. □ □ □ □
10. At times I think I am no good at all. □ □ □ □
 V1 E1

V2 E2

V3 E3

V4 E4

V5 E5
self-esteem
V6 E6

V7 E7

V8 E8

V9 E9
4
V10 E10
 Guidelines in Development of a Scale

 Step 1：Determine clearly what it is you

want to measure
 Step 2：Generate an item pool
 Step 3: Have initial item pool reviewed by
experts
 Step 4：Preliminary evaluate the scale
psychometric testing: validity and reliability
 Step 5：Maintain and application 5
Step 1：Determine clearly what it is you want to measure

 Theory

 Specificity

 Being clear about what include in a measure

 the construct wish to measure is distinct from other

constructs (specific attributes)

 Operational definition

making a fuzzy concept measurable

6
multi-item measures
 Diabetes Self-Care behaviors
 Conceptual definition：actions that an individual
take to cope with diabetes to improve his or her
health.

 Operational definition：The daily-activities that

performed by patients with diabetes to achieve
optimal diabetes control. The activities include diet
control, exercise, taking medication, and blood
7
sugar monitoring.
 Dimensions Indicators
Item 1
Diet Item 2
Item 3
Item 4
Exercise Item 5
Item 6
Self-care
Item 7
BS testing Item 8
Item 9
Item 10
Medication Item 11
8
Item 12
 Step 2：Generate an item pool

 Item generation and wording

 reflect the construct of concept

 the larger the item pool, the better

 homogenous, comprehensive, sensitive

 clarity, unambiguous, easy reading

 positively and negatively worded items

9
 Method of generating item pool
 published literature, focus group, one-to-one
interview, expert’s opinion, concept analysis

 Item intensity

 Format for measurement

 Likert Scale

 Item time frames

10
 Response options
 Strongly disagree, disagree, agree, strongly agree

 Disagree strongly, disagree moderately, disagree slightly,

agree slightly, agree moderately, agree strongly

 Never, rarely (or seldom), occasionally (or sometimes),

frequently (or usually), always

 Very important, important, somewhat important, of little

importance, unimportant

 Definitely not, probably not, possibly, probably, very

11
probably, definitely
Step 3：Have initial item pool reviewed by experts

 Relevant to what intend to measure

 Clarity and conciseness

 Pointing out the phenomenon that you have failed to include

 Accept or reject is your responsibility

12
13
14
 Content Validity (1/2)
 CVI ( content validity index)

1= not relevant 2= some what relevant

(representativeness)
3= quite relevant 4=highly relevant

 I-CVI (item level CVI)

 No. of experts giving a rating of 3 or 4 /No of experts
 I-CVI
 3~5 experts: 1.00 (Lynn,1986)
15
 ＞5 experts: 0.83
16
 Content Validity (2/2)
 S-CVI (scale level)
 S-CVI/Ave: Sum of I-CVIs /No. of items
 S-CVI /UA (universal agreement)
No of items giving a rating of 3 or 4 by all experts /No. of items
in total scale
 S-CVI /UA＞0.8
 Recommendation (Polite, Beck, Owen, 2007).
 I-CVIs : 0.78
 S-CVI/Ave : 0.90
regardless of the number of experts 17
Step 4：Preliminary evaluate the scale

 Sample

large, representativeness

 Item-analysis : delete redundant items

 Reliability

 internal consistency: the consistence of the items within the

measure

 test-retest reliability: the consistency of a measure over time

18
 inter-rater reliability
 Validity: evidence that the measure actually measures
what it is intended to measure

 construct validity

 criterion-related validity

19
 Item analysis
 Missing data
 Item-scale (item-total) correlations
Correlation between an individual item and the total
score(without that item)
 Item variances (standard deviation)
 Item means （mean ±1.5 SD)
 Skewness and Kurtosis -2.00 (-1.00) to 2.00 (1.00)
 Corrected item-total correlation ≧0.30
 Absolute value of Inter-item correlation＜0.3 or ＞0.7
should be considered for removal 20
Negatively skewed Positively skewed Symmetrical

skewness

21
leptokurtic platykurtic mesokurtic

kurtosis

22
 Comparisons of extreme groups
 Participants are separated into high score group (top 27%) and low score group
( bottom 27%) based on the total scores (criteria group). Comparing (t or F test) the
mean scores of each item between high score group and low score groups. If a item
mean significantly differ between high score group and low score, we can say this
item is useful to discriminate high and low scores.

23
27% 73%
Item-total
Consider to be deleted
correlation

24
total scale

25
No. % of Mean SD Skewness F test of extreme item-total factor
Content of Items missing groups correlation loadings
data comparisons)
1 X X X X X X X 2.2% 2.77 .93 -.21 206.0 .64 .66
2 X X X X X X X .4% 2.86 .89 -.29 160.6 .63 .65
6 X X X X X X X 3.6% 1.80 .85 .64 15.1 .29 .27
7 X X X X X X X 2.2% 2.58 .97 -.02 144.4 .62 .65
8 X X X X X X X .4% 2.53 .98 .10 111.7 .60 .61
11 X X X X X X X .0% 2.09 .94 .34 25.5 .28 .28
12 X X X X X X X 1.3% 2.60 .94 -.20 40.8 .38 .44
13 X X X X X X X .9% 1.94 .86 .59 4.3 .31 .27
14 X X X X X X X .4% 2.39 .88 -.16 15.0 .29 .26
15 X X X X X X X .4% 2.29 .89 .12 17.7 .27 .23
16 X X X X X X X 2.7% 2.82 .89 -.24 64.5 .50 .50
17 X X X X X X X .4% 2.87 .85 -.43 70.2 .56 .57
18 X X X X X X X 1.8% 2.83 .91 -.33 129.6 .60 .62
19 X X X X X X X 6.3% 2.32 .96 .24 55.6 .46 .51
20 X X X X X X X 7.2% 2.70 .94 -.33 3.0 .13 .08
21 X X X X X X X 4.9% 2.23 .98 .24 16.3 .30 .28
22 X X X X X X X 1.8% 2.79 .89 -.29 65.2 .50 .50
23 X X X X X X X 2.2% 2.98 .80 -.45 137.5 .64 .69
24 X X X X X X X 1.3% 3.12 .78 -.74 102.9 .59 .66
25 X X X X X X X 1.3% 3.20 .79 -.77 98.9 .60 .67
26 X X X X X X X .9% 3.15 .85 -.82 85.5 .51 .58
Total
scale
2.2% 2.71 .47 -.10 26
Reliability and validity (Howitt & Cramer, 2005)

 Reliability and validity are not inherent characteristics

of measures. They are affected by the context and
purpose of the measurement.

 A measure that is valid for one purpose may not be

valid for another purpose.

27
 Reliability (internal consistency)

 Coefficient alpha (Cronbach’s α )

 ＜.60 unacceptable
 .60~.65 undesirable
 .65~.70 minimally acceptable
 .70~.80 respectable
acceptable
 .80~.90 very good
 ＞.95 consider shortening the scale
28
 Validity
What the scale measures and how well it does so
The measure actually measures what it is intended to measure
The degree of measuring the theoretical concept or attributes
 Construct validity

 Whether an instrument measures the unobservable construct

 How well the items in the questionnaire represent to
underlying conceptual structure
 Factor analysis

Exploratory Factor Analysis (EFA)

Confirmatory Factor Analysis (CFA)
29
 Factor analysis
 Allows researchers to investigate concepts that are not
easily measured directly by collapsing a large number
V1 E1
of variables into a few interpretable underlying
V2 E2
factors. V3 E3

 latent variable-are variables that are V4 E4

self-esteem V5 E5
not directly observed but are
V6 E6
rather inferred from observed variables.
V7 E7
 observed variable- variables that can be V8 E8
V9 E9
observed and directly measured.
V10 E10
30
Self-care behavior for diabetes
Item 1
Diet Item 2
Item 3
Item 4

Exercise Item 5
Item 6
Item 7

BS testing Item 8
Item 9
Item 10
Medication Item 11
31
Item 12
 Rosenberg's Self-Esteem Scale
Strongly Agree Disagree Strongly
Agree Disagree
1. I feel that I am a person of worth, at □ □ □ □
least on an equal plane with others.
2. I feel that I have a number of good □ □ □ □
qualities.
3. All in all, I am inclined to feel that I □ □ □ □
am a failure.
4. I am able to do things as well most □ □ □ □
other people.
5. I feel I do not have much to be proud □ □ □ □
of.
6. I take a positive attitude toward myself. □ □ □ □
7. On the whole, I am satisfied with myself. □ □ □ □
8. I wish I could have more respect for myself. □ □ □ □
32
9. I certainly feel useless at times. □ □ □ □
10. At times I think I am no good at all. □ □ □ □
 Y = b1X1 + b2X2 + b3X3 + ... ... + b10X10 + U

Factor Analysis Coefficient Can not be estimated

33
 How much the item
measures the concept
Item 1
Item 5 Item 2
Concept

Item 4 Item 3
34
Exploratory Factor Analysis, EFA

※ Find common attributes from a group of variables

※ data-driven approach, inductive 【what's the best 】

35
 Confirmatory Factor Analysis, CFA
concept-based, deductive【Whether fit the data 】

※ test factors that have been developed a prior.

※ a process for testing what you already think you know

(based on the concept)

※ a theoretical factor structure is specified and tested for
its degree of correspondence with the observed
covariances among the items in the factors.
36
 Self-care behavior for diabetes
Item 1
Diet Item 2
Item 3
Item 4

Exercise Item 5

Self-care Item 6
Item 7

BS testing Item 8
Item 9
Item 10
Medication Item 11
37
Item 12
38
 Factor loadings
 .71（50％）excellent

 .63（40％）very good

 .55（30％）good

 .45（20％）acceptable

 .32（10％）undesirable

39
 Criterion-related Validity

 Concurrent validity

 Predictive validity

 Known-groups validity

 Convergent validity

 Divergent validity

 Discriminant validity
40
41
 Instrument Selection Process(1/2)

1. Be sure that the content area of the is instrument suits the

purpose of your study.

2. Be aware of what it was developed for and whom it was

developed on and not just assume that it will work on your
sample. Be aware of cultural differences.

3. Check that appropriate item selection and reduction process

were used and that the final number of items in the
instrument is not too large as to represent a burden to
respondents. 42
 Instrument Selection Process(2/2)

4. Check the scaling for whether adding scores is justified

statistically.

5. Check that the validity and reliability of the instrument are

adequate for your purposes.

6. Check for useful interpretation and responsiveness data that

correspond to your intended purpose.

43
44
 Example 1: Developing a new instrument
Chen, M. F., Wang, R. H*., Cheng, C. P., Chin,C.C., Stocker, J.,
Tang, S. M., & Chen, S. W. (2011). Diabetes Empowerment
Process Scale: development and psychometric testing of the
Chinese version. Journal of Advanced Nursing, 67(1), 204-214.

45
 Method
 Stage1:
 Item generation
 Concept analysis, relevant instruments, interview
 Operational definition: a series actions of healthcare professiona
assist people living with diabetes in developing and using their
ability and rights to take control their diabetes
 mutual participation
 raising awareness
 providing necessary information
 open communication
27 item, 5-point Likert scale [strongly disagree (1) to 46

strongly agree (5) ]

 Content validity
 Six experts S-CVI/UV: 0.9
4 items were deleted because there were
similar to other items 23 items
 Evaluated by 10 people living with diabetes
22 items

47
 Method
 Stage 2: item analysis

 Sample size estimation

5 + cases per variables (min)

20+ cases per variable (ideal)

N > 200

 211 convenience samples

aged ≧20 yrs, type 1 or type 2 diabetes can perform self-care.

 Item-total correlation: 0.3~0.7

48
 Method
 Stage 3: validity testing

construct validity

 EFA with promax rotation

 eigenvalue ＞1, factor loadings ≧ 0.4

 CFA

 X2/df ＜3; CFI,NNFI, ≧0.9; RMSEA ＜0.8

49
 Method
 Stage 3: validity testing
Concurrent validity (used criteria should have good reliability
and validity)
 Summary of Diabetes Self-care Activities
diet (4 items), exercise (2 items), blood glucose testing (2
items)
 Diabetes Empowerment Scale
Ⅰ: managing the psychosocial aspects of diabetes
Ⅱ:assessing dissatisfaction and readness to
change 50

Ⅲ:setting and achieving diabetes goal

 Method
 Stage 4: reliability testing

 Cronbach’s α: each subscale and global scale

 Test-retest reliability: 30 participants, 2-week interval

51
 Results
 Participant demographics and disease/treatment characteristics (N=211)
Variable N (%)
Age (years)
≤ 35 10 ( 4·7)
36-45 40 (19·0)
46-55 52 (24·7)
56-65 68 (32·2)
＞65 41 (19·4)
Range: 20-87 mean (SD): 59·3 (13·4)
Gender
Male 105 (49·8)
Female 106 (50·2)
Education
No formal education 19 ( 9·0)
Elementary school 47 (22·4)
Junior high school 54 (25·6)
Senior high school 41 (19·4)
College 25 (11·8)
Graduate school 25 (11·8) 52
 Results
Variable N (%)
Marital status
Single 13 ( 6·2)
Married 188 (89·1)
Widowed 10 ( 4·7)
Diabetes type
Type 1 11 ( 5·3)
Type 2 200 (94·7)
Duration of diabetes (years)
≤5 124 (58·8)
6-10 64 (30·3)
11-15 8 ( 3·8)
＞15 15 ( 7·1)
Range: 0·1-33 mean (SD): 6·7(6·4)
Diagnosis with complications
No 181 (85·8)
Yes 30 (14·2)
Treatment
Oral hypoglycemic agent 177 (83·9)
53
Insulin 11 ( 5·2)
Oral hypoglycemic agent and insulin 23 (10·9)
 Results
 Item analysis

 22 items→ 15 items
( 2 items variance <0.5 and mean < 2, 5 items item-total
correlation <0.3 or >0.7 )

 Construct validity

 EFA

4 factors, explained 40.42%, 8.96 %, 8.35%,

6.67% of variance in the C-DEPS
54
55
Second-order Confirmatory Factor Analysis

56
 Table 3 Correlation between Chinese version of the Diabetes Empowerment
Process Scale, Self-Care Behavior Scale and Diabetes Empowerment Scale
Chinese version of the diabetes empowerment process scale
Mutual Raising Providing Open Global
participation awareness necessary communicatio scale
information n
Self-care behavior scale
Diet 0·31** 0·30** 0·29** 0·23** 0·35**
Exercise 0·30** 0·28** 0·22** 0·21** 0·31**
Blood sugar testing 0·35** 0·31** 0·28** 0·27** 0·37**
Global scale 0·34** 0·37** 0·32** 0·26** 0·40**
Diabetes empowerment scale
Subscale Ⅰ 0·45** 0·57** 0·52** 0·47** 0·61**
Subscale Ⅱ 0·35** 0·47** 0·48** 0·37** 0·51**
Subscale Ⅲ 0·45** 0·60** 0·52** 0·44** 0·62**
Global scale 0·46** 0·61** 0·56** 0·48** 0·65**
Note
Subscale Ⅰ: Managing the psychosocial aspects of diabetes 57
Subscale Ⅱ: Assessing dissatisfaction and readiness to change
Subscale Ⅲ: Setting and achieving diabetes goals
**p<0·01
 Cronbach’s α
mutual participation: 0.82
raising awareness: 0.80
providing necessary information: 0.73
open communication: 0.79
global:0.91
 Test-retest reliability
mutual participation: 0.77
raising awareness: 0.75
providing necessary information: 0.75
open communication: 0.83
global: 0.77
58
 Example 2: shorten a developed scale (2)

Hsu H. C., Chang Y. H., Lee P. J., Chen S. U. Hsieh C.

H., Lee Y. J. & Wang R. H*.(2013). Develop and

psychometric testing of a short form problem areas in

diabetes scale in Chinese patient. The Journal of

Nursing Research, 21(3), 212-217.

59
 Problem Areas in Diabetes scale (PAID) (Polonsky et al.,
1995)

 PAID 20
 diabetes-related emotional problems

 treatment-related problems

 food-related problems

 social support-related problems

 PAID 5 (McGuire et al., 2010)

60
20-item Chinese version of PAID (PAID-C)
9 items Kurtosis, Skewness ＞2
2 items item-total correlation ＜0.4

EFA on group 1 (n=428)

(1 item was deleted, factor loading ＜0.7)

8 items SF-PAID-C, CFA on group 2 (n=427)

(two-correlated first-order factors)

61
Table 1. Exploratory factor analysis and confirmatory factor analysis of the 8-
item short form Chinese version of the problem areas in diabetes scale

Factor loadings Factor loadings of

Item of EFA CFA
Diabetes-related emotional problems
2.Feeling scared when you think about living with diabetes 0.73 0.63
6.Feeling guilty or anxious when you get off track with
0.71 0.65
your diabetes management
8.Feeling that diabetes is taking up too much mental and
0.71 0.66
physical energy
9.Coping with complications of diabetes 0.73 0.64
10.Feeling overwhelmed by your diabetes regimen 0.76 0.73
11.Feeling depressed when you think about living with
0.79 0.77
diabetes
Food-related problems
17.Feelings of deprivation regarding food and meals 0.84 0.66
18.Feeling constantly concerned about food and eating 0.87 0.92

62
Note. EFA, exploratory factor analysis; CFA, confirmatory factor analysis.
 CFA results:
 χ2 = 26.21, df = 19, p <.001; χ2/df ratio = 1.38, NFI= 0.99,
NNFI = 0.99; CFI =0.99; RMSEA = 0.033 (90% CI: 0.001~
0.055).
 Concurrent validity
 with PAID-C, r = .941, p <.001
 with Baseline HbA1c, r = .148, p <.001

 Predictive validity
 with 12-month HbA1c, r = .142, p <.001
 Cronbach’s α : 0.85
 ICCs: 0.93 ( 24 patients, 2- week interval)

 Sensitivity & specificity: 93.2%, 94.2%

63
64
Source: Kuo C.-L., Turton M.A., Jane L.-H., Tseng H.-F., Hsu C.-L.(2007).Measuring peer caring 65

behaviors of nursing students: Scale development. International Journal of Nursing Studies, 44:105-114
Source: Kuo C.-L., Turton M.A., Jane L.-H., Tseng H.-F., Hsu C.-L.(2007).Measuring peer caring behaviors
of nursing students: Scale development. International Journal of Nursing Studies, 44:105-114

66
 References
 Rattray J., Jones M.C. (2007). Essential elements of questionnaire design and
development. Journal of Clinical Nursing, 16: 234-243.
 Polite D.F., Beck C.T., Owen S.V. (2007). Focus on Research Methods: Is the CVI
an Acceptable Indicator of Content Validity? Appraisal and Recommendations.
Research in Nursing & Health, 30: 459-467 .
 Hsu H.C., Chang Y.H., Lee P.J., Chen S.Y., Hsieh C.H., Lee Y.J., Wang R.H.*
(2013, Sep). Developing and psychometric testing of a short-form problem areas
in diabetes scale in Chinese patients. The Journal of Nursing Research, 21(3),
212-217.
 Pesudovs K., Burr J.M., Harley C., Elliott D.B. (2007). The development,
Assessment, and Selection of Questionnaires. Optometry and Vision Science, 84(8):
663-674.
 Chen, M. F., Wang, R. H., Cheng, C. P., Chin, C. C., Stocker, J., Tang, S. M., &
Chen, S. W. (2011). Diabetes Empowerment Process Scale: development and
67
psychometric testing of the Chinese version. J Adv Nurs, 67(1), 204-214.
TERIMA KASIH A.K.A Thank you 68