Journal of Conmlting and Clinical Psychology Copyright 1987 by the American PsychologicalAsaociafitm, Inc.

1987. Vol. 55. No. 2, 218-223 O022-1X~X/87/$O0.75

Primary Therapist Response Modes: Comparison of Six Rating Systems

Robert Elliott Clara E. Hill
University o f Toledo University o f Maryland
William B. Stiles Myrna L. Friedlander
Miami University State University of New York at Albany
Alvin R. Mahrer Frank R. Margison
University o f Ottawa, Canada Central Manchester Health Authority, England

Six therapist response-mode rating systems were compared in order to delineate a set of primary
modes that would best summarize the domain of therapist actions. Ratings of seven diverse therapy
sessions showed that, in spite of differences in measurement assumptions and rater characteristics,
interrater reliabilitiesgenerally were similar. When categories in different rating systems were col-
lapsed to the same level of specificity, moderate to strong convergence was found for the six modes
rated in all systems: question, information, advisement, reflection, interpretation, and self-disclo-
sure. These modes discriminated among the seven contrasting therapeutic approaches. Each thera-
pist was characterized by a unique pattern of response modes that differed significantly from the
others. Researchers interested in assessing therapist in-session behaviors should consider incorporat-
ing measures that include these six modes.

The psychotherapy or counseling process can be divided into
four aspects: content (what is said), action (what is done), style
(how it is said or done), and quality (how well it is carried out)
(Elliott, 1984; Russell & Stiles, 1979). Categories or dimensions
of verbal action are referred to by linguists and philosophers as
speech acts (Searle, 1969) and by process researchers as re-
sponse modes (Goodman & Dooley, 1976).
Measures of therapist response modes have gradually accu-
mulated, resulting in some 20 or 30 systems to date (EUiott et
al., 1982; Goodman & Dooley, 1976; Hill, 1982; Russell &
Stiles, 1979). Some of these systems have been incorporated
into skills training packages for beginning counselors (e.g., Ivey
& Gluckstern, 1974), whereas others have been used to discrim-
inate between treatments or to predict therapy outcome (e.g.,
Sloane, Staples, Cristol, Yorkston, & Whipple, 1975).
The multiplicity of rating systems makes it difficult to com-
pare studies or to tell whether differences between studies are
due to rating systems or to the interviews rated. Furthermore,
to our knowledge, no published studies have compared different
response-mode rating systems.
Barriers to the comparison o f response-mode systems in-
clude (a) the use o f different labels for similar categories; (b)
the failure to report reliabilities for specific response modes,
An earlier version of this article was presented at the annual meeting
of the Society for Psychotherapy Research, Lake Louise, Alberta, Can-
ada, June 1984. The authors thank the six teams of raters, as well as
Henya Rachmiel and Mee-Ok Clio, for their assistance in data entry
and analysis. This article was written while the first and third authors
were Visiting Researchers at the Social and Applied Psychology Unit at
the University of Sheffield, Sheffield, United Kingdom.
Correspondence concerning this article should be addressed to Rob-
ert Elliott, Department of Psychology, University of Toledo, Toledo,
Ohio 43606.
218
making it difficuR to evaluate specific categories within and
across systems; (c) different measurement assumptions and rat-
ing procedures (e.g., differences in scoring units); and (d) theo-
retical biases, resulting in overemphasis on certain verbal be-
haviors and restricted applicability to therapist behaviors of a
particular orientation. Thus, a need exists for comparing re-
sponse-mode systems on a single sample o f therapy sessions.
In this study, six developers o f rating systems collaborated
to rate a common set of actual therapy sessions. The sessions
represented a range of theoretical orientations, a range o f client
types, and a mixture o f initial and later sessions, providing a
variety of therapist verbal behaviors.
Our goals were (a) to compare interrater reliabilities, (b) to
seek a common set of primary modes, and (c) to assess the dis-
criminant validity of the primary modes by contrasting re-
sponse use by different therapists.
Method
Sample
Seven therapy sessions were rated: (a) John Paul Brady: Behavioral
Treatment of Stuttering(Brady, 1983), an initial session demonstrating
deconditioning of stuttering with a young woman; (b) Albert Ellis: John
Jones(Ellis, 1983), the 15th session of rational-emotive therapy with a
young male homosexual; (c) Clara Hill, the 5th session of a 12-session,
time-limited, relationship-insight-oriented treatment with a female col-
lege student (Hill, Carter, & O'Farrell, 1983); (d) Robert Hobson (Hob-
son, 1982), an initial session of conversational therapy (a relationship--
dynamic treatment conducted by its originator, a British psychiatrist)
with a young woman with interpersonal problems; (e) Ira Progoff: Gregg
(Progoff, 1983), Jungian dream analysis with a male client (the session
appears to be one of the last from a long-term therapy); (f) CarIRogers:
Miss Munn (Rogers, 1983), the 17th session of a client-centered treat-
ment with a young woman; 0g) Faith Tanney, an intake session with
a male client with procrastination problems that was conducted by a
 PRIMARY RESPONSE MODES
counseling center therapist with a gestalt-dynamic orientation (Hill,
1978).
Response Mode Systems
The six response mode systems used in the study are summarized
below and in Table 1.
1. Hill's Counselor Verbal Response Mode Category System (Hill,
1978) consists of 14 mutually exclusive categories. Response modes are
judged for each response unit, which is defined as a grammatical sen-
tence that has been unitized separately (brief phrases such as "mm-
hmm'" and "yes" are also treated as separate units). Unitized transcripts
were rated independently by three trained undergraduates. Final ratings
were based on agreement by two of the three judges; three-way disagree-
ments were resolved by discussion.
2. Friedlander's (1982) refinement of Hill's (1978) rating system in-
eludes nine mutually exclusive categories, combining several of Hill's
categories (e.g., "open question" and "closed question" are combined
as "information seeking"). The scoring unit is generally the same as
Hill's, except that each unit must minimally contain a verb phrase (i.e.,
"uh-huh" is not rated and compound predicates are scored separately).
Ratings were done from unitized transcripts. Three raters were used.
Procedures for final ratings and resolution of disagreements were identi-
cal to those of Hill (1978).
3. Stiles' Verbal Response Mode System (Stiles, 1978, 1979) consists
of eight mutually exclusive categories. Verbal form (literal meaning) and
speaker intention (meaning intended in the situation) are rated sepa-
rately, and the unit is defined as the independent clause or nonrestrictive
dependent clause. Three trained undergraduate students unitized and
rated transcripts. A two-out-of-three convention was used for resolving
disagreements; three-way disagreements were defined as unclassifiable.
4. Elliott's Response Mode Rating System (EUiott, 1985) consists of
10 nonmutually exclusive dimensions that are rated using 0-3 confi-
dence ratings. The unit is flexible, but in this study Hill's (1978) verbal
sentence units were used. Ratings were made from unitized transcripts
and tapes. Final ratings were achieved by rescaling confidence ratings
to 0-1 scales, then averaging ratings across the four raters (three under-
graduates and coauthor Friedlander).
5. The Conversational Therapy Rating System (Goldberg, Hobson,
Maguire, Margison, O'Dowd, Osborn, & Moss, 1984; referred to here
as the Margison system) was developed to rate the therapist behaviors
described in Hobson's Conversational Model of Therapy (Hobson,
1985). The system includes 11 mutually exclusive function categories,
rigidly defined by formal cues, with particular emphasis on types of
questions and advisement. The final ratings represent a combination
of ratings by two judges (a psychology research assistant and coauthor
Margison).
6. Mahrer's Taxonomy of Procedures and Operations in Psychother-
apy (Mahrer, 1983) contains 35 mutually exclusive categories. The unit
is defined as the therapist's speaking turn. Disagreements (less than 50%
agreement) are resolved by rerating responses or by labeling responses
as unclassifiable. Between eight and 12 raters (graduate students and
coauthor Mahrer) were used on each session. Ratings were made from
tapes and transcripts.
Results
Reliability Analyses
To compare systems, the phi statistic, a simplification of the
product-moment correlation (for pairs of dichotomous vari-
ables) was used. The response-mode systems using nominal
scale measurement (all except EUiott's) were treated as sets of
dichotomous dimensions, each corresponding to separate re-
219
sponse-mode categories. Correlations were then calculated be-
tween each pair of raters for each category or dimension in each
system. The means of these correlations, provided in Table 1,
can be interpreted as the reliability of the average rater for each
category or dimension within each rating system. The median
values (across categories) for five oftbe seven systems were quite
similar (Hill = .61, Friedlander -- .59, Elliott = .56, Stiles-
Intent = .55, and Mahrer = .52, or .57 if the 12 categories with
zero or near zero base-rates are excluded). These small differ-
ences probably reflect differences in the training and sophistica-
tion of the raters. The median reliability for Stiles-Form ratings
was .73, presumably because response form is easier to rate
than speaker intent. (The other response-mode systems are pri-
marily intent based.) The typical values for the Margison sys-
tem were higher yet (Mdn = .88), probably because these cate-
gories have been rigidly defined and require highly experienced
raters. When median reliability values were grouped by mode,
question categories were rated most reliably (.71), followed by
advisement (.66), information (.64), self-disclosure (.61), reas-
surance (.58), interpretation (.56), reflection (.53), confronta-
tion (.48), and other (.37).
Alpha reliability coefficients were also calculated in order to
obtain a measure of actual reliability. These were substantially
higher than the average rater figures because they were averages
o f 2 - 1 2 ratings. Only 19 of the 97 different categories or dimen-
sions had alpha reliabilities less than .70. All of these involved
either zero or very low base-rate categories (13 instances, most
in the Mahrer system); other or unclassifiable categories (3 in-
stances); or had alphas between .60 a n d . 70, indicating that the
data were still usable (3 instances).
Comparison of Rating Systems on
Primary-Response Modes
The final ratings were assembled into a master file, first using
Hill's response units and then subdividing to accommodate the
smaller units used by Stiles and (occasionally) by Friedlander.
If Stiles' raters divided a Hill unit into smaller parts, then the
other systems' ratings were duplicated over the same parts o f the
response. The exception occurred when Stiles or Hill created a
unit for something not rated in other systems (e.g., initial "yes,'
"well," "you know"); in this case, a missing value was entered
for the system in which the unit was not rateable. Finally, be-
cause there was disagreement among systems as to whether ac-
knowledgments or minimal encouragers (e.g., "uh-huh") and
silence should be considered units, these were dropped from the
analyses.
In the initial comparisons among systems, the original cate-
gories were used. It became clear, however, that differences in
"fine-grainedness" of categories were distorting the results. For
example, Friedlander's reflection-restatement category is su-
perordinate to Hill's reflection, restatement, and nonverbal ref-
erent categories. To compare categories across systems, similar
categories were identified by careful checking of category defi-
nitions and examples and by initial correlational analyses. We
found six primary response modes (question, information, ad-
visement, interpretation, reflection, and self-disclosure) that
were measured by all the systems and two modes (reassurance
and confrontation) that were common to four of the systems.
220 ELLIOTT, HILL, STILES, FRIEDLANDER, MAHRER, MARGISON

,.,2-
~ ~ ~ ,..2,
•r. _~
.,2. ~ ~ ~ ~ I. ~

,. .8

~ ~ °~'
~.~_~ ~ ~ ~ ~ ~ o ~~

• ~ ~

g ~5

~ i i~
,-~.~ ~ ~ ~~ -~ ~i

L.

r~

oo e~

•-= 8

.e,

~.~ ~

E
E
J~
 PRIMARY RESPONSE MODES 221

Table 2
Proportions and Item-Total Correlationsfor Primary Response Modes
Rating system Question Information Advisement Reflection Interpretation Self-disclosure
Proportions
Hill .14 .24 .02 .12 .27 .0l
Friedlander .11 .37 .05 .17 .15 .01
Stiles-form .13 .32 .04 .16 .07 .16
Stiles-intent .15 .15 .06 .18 .22 .07
Eiliott .17 .16 .10 .20 .31 .05
Mar#son .13 .27 .07 .19 .19 .29
Mahrer .18 .18 .07 .08 .38 .02
Item-total correlations
Hill .82 .58 .50 .58 .62 .32
Friedlander .76 .51 .73 .61 .49 .34
Stiles-form .73 .32 .58 .49 .17 .59
Stiles-intent .84 .62 .73 .70 .45 .57
Elliott .84 .68 .67 .66 .67 .48
Mar#son .73 .37 .46 .35 .26 .52
Mahrer .49 .51 .33 .47 .52 .39
Note. Categories are treated as separate dimensions within each system. The item-total correlation is the correlation between a category in a system
and the proportion for the category for all the remaining systems.

In all, 138 tests of convergence (correlations between same
primary modes across different systems) were performed on 50
measures of primary-response modes.I The very large sample
size (N --- 1,947) and the nonindependence of observations
(units within sessions) rendered the usual statistical significance
criteria meaningless. However, 117 of these tests (85%) had cor-
relations of at least .20 and 98 (71%) of the correlations ex-
ceeded .30. An estimate of discriminant validity was given by
comparing these with the 930 correlations of different modes in
different systems; only 10 exceeded .20 (1.1%), and none ex-
ceeded .30.
Mean intercorrelations showed that convergence across sys-
tems for comparable modes was highest for question and reas-
surance (.61 and .52, respectively), and was moderate for infor-
mation (.40), reflection (.38), advisement (.35), self-disclosure
(.33), confrontation (.31), and interpretation (.28). These con-
vergence figures, interestingly, average only about .20 lower than
the reliability estimates for categories within systems.
To provide an index of each response-mode system's conver-
gent validity, each was compared with an index consisting of all
other measures of the same mode. These corrected item-total
correlations are presented in Table 2 for the six primary modes.
(The comparable values for reassurance are Hill, .68; Fried-
lander, .62; Elliott, .73; and Mahrer, .52. For confrontation,
comparable values are Hill, .47; Friedlander, .51; Elliott, .45;
and Mahrer, .29.) Hill's system showed particularly good con-
vergence relative to other systems for its measures of question,
interpretation, and confrontation, whereas its measure of self-
disclosure appeared to differ from that of the other systems.
Friedlander's version of the Hill system showed good conver-
gence on measures of advisement and confrontation but not on
measures of self-disclosure. Elliott's system converged rela-
tively strongly in its measures of question, information, reflec-
tion, and interpretation. Stiles-Intent converged well for ques-
tion, information, advisement, reflection, and self-disclosure.
Stiles-Form converged well only for self-disclosure and diverged
somewhat for information and interpretation. The Mahrer and
Margison systems appeared to be most divergent from the other
systems, generally showing lower item-total correlations.
Discrimination of Therapists by Response-Mode Use
As a further test of the discriminant validity of the primary
response modes, we compared the proportions of response
modes used by the seven therapists. We developed a composite
response-mode index, calculated by finding the proportion of
systems scoring the given mode as present for a given unit.
These composite indices were then averaged across the 1,947
units (see Table 3). Finally, dummy variable correlations were
used for each mode to compare each therapist with all other
therapists. Effect sizes (rs > . 10, .20, .30) are also given in Table
3 because conventional significance levels were not useful (for
r = .10,p < .001).
Each therapist was characterized by a unique profile of re-
sponse-mode use: Brady (a behavior therapist) used more infor-
mation and advisement but was low on reflection, interpreta-
tion, and confrontation. In comparison, Progoffalso used more
information but less advisement and confrontation. Tanney (a
gestalt]dynamic therapist) frequently used information, self-
disclosure, and question and avoided use of reflection and inter-
pretation. Consistent with his treatment model, Rogers used
more reflection than did any other therapist for that mode or,
in fact, for any one mode. Hill's relatively high use of interpreta-
tion and reflection modes is consistent with her bridging of the
relationship and dynamic therapy traditions. Ellis was unique
in his high use of both reassurance and confrontation and gave
fewer information and self-disclosure responses. Hobson was
' These data are available from the first author.
222 ELLIOTT, HILL, STILES, FRIEDLANDER, MAHRER, MARGISON

Table 3
Mean Composite Ratings by Therapist for Response Modes
Response mode Total Brady Ellis Hill Hobson Progoff Rogers Tanney

Question .15 .20 .17 .06" .14 .11 .10 .21"

Information .24 .38" .14" .17" .13" .31" .02" .33"
Advisement .06 .20 c .03 .03 .09 .01" .00 .06
Reflection .16 .05" .14 .22" .18 .13 .71 c .I0"
Interpretation .23 .03 b .28 .38 b .28 .26 .14 .13 b
Self-disclosure .09 .07 .01" .10 .11 .07 .04 .13"
Reassurance .05 .03 .09" .02 .03 .07 .02 .04
Confrontation .05 .00" .16" .06 .02 .01" .00 .05
Other .03 .03 .04 .01 .03 .03 .00 .03
N 1,947 216 269 357 239 364 65 437
Note. Values in the table are mean composite response ratings (proportion of systems scoring a unit as a given mode, averaged across all units).
Dummy variable correlations were used to compare each therapist with all others for each mode (correlating composite response mode ratings with
a set of variables coded 1 for each therapist vs. 0 for all other therapists). Effect sizes are given because conventional significance levels were not
useful (for r = .10,p < .001).
°r>.10. br>.20. Or>.30.

not characterized by relatively greater use of any response
mode, although he did use less information.
Discussion
The present results suggest that a set of fundamental re-
sponse-mode categories or dimensions underlies a variety of
systems with different origins and purposes. We have reported
evidence for the convergent and discriminant validity of six pri-
mary response modes (question, advisement, information, re-
flection, interpretation, and self-disclosure) rated in all the sys-
tems studied.
Convergence was obtained despite a variety of methodologi-
cal differences between rating systems, including levels of rater
training experience (a few weeks to several years), rater sophisti-
cation (advanced undergraduates to seasoned therapy research-
ers), rating units (clauses to complete responses), thoroughness
of definitions and examples (brief descriptions to complete ex-
positions), and measurement structure (nominal scale vs. set of
rating dimensions).
The factors that did seem to affect interrater reliability or
convergent validity were (a) the category itself, with question
being the easiest category to rate reliably; (b) the form versus
intention distinction introduced by Goodman and Dooley
(1976) and incorporated by Stiles (1978, 1979); and (c) whether
or not the system was designed to measure a particular ap-
proach to therapy.
At the same time, the results make it clear that the response-
mode measures do not converge entirely. The intersystem valid-
ity coefficients are still about .20 lower than the typical inter-
rater correlations within systems (i.e., .30-.40 vs..55-.60). Al-
though the systems measure the same primary modes, they are
still defined somewhat differently.
We conclude that there generally is not a best response-mode
rating system. Reliability values were comparable and conver-
gent validity varied by category, with no one system clearly
proving best on all modes. However, to promote comparability
across studies, researchers measuring therapist speech acts
should examine the six primary modes and may also want to
consider measuring reassurance, confrontation, or acknowledg-
ment. Researchers should select or adjust response-mode sys-
tems according to their needs in relation to the features and
strengths of particular systems.
At present, the therapist response modes are probably the
most widely studied therapist process variables. Response
modes are conceptually clear and can be thoroughly specified;
they also discriminate between different approaches to therapy.
On the other hand, therapist response modes predict outcome
or therapist effectiveness only weakly or moderately (Elliott et
al., 1982). The response modes measure only one aspect of ther-
apist responses: action. A more complete description of thera-
pist responses requires additional ratings of content, style, qual-
ity, or effectiveness (Elliott, 1984; Russell & Stiles, 1979). Fi-
nally, therapist response modes also take their meaning and
impact from context, in particular, from background character-
istics of the client, from the current status of the relationship,
from the current important helping tasks, and from the client's
immediately prior statement (EUiott, 1984).
References
Brady, J. P. (1983). Behavioral treatment of stuttering (AAP Tape Li-
brary Catalogue, Tape No. 38). Salt Lake City, UT: American Acad-
emy of Psychotherapists.
Elliott, R. (1984). A discovery-oriented approach to significant events
in psychotherapy: Interpersonal process recall and comprehensive
process analysis. In L Rice & L. Greenberg (Eds.), Patternsof change
(pp. 249-286). New York: Guilford Press.
EUiott, R. (1985). Helpful and nonhelpful events in brief counseling
interviews: An empirical taxonomy. Journal of Counseling Psychol-
ogy, 32, 307-322.
Elliott, R., Stiles, W. B., Shiffman, S., Barker, C. B., Burstein, B., &
Goodman, G. (1982). The empirical analysis of helping communica-
tion: Conceptual framework and recent research. In T. A. Wills (Ed.),
Basic processes in helping relationships (pp. 333-356). New York:
Academic Press.
Ellis, A. (1983 ). John Jones (AAP Tape Libary Catalogue, Tape No. 11).
Salt Lake City, UT: American Academy of Psychotherapists.
Friedlander, M. L. (1982). Counseling discourse as a speech event: Revi-
 PRIMARY RESPONSE MODES
sion and extension of the Hill Counselor Verbal Response Category
System. Journal of Counseling Psychology, 29, 425--429.
Goldberg, D. P., Hobson, R. E, Maguire, G. P., Marglson, E R.,
O'Dowd, T., Osborn, M., & Moss, S. (1984). The clarification and
assessment of a method of psychotherapy. British Journal of Psychia-
t ~ 144, 567-580.
Goodman, G., & Dooley, D. (1976). A framework for help-intended
communication. Psychotherapy: Theo~ Research and Practice, 13,
106-117.
Hill, C. E. (1978). The development of a system for classifying counselor
responses. Journal of Counseling Psychology, 25, 461-468.
Hill, C. E. (1982). Counseling process research: Philosophical and
methodological dilemmas. Counseling Psychologist, 10, 7-19.
Hill, C. E., Carter, J. A., & O'Farrell, M. K. (1983). A case study of the
process and outcome of time-limited counseling. Journal of Counsel-
ing Psychology,, 30, 3-18.
Hobson, R. (1982). Louise R. Unpublished tape, Central Manchester
Health Authority, Manchester, England.
Hobson, R. F. (1985). Forms offeeling. London: Tavistock.
Ivey, A. E., & Gluckstern, N. B. (1974). Basic attending skills: Partici-
pant manual. North Amherst, MA: Microtraining Associates.
223
Mahrer, A. R. (1983). Taxonomy of procedures and operations in psy-
chotherapy Unpublished manuscript, University of Ottawa, Canada.
Progoff, I. (1983). Gregg. (AAP Tape Libary Catalogue, Tape No. 12).
Salt Lake City, UT: American Academy of Psychotherapists.
Rogers, C. R. (1983). Miss Munn (AAP Tape Libary Catalogue, Tape
No. 5). Salt Lake City, UT: American Academy of Psychotherapists.
Russell, R. L., & Stiles, W. B. (1979). Categories for classifying language
in psychotherapy. Psychological Bulletin, 86, 404-419.
Searle, J. R. (1969). Speech acts:An essay in the philosophy of language.
Cambridge, England: Cambridge University Press.
Sloane, R. B., Staples, E R., Cristol, A. H., Yorkston, N. J., & Whipple,
K. (1975). Psychotherapy versus behavior therapy. Cambridge, MA:
Harvard University Press.
Stiles, W. B. (1978). Verbal response modes and dimensions of interper-
sonal roles: A method of discourse analysis. Journal of Personality
and Social Psychology, 36, 693-703.
Stiles, W. B. (1979). Verbal response modes and psycbotherapeutic tech-
nique. Psychiat~ 42, 49-62.
Received March 7, 1986
Revision received July 22, 1986 m