Report of the Boulder Faculty Assembly

Administrator Appraisal Committee Concerning

Phil DiStefano, Chancellor
Spring, 2015
Administrator Appraisal Program
The Administrator Appraisal Program (AAP) of the Boulder Faculty Assembly
(BFA) seeks to provide substantive feedback about administrators based on a high
rate of response reflecting a representative survey of the faculty. Faculty members
have the opportunity to provide AAP feedback to the review/reappointment
process when the president, chancellor, provost, or the dean of their school or
college is undergoing the third- or fifth-year review.
This was the fifth-year review of Phil DiStefano, Chancellor of the Boulder campus.

Methods of Review
The Office of Planning Budget and Analysis (PBA), under the direction of the
AAP committee, administers an online questionnaire, including the option for
open-ended responses, to the faculty under the administrator and to faculty
nominated by the administrator. Details about the survey are given in Appendix A.
All faculty members of campus (hereafter referred to as the Population),
designated as eligible by the BFA were sent an e-mail requesting that they complete
the online administrator appraisal questionnaire. In keeping with past practices,
the AAP Committee requested the Chancellor to nominate as raters people he
judged especially likely to be knowledgeable about his role. These will be referred
to as the Administrator-nominated (AN) raters.
The Chancellor nominated 15 faculty as those who are knowledge about him
and his duties; of these, 14 (93%) responded to the survey. Of the 14, four (29%)
described themselves as somewhat familiar with the Chancellor and his performance
with the remainder saying they were very familiar with him.
There were 2,458 invitations sent to the remainder of the faculty, and there
were 1,286 respondents (52%). Of these, 716 (56%) said that they were not at all
familiar with the Chancellor; neither item ratings nor comments were gathered on
these faculty. Four hundred fifty-six (35% of the general population and 80% of the
general population who provided ratings) regarded themselves as somewhat
familiar with the rest rating their knowledge as very familiar.
The AAP Committee and BFA have agreed, on the advice of faculty who
specialize in survey methods, that a 60% return rate is needed for a representative
statistical study. This survey does not meet this requirement.

Item Responses
Figure 1 presents abbreviated item content and item means and categories
separately for general population and AN faculty. The items are ordered by the
average of the general population and AN means. There was strong agreement in
item means between the general population and the AN faculty (correlation
coefficient = 0.75).
As with past AAP surveys, AN gave significantly higher (i.e., more favorable)
ratings than general population faculty. All of the items rated by AN faculty fall into
the category of strengths to build on. There were no items judged as areas that
require improvement according to the general population. Given the correlation
between the AN and the general population means, it is reasonable to conclude that
those items with high means in both are areas of real strength. They include the
Chancellors integrity, knowledge about the University, and responsiveness to
student concerns.
Areas with low means from both sets of raters should not be considered
weaknesses but rather aspects that the general population considers as meeting
expectations but perhaps just barely. These include the Chancellors ability and
vision to lead, earning trust from and collaborating with faculty, and equitable
faculty salary distributions.
Consistent with all past AAP surveys, there was a very large general factor for
the correlation matrix among all items.1 This suggests that raters varied on a
general tendency to rate the Chancellor the same across all items. Those favoring
him tended to give high ratings across items; those who do not view him favorably
gave uniform low ratings; the majority giving ratings toward the middle of the
scale. Consequently, an average item response for each rater was computed as the
sum of all non-missing items divided by the number of non-missing items.2
There was no association between self-described familiarity and average item
response for the Chancellor. The mean for those somewhat familiar was 3.5 and
the mean for those very familiar was 3.4.
There was an association between average item response and comments.
Those who provided comments gave the Chancellor lower ratings (mean = 3.2) than
those who did not comment (mean = 3.5).

There was only one eigenvalue greater than 1 in the correlation calculated using
pairwise deletion. This vector accounted for 72% of the variance in the matrix. In
comparison, the second vector accounted for 5% of the variance
2 The calculation was performed on only those raters who had five or more nonmissing items. The distribution of the average item responses was unimodal with a
negative skew and a ceiling effect at the high positive end.

Figure 1. Mean item and item categories for Chancellor DiStefano. AN =

administrator-nominated raters; POP = general population raters. Note item
numbers correspond to consensus items (see Appendix A).
Administrator Appraisal Program 2014-15
Survey Evaluation of Philip DiStefano, Chancellor
Summary of Findings
Means
POP

AN

POP

8 Solid undrstndng: fac gov, Univ pol, & budget

3.8

4.8

1 Acts with integrity

3.8

4.7

5 Acts constructively on student concerns

3.7

4.8

10 Makes decisions in a timely manner

3.6

4.6

20 Responds rspctflly, tmly manner: fac inquiries

3.6

4.7

11 Promotes diversity

3.5

4.8

3.5

4.6

3.4

4.8

3.4

4.7

3.3

4.5

4 Acts constructively on staff concerns

12 Supprts recruitmt/reten of underrep faculty
3 Acts constructively on faculty concerns
19 Rewards high quality service

3.2

4.5

2 Involves faculty in decision making

3.2

4.5

6 Earns faculty trust

3.2

4.4

9 Leads the industry/profession

3.1

4.3

POP: Faculty that answered the survey items (N=567)

AN: Raters nominated by the provost (N=14; not included in the POP)
Scale ranges from 1 (Strongly disagree) to 5 (Strongly agree)

KEY:
Strengths to build on: 60% or more ratings of 4 or 5
Assets to protect: 50%-59% ratings of 4 or 5
Issues to be mindful of: 25%-39% ratings of 1 or 2
Areas that need improvement: > 40% ratings of 1 or 2
Bimodal (more than one of the above)
No consensus
Analysis of each item excluded raters who answered "Don't know/not applicable" or who
skipped the item. In the general pop, % answering dk/na ranged from 10% to 50% across
items; median=19.5%, modes=10% and 16%. % skipping an item ranged from 1% to 3%.
Among nominated raters, % answering dk/na ranged from 0% to 43% across items;
median=7%, mode=0%. No nominated raters skipped items.

AN

There were both college and departmental effects on average item response3.
Ten percent of the variance was accounted for by College or School and a further
22% by Department within College/School. The Department effect would be
considered large in behavioral survey research.

Comments
There were 150 comments, 145 from the general population and five from AN
faculty. Two-thirds of the all raters who commented said that they were somewhat
familiar with the Chancellor, the remaining one-third stating that they were very
familiar.
Comments were rated as positive, mixed, or negative in 7 categories: (1)
General, (2) Recent Controversies, (3) Athletics, (4) Academic Support, (5) Diversity,
(6) Leadership, (7) Staff, and (8) Other. A single comment could fall into more than
one category and a single comment could be rated as positive, mixed, or negative in
each category. For example, one faculty member was very positive about the
Chancellor but regretted his involvement in the Philosophy Department issues. This
was rated as a positive comment in the General category but a negative one in
Recent Controversies.
It is crucial to recognize that individuals who commentas well as the nature
of their comments--are self-selected and do not necessarily reflect a consensus of
faculty opinion. For example, all comments mentioning athletics were negative. A
true barometer of faculty opinion on athletics would ask each and every faculty to
respond to one or more questions about athletics. Nevertheless, the comments can
provide important feedback to both an administrator and the general CU community
about areas of strength and weakness in the University.
Also, some comments take the form of recommendations. The AAP committee
members who categorized the comments, lacking the prescience to judge the faculty
members intention, considered these as Mixed.
General
General comments were those containing an evaluation of the Chancellor but
not specifying particular areas for those evaluations. Twenty comments were
General. Fifteen (75%) were positive (and usually quite short), one (5%) mixed and
four (20%) negative. Many of the positive comments extolled the Chancellors
affability and interpersonal skills. Examples are:
supportable and approachable
Chancellor DiStefano is an incredible gift. His sensitivity to concerns from
all and sincere responsiveness are a joy.
has a hellishly difficult job and has served with real distinction.
a fine Chancellor in every way.
sincere, dedicated, and caring.

Only general population raters (N = 516) were included in this analysis.

 Phil has been a quite good Chancellor in a difficult time for CU. His long time
knowledge of the Boulder campus has been very effectively utilized.
overly cautious. gives too much power to non-academic side.
[he] protect himself and his own interests over those of the university.
With that said, in a period of difficult financials, he has done a good job keeping this
campus moving forward.
Leadership
Twenty-one comments mentioned Leadership. Of these seven (33%) were
positive, four (19%) mixed, and 10 (48%) negative. The positive comments largely
praised the Chancellor for his knowledge about and his dedication to the University.
Like the General comments, many were brief:
Phil has done a fine job in defining his forward-looking, mission-critical
priorities for the campus.
I admire his long leadership for the campus; it has sustained us.
The mixed comments all regarded the Chancellor is a nice guy but expressed
a desire for stronger and more forceful direction:
a very decent, but largely ineffectual leader. He seems to be more
concerned about not ruffling feathers than preparing a major research university for
the challenges of the 21st century.
Phil DiStefano knows the university very well and is ethical. I like him. I do
not see him as brave, however, and in my opinion these very difficult times for
universities call for strong leadership. I think he is too timid about telling faculty
when changes need to be made.
The negative comments express concerns that campus leadership is not
improving the status of the University, some even worrying that our reputation is
suffering:
His policies are turning the University of Colorado Boulder into a secondrate version of Michigan State.
This university is just following the profession instead of leading it.
CU is shifting away from one of research excellence, toward more of a
middle of the road state university that justifies its keep in the state without really
having any of the "buzz" that brought me to CU 1-2 decades ago, or the desire for
excellence.
Diversity
Twelve comments addressed diversity, one positive, two mixed and nine
negative.
He carefully listens to faculty and staff, cares deeply about diversity
I would give Chancellor DiStefano a higher rating on diversity issues, and
higher rating in general, except that recently the highest-level administrators he has
appointed are white males (except in places that require leaders from a diverse
background, e.g. ODECE)

 The university as a whole, and the chancellor in particular, need to do a

better job at creating a campus climate that is safe, welcoming, and nurturing for
students and faculty of color.
Recent Controversies
Twenty-one comments mentioned recent highly publicized controversies
involving Professors Patricia Adler, Dan Kauffman, David Barnett and the
Department of Philosophy. All but one was negative. Many comments were directed
at other administrators as well as the Chancellor. The major themes were an
absence of forethought and a rush to judgment. A few questioned whether campus
policy was followed and more than one cited a lack of communication. Several
expressed concern that the perceived mishandling has had a chilling effect on
discourse and has induced a climate of fear. Examples are:
CU has acted quickly in response to a claim, apparently overreacted, and
then had to try to execute damage control. "Ready, Fire, Aim" seems to be the CU
strategy
The whole handling of the Philosophy Department was done with a
hammer and not a scalpel.
The veiled process involving the Office of Discrimination and Harrassment
[sic] is worrying and has NOT been addressed well or even at all by campus
leadership.
that he rushed to judgment, that people were smeared in the press and
humiliated in other ways, and that he failed to communicate effectively with the
larger faculty as to what was going on or what procedures he was following.
Athletics
Ten comments, all negative, mentioned Athletics. They implicitly referred to
intercollegiate athletics and not to club sports. All addressed a single themethe
amount of resources devoted to athletics as opposed to academics. In addition, two
of the nine complained about disparity in salaries between athletics staff and faculty.
Examples are:
It seems that most major investments are happening around sports facilities
rather than improving educational buildings
supports football at the expense of academic program [sic]
These are significant internal funding problems that are overlooked - even
as sports venues are overhauled and our sports leaders earn an order of magnitude
more than our teaching and research faculty - and two orders more than our adjunct
faculty.
Academic Support
In addition to comments expressing concern about the resources devoted to
athletics versus academics, thirteen comments, all but one negative, articulated
reservations about resource allocation within broad academic disciplines. A
consistent theme was that the arts and humanities were being shortchanged in
favor of hard sciences and technology. Examples are:

 the humanities classes are falling apart.

Disciplinary areas in the Humanities continue to receive short shrift on the
CU Boulder campus. In addition, the dearth of languages offered at CU continue to
make the campus notoriously provincial.
I also don't see that he [the Chancellor] has any respect for faculty -especially those who aren't in the hard sciences.
your humanities faculty are disgruntled and have no faith whatsoever in
your leadership or those of our deans.
In addition, two comments criticized an unequal distribution of Esteemed
Scholars among academic disciplines, both claiming a disparity between the College
of Arts and Sciences and the College of Engineering.
Staff
Eight comments were directed at the Chancellors ability to assemble an
effective staff. All were negative. Examples are:
the Chancellor has populated Regent Hall with what seems like an
innumerable number of assistant, associates, and other vice chancellors who have
served little than to create more distance between the faculty and administration.
The main weakness that I have observed with the Chancellor is that he has
not always made good choices of VC and AVC personnel in support of him and the
campus.
Other
The Other category is a potpourri of items with few themes. Although the
category is large (62 of the 150 comments), many also fell into one of the categories
detailed above. Of the Other comments, 15 (24%) were positive, nine (15%) were
mixed, and 38 (61%) were negative.
The most thematic comments were negative complaints about salaries. Salary
equity, poor instructor pay, and salary relative to the cost of living in the Boulder
area were the major issues. Three comments reported that the Chancellor was more
concerned with external stakeholders than with faculty matters.
Most comments in this category, however, involved single-issue items. For
example, in a mixed response, a rater praised the Chancellors public persona but
wrote of problems about OCG. Other examples include: concern that an over
reliance on information technology is turning everything into an App so that
students are no longer challenged to think for themselves; an inadequate CU
retirement plan; poor progress in getting guns off the campus; a call for
establishing a central faculty lounge; the dependent tuition benefit; a lament about
the cost of the rec center; and concern over the AAP process itself (Im afraid it [is]
a popularity barometer). Several other single issues cannot be outlined here lest
they compromise the confidentiality of the rater.

Summary
Overall, the faculty viewed the Chancellor in a positive vein, particularly with
respect to friendliness and interpersonal skills. Average item rating were high--3.5
from the Population and a substantially higher 4.4 rating from administrator-nominated
facultysuggesting overall satisfaction. His knowledge of the University and integrity
are particularly strong assets. There were no glaring weakness, but the Chancellor is
urged to pay closer attention to overall leadership, earning faculty trust, involving
faculty in decision-making, and salary equity.
There were many negative comments. Recall, however, that positive comments
tended to be short, pithy and nonspecific. Negative comments were longer, directed
at specific issues, fitted into more than one category, and confounded
recommendations for better performance with negative job evaluation. Also,
commenting itself was associated with lower ratings.
The committee considers these comments as important but more in the way of
pointing out areas of faculty and campus life that could have been better managed in
the past (e.g., Recent Controversies) or require constant vigilance (e.g., Diversity)
than as beacons of the Chancellors overall performance. Still, the administration is
urged to be sensitive to following issues that were thematic in the comments:
equitable salary distribution; resource allocation, especially concerning hard
sciences and STEM versus other disciplines; and diversity. The administration and
campus would also profit from a stronger vision of its commitment to excellence or
at least more effective communication of that vision to the faculty.
One important but enigmatic finding from the survey was the fact that onethird of the variance in ratings was associated with Colleges/Schools and
Departments within Colleges/Schools. This is an unusually large effect, and it
implies that there are strong pockets of satisfaction and dissatisfaction with the
Chancellor. It will be very important, albeit challenging, to identify and rectify
discontent in the relevant units.
The AAP vote for the Chancellor was two for exceeds expectations and four
for meets expectations.

Appendix A: Procedures
The administrator appraisal survey (posted at
http://www.colorado.edu/pba/aap/index.htm) contained 19 core items
addressing the effectiveness of the administrators performance in key areas, such
as administrative/leadership style; support for teaching, research, and service;
meeting faculty, staff, and student concerns; and making progress toward diversity
goals. Administrators have the option of submitting additional items on topics they
consider of importance to their roles and performance. One item was added to the
survey (acts constructively on staff concerns). The Chancellor thought that six of
the items were more appropriate for the Provost and the AAP committee voted to
delete these six items.
Faculty members responded to the items using a 5-point Likert type scale (5 =
Strongly agree, 4= Agree, 3= Neither agree nor disagree, 2= Disagree, 1= Strongly
disagree), with the option of answering Dont know/not applicable. Higher
scores indicate a more positive evaluation.
In addition, space was provided for respondents to write open-ended comments
about the administrators performance. The rater had an option to have the openended comments transcribed and transmitted to the administrator or to have
them available only to the AAP committee. The questionnaire was completed
online.
In keeping with past AAP practices, item responses were categorized as:
a) Strengths to build on: items rated agree or strongly agree by a
substantial majority of the faculty (60% or higher of respondents gave a
rating of 4 or 5 on the scale)
b) Assets to protect: items where at least half of the respondents
responded agree or strongly agree (50-59% of respondents gave a
rating of 4 or 5 on the scale)
c) Issues to be mindful of: items rated disagree or strongly disagree by a
sizeable minority of respondents (25-39% of respondents gave a rating of 1
or 2 on the scale)
d) Areas that need improvement: items rated disagree or strongly
disagree by a large portion of respondents (40% or more of
respondents gave a rating of 1 or 2 on the scale)
e) Bimodal: items that meet criteria for two of the above categories. Note
that one of these categories must be (a) or (b) and the other (c) or (d).
Hence, a bimodal category implies significant disagreement among the
group about the administrators strength or weakness in an area.
The percentages used to construct categories were based on all non-missing
values and excluded raters who responded to an item with Dont know/not
applicable.

Members of the AAP committee were Gregory Carey (chair), Bud Coleman, Roger
Buzz King, Kai Larsen, Susan Nevelow Mart, Chuck Rogers, Lorrie Shepard, and
James Williams. Frances Costa was the PBA liaison and Cathy Kerry was the PBA
analyst and data manager.