Professional Documents
Culture Documents
Prepared by:
Center for Research, Evaluation, Assessment, and Measurement
University of South Florida
George MacDonald is the Director for the Center for Research, Evaluation, Assessment, and
Measurement (CREAM) in the College of Education at the University of South Florida. George
holds a Ph.D. in Curriculum and Instruction with an emphasis in Measurement and Evaluation.
MacDonald is a Co-Principal Investigator for the National Science Foundation (NSF) funded
study, The Effects of Social Capital and Cultural Models on the Retention and Degree
Attainment of Women and Minority Engineering Undergraduates. He is responsible for a
number of program evaluations, and conducts the Assisted Living Facilities and State-Wide
Public Guardianship certification exams for the Florida Department of Elder Affairs. His
research agenda focuses on cognitive diagnostic assessment employing the linear logistic test
model particularly as it applies to mathematics education.
ii
Table of Contents
iii
CREAM Overview
The Center for Research, Evaluation, Assessment and Measurement (CREAM) is a hub of
research activity in the College of Education. It houses all funded research projects in the
College, provides a venue for faculty engagement in interdisciplinary research efforts, and
supports faculty engagement in a broad range of research activities (e.g., qualitative, quantitative,
and mixed methods). In addition, CREAM utilizes the talents of faculty in the College of
Education to provide a broad spectrum of research-related services (e.g., program evaluation,
analytical inquiry, and technical consultation) at the local, regional, national, and international
level.
Mission
Objectives:
iv
Executive Summary
The University of South Florida, Center for Research, Evaluation, Assessment and Measurement
(CREAM), was contracted by the Hernando County School Board (HCSC) to develop, administer
electronically, and analyze a Superintendents 360 Degree Feedback Survey instrument.
CREAM was provided names and/or email addresses for 3,051 employees. Responses were
received from 942 individuals. Of those who began the survey CREAM received data from 4
school board members, 40 School-based Administrators, 548 School-based Instructional Staff, 185
School-based non-Instructional Staff, 109 District Staff, 26 School-based Administrators, and the
Superintendent.
There were 52 Likert type items on the online Survey and two open ended items. The items were
developed employing Danielle Stufflebeams Superintendents Duties (Stufflebeam, 1995). The
items were mapped to the nine domains of the Superintendent Competencies of the American
Association of School Administrators (AASA).
We examined the average respondent rating by item. The results demonstrate the Superintendent
received average scores between 3.01 and 3.49 on 18 of the 52 items and was scored between 2.0
and 2.99 on 33 items. One of the items was exactly on the mean. In sum, the performance of the
Superintendent as reported by employees of the Hernando County School District is 2.86 which
falls between slightly to moderately effective on the survey scale.
Next we analyzed the item responses by AASA domain using cognitive diagnostic assessment
modeling. We employed the approach used by MacDonald (2013). The non-linear fixed
procedure fits the model by maximizing an approximation to the likelihood integrated over the
random effects (SAS PROC NLMIXED). These results demonstrate a similar pattern as seen in
the item means. Results indicated her strongest areas of performance was Values and Ethics of
Leadership and her weakest area was Communication and Community Relations. According to
Cohen (1986) both of these fall into the range of medium sized effects (.50 - .79). Results also
demonstrate that School-based Administrators and District Administrators rate the Superintends
performance higher than Instructional, non-instructional and District Staff. In brief, her overall
performance ratings as reported by employees of the District is average with one domain below
average and another above average.
The Qualitative results as reported in 90 pages of written response to the two open-ended items
indicate some believe she has strong beliefs, is guided by deeply held principles and believes in her
students and their success. They believe she has a vision, establishes high expectations for all staff
and is a data driven decision maker. However, others have strong feelings she leads a fear-based
administration, is vindictive when solving disagreements, communicates poorly with all groups
and retaliates against those who disagree with her. Many are now discouraged with her job
performance and believe she has broken their trust. These people believe it is time for a change in
Superintendents.
Background
The University of South Florida, Center for Research, Evaluation, Assessment and Measurement
(CREAM), was contracted by the Hernando County School Board (HCSC) to develop, administer
electronically, and analyze a Superintendents 360 Degree Feedback Survey instrument. The
development process was initiated following a psychometric consultation (Dr. George MacDonald,
2016) to the Hernando County School Board chaired by First Last. In an effort to reduce costs to
the school district the survey was administered electronically.
CREAM consulted with the school board, former Deputy Superintendent Eric Willams, and
Superintendent Lori Romano. Protocols were developed to conduct interviews and focus groups
with teachers, principals, parents, and district staff. Visits to an elementary and a high school
afforded opportunities to learn what the various constituencies considered important in the survey
instrument. Development of the survey was an iterative process with changes made to the
instrument at each step. The survey was piloted with a group of high school teachers in a paper and
pencil format. The survey was presented to Superintendent Romano in a meeting at the district
office the day before Thanksgiving 2016. The Superintendent shared the survey with her cabinet
and solicited their feedback. Their comments were considered and a request to meet with the
School Board was made of the Superintendent. Dr. MacDonald presented the survey to a School
Board Workshop on January 24th, 2017. Following the workshop final changes were made to the
survey as requested by the School Board members.
To assure respondents anonymity the final survey was administered through CREAM and
responses secured on a password protected server at the University of South Florida.
Psychometrics
Survey Validity
It is necessary to establish that the survey is reliable or a consistent measure of the performance
domains required by a school district superintendent. It is possible for a survey to be reliable and
not be a valid measure. Evidence of validity is important in determining the appropriate use of the
survey as a tool to add value to the annual evaluation of a superintendent. There are four types of
validity to consider in assessing the validity of the survey instrument:
1. Content
2. Criterion-related
3. Construct
4. Consequential
Content validity considers the alignment between survey questions and performance domains. We
conducted individual and focus group interviews with two school principals, two groups of
teachers (elementary and high school), three school board members, and Superintendent Romano
during the survey development phase to judge content validity. This process included both face
validity in that survey items appear to measure a construct (e.g., policy and governance). Updated
2
content related evidence of validity came from the judgments of district staff and school board
members feedback following a January Board Workshop. An additional step in determining
content validity evaluated the survey items from the perspective of Stufflebeams Evaluation of
Superintendent Performance: Toward a General Model (1995).
Construct validity refers to whether the survey items measure domains that capture the intersection
of superintendent competencies (AASA) and superintendent of a school district superintendent.
Questions permitted respondents to provide their level of agreement or perceived level of
effectiveness to the specific questions mapped to the domains of interest. Appendix B presents
Stufflebeams matrix.
Survey Reliability
The Superintendents 360 Degree Feedback Survey is a 52-question survey with questions mapped
to nine domains. Responses were dichotomized at the mean to obtain parameter estimates using De
Boeck and Wilsons (2004) extension of Fischers Linear Logistic Test Model (1973).
Cronbachs alpha is a measure of internal consistency. The reliability estimate for the total items
on the scale is 0.99. Values above 0.7 are considered acceptable, and values approaching 0.9 are
considered good (Cronbach, 1951). The Cronbach alpha values for the individual domains range
between 0.80 and 0.98 which provides strong evidence that the survey is reliably targeting the nine
domains.
When considering the use of cognitive diagnostic assessement, reliability estimates stabilize once
the sample size has reached approximately 600 respondents (MacDonald, 2013). A sample size
above 600 per domain provides strong evidence the parameters estimates for this survey are stable.
Therefore we can have confidence the parameter estimates on the nine domains ranging from 0.71
to -0.68 are reliable. Summary statistics for the full sample are found in Tables 4 and 5.
Method
Responses form a row in the person response matrix which is transposed so that person becomes an
item group and item responses are nested in the person (group) and the q-matrix. The person ability
estimate (theta) was generated (approximately distributed) from a normal distribution with a mean
0, and standard deviation 1.
A Q-matrix (a.k.a., weight matrix) was created to provide the initial cognitive component values,
the item-response matrix was developed from survey responses from Hernando County School
District Superintendents 360 Degree Feedback Survey.
Sample. A sample of Hernando County School District employees, school board members and the
Superintendent was developed from responses to a USF-CREAM administered online survey (942
respondents, 52 survey Likert-type items, scored dichotomously).
Procedure. The analysis of the survey sample followed the approach used by MacDonald (2013).
SASs NLMixed Procedure was employed to fit a non-linear mixed model that maximizes an
approximation to the likelihood integrated over the random effects. When the marginal likelihood
is formed in this way, the undesirable limitation on person inferences is avoided.
Item Performance
Individual survey item responses are presented in Tables 4 and 5 using the percentage responding
above the mean rating as agree and effective. High percentages indicate that for a particular survey
question respondents were more likely to respond favorably (agree, effective) while low
percentages indicate that respondents were less likely to respond favorably (disagree, not
effective). The average rating was above three on 18 of the 52 questions (34.5 %) indicating that
respondents were more likely to agree or view the Superintendent as effective. The average rating
was below three on 33 of the 52 questions (63.4 %) indicating these respondents were less likely to
agree or view the Superintendent as effective. There was one question with a mean rating of three
(1.9 %).
The Linear Logistic test Model (LLTM) (Fisher, 1973; MacDonald, 2013), is a cognitive
diagnostic assessment approach, used to derive the estimates to evaluate the validity of the
domains surveyed.
A negative estimate provides evidence that respondents were more likely to endorse an item.
Positive value indicate respondents were less likely to endorse an item.
These estimates allow us to see differences between performances ratings for school-based
administrators, district administrators, school-based instructional staff, school-based non-
instructional staff, school board members and district staff.
Before estimation, every item is dichotomized. The Likert items are dichotomized at the mean
with 1 indicating above mean in agree/effective direction, and 0 below mean in disagree/ineffective
direction.
exp p i
Pr p
1 exp p i
Pr p p
where is the probability of the respondent giving a 1 response to item i, is the ability
of the respondent, and a i is the difficulty of item i. Domains enter into the model as determinants
of the difficulty of each item, as expressed in the following equation
p
i ij j c
j 1
j ij
where is the parameter estimate for the cognitive component (CKS) j, is the given weight
j
of with respect to the difficulty of item i, and c is an arbitrary normalization constant. In this
survey there are nine domains and each item has a weight on each of the nine domains that is either
1 or 0 as determined by researcher judgment and represented in the Q matrix.
4
If we substitute the second equation into the first we get:
p
exp p ij j c
Pr p j 1
p
1 exp p ij j c
j 1
From inspection of this equation we can see that a negative parameter estimate for a given j
makes the term in parenthesis larger and so increases the probability of a 1 response to the item. A
positive parameter estimate for a given j makes the term smaller and so decreases the probability
of a 1 response. In the original context of testing where a 1 response means person i was correct
on item j, negative coefficients mean easier items (higher probabilities of superintendent
effectiveness and positive coefficients mean harder items (lower probabilities of superintendent
effectiveness).
Results
Sample Results
Data were collected from two sources provided by Hernando County School Board staff. We were
provided a list of names and email addresses. These individuals were emailed an individual link to
access the survey. We were also provided a list of names and work units for individuals that did
not have a school district email address. Individuals whose names were provided in this manner
were directed to specific school sites where they could access the survey portal through the school
librarian (See table 1).
Participants were presented with a letter authorized by the Hernando School Board and
Superintendent Lori Romano describing the purpose of the survey, the intended use of the survey
results, and assurances regarding the confidentiality of their responses.
Respondents were asked to provide the position that best described their role with the Hernando
County School District from among seven options. The roles, number and percent are presented in
table 2.
The coefficients presented in Table 3 indicate the Superintendent performance as rated by the
respondents. The estimates use a logit scale (-4 to +4).
Principals, assistant principals, teachers, and non-instructional staff comprise the school-based
respondents. In the context of 360 degree feedback survey we might think of the four groups as
school levels each a bit further from direct supervision or influence from the Superintendent. In the
Hernando County School District the Superintendent recommends individuals for the principal role
and these individuals serve at the pleasure of the Superintendent. In turn the principals are
responsible for sharing the Superintendents vision with the staff in their respective schools.
Results for school-based administrators suggest they found it easier to endorse the effectiveness of
the Superintendents performance when compared to instructional and non-instructional staff for
6
seven of the nine domains surveyed. With the exception of Leadership and District Culture and
Human Resources Management school-based administrators were more likely to endorse the
effectiveness of the superintendent than instructional or non-instructional staff. Table 3 presents
the results of the Cognitive Diagnostic Assessment for the Superintendent Competencies by sample
size and role group.
Table 3. Linear Logistic Test Model Domain Estimates for Sample and Role Groups
Survey Domain Sample 2 3 4 5 6
Leadership and District Culture 0.13 0.1 0.03 0.22 0.39 0.21
Policy and Governance 0.13 0.13 0.17 0.07 0.31 0.49
Communication & Community Relations 0.71 0.11 0.94 0.34 0.26 0.38
Organizational Management 0.07 0.35 0.22 0.31 0.04 0.31
Curriculum Planning and Development 0.29 0.32 0.46 0.34 0.29 0.05
Instructional Leadership -0.07 0.36 0.11 0.22 0.04 0.56
Human Resources Management 0.32 0.71 0.42 0.17 0.2 0.44
Values and Ethics of Leadership 0.68 1.08 0.69 0.58 0.59 0.83
Labor Relations 0.17 0.94 0.26 0.33 0.21 0.52
Note: 1 School Board members (omitted), 2Schoolbased Administrators, 3Schoolbased Instructional
staff, 4Schoolbased NonInstructional staff, 5District Staff, 6Districtbased Administrators
Table 4 contains the individual items on which the Superintendent was rated above the mean. The
values in table 4 are interpreted as the proportion agreeing, or judging the superintendent to be
effective for individual survey questions.
In brief the respondents rated the superintendent highest for promoting high standards for staff
performance (3.49). In absolute terms, however, the Superintendents highest rating of 3.49
provides evidence she is judged to be slightly above neutral and just below agree on
Superintendent effectiveness for the items contained in table 4.
Table 4. Number, Mean, Standard Deviation & Item-to-total Correlation for Items above the Mean
Item N Mean S.D. ItemTotal
Correlation
Promotes high standards for staff performance 759 3.49 1.41 0.74
Delegates responsibility 655 3.49 1.39 0.69
Promotes high standards for teaching 743 3.48 1.40 0.74
Ensures use of student data to adapt practices 637 3.48 1.39 0.77
Promotes the school district's mission and vision 762 3.44 1.41 0.77
Promotes high standards for learning 750 3.42 1.43 0.76
Ensures professional development for staff 668 3.35 1.44 0.75
Participates in community events and activities 610 3.34 1.45 0.77
Ensures the evaluation of all staff 627 3.34 1.44 0.73
Ensures use of current research to adapt practices 615 3.21 1.41 0.82
Ensures school district compliance with federallyfunded 550 3.17 1.30 0.85
education program requirements
Models appropriate moral leadership 743 3.14 1.48 0.79
Ensures high quality curriculum 652 3.11 1.44 0.80
Ensures school district compliance with legal requirements 559 3.10 1.34 0.85
7
Table 5 contains the individual items on which the Superintendent was rated below the mean. The
values in table 5 are interpreted as the proportion dis-agreeing, or judging the superintendent to be
ineffective for individual survey questions.
In brief the respondents rated the superintendent lowest for communication with school volunteers
(2.00) In absolute terms the Superintendents lowest rating of 2.00 provides evidence she is judged
to be somewhat or slightly effective. When the scale is agreement a low mean suggest the
respondents somewhat disagree the superintendent is effective.
In brief, the respondents rated the superintendent as ineffective in communications with almost
every group.
Table 5. Number, Mean, Standard Deviation & Item-to-total Correlation for Items below the Mean
Item N Mean S.D. ItemTotal
Ensures maintenance of district facilities 612 2.96 1.47 0.78
Ensures development of district technology plan 610 2.95 1.47 0.83
Ensures effective culturally diverse policies and practices 569 2.95 1.37 0.87
Recommends policies to the school board 524 2.94 1.27 0.88
Ensures school district compliance with special education 577 2.90 1.36 0.83
Communicates with School Administrators 479 2.86 1.40 0.91
Shares authority 666 2.85 1.44 0.76
Empowers leadership of others 710 2.83 1.48 0.84
Balances community concerns and the best interest of 645 2.78 1.46 0.92
Provides timely information 641 2.76 1.44 0.87
Takes risks to encourage positive change 729 2.76 1.44 0.83
Communicates with Business and community leaders 441 2.74 1.47 0.85
Supports new ideas 633 2.74 1.46 0.88
Leads by example 670 2.71 1.54 0.91
Shares opportunities to problem solve 617 2.62 1.50 0.91
Communicates about the Information parents need 624 2.58 1.40 0.88
Actively listens to diverse opinions and interests 636 2.57 1.50 0.88
Ensures approaches to attract highly qualified professionals 635 2.57 1.48 0.87
Supports independent action 686 2.55 1.42 0.84
Listens to needs and concerns 674 2.55 1.53 0.87
Maintains effective working relationships 633 2.55 1.50 0.90
Communicates with the groups 544 2.51 1.47 0.87
Explains budget issues and budget decisions to community 608 2.49 1.42 0.86
Works effectively to build consensus 740 2.48 1.45 0.87
Communicates with the News media 538 2.45 1.40 0.86
Explains budget issues and budget decisions to school 636 2.43 1.41 0.87
Communicates about the needs of the school district 672 2.38 1.43 0.89
Communicates about the information staff need 635 2.31 1.41 0.91
Communicates about the information teachers need 675 2.25 1.40 0.89
School-based Respondents
Figure 1 presents graphically the parameter estimates in logits for school-based respondents,
administrators, instructional staff, and non-instructional staff. High negative logits indicate groups
were more likely endorse the Superintendents performance on a domain. High positive logits
indicates that respondents were less likely to endorse the Superintendents performance on a
domain. The parameter estimates presented in Table 3 are described below.
School administrators were more likely to endorse the Superintendents performance on the Values
and Ethics of Leadership (-1.08), Labor Relations (-0.94), Instructional Leadership, (-0.36),
Organizational Management (-0.35), and Curriculum Planning, and Development (-0.33) domains.
School administrators were less likely to endorse the Superintendents performance on Human
Relations Management (0.71).
School-based instructional staff were more likely to endorse the Superintendents performance on
the Values and Ethics of Leadership (-0.69) domain. School-based instructional staff were less
likely to endorse the Superintendents performance on the Communications and Community
Relations (0.94), Curriculum Planning, and Development (0.46), Human Relations Management
(0.42), and Labor Relations (0.26) domains.
Non-instructional staff were more likely to endorse the Superintendents performance on two
domains, Values and Ethics of Leadership (-0.58) and Organizational Management (-0.31) and
were less likely to endorse the Superintendents performance on the Curriculum Planning, and
Development (0.34), Communications and Community Relations (0.34), and Labor Relations
(0.33) domains.
Figure 1. Linear Logistic Test Model Parameter Estimates for Schoolbased Individuals
9
Districtbased Respondents
Figure 2 presents graphically the parameter estimates in logits for district-based respondents,
district-based administrators, district staff, and school board members. As in the previous section,
high negative logits indicate that respondents were more likely to endorse the Superintendents
performance on the domains. High positive logits indicates that respondents were less likely to
endorse the Superintendents performance on a domain. The parameter estimates presented in
Table 3 are discussed below.
District staff were more likely to endorse the Superintendents performance on three domains,
Values and Ethics of Leadership (-0.59), Leadership and District Culture (-0.39) and Curriculum
Planning, and Development (-0.29). District staff were less likely to endorse the Superintendents
performance on the Policy and Governance (0.31), and Communications and Community Relations
(0.26) domains.
District-based administrators were more likely to endorse the Superintendents performance on the
Values and Ethics of Leadership (-0.83), Instructional Leadership (-0.56), Labor Relations (-0.52),
and Organizational Management (-0.31) domains. District-based administrators were less likely to
endorse the Superintendents performance on the Policy and Governance (0.48), Human Resources
Management (0.43) and Communications and Community Relations (0.38) domains.
Figure 2. Linear Logistic Test Model estimates for Districtbased Staff & School Board Members
Figure 3 presents graphically the parameter estimates in logits for the three organizational units,
academic services, business services, and support operations. As mentioned previously, high
negative logits indicate that respondents were more likely to endorse the Superintendents
performance on the domains. High positive logits indicates that respondents were less likely to
endorse the Superintendents performance on a domain. Table 6 presents the parameter estimates
for the full sample and the organizational units.
10
Academic Services respondents were more likely to endorse the Superintendents performance on
the Values and Ethics of Leadership (-0.35) and Labor Relations (-0.26) domains. Academic
Services respondents were less likely to endorse the Superintendents performance on the Policy
and Governance (0.37), and Communications and Community Relations (0.23) domains.
Business Services respondents were more likely to endorse the Superintendents performance on
the Values and Ethics of Leadership (-1.08) and Organizational Management (-0.79) domains.
Business Services respondents were less likely to endorse the Superintendents performance on the
Labor relations (0.93), Leadership and District Culture (0.52), and Curriculum Planning and
Development (0.45) domains.
Support Operations respondents were more likely to endorse the Superintendents performance on
the Values and Ethics of Leadership (-0.77) and Leadership and District Culture (-0.51) domains.
Support Operations respondents were less likely to endorse the Superintendents performance on
the Curriculum Planning and Development (1.20) and Human Resources Management (0.52), and
Labor Relations (0.26) domains.
Figure 3. Linear Logistic Test Model Parameter Estimates for Organizational Units
Table 6. Linear Logistic Test Model Estimates for Total Sample and Organizational Units
Survey Domain Sample AS BS SO
Leadership and District Culture 0.13 0.13 0.52 0.51
Policy and Governance 0.13 0.37 0.20 0.08
Communication and Community Relations 0.71 0.23 0.03 0.18
Organizational Management 0.07 0.00 0.79 0.21
Curriculum Planning and Development 0.29 0.01 0.45 1.20
Instructional Leadership -0.07 0.01 0.19 0.07
Human Resources Management 0.32 0.14 0.16 0.52
11
The parameter estimate for this domain (-0.13) suggests an average rating for her performance.
Fourteen items mapped to this domain with the proportion greater than 0.50 for nine items. Nearly
sixty percent of respondents agreed that the Superintendent promotes high standards for teaching
the highest rated item on this domain. Similar proportions of respondents agreed that the
Superintendent shares authority (57.3), promotes high standards for staff performance (57.2%),
promotes the school district's mission and vision (56.9%), and promotes high standards for learning
(56.1%). Only 42 % agreed that that the Superintendent recognizes others for their professional
efforts.
The parameter estimate for this domain (0.13) suggests an average rating for her performance.
Nine items mapped to the Policy and Governance domain with the proportion greater than 0.50 for
three items. Almost twothirds of respondents rated the Superintendents level of effectiveness
highly in response to the question, recommends policies to the school board (64.88%). Slightly
12
fewer than 40 percent of respondents rated the Superintendent as effectively communicating the
needs of the school district (38.09%).
The parameter estimate for this domain (0.71) is statistically significant and a moderate effect. It
provides evidence that this domain is an area of concern for the Superintendents performance.
Nineteen items mapped to the Communications and Community Relations domain with the
proportion greater than 0.50 for seven items. Over sixty percent of respondents reported that the
Superintendent effectively communicates information that parents need (61.1%). Only 28 percent
agreed that the superintendent effectively communicates with teachers.
Organizational Management
The parameter estimate for this domain (0.07) suggests an average rating for her performance.
Sixteen items mapped to this domain with the proportion greater than 0.50 for six items. Almost
sixty percent of respondents reported that the Superintendent ensures maintenance of district
facilities (58.5%), ensures development of district technology plan (57.4%), and shares authority
(57.4%). Smaller proportions of respondents report that the Superintendent communicates
information teachers need (34.5%), communicates information staff need (35.9%), or
communicates about the needs of the school district (38.1%).
The parameter estimate for this domain (0.29) suggests an average rating for her performance. Four
items mapped to this domain with the proportion greater than 0.50 for a single item. Respondents
report that the Superintendent ensures use of student data to adapt practices (57.4%), however a
smaller proportion report that the Superintendent ensures improvement in student achievement
(43.7%)
14
Instructional Management
The parameter estimate for this domain (-0.07) suggests an average rating for her performance.
Eleven items mapped to this domain with the proportion greater than 0.50 for five items.
Respondents reported that the Superintendent promotes high standards for teaching (58.0%),
ensures use of student data to adapt practices (57.4%), and promotes high standards for learning
(56.1%). However, less than thirty percent of respondents report that the Superintendent
communicates with teachers (28.1%).
The parameter estimate for this domain (0.32) suggests an average rating for her performance.
Seven items mapped to this domain with the proportion greater than 0.50 for three items. Over
sixty percent reported that the Superintendent ensures effective culturally diverse policies and
practices (62.7%). A majority of respondents reported that the Superintendent ensures professional
development for staff (54.8%) and ensures the evaluation of all staff (51.2%). However, close to a
third of respondents reported the Superintendent does not communicate with school support and
facilities staff (30.9%) or communicate with teachers (28.1%).
The parameter estimate for this domain (-0.68) is statistically significant and a moderate effect. It
provides evidence that this domain is an area of effectiveness for the Superintendents
performance. Four items mapped to this domain with the proportion greater than 0.50 for three
15
items. Respondents report the Superintendent balances community concerns and the best interest of
students (53.6%) and promotes the school districts mission and vision (56.9%). Less than half of
respndents report that the Superintendent models appropriate moral leadership (46.0%).
Labor Relations
The parameter estimate for this domain (0.32) suggests an average rating for her performance.
Three items mapped to this domain with the proportion greater than 0.50 for one item. Less than
half of respondents reported that the Superintendent maintains effective working relationships
(44.3%) or listens to needs and concerns (43.9%).
16
The online survey included two open ended questions to obtain information from respondents that
could be used to provide additional feedback to the school board in addition to the quantitative data
shared in the report. The items received 412 responses out of 917 respondents. Using this count
45% of the survey respondents commented. USF will retain the raw responses and condensed
reports from these open ended items but due to the requests for anonymity from Hernando County
School Board and survey respondents we will not be providing direct quotes.
The most common response to this item was that the Superintendent had no areas in which
she was especially effective.
Many responded they had no intensive interaction or no experience with the
Superintendent.
There were positive responses to the Superintendents effectiveness, which can be summarized as:
17
Dr. Romano has established a vision for the School District which impresses some school
based administrators. They trust her capacity as Superintendent and believe she is effective
in her role.
Many feel Dr. Romano is effective at building partnerships with local government agencies,
community stakeholders and business partners for the benefit of the School District.
Some feel she is a has made unpopular but needed changes for the betterment of the school
district.
Many feel she has a positive attitude as she aims the School District toward becoming an
A grade District.
Some feel she is excellent at speaking with members of the Press.
Some feel Dr. Romano has a polite and respectful demeanor.
Many feel she is highly effective within the community.
There were mixed responses to the Superintendents effectiveness, which can be summarized as:
18
Many believe they are on notice about their job performance and therefore do not disagree
with or offer suggestions to the Superintendent.
Many feel the Superintendent is responsible for those above the instructional level being
clearly closed to feedback from those of us in the classroom.
Many feel the Superintendent is a very poor communicator and organizer.
That she is effective in driving down the morale in the School District.
Some report that when serious events are reported to the Superintendent they disappear for
caring enough to try and trust her or her confidants.
Many feel the superintendent does not directly engage with teachers, parents and staff.
A few believe the Superintendent is dishonest.
Some feel the Superintendent presents well to those outside the district but does not follow
through on issues or concerns.
Some feel she deceives the public and school employees by presenting information in one
manner but once she achieves her goals she proceeds with her own plan.
Some community members feel the Superintendent does not take part in community events.
Many feel the Superintendent does not serve the needs of students and is selfishly oriented
toward her own career.
Some feel she effectively manipulates data to give the appearance of improvement.
Many feel the Superintendent micromanagement leadership style is adversely affecting the
District.
Many feel the Superintendent fires good employees and that a select few get re-assigned to
these positions with fancy new titles and raises.
Many feel the Superintendent pretends to listen and will do whatever it takes to appease
complaining parents.
Many feel the Superintendent is not concerned about what goes on in the classroom
focusing instead on measures of student success.
Many feel she does not respond to emails, letters or other types of communication.
An open ended question was administered and responded which received 512 responses out of 975
respondents. Using this count 53% of the survey respondents commented. USF will retain the 31
pages of raw responses from this item but due to the requests for anonymity from Hernando
County School Board and survey respondents we will not be providing direct quotes. 512/975
Many feel she has eliminated competent staff at the district level for no good reason
because she was threatened by them or disagreed with them.
Many feel she was effective at enabling a considerable raise for herself but teacher only
received a small increase in pay.
Some feel that student programs have been cut without notice or explanation.
Many feel the superintendent hears the concerns of teacher and should listen and address
major issues.
19
In sum, the comments indicate the Superintendent is polite, well presented publically and excellent
when dealing with the Press. She has strong beliefs, is guided by deeply held principles and
20
believes in her students and their success. She has a vision, establishes high expectations for all
staff and is a data driven decision maker. She is working hard to build effective partnerships with
local government agencies, community stakeholders and business partners.
The comments further indicate she listens without following up on concerns or issues, that she
eliminates competent staff at the district level when feeling threatened and has advocated for her
own salary increase without considering others in the District.
Finally, the responses indicate the Superintendent does not visit the schools as she did early in her
tenure, her administration should be more transparent and she should improve her job performance
in all areas. They feel she does not communicate well and has created an atmosphere of fear and
retaliation for those that speak out against her, which has resulted in low morale within the District.
The teachers within the District feel she does not listen to their concerns. Many are now
discouraged with her job performance and believe she has created a fear-based administration
which has broken trust. That group does not believe the trust can be regained and think the time
has come for her to leave the district or be relieved of duties.
21
References
MacDonald,G.(2014).TheperformanceofthelinearlogistictestmodelwhentheQmatrixismisspecified:Asimulation
study.UniversityofSouthFlorida,ProQuest,UMIDissertationsPublishing,2014.3618477.
Stufflebeam,D.L.(1995).EvaluationofSuperintendentPerformance:TowardaGeneralModel.StudiesinEducational
Evaluation,21,153225
Stufflebeam,D.L.,(1995).APortfolioforEvaluationofSchoolSuperintendents.CenterforResearchinEducational
AccountabilityandTeacherEvaluation(CREATE),Kalamazoo,MI.
22
Appendices
23
HernandoCountySchoolDistrictSuperintendent's360DegreeFeedbackSurveyApril2017
DearHernandoCountySchoolBoardemployee,
TheUniversityofSouthFloridaisadministeringaSuperintendents360DegreeFeedbackSurveyonbehalf
oftheHernandoCountySchoolBoard. Byparticipatinginthesurvey,youwillcontributetotheHernando
CountySchoolBoardsannualevaluationoftheschooldistrictsuperintendent.Theinformationthatyou
providewillassisttheHernandoCountySchoolBoardmembersinevaluatingtheSuperintendent's
performanceoverthepastyear. Yourindividualresponsestothesurveywillbekeptconfidential. Only
membersoftheUSFsurveyteamwillhaveaccesstoyourinformationandresponses.Pleasebeassured
thatIndividuallyidentifiableresponseswillnotbeprovidedinanysurveyreports. Tobeeligibleto
participateinthissurvey,youhavereceivedasurveylinkaddressedtoanemailaddressprovidedtoUSFby
theHernandoCountyschooldistrict.Responseswillbeencryptedandtransmittedelectronicallytobe
storedonapasswordprotectedserverhousedattheUniversityofSouthFloridainTampa. Pleasebe
honestinyourresponsesandprovidedescriptiveanswerswhereasked.Wethank
youforrespondingtothesurvey,andforparticipatingintheevaluationprocess.Ifyouhavequestions
aboutyourparticipationinthissurvey,pleasecallDr.GeorgeMacDonaldat(813)9745977.
Sincerely,
BethNaverud(District3)BoardChairperson
MarkC.Johnson(District1)BoardViceChairperson
SusanDuval(District5)
GusGuadagnino(District4)LindaK.Prescott(District2)BoardMembers
LoriRomano,Superintendent
24
WhichoneofthefollowingpositionsbestdescribesyourrolewiththeHernandoCountySchool
District?
SchoolBoardMember(1)
SchoolbasedAdministrator(2)
SchoolbasedInstructionalStaff(3)
SchoolbasedNonInstructionalStaff(4)
DistrictStaff(5)
DistrictbasedAdministrator(6)
Superintendent(7)
Ifyouareschoolbasedemployeepleaseindicateschoolsite.
BrooksvilleElementary(1)
ChocachattiElementary(2)
DeltonaElementary(3)
EastsideElementary(4)
J.D.FloydElementary(5)
MotonElementary(6)
PineGroveElementary(7)
SpringHillElementary(8)
SuncoastElementary(9)
WestsideElementary(10)
ChallengerK8(11)
ExplorerK8(12)
WindingWatersK8(13)
D.S.ParrottMiddle(14)
FoxChapelMiddle(15)
PowellMiddle(16)
WestHernandoMiddle(17)
CentralHigh(18)
HernandoHigh(19)
F.W.SpringsteadHigh(20)
NatureCoastTechnicalHigh(21)
WeekiWacheeHigh(22)
CatapultAcademy(23)
EndeavorAcademy(24)
QuestAcademy(25)
HomeSchool(26)
FLVirtualSchoolFulltime(27)
OtherCenterorCharter(28)
25
Ifyouaredistrictbasedemployeepleaseindicateyourorganizationalunit.
Schools(1)
AcademicServicesDivision(2)
BusinessServicesDivision(3)
SupportOperationsDivision(4)
HowmanyyearshaveyoulivedwithintheHernandoCountySchoolDistrictboundaries?
Lessthanoneyear(1)
Oneyear(2)
Twoyears(3)
Threeyears(4)
Fouryears(5)
Fiveormoreyears(6)
Donotlivewithinschooldistrictboundaries(7)
26
PleaseindicateyourlevelofagreementwiththestatementsbelowregardingtheSuperintendent.
Strongly
Neither Notableto
Somewhat disagree Somewhat Strongly evaluate(6)
disagree(1)
disagree(2) noragree agree(4) agree(5)
(3)
Modelsappropriatemoral
leadership(1)
Promotestheschooldistrict's
missionandvision(2)
Promoteshighstandardsfor
teaching(3)
Promoteshighstandardsfor
learning(4)
Promoteshighstandardsforstaff
performance(5)
Delegatesresponsibility(6)
Empowersleadershipofothers
(7)
Sharesauthority(8)Supports
independentaction(9)Recognizes
othersfortheir
professionalefforts(10)
Takesriskstoencouragepositive
change(11)
Workseffectivelytobuild
consensus(12)
27
Pleaseindicate theSuperintendent'slevelofeffectivenessforeachofthestatementsbelow.
Notableto
Not Slightly Very Extremely
Moderately evaluate(6)
effectiveateffective(2) effective(4) effective(5)
effective(3)
all(1)
Recommendspoliciestotheschool
board(1)
Implementsschoolboardpolicies
andpractices(2)
Ensuresschooldistrictcompliance
withfederallyfundededucation
programrequirements(3)
Ensuresschooldistrictcompliance
withspecialeducationprogram
requirements(4)
Ensuresschooldistrictcompliance
withlegalrequirements(5)
Ensureseffectiveculturallydiverse
policiesandpractices(6)
Explainsbudgetissuesandbudget
decisionstoschooldistrictstaff(7)
Explainsbudgetissuesandbudget
decisionstocommunitymembers(8)
28
Somewhat Notableto
Not Mostly Very
effective Effective evaluate
effective effective effective
(2) (3) (6)
(1) (4) (5)
SchoolBoardmembers(1)SchoolDistrict
staff(2)SchoolAdministrators(3)Teachers
(4)
Schoolsupportandfacilitiesstaff(5)
Schoolvolunteers(6)Parentsandstudents
(7)
Businessandcommunityleaders(8)
Newsmedia(9)
PleaseindicatehoweffectivelytheSuperintendentcommunicatesaboutthetopicslistedbelow.
Notableto
Not Somewhat Mostly Very
Effective evaluate
effective effective(2) effective effective
(3) (6)
(1) (4) (5)
Informationparentsneed(1)Information
teachersneed(2)Informationstaffneed(3)
Needsoftheschooldistrict(4)
29
Pleaseindicateyourlevelofagreementwiththestatementsbelowregardingtheperformanceofthe
Superintendent.
Neither Notableto
Strongly Somewhat disagree Strongly
Somewhat evaluate
disagree disagree(2) noragree agree(5)
agree(4) (6)
(1) (3)
Activelylistenstodiverseopinionsand
interests(1)
Supportsnewideas(2)Leadsbyexample
(3)
Providestimelyinformation(4)
Participatesincommunityeventsand
activities(5)
Balancescommunityconcernsandthe
bestinterestofstudents(6)
Maintainseffectiveworkingrelationships
(7)
Sharesopportunitiestoproblemsolve
(8)
Listenstoneedsandconcerns(9)
30
Neither Notableto
Strongly Somewhat disagree Strongly
Somewhat evaluate
disagree disagree(2) noragree agree(5)
agree(4) (6)
(1) (3)
Ensureshighqualitycurriculum(1)
Ensuresuseofstudentdatatoadapt
practices(2)
Ensuresuseofcurrentresearchtoadapt
practices(3)
Ensuresprofessionaldevelopmentfor
staff(4)
Ensuresleadershipdevelopmentforstaff
(5)
Ensuresmaintenanceofdistrictfacilities
(6)
Ensuresdevelopmentofdistrict
technologyplan(7)
Ensuresapproachestoattracthighly
qualifiedprofessionals(8)
Ensurestheevaluationofallstaff(9)
Ensuresimprovementinstudent
achievement(10)
PleaseconsideryourexperiencewiththeSuperintendentandrespondtothefollowingquestionsinthe
spaceprovided.
InwhatareasistheSuperintendentespeciallyeffective?
InwhatareasshouldtheSuperintendentimprovehisorherperformance?
31
Foster Improvement
of Classroom
Instruction X X X
Lead and Manage
Personnel X X X
Manage District
Resources X X X X
Foster Positive
Student Conduct X X X X X
Foster Effective
School Community
Relations X X X X
Engage in
Professional
Development X X X X X X
Relate Effectively to
the Board X X X X X
Source:Stufflebeam,D.L.(1995).EvaluationofSuperintendentPerformance:TwardaGeneralModel.StudiesinEducationalEvaluation,21,153225
32
33