Studies in Educational Evaluation 49 (2016) 15–29

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journal homepage: www.elsevier.com/stueduc

Classroom observation for evaluating and improving teaching: An

international perspective
Felipe Martineza , Sandy Tautb,* , Kevin Schaafa
a
University of California, Los Angeles, United States
b
Pontificia Universidad Católica de Chile, Chile

A R T I C L E I N F O A B S T R A C T

Article history:
Received 21 July 2015 Teacher evaluation and development policies around the world are undergoing significant reform.
Received in revised form 14 March 2016 Classroom observation often carries a considerable weight in teacher appraisal and improvement
Accepted 18 March 2016 systems, and provides the critical formative anchor informing professional development. This study
Available online 15 April 2016 examined a purposively selected sample of sixteen classroom observation systems in six countries,
including high performing Singapore and Japan, regional exemplar Chile, the three largest school districts
Keywords: in the United States, and other interesting examples in Australia, Germany and the United States to add
Classroom observation diversity to the sample. The study offers an analytic framework for understanding classroom observation
Teacher appraisal
systems across contexts, distinguishing conceptual, methodological and policy aspects that shape these
Teacher development
systems. The sixteen systems were remarkably consistent in their stated overall purposes, but there was
Instructional quality
Professional development variation in terms of how they operationalized good teaching, the degree of standardization of the
Evaluation systems observation process, emphasis on validation, and information uses. The paper describes and discusses
these characteristics in order to help researchers and policymakers reflect on the available options and
take more informed decisions in designing classroom observation for evaluating and improving teaching.
ã 2016 Elsevier Ltd. All rights reserved.

1. Introduction comprehensive approaches for evaluating teacher effectiveness,

Teacher evaluation and development policies are in the midst of
substantial reform in education systems around the world. Such
calls for reform tend to reflect continuing concerns about
shortcomings in educational quality and equity, reflected in state,
national, or international student achievement testing results,
either in absolute terms or in relation to peer districts, states, or
countries (Feuer, 2012). International reviews also reflect growing
interest in revamping teacher evaluation and development at local,
state, and national levels, with a focus on supporting valid
inferences about teachers and providing useful feedback to help
them improve their practices in the classroom (OECD, 2013a,
2013b; Santiago & Benavides, 2009). In the United States, in
particular, growing interest in teacher evaluation reflects policy
shifts away from highly qualified teachers to a notion of highly
effective teachers (Goe, Bell, & Little, 2008). In the context of federal
guidelines that require that districts and states develop
* Corresponding author at: Pontificia Universidad Católica de Chile, Escuela de
Psicología, Centro de Medición MIDE UC, Avda Vicuña Mackenna 4860, Macul,
Santiago, Chile.
E-mail address: staut@ucla.edu (S. Taut).
a great deal of attention and discussion among educators,
statisticians, and policymakers has focused on the use of
aggregates of student achievement to evaluate teachers, typically
through so-called Value Added Models, or VAMs (for reviews see
e.g. Baker et al., 2010; Braun, 2005).
While the attention paid to VAMs in the research literature and
the press could suggest otherwise, however, student achievement
is rarely the driving indicator for teacher evaluation. In most cases,
measures of teacher practice based on direct observation of
teachers in classrooms receive an equal or larger weight. A
common assumption to most teacher evaluation systems around
the world is that classroom practice is the key mediator between
education policies and student outcomes (Schleicher, 2011), and
classroom observation remains the method of choice (a de facto
gold standard) for gaining systematic insight into these practices in
their natural setting (Kennedy, 2010; Stallings, 1977). In the context
of teacher evaluation policies, observation-based assessments of
teachers in the classroom are seen as key both for understanding
the mechanisms linking classroom processes and desired improve-
ments in student outcomes, and for informing formative and
developmental feedback to guide teacher improvement efforts

http://dx.doi.org/10.1016/j.stueduc.2016.03.002
0191-491X/ã 2016 Elsevier Ltd. All rights reserved.
16 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
(Kane & Staiger, 2012; OECD, 2013a, 2013b; Reform Support
Network, 2012).
Yet, even as teacher development and evaluation policies
receive unprecedented attention around the world, systematic
study of classroom observation in this context has been scarce.
Direct observation of the work of teachers in the classroom has
been a staple of research on teaching for nearly a century (see
Medley & Mitzel, 1963). However, educators and policymakers
need to understand how evaluative contexts influence the
information collected through classroom observation, and how
it is used to inform inferences about, and feedback to teachers.
With this paper we start to fill this gap by investigating how a
sample of notable and varied regional and national education
systems around the world apply classroom observation to inform
teacher evaluation and professional development. Specifically, by
adopting a comparative lens for data collection and analysis, we
seek to a) outline an analytic framework for classroom observation
systems based on the conceptual, methodological, and contextual
commonalities and differences to help bring cohesion to the
research literature, and b) to support policymakers and system
designers in systematically considering key questions involved in
the design of classroom observation systems.
2. Theoretical background
Large-scale policy initiatives on teacher evaluation and
professional development have traditionally been informed by
two different models. The standards-based model emphasizes
explicit frameworks to model quality instruction and classroom
practice (Peterson, 2000), while the outcome-based model
privileges productivity in terms of student achievement and other
relevant outcomes (Kennedy, 2010). These two models have
started to converge in recent years, as a new crop of systems of
teacher evaluation and development combines an emphasis on
student achievement with explicit and detailed models of
instructional practice (Kane & Staiger, 2012). Importantly, despite
the growing international consensus around the importance of
evaluating and improving teaching, the specific characteristics of a
system depend on a series of factors that include assumptions
about the goals and purposes, stakes and incentives attached to the
evaluation results, frameworks or models that define the domains
and criteria to be assessed, and methods used to collect and
analyze information about these domains (OECD, 2013a, 2013b).
A common feature in recent efforts to revamp teacher
evaluation across countries and contexts is a focus on formative
uses of information. For example, the federal Race to the Top
legislation in the U.S. calls for data systems “that inform teachers
and principals about how they can improve instruction” (U.S.
Department of Education, 2009), while in Australia the national
agreement on teacher performance seeks to promote professional
conversations that improve teaching (Australian Institute for
Teaching and School Leadership (AITSL), 2016). Similarly, Chilean
legislation sets up a formative teacher evaluation system focused
on “improving the pedagogical work of educators and promoting their
continuous professional development” (Law, 2004, Law No. 19.961).
This formative focus has renewed interest in classroom observa-
tion as a method for collecting information to support improve-
ment efforts regarding teaching quality.
Classroom observation is a powerful tool that offers an
unobstructed view of classroom practice (Millman & Darling-
Hammond, 1990) and allows us to understand how teachers teach
within a realistic context (Putnam & Borko, 2000). Although
observation protocols can be found in the research literature since
at least the 1920s (Stallings, 1977), classroom observation enjoyed
widespread popularity as a method of large scale data collection in
the 1950s and 1970s, when studies investigating the validity of
observation protocols became a staple of education research (see
e.g. Brophy & Good, 1986; Medley & Mitzel, 1963; Shavelson, Webb,
& Burstein, 1986; Stallings, 1977). Validation research is again
being conducted, as a new generation of researchers works to
develop observation protocols and instruments to help us
understand the relationship between classroom practices, teacher
effectiveness, and student achievement (Bell et al., 2012; Hill,
Charalambous, & Kraft, 2012; Mihaly, McCaffrey, Staiger, &
Lockwood, 2013).
Growing interest in classroom observation also crucially
reflects the widening reach of teacher evaluation policies around
the world (Bruns & Luque, 2014). Despite the ubiquity of efforts to
use student achievement to evaluate teachers, in practice most
teachers in the United States are evaluated through procedures
that rely heavily on classroom observation (Kane & Staiger, 2012;
Loup, Garland, Ellett, & Rugutt, 1996). Indeed, all states granted
funding under the new Race To The Top legislation in the United
States included a new or redesigned classroom observation
component for teacher evaluation (Reform Support Network,
2012), and a similar trend is evident in international reviews (see
e.g., OECD, 2013a, 2013b; Santiago & Benavides, 2009). In the
2007 Teaching and Learning International Survey (TALIS), for
example, over 70% of teachers in participant countries reported
that classroom observation was a component in performance
evaluation and feedback (OECD, 2009). This is not unexpected
because classroom observation enjoys considerable face validity
among educators and policymakers, and is seen as the key source
of information supporting teacher formative evaluation and
feedback (Danielson, 1996; Goe et al., 2008; Pianta, La Paro, &
Hamre, 2007; see also Chait, 2010; Weisberg, Sexton, Mulhern, &
Keeling, 2009), teacher self-reflection (Richards, 1991), and the
diagnosis of areas of strength and those in need of improvement
(Protheroe, 2002).
As with teacher evaluation more generally, however, what it
means to do classroom observation for teacher evaluation can
differ across systems and contexts. Below we outline an analytic
framework for systems of classroom observation that involves
three main dimensions, related to their basic conceptualization of
instructional practice and teacher effectiveness (i.e., conceptual
issues), the sources of evidence and methods used to gather
information about these constructs (i.e., methodological issues),
and policy context, processes, and decisions that shape the
evaluation (i.e., policy issues).
2.1. Conceptual issues
The first step in designing an observation system is to decide
upon the theoretical or conceptual underpinnings that will provide
the basis for understanding, describing, and assessing teacher
practice. Standards of effective teaching have become widespread
in the past decade (e.g., Campbell, Kyriakides, Muijs, & Robinson,
2004; Danielson, 1996) and many education systems are stan-
dardizing their approaches to classroom observation to align them
to such frameworks. The expectation is that clear and explicit
teaching standards, and matching observation rubrics, will provide
useful guidance to teachers and administrators for understanding
and promoting high-quality instruction (Darling-Hammond, 2013;
Ingvarson & Rowe, 2008). One example is the Framework for
Teaching (Danielson, 1996, 2011) adapted in a number of school
districts in the United States (e.g., New York, Los Angeles,
Cincinnati) and abroad,—e.g., as “Marco para la Buena Enseñanza”
in Chile (Ministry of Education, 2004). Within each of these sets of
standards, protocols and rubrics are typically developed targeting a
narrower set of dimensions, deemed most amenable to direct
observation in classrooms. These protocols may characterize
teacher performance in general pedagogical aspects of instruction
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
(e.g. time and classroom management; differentiation; teacher-
student interactions) and content- or subject-specific aspects (e.g.
cognitive challenge of subjects; alignment of instruction and
learning goals).
Systems can also target different groups of teachers for
observation—e.g. some systems seek to monitor and mentor
novice teachers, others are meant as a general approach for
assessing teacher practice, irrespective of status or seniority. They
can also give different weights to the information collected
through observation in informing broader judgments about
teacher practice and effectiveness—effectively changing the
definition of the teacher construct(s) targeted. Finally, systems
can differ in terms of the extent and nature of feedback offered to
teachers; for example, feedback can be more or less explicit (Hattie
& Timperley, 2007), linked to concrete avenues for positive change
(Kimball, 2002), or based on single or multiple sources (Seifert,
Yukl, & McDonald, 2003). Notably, various actors and stakeholders
may be expected to use the information collected through
classroom observations. For example, administrators or observers
(or both) may have a role in selecting and designing professional
development activities based on diagnosed shortcomings, and
inform personnel decisions related to promotion, allocation, hiring
and firing (Halverson, Kelley, & Kimball, 2004; Jacob & Lefgren,
2005).
2.2. Methodological issues
A range of methodological issues and choices are involved in
designing and implementing observation systems for teacher
evaluation and development. These include prominently, the
identity and qualifications of the observers; the evidence available
suggests that trained administrators, peers, and district personnel
are capable of similarly reliable ratings (Bell et al., 2014; Ho & Kane,
2013). Second, the modes of observation, and within these, the
type and extent of training provided to observers for certification,
has been found to be a critical component of accurate scoring
(Casabianca et al., 2013; Leahy, 2012; McKay & Silva, 2015). Third,
the number of observations per teacher, the approach for sampling
lessons for observation, and the appropriate combination of
announced and unannounced, formal and informal classroom
visits also need to be considered. The preponderance of evidence in
this regard suggests that a reliable assessment of teaching requires
multiple observations, but beyond that general level the results are
far from conclusive. Recent evidence from the influential Measures
of Effective Teaching (MET) project in the U.S. suggests that at least
two different observers, each observing four lessons may be
needed (Ho & Kane, 2013), while earlier work by Erlich and
Shavelson (1978) suggests that as many as seven or more lessons
may be required for generalizations involving a single observer.
Ultimately, no broad guidance is available about the number of
observers or observations; any such judgment would have to be
local and take into consideration the purpose and context of the
observational system, and the variety of factors that might affect
the reliability of the results. However, the literature does seem to
suggest that the reliability of indicators of teaching practice based
on classroom observation is generally lower than that traditionally
expected of large-scale standardized paper and pencil tests. Even
reliability coefficients that would be considered mediocre by
traditional standards can be difficult and expensive to attain,
requiring extensive training, and multiple visits to the classroom
by multiple observers (Martinez, 2013; Ho & Kane, 2013). The
precision of measures has not been examined extensively, but
what evidence exists suggests the measures can differentiate
between the upper and lower ends of a performance distribution,
while finer distinctions are more difficult to establish (Ho & Kane,
2013; Taylor & Tyler, 2012).
17
Finally, it is well established in the measurement literature that
in addition to adequate reliability, developers need to ensure that
sufficient evidence is available to support the validity of intended
inferences about teacher practice based on information collected
through observation (American Educational Research Association,
American Psychological Association, & National Council on
Measurement in Education, 2014; Bell et al., 2012). As with
reliability, variation can also be expected in the extent to which
teacher evaluation and improvement systems will study and
provide support for the validity of the inferences derived from
classroom observation indicators (Taut, Santelices, & Stecher,
2012).
2.3. Policy issues
Policy-related factors are as important as methodological and
conceptual considerations for influencing the design, credibility,
and sustainability of a teacher evaluation and development system
(Herman & Baker, 2009). Critical policy factors include the actors
involved in creating and designing the system, buy-in from key
stakeholders, and public narratives around the system’s motiva-
tion, credibility, and predicted consequences. The literature
consistently suggests that contextual factors are particularly
relevant where the political context is charged, and objective
technical guidance for practitioners and policymakers is scarce
(Elmore, 2008; Engel, Williams, & Feuer, 2012; Fullan, 1992; Joint
Committee on Standards for Educational Evaluation, 2009;
Peterson, 2000). Here we consider how three key policy factors
may inform and explain the conceptual and technical features of
observation systems we have examined.
The first factor relates to the stakes attached to teacher
evaluation in general, and to classroom observation in particular.
Systems that attach concrete consequences to the evaluation
intend to exert maximum influence over teacher practice through
detailed indicators and rubrics. Developers get maximum control
over what gets measured and how, but need to choose and
operationalize the indicators of practice of interest in advance, and
to carefully consider issues of standardization, reliability, and
fairness in measurement. These systems can offer incentives to
teachers for artificially improving performance, raising questions
about authenticity of the measures, and compromising their
formative value. At the other end, low stakes systems, while less
effective at driving behavior can do better at serving formative
purposes and ease concerns about resistance to the system and
challenges to its legitimacy as a policy (City, Elmore, Fiarman, &
Teitel, 2009). Notably, however, even within low stakes systems
valid and effective feedback requires attention to the quality and
fairness of the measures.
The second dimension is locus of control—who implements the
observation on the ground. Systems can be locally defined and
implemented at the school level with school leaders serving as
observers and evaluators, or centrally organized and implemented
at the district, regional, or national level—issues of standardization
and measurement are more salient in the latter case, as
observation tools gain in complexity, and the stakes tend to
become higher (OECD, 2013b). Local control often goes hand in
hand with a desire to strengthen school-level autonomy, but this
requires a minimum of capacity in all schools. As a result, lower-
performing systems with lower professional capacity tend towards
centralization and standardization in evaluation (Mourshed,
Chijioke, & Barber, 2010).
The third dimension addresses the model of accountability that
is dominant in each context (Abelmann & Elmore, 1999). As
proposed by Levitt, Janta, and Wegrich (2008) we differentiate
between self-regulating, professional, and organizational models
of educational accountability. Self-regulated accountability
18 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29

normally precludes teacher evaluation as a formal policy tool, since
teachers are trusted to hold themselves accountable based on
strong feelings of moral and ethical responsibility for the common
good. Professional accountability systems see teachers as trusted
professionals with peers and school level personnel playing the
key role in teacher development and evaluation—while explicit
standards and regulations may provide a basis for accountability,
monitoring compliance with these standards is more implicit and
locally shaped. Finally, organizational accountability models tend
to rely on explicit, bureaucratic, centrally implemented mecha-
nisms to monitor teacher practice and effectiveness, with a focus
on ensuring compliance with standards and regulations, and the
expectation that stronger control and more pressure will result in
improving desired outcomes (such as student achievement).
Where teacher status is high, systems may lean towards self-
regulated or professional in-service evaluation; while lower
professional status for teachers may be coupled with organiza-
tional mechanisms of in-service evaluation (Tatto, Schwille,
Ingvarson, Rowley, & Peck, 2012).
3. Methods
Our paper examines classroom observation systems in a diverse
cross-section of local, regional and national educational juris-
dictions in six countries. The overarching research question is
focused on understanding how different systems around the world
use classroom observation to inform teacher evaluation and
professional development. More specifically, we view similarities
and differences across observation systems in relation to the
analytic framework outlined above, which includes the conceptu-
alizations of classroom practice underlying the observation,
methodological approaches employed to collect high quality data,
and contextual and policy factors that interact with the two.
3.1. Sample
We purposively selected observation systems designed in the
context of teacher professional development and appraisal, or
school evaluation, for which we were able to obtain sufficiently
detailed information from available documents (either publicly
accessible or made available to us by program staff). We did not
seek to statistically represent a hypothetical target population of
observation systems, but instead to demonstrate and describe the
variety of uses of and approaches to classroom observation in a
variety of national, regional, and local contexts around the world.
We thus sought to construct a sample to maximize variation
(Patton, 2002) in terms of the three key dimensions in our analytic
framework, namely conceptualization (model of instructional
practice and professional development; target constructs; weight
and uses of observation data), methodology (identity of observers
and observed, mechanisms of data collection, standardization and
data quality), and policy context (system size; locus of control;
stakes for participants). The resulting sample comprises 16 class-
room observation systems in six countries, including two of the
highest performing education systems in the world (Singapore and
Fukuoka Province, Japan), the highest performing and one of the
fastest improving system in Latin America (Chile), and regional
systems in Australia (Barwon South Western Region, Victoria) and
Germany (Hamburg and Saxony) that add to the diversity of the
sample. The study also included a variety of state and local systems
in the United States, including the three largest school systems in
the country (New York City, Los Angeles, Chicago), one state
(Tennessee), and notable examples in smaller districts (Toledo,
Cincinnati, Pittsburgh, Santa Monica), along with two high profile
independent systems that reach a large number of teachers across
the country (the observation component used for training and
certification in Teach for America, and the National Board for
Professional Teaching Standards) (see Table 1).
3.2. Data collection
Data was collected from multiple sources and through a variety
of methods. First, we collected general information about the
systems in our sample in publicly available reports, online
documentation, and relevant publications. Second, lead personnel
at each participating system provided a range of additional

Table 1
Classroom observation systems included in the sample.

System Country (region) Number of Students* Year of Inception

Instructional Rounds Australia (Barwon South Western Region, 541,990 (2015) 2008
Victoria)
Sistema de Evaluación del Desempeño Profesional Docente, Docentemás Chile 1,120,000 (2014) 2003
Schulinspektion Hamburg Germany (Hamburg) 188,000 (2015) 2007
Externe Schulevaluation Sachsen Germany (Saxony) 350,000 (2014) 2007
Performance Evaluation System Japan (Fukuoka) 481,092 (2013) 2006
Enhanced Performance Management System Singapore 510,700 2001
(2012)
National Board of Professional Teaching Standards USA 110,000+ teachers 1987
(2014)
Teaching As Leadership Performance System (Teach For America, USA 50,000+ teachers 2001
TFA) (2015)
Excellence in Teaching System USA (Chicago) 399,680 2013
(2015)
Teacher Evaluation System USA (Cincinnati) 34,000 1999
(2015)
Educator Growth and Development Cycle USA (Los Angeles) 655,490 (2014) 2012
Annual Professional Performance Review USA (NYC) 1,100,000 2013
(2012)
Research-based Inclusive System of Evaluation USA (Pittsburgh) 25,000 2010
(n.d.)
Standards-based Teacher Evaluation System USA (Santa Monica) 11,400 (2011) 2005
Tennessee Educator Acceleration Model USA (Tennessee) 995,890 (2015) 2011
The Toledo Intern Plan USA (Toledo) 22,200 1981
(2014)
*
Note: in parentheses included is the year the information represents.
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29 19

materials and documents, including unpublished technical reports, commonalities or differences across systems. Such contextualized
internal research documents, policy guidelines and directives, qualitative summary was nevertheless necessary, given the scope
scoring rubrics, and observation manuals, among others. Finally, and variety of the observation systems we review and synthesize in
we conducted structured interviews (by telephone or in person) this paper.
with key personnel, or experts with direct knowledge of each
system; these interviews allowed us to fill remaining gaps in the
4. Contextual information on the sample of classroom
information available in the materials, and obtain additional detail
observation systems
about the historical and policy context for each participating
system.1
Before presenting the results we offer a brief description of the
broader teacher assessment and development systems in order to
3.3. Analytic approach
help situate and contextualize our subsequent analysis of
similarities and differences among classroom observation systems.
International comparative educational research emphasizes the
The three largest school districts in the United States (New York,
value of analyzing similar experiences across different contexts in
Los Angeles and Chicago) launched comprehensive efforts in
order to derive insights that can inform educational practice and
recent years to redesign their teacher evaluation systems to align
policy-making. Our study sought to understand, compare, and
with guidelines of the federal Race to the Top legislation, which
classify a sample of existing systems of classroom observation in
requires a comprehensive approach that integrates evidence of
educational jurisdictions around the world, along dimensions of
student academic growth, and other indicators of teacher practice
conceptual, methodological, and policy variation. Importantly, as
and performance (U.S. Department of Education, 2009). Indeed, in
discussed above this is not an international comparative study in
addition to adopting some form of value added model for assessing
the sense of contrasting representative sets of units (in this case,
teacher contributions to student achievement, the three districts
observation systems) across countries. Instead, observation
replaced informal and largely perfunctory classroom observation,
systems were selected based on the analytic framework outlined
with a standardized large-scale observation system tied to models
above, and examined with attention to how they exist within their
of instruction and teacher practice adapted from Danielson
national, regional and local contexts. Our analyses seek not to
(Danielson, 2011).
explain why certain countries (or districts) adopted the systems
The state of Tennessee also recently redesigned its approach to
they have, but to outline an analytic framework of classroom
teacher evaluation in order to secure funding under the new
observation systems based on conceptual, methodological, and
federal guidelines. Unlike other examples, however, Tennessee’s
contextual commonalities and differences. By doing so we intend
redesign did not entail adding indicators of student achievement to
to help bring cohesion to the research literature and help support
a stagnant observation system. Instead, it strengthened its long-
policymakers in systematically considering key questions involved
standing value-added accountability system (the first imple-
in the design of classroom observation systems.
mented in the country in the 1990s) by adding a robust
The analytic approach involved qualitative examination and
standardized classroom observation component. Districts can
synthesis of information from multiple sources described in the
choose from four state-approved frameworks to guide classroom
previous section. In gathering data about these systems, we sought
observation, of which the most widely adopted is the TAPTM model
to triangulate or “cross-examine” the initial descriptions gained
(originally known as the Teacher Advancement Program) from the
from publicly available documents by interviewing key district/
National Institute for Excellence in Teaching (Barnett, Wills,
system personnel, in some cases multiple people, and by
Hudgens, & Alexander, 2015).
examining internal documents made available to us. This
Cincinnati and Toledo exemplify a long-standing model of
triangulation of sources provided a more in-depth understanding
formative teacher evaluation that relies on classroom observation,
of observation systems and a clearer sense of how they are
and focused feedback to teachers from peers, experts and
implemented on the ground (as opposed to how they are described
administrators—while still including high-stakes summative
on paper). We organized the analysis around the analytic
evaluation for tenure and dismissal (Danielson, 2011; Koppich,
framework outlined earlier, which comprises conceptual aspects
2009). Notably, while these systems do not include student
(e.g. models and dimensions of instruction), methodological issues
achievement as a factor in high stakes decisions, research
(e.g. structure of observations, sources of evidence), and policy
undertaken in Cincinnati schools offers empirical evidence that
context (e.g. type of accountability model, stakes). These dimen-
observation scores are associated with value added indicators of
sions were the basis for classifying systems and highlighting areas
teacher effectiveness (Milanowski, 2004) and that the observation
of conceptual and methodological commonality and difference,
process leads to improvements in teachers’ value-added scores
within specific policy contexts. To examine systems along these
(Taylor & Tyler, 2012). Similarly, Pittsburgh recently redesigned its
dimensions we used within- and cross-case analyses (Miles &
evaluation system to incorporate formative evaluation based on
Huberman, 1994) to generate case summaries for each system, and
classroom observations in all personnel steps from teacher
devise classification schema that were then revised and refined
selection and training to advancement and promotion (without
over multiple rounds of discussion. Based on these discussions our
stakes for teachers). A collaborative effort brought together district
three-person research team developed an in-depth understanding
leadership, administrators, and the teacher union, an exception to
of the characteristics of each system, and ultimately reached a
the contentious process common in other districts.
consensus classification decision. Also, this classification neces-
The system in place in the Santa Monica Unified School District
sarily involved a qualitative data reduction exercise where
exemplifies a common model in small and mid-size districts in the
complex features of systems were summarized and matched to
U.S., where detailed rubrics are available to guide standardized
dimensions in our framework. The analyses are thus subject to
observation, but school principals are given ample discretion to
human judgment error and inconsistency, and involve a certain
implement them in the field, and use them to provide feedback to
degree of arbitrariness in the decision to emphasize certain
teachers (Ellett & Garland, 1987; Loup et al., 1996). This is also the
more common approach internationally, if compared to the large-
scale standardized observation systems emerging in recent years
1
A complete list of documents and resources consulted for each participating in the United States.
system is available in Appendix A.
20 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29

Table 2
Summary of key conceptual aspects for selected teacher evaluation systems.

Framework or Standards of Practice Emphasis of Target of Weight in Role in Professional Development

Observation Observation evaluation
Chicago Illinois Professional Teaching Classroom Tenured and 65-70% of total Mandatory feedback after observations (and post-
Standard + CPS Framework for Teaching Environment and Non-tenured summative obs. conferences). Observer designs PD plan for
(Synthetic Danielson: 4 Domains, Instruction Teachers score areas in need of improvement. Online and live
19Components) mentoring aligned to CPS is available
Tennessee “TAP” Model (National Institute for Planning, Apprentice & 50% of total Observer offers feedback in writing & in-person
Excellence in Teaching). 4 Domains, 19 Instruction, Professional summative within a week focusing on areas identified for
Indicators Environment teachers score development. Final yearly summary rating
Teach For Teaching as Leadership (TAL) Rubric. Goals, Planning, Teachers in first 0%: Information from observations, and pre- and post-
America Internally developed: Execution, Working two years of Summative; conferences helps determine the kind and level of
observations + interviews of TFA teachers relentlessly career 100%: support needed
with largest student achievement gains Formative
evaluation/PD
Toledo Toledo Plan, internally developed: joint Procedures, CR Teachers in 1st- 100%: Each CT is responsible for mentoring and evaluating
union/management committee management, 2nd year or Evaluation is a small number of teachers. Mentoring relies on
subject knowledge, identified for based entirely observation, but can vary at discretion of CT
professionalism intervention on observation
Chile Adapted for local use from the Danielson Learning Public school 43% of Automated report describes performance level for
FFT (1996): 4 Domains, 20 criteria environment, teachers starting portfolio score each dimension and indicator. Municipalities
Lesson structure, in 3rd year of (26% of overall determine PD activities based on evaluation results
Pedagogical service score)
interaction
Singapore Singapore Teaching Competency Model. Nurturing the All teachers Not specified: Prior results guide observation. Verbal feedback
Developed internally w/support from US- whole child One piece in a after each observation, mid- and end-of year review.
based HR firm (13 Competencies, in holistic Observer & teachers discuss PD/mentoring needs,
5 Clusters) evaluation and activities
Fukuoka Developed by Board of Education. 3 Counseling skills; All teachers Not Specified: Holistic, at discretion of teacher and principal
Domains, 18 criteria Instructional skills; One piece in a
Others holistic
evaluation

The sample also includes two non-traditional systems with far
reach across districts and states in the United States. The National
Board for Professional Teaching Standards (NBPTS) offers subject-
and grade-specific certificates for teachers through a portfolio-
based evaluation model (Hakel, Koenig, & Elliott, 2008), intended
to identify and recognize the best teachers in the country, and to
inform the broader discussion around the characteristics of
effective teachers. On the other hand, Teach for America is an
independent organization seeking to improve school districts’
access to qualified teachers, by training and deploying a corps of
high-achieving young professionals to teach in schools that serve
high proportions of disadvantaged students.
The Chilean national teacher evaluation system has received
attention as an exemplar at regional level (Santiago, Benavides,
Danielson, Goe & Nusche, 2013; Bruns & Luque, 2014). The system
collects teacher portfolios that include written evidence of work
and a video of one lesson, along with supervisor questionnaires,
peer interviews, and a self-evaluation (Taut & Sun, 2014; Manzi,
Gonzalez, & Sun, 2011). Teachers undergo evaluation every four
years and may be eligible for significant monetary incentives, or
subject to mandated professional development, and even removal
if underperformance persists. Formative feedback to teachers
relies to an important extent on a professionally videotaped lesson
that is contrasted to a model of good instruction (Marco para la
Buena Enseñanza) adapted from the Danielson Framework
(Danielson, 1996). Australia is in the process of developing national
teaching standards (Australian Institute for Teaching and School
Leadership (AITSL), 2012) and a comprehensive teacher develop-
ment and appraisal system. Here, we studied teacher evaluation in
one district in the territory of Victoria, where instructional rounds
(City et al., 2009) are used to guide both school improvement and
teacher development. Wide leeway is given to each local district
and school to adapt the approach to their local needs. Novice
teachers often are assigned a mentor, while a team of school
leaders and peers conducts periodic classroom observations to
support teacher improvement efforts in a direct and contextual-
ized way.
Singapore’s education system is notable for the high level of
achievement of its students in international assessments, and its
teacher evaluation approach for its low stakes and dual focus on
excellence and professional growth (Sclafani & Lim, 2008). Teacher
evaluation is largely based on multiple classroom observations
throughout the year by peers, experts, and administrators, using
the local Enhanced Performance Management System (EPMS) as a
model of reference for classroom practice and teacher competence.
Similarly, teacher evaluation in Fukuoka, Japan emphasizes helping
beginning teachers develop the skills and practices of senior
faculty.2 Principals are trained in classroom observation by the
Board of Education and given wide leeway to determine the
frequency, focus, and use of observation to inform instructional
assessment and improvement. Principals typically involve senior
teachers in the school in the process of observing and providing
feedback to novice teachers.
In Germany, teacher candidates are observed frequently and
offered detailed formative feedback by their mentors during initial
2
Our discussion refers specifically to the Fukuoka prefecture but this system is
largely representative of how teacher evaluation is carried out in other locales in
Japan (Ministry of Education et al., 2016).
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29 21

Table 3
Frameworks for teacher practice in selected international teacher evaluation systems.

Fukuoka (Japan) Singapore Teach For America Danielson (USA)

Student development & Counseling
Consider pupils’ languages, values,
and human rights
Build trust and develop pupils;
Address student incidents
Consider health and safety
Facilitate learning environment
Provide appropriate counseling
Support co-curricular activities
Instructional skills
Develop annual plan based on
standardized curriculum
Facilitate an organized class
Set appropriate objectives
Facilitate learning creatively
Understand learning conditions
Provide supports to students
Conduct evaluations to encourage
learning
Have and improve content knowledge Working with Others
and skills
Others
Cooperate and collaborate to achieve
school missions
Build relationships with community
and parents
Practice risk management
Discharge duties as public official
Nurturing the Whole Child
Share values with student
Take action to develop student
Act in student’s interest
Winning Hearts and Minds Understand Create a welcoming environment
the environment
Develop others
Knowing Self and Others
Emotional intelligence
Partner with parents
Work in teams
Cultivating Knowledge
Subject mastery
Analytical thinking
Initiative
Teach creatively
Set Big Goals Planning and Preparation
Ambitious, feasible, aligned goals Knowledge (content + pedagogy)
Knowledge of students
Invest Students and Influencers to Work for Select instructional goals
the Big Goal Knowledge of resources
Instill “I can” and “I will” message Design coherent instruction
Reinforce effort and mastery Assess student learning
Model persistence and success
Classroom Environment
Create environment of respect
Plan Purposefully Establish a culture for learning
Assess progress toward big goal Manage classroom procedures Manage
Design long-term and unit plans student behavior
Develop aligned lesson plans Organize physical space
Differentiate
Develop rules and procedures Instruction
Communicate clearly
Execute Effectively Use questioning strategies
Clearly present academic content Monitor Engage students in learning
student understanding
Rules, consequences Provide feedback to students
Time-saving procedures Be flexible and responsive
Continuously Increase Effectiveness Professional Responsibilities
Identify progress for subgroups Reflect on teaching
Identify key actions (stdnt/tchr) Maintain accurate records
Adjust to solve problems Communicate with families Contribute
Build on strengths to school/district
Grow/develop professionally
Work Relentlessly Show professionalism
Persist in the face of challenges
Solve time/resource constraints

training (Referendariat). Moreover, certification and tenure require
that teachers be observed and assessed teaching a lesson by a
board of examiners. Otherwise observation is given a marginal role
in career promotion, but mostly as a pro-forma exercise with little
feedback for teachers or tie to models of instruction or professional
practice. On the other hand, classroom observation is used in the
context of school evaluation or inspection (following the traditions
in the Netherlands and England). This approach relies on highly
knowledgeable inspectors or trained teacher evaluators, and state
frameworks for classroom and school quality. It considers
classroom observation results in the aggregate by school, with
no feedback provided to individual teachers (Müller, Pietsch, & Bos,
2011). We discuss examples in two German states (Hamburg and
Saxony).
5. Results
We analyzed the characteristics of the 16 systems of classroom
observation in relation to the three components of our theoretical
framework: conceptual aspects, methodological issues, and policy
context. For clarity of exposition, the tables in the Results section only
contain a subset of seven systems (identified in bold in Table 1),
chosen to eliminate redundancy and illustrate key types of systems
and dimensions of variation. The subset includes national systems in
Chile and Singapore, state systems in Tennessee (United States) and
Fukuoka (Japan), and local systems in Chicago Public Schools and
Toledo (both in the United States), along with an independent U.S.
based system (Teach for America). Information for the full sample of
16 systems is available in Appendix B.
5.1. Conceptual aspects
In order to investigate how different systems conceptualize
classroom observation, we prioritized aspects of our framework
that are a particular focus in current policy reform efforts. These
include the model or framework of instructional practice; the
teachers targeted for observation; the weight assigned to
observation in the evaluation; and the role of observation in
informing feedback and professional development. Table 2 sum-
marizes this information for the illustrative subset of seven
systems.
5.1.1. Models and dimensions
What observers look for in the classroom varies across systems
in our sample, from a few broad dimensions of instruction, to
detailed outlines including dozens of specific features of teacher
practice. For example, a number of districts in the U.S. (including
the three largest in New York, Los Angeles and Chicago) and
internationally (e.g. Chile) have adopted the Danielson Framework
for Teaching (Danielson, 1996), which codifies the work of teachers
in great detail in terms of dozens of specifics elements of classroom
practice. While use of the Danielson framework is becoming
widespread, a great variety of frameworks are in use in other U.S.
districts and internationally. Table 3 shows the dimensions of four
models of teaching practice used around the world. Not only do the
numbers of domains and dimensions vary, but definitions of good
teaching practice are also to some extent local and culture–specific.
As we might expect, there are areas of considerable overlap across
systems and countries; for example all systems consider teachers’
content knowledge, and their ability to plan and set teaching and
learning goals. Similarly, every model includes in some form
teachers’ consideration of student characteristics, and the use of
appropriate assessment practices in the classroom as a key
dimension of high quality instructional practice (Brookhart, 2011).
At the same time, there are differences in the focus and breadth
of areas targeted for observation across systems. Classroom
observation in Singapore and Japan leverages information
obtained through frequent observation of classroom practice
throughout the year, to infer other underlying competencies and
attitudes (e.g. nurturing the whole child,winning hearts and minds;
22 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
pupils’ values and human rights, pupil trust) which tend to receive
less attention in American frameworks, where technical and
procedural aspects of instruction play a larger role (e.g. questioning
techniques, classroom management, identify progress for subgroups).
In Singapore, supervisors monitor each teacher’s progress on these
competency goals, informally observing and conferring with
teachers frequently throughout the year (and providing feedback
and guidance when needed). The emphasis and attention paid to
areas of student growth and interests also differs: Singapore
broadly directs teachers to share values with students and act in
their best interest (e.g. recognizing individual potential and
developing self-confidence); Danielson references knowledge of
students as part of planning for quality instruction and later to use
questioning strategies effectively; a third of the Fukuoka frame-
work centers on student development and counseling, in relation
to values, trust, language, health, and safety; finally, the Teach for
America model is based on the Teaching as Leadership frame-
work—which perhaps more than the others emphasizes concrete
behavioral aspects of classroom practice (reinforce rules and
consequences, identify progress for subgroups), as well as attitudinal
and motivational targets like personal effort and perseverance.
One important note of caution is in order concerning the grain
size of the analysis of dimensions. Because we did not have access
to detailed observation protocols for all systems, we analyzed the
general definitions and descriptions of the dimensions included in
the frameworks. However, we did not examine in detail the specific
classroom behaviors and processes that may be comprised in the
operationalization of these dimensions. Thus, even dimensions
that seem to be similar could differ in terms of the specific
classroom processes and interactions included or emphasized in
their operationalization—the specific features of instruction that
an observer would attend to.
5.1.2. Target of observation
The systems we examined vary considerably in terms of the
specific teachers they observe in the classroom. Systems like
Toledo, or those of German states focus primarily on novice or in-
training teachers. After this initial intensive period, classroom
observation for tenured and experienced teachers in these systems
becomes infrequent and somewhat routinized. Singapore empha-
sizes summative aspects of evaluation for teachers in training and
hiring decisions; once hired, teachers continue to be observed and
evaluated, but punitive consequences are rare since the focus shifts
toward professional growth and shaping teachers’ career paths. In
these systems in general, veteran teachers are observed less
frequently, feedback is less systematic, and the consequences of
the evaluation are less critical. Finally, the Teach for America
system in the United States focuses resources entirely on teachers
in their first two years, after which they are considered alumni of
the program and no longer receive formalized support or
mentoring from the organization.
Santa Monica exemplifies a decentralized, semi-standardized
system common to smaller districts in the United States, where
principals exert a great deal of discretion in selecting the teachers
that will be target of observation—and the aspects of practice that
will be the focus of observation. Thus, one principal may decide to
observe all teachers focusing on one dimension of practice across
the school, while another may decide to observe only novice or
struggling teachers, and tailor observations to focus on specific
dimensions of interest for individual teachers. Their choices
exemplify an effort to balance depth, detail, and comparability
that is present in one form or another across systems.
The new generation of systems in development in large school
districts in the United States expands standardized observation to
all teachers throughout the career cycle (e.g. New York City and Los
Angeles explicitly aim to include all teachers in classroom
observation). While some allowances are made for less frequent
observation of teachers exhibiting adequate performance, even
veteran teachers are observed in regular cycles, and at key points of
career advancement. The Tennessee or Chicago systems, for
example include all teachers, but prescribe different schedules
of observations at different career stages (apprentice vs. profession-
al; tenured vs. non-tenured).
5.1.3. Weights and roles of observation
In most of the evaluation systems we reviewed classroom
observation is the most important component, both in terms of
summative weight and value in informing professional develop-
ment. Where information from various evaluation components is
combined into an overall indicator of effectiveness, observation
typically receives the largest weight, ranging from 40% to as much
as 100% of the overall score. In Chicago, for example, 65% to 70% of
the total summative score assigned to teachers is based on the
observation ratings. In Los Angeles, New York, and Tennessee the
weight ranges from 50% to 60%. In other cases, the information
collected through observations is also weighted highly but
combined in a qualitative or holistic process, without explicit
mathematical rules or algorithms. This is the case in Santa Monica
where the observation is the main source of evidence in a holistic
evaluation. Similarly, in Singapore evidence of performance is
collected through intensive observation and is considered holisti-
cally for evaluating (mostly early career) teachers. Chile presents an
intermediate case where the information from observation is
incorporated through explicit quantitative guidelines and makes
up almost half of the portfolio, but its weight is minor when
combining scores on four different instruments to determine
teachers’ final performance category.
Lastly, while all systems share a discourse of formative
evaluation focused on improving teaching, important differences
emerge in how observations are structured and used specifically to
inform teacher professional development. Teachers in Chile
receive a written, automated report that describes their perfor-
mance on the seven portfolio dimensions and respective indicators
(based on the written evidence as well as the recorded video).
Chilean school districts use these results to assign federal funds to
professional development plans for low-performing teachers. In
Japan principals are free to use the information collected through
standardized observation rubrics holistically to determine profes-
sional development needs and plans for each individual teacher.
Finally, in Germany the link operates at the level of the school, not
the individual teacher; inspectors may give feedback to the school
community emphasizing issues observed during classroom
observation, which the principal can then further discuss with
teachers in order to delineate school professional development
plans.
Most of the new large-scale observation systems in the U.S.
require that trained observers (and/or school administrators) meet
with teachers, both before the observation to discuss focus areas,
and after observation. Post-observation meetings are used to
debrief teachers on the results of the observation, discuss them in
more detail, and delineate plans for moving the teacher forward,
including appropriate professional development. This is the case
for example in Chicago, Tennessee, and Los Angeles. On the other
hand, traditional peer assistance models like Toledo rely on a
trained Consulting Teacher (or teachers) for mentoring and
evaluating small cohorts of mainly novice teachers using
information they collect through classroom observations.
5.2. Methodological aspects
As with conceptual features, classroom observation systems
can also differ in important ways in terms of how the observation is
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29 23

carried out in practice. For our analysis we focus on three key and improve instruction among teachers in training, and to inform
aspects of methodological variation: the frequency, number, and tenure and promotion decisions.
format of observations; the background and qualifications of those The number of observations carried out within these observa-
observing; and the procedures followed to examine the reliability tion cycles varies across systems. We found the lowest number
of the data collected and the validity of the inferences drawn. (one observation per cycle) in Chile and Santa Monica; other
Table 4 presents this information for the same sub-set of seven systems require two observations per cycle (Los Angeles), four
systems presented previously. (Pittsburgh, Tennessee), or more (New York City). Toledo on the
other hand focuses on the volume of evidence from observation,
not frequency; the guidelines call for teachers to be observed for a
5.2.1. Frequency and number of observations
total of 20 h of instruction throughout the year. As before, the
As discussed above, most of the systems in our sample establish
number of observations can also vary within systems depending on
different cycles of observation for teachers in different career
the stage of career advancement or prior performance: Chicago for
stages. Generally speaking novice teachers are observed more
example requires two observations for tenured teachers, and three
frequently than tenured and experienced teachers; this includes
for those without tenure or identified for improvement, and New
both the cycle of observation and how many times they are
York requires two and six observations, respectively, for the same
observed in an observation cycle. Some systems aim to observe all
groups; similar differential criteria are set in Los Angeles,
teachers every year; in our sample these include New York and
Tennessee, and Pittsburgh.
Tennessee in the U.S., and Fukuoka and Singapore internationally—
Another dimension of variation is whether classroom visits are
although in these latter cases principals have discretion for
arranged with or otherwise known in advance to teachers (Los
deciding who is observed and how often. Among the remaining
Angeles, Chile), or involve a combination of announced and
systems, yearly observation of beginning teachers is the norm (Los
unannounced visits (Tennessee, New York, Singapore). None of the
Angeles, Chicago, Santa Monica, Toledo). More experienced
systems in our sample uses only unannounced visits for formal
teachers are observed in cycles of two years (Chicago), three years
assessment, but a number use informal unannounced observations
(Los Angeles), or four years (Chile). The Chilean system is
to complement the formal planned observations and enrich
interesting in that the cycle of evaluation (and thus, observation)
formative feedback and professional development (Chicago,
is not tied to career stage but to past performance: teachers rated
Cincinnati). A distinction is needed here between visits that are
as unsatisfactory are evaluated the following year, and those rated
announced and those that are negotiated (i.e. the teacher and the
as basic after two years; everyone else is evaluated in four-year
principal or another observer jointly determine which lessons will
cycles. Examples where teachers are not observed in regular
be observed) or self-selected (i.e. the teacher determines which
intervals beyond the early career stages seem to be less frequent; in
lessons will be observed). A similar distinction is needed between
our sample these were limited to Toledo in the U.S., and Germany,
unannounced observation (i.e. the teacher does not know which
where observation of individual teachers is used mostly to assess
lessons will be observed) and informal observation. In the latter

Table 4
Summary of key methodological aspects for select teacher evaluation systems.

Who is observed? How Who observes? How? What training/certification do observers Research on quality of the observation data
often? Internal v external Announced receive?
(A)/Unann.
(U)
Chicago Tenured: 1–2 every 1 or 2: A (1 informal 50 hours 1) Proficiency test on Internal: agreement w/master raters;
other year; Principal + Calibrators observation framework + Agreement w/master video dimensionality; VAM; qualitative; fairness
Non-tenured: 4/yr in a small% of CR may be U). rating 2) Ability to communicate and develop reviews when observation and VAM differ
PD plan
Tennessee Apprentices: 6/yr Always 2, 2A + 2U 32 + hours. Certification: agreement with Developer studies: agreement w/master
Professionals: 4/yr Principal + AP; master video rating raters; correlation w/VAM
external teams in 20%
of districts
Teach For 1st and 2nd year External Manager of A Varies by region. Rubric norming on teacher Internal: Revision of TAL Framework in
America teachers, as often as Teacher Leadership video & docs; book clubs, webinars, mini- process
determined by Development retreats
observer
Toledo 1st and 2nd year Y1/Intervention: discretion of Ad-hoc training by veterans; CTs share office; N/A; Qualitative
teachers and teachers External Teacher observer new and veteran CT observe together and
in intervention consultants compare notes
Y2: Principal
Chile All public school Externally selected & A (Video) 24 hours, 3-day trial scoring period (scoring Generalizability studies, factor analyses,
teachers, freq. trained teacher raters supervisors are trained 64 hours) in-depth group comparison studies,
depends on prior correlation with VAM, consequential
evaluation validity
Singapore All teachers 2+: Internal. AP, Dept 1 A & 2+ U 100+ hrs/yr (PD to improve teaching, incl. Internal
Head, Senior Teachers, doing observation)
Principal
Fukuoka All teachers Internal: Principal and discretion of Board of Education makes manual and Internal
possibly vice principal principal demonstrates to principals
(s)
24 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29

case, the teacher may or may not know which lessons will be
observed, but the information collected is either a) not part of the
formal evaluation and used only formatively for feedback, or b)
used by the principal or observer to inform their judgment outside
of the observation scoring rubric.
Little variation is apparent in terms of the format of
observation, since in most systems we find direct observation in
classrooms, as is currently the case in most districts in the United
States. Only two systems in our sample the Chilean national
teacher evaluation system, and NBPTS for teacher certification
use video-recordings of classroom teaching. Interestingly, video-
taping is garnering increasing attention as an alternative to live
classroom observations (Gates, 2013), as districts start to bear the
full cost of large-scale classroom observation. Video can be
important for supporting high stakes inferences and decisions,
and could also offer cost savings by allowing remote asynchronous
review—especially if the cost of videographers can be reduced.
5.2.2. Observer background and qualifications
Most systems in our sample involve school administrators as
observers and sources of feedback for teachers, either on their own
(Santa Monica, Pittsburgh), or jointly with other administrators
(Tennessee, New York, Japan, Germany). A handful of systems, like
Chile, Chicago, and Toledo, relies entirely on external raters—
typically trained experienced teachers. In Australia there are two
observation tracks: principals are tasked with observation for
performance appraisal and professional development, whereas
external observers are used for certification of accomplished
teaching.
Rater training is critical in large-scale observation for teacher
evaluation and development. In all systems in our sample
observers undergo training before they can conduct official
observation ratings. Training ranges from 20 to 100+ hours, and
may take the form of formal calibration or norming sessions (Los
Angeles, TFA, Chile), train the trainer approaches (Pittsburgh), or
shadowing experienced raters during observation (Toledo). Most
systems establish explicit criteria for certifying external observers;
on the other hand, systems that rely on principals or inspectors as
observers often lack any explicit certification requirements (e.g.
Santa Monica, Saxony, Japan). Certification typically requires a
minimum level of agreement with master ratings during training
(Los Angeles, Chile), and may also involve tests of content or
pedagogical knowledge (Chicago), or open ended assessments
measuring the ability to design appropriate professional develop-
ment plans (New York, Singapore).
5.2.3. Reliability and validity
As could be expected from their diverse goals, designs, and
contexts, the systems in our sample also vary in the extent and
kinds of empirical evidence regarding the quality of information
they make available. Of 16 systems in our sample, we found that
only about half had published or otherwise made available reports
explicitly examining the reliability of the indicators generated
through classroom observation, and only four presented empirical
evidence supporting the validity of the resulting inferences about
teachers and teaching practice.1 The extent and nature of reliability
and validity research varies depending on system size and
resources, as well as the stakes for teachers: a small number of
established systems in our sample have implemented standard-
ized classroom observation systems on a large scale, with serious
stakes for participant teachers, presenting the strongest incentives
and needs for providing empirical evidence supporting the validity
of their inferences (Taut et al., 2010). Not surprisingly, these
systems (e.g., Chile, Chicago, NBPTS) offer the most extensive
empirical evidence of reliability and validity for their indicators (
Taut et al., 2012). These systems also tend to have longer histories
and access to technical expertise to use modern psychometric
techniques like Generalizability Theory or Item Response Theory to
investigate the properties of indicators and resulting inferences,
and a comprehensive understanding of validation (Ho & Kane,
2013; Kane, 2006).
In the smaller school districts included in our sample, on the
other hand, the stakes of teacher evaluation tended to be lower and
the need for standardization, as well as reliability and validity
evidence, was perceived as less acute. Thus, systems like Santa
Monica, Toledo, or Fukuoka offer little or no empirical evidence of
the reliability of observation indicators. Evidence supporting the
validity of the intended inferences is even more limited. These
smaller systems provide more space for the professional judgment
and expertise of school leaders to play an evaluative role, so that
empirical research on the reliability and validity of quantitative
standardized measures seems less applicable. Finally, a number of
systems that were recently created or re-designed (e.g. Los
Angeles, New York, Pittsburgh, Tennessee) are in the process of
conducting studies to collect evidence of reliability and validity,
which will be critical for supporting inferences as these systems
become operational.
5.3. Policy aspects
The final dimensions of variation we examined for our sample
of systems centers on their goals and policy context. As outlined in

Table 5
Summary of key policy aspects for select teacher evaluation systems.

Accountability Locus of control Stakes for participant teachers

model
Chicago Organizational Central District 1) Tenure decisions. 2) Formative Evaluation. Mandatory Mentoring PD focused on identified framework
components if unsatisfactory summative rating. Hearing and potential termination for those who fail to
improve.
Tennessee Organizational State and District level Used to inform human capital decisions, including mandatory individual and group professional development,
hiring and promotion decisions, tenure and dismissal, recognition and compensation.
Teach For Organizational Regional level; Discretion Observations are used to support teachers and identify the practices of top performers but performance is
America to Evaluators defined exclusively by student gains.
Toledo Professional Discretion to Evaluators Unsatisfactory teachers can be recommended for non-renewal after 2 semesters without improvement.
Chile Organizational National level Basic/unsatisfactory performance: mandatory professional development. Teachers may be removed with
repeat Basic/Unsatisfactory performance, or receive salary bonus if competent/outstanding AND successful on
subject/pedagogical knowledge test.
Singapore Professional School level Identify candidates for leadership and research roles. Determine annual bonuses. Dismissal in rare instances.
Fukuoka Professional School level Principal tells teachers the results of the observations, advises each teacher how to improve his/her teaching
based on the observation. Generally speaking, observation results not used to fire teachers or give bonuses.
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
our framework, we focus on three relevant aspects of the policy
context, namely the stakes associated to the evaluation for
teachers; the locus of control for the observation or the degree
of local discretion; and the type of accountability model
underlying the approach to observation and teacher evaluation.
Table 5 condenses this information for the same illustrative subset
of systems presented throughout the paper.
5.3.1. Stakes of evaluation for teachers
The stakes attached to classroom observation for participant
teachers differ across systems in our sample. Importantly, the
frequency and stakes of the observation can differ for novice and
tenured teachers, within the same system. Novice teachers are
often observed more frequently (e.g. Chicago, Tennessee) and in
most systems face higher stakes in the form of a tenure decision in
the first few years of their teaching careers (Toledo, New York,
Santa Monica). In addition, the stakes of the overall evaluation are
not necessarily always the same as the stakes of the classroom
observation. In the Teach for America system, for example,
observation carries no summative weight in evaluation and is
used only for formative purposes (i.e. to improve teaching in ways
conducive to improving the key outcome of interest: student
achievement).
For experienced teachers there are non-punitive consequences
in Singapore, Germany, and Santa Monica, where the information
collected through observation is mostly used for formative (or pro
forma) purposes. This contrasts with high-stakes systems like
Tennessee, or Chicago, where the results of observation keep
carrying 50% to 70% of the weight in an overall summative
evaluation score used to determine personnel decisions, in
addition to being used for formative purposes. In fact, most
systems combine formative and summative purposes but they
differ in the extent to which the stakes attached are punitive and/
or rewarding (e.g., dismissal, salary bonuses). The Singapore
system offers an interesting case: while punitive stakes are low, the
results of the evaluation can have important positive consequences
for teachers. Specifically, high performing teachers in Singapore
can be placed in a track to become school leaders, and even
education researchers (with assignment to the National Institute of
Teachers).
5.3.2. Locus of control for the observation (degree of local discretion)
Related to this point, systems differ in how much control over
the observation process rests at school level and how much control
stays with a central authority that determines the methodological
details. For example, the fact that principals act as evaluators
means very different things in Japan and New York City: In Japan,
the principals decide much about how to carry out the
observations while in New York City they have to complete an
online training and pass a certification test to assure calibration. In
Chile, the Ministry of Education sub-contracts implementation to a
specialized university center, but districts are involved at various
steps during the evaluation process. A local evaluation commission
has the legal right to ratify or modify the final evaluation results of
their teachers, taking into account contextual information that
could otherwise not be considered in this national standardized
assessment system. In Victoria, Australia each region or even each
school within each region defines how it will use the so-called
“instructional rounds” approach to classroom observation for their
benefit and improvement, so here the control over the purposes,
methods and uses of observation rests entirely at the local or
school level.
5.3.3. Accountability model
The observation systems sampled differ in terms of the
accountability models in which they are immersed. Singapore
25
and Toledo, for example, exist within professional models of
accountability that are less standardized, with generally lower
(punitive) stakes attached and give flexibility to the local
professional community. On the other hand, observation systems
being developed in large school districts around the United States
in recent years seem to reflect an organizational model of
accountability, which raises the stakes for teachers and aims to
standardize the process as much as possible in order to reduce
discretion (subjectivity) from the evaluators. Not surprisingly,
these systems show most emphasis on establishing the reliability
of the observation processes and resulting measures. Finally, none
of the systems we studied relied on a strictly self-regulated model
of accountability (systems with this model of accountability would
likely not have any systematic classroom observation system);
while the systems often emphasize the value of teacher self-
reflection as a mechanism for improving teaching, in all cases
professional or organizational accountability, or a combination,
was in place or being developed.
6. Designing classroom observation systems for teacher
development and evaluation
Our project aimed to characterize the diversity of approaches to
classroom observation implemented in education systems around
the world for the purpose of teacher professional development and
evaluation. We assembled a purposive sample of 16 observation
systems that comprises two of the highest performing education
systems in the world (Singapore and Japan), the highest perform-
ing and fastest improving system in Latin America (Chile), and
regional systems in Australia and Germany that add to the diversity
of the sample. The study also included a variety of state and local
systems in the United States, including the three largest school
systems in the country (New York City, Los Angeles, Chicago), and
notable examples in smaller districts (Toledo, Cincinnati, Pitts-
burgh), along with two high profile independent systems that
observe large numbers of teachers across the country (TFA, NBPTS).
While the sample is diverse, as the basis for our study it also
poses a number of challenges and limitations. First, there is over-
reliance on U.S. systems, partly because a wider range of distinctive
approaches have been used in that country, but also because of
reasons of access, opportunity, and availability of information.
Overall, the sample includes systems in six countries, but
undoubtedly leaves out others that would yield valuable insights
if examined. For instance, for maximum variation it might have
been advisable to include an example like Norway, where there are
neither national nor regional formalized systems of classroom
observation for teacher appraisal, development or school improve-
ment, while such efforts do exist to varying extent at school (and,
rarely, municipal) levels (Nusche et al., 2011). We recognized early
on that inclusiveness and representativeness would be elusive
goals and instead focused on procuring a sample of systems for
which sufficient information could be collected, and that would
represent a broad enough slice of variation along our key
dimensions of analysis. We believe the range and nature of the
sample of systems we examined can spearhead a broader
conversation around the conceptualization, design, and use of
classroom observation systems for improving teaching practice
around the world, but it undoubtedly can be further extended. In
this context it is also important to note that some of the countries
seen as exemplars in terms of the quality of teachers and teaching,
have not traditionally used systematic classroom observation as a
tool for professional development.
Along with maximum variation, accessibility of information
was another key concern in the study. This included having to
overcome language barriers (the research team covers English,
Spanish, German, and French), as well as establishing rapport with
26 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
key informants. These considerations determined in part which
systems were included in the study, and thus constitute another
limitation of the study. Still, available information varies across
systems, with national, long-standing and higher stakes systems
offering more complete and publicly available information, while
for smaller, local and newly developed systems we had to rely more
on interview data and documentation that was not always publicly
available.
A third limitation has to do with the framework we used for the
analysis. The emerging framework we have offered in the previous
sections is inherently theory- and sample-dependent and does not
purport to be exhaustive or definitive. In fact, the framework we
outline involves an important component of informed judgment
on the part of the researchers, based on dimensions that are
conceptually derived, empirically refined, and far from orthogonal
(a number of areas of overlap or common influences were
discussed in the paper). It might seem difficult to extract specific
conclusions from comparisons across abstract dimensions span-
ning vastly different contexts. However, in practice international
comparative study of a diversity of approaches to classroom
observation is important for informing our views about the
advantages and limitations of this long standing research
technique for capturing instructional practice, and can be useful
for considering the possibilities observation affords for evaluative
purposes in specific policy contexts.
In this section we discuss the implications of our emerging
findings for informing the design of classroom observation systems
for teacher evaluation and development. In particular, we focus on
four issues, related to the choice of framework or model of
instruction; the degree of standardization and locus of control; the
reliability and validity of observation measures; and the appropri-
ate uses of the information collected through observation.
6.1. Model of instruction
A wide variety of frameworks or standards are used in districts
and states in the United States and around the world to
characterize high quality instruction. One conclusion that can be
drawn from our analysis is that the definition of quality teaching is
dependent on local context and culture, and so is the process by
which frameworks are developed and examined. However, there
are some core aspects of quality teaching that can be found in all
systems, for example some form of student orientation; instruc-
tional or pedagogical skills; and professional duties. Despite these
commonalities the complexity of the underlying construct the
quality of teaching leaves room for different emphases and
operationalizations of these shared dimensions. The U.S. systems
we examined tend to have a behavioral focus and emphasize
narrowly defined technical aspects of general or content-specific
instructional practice, supported (to greatly varying extents) by
empirical research linking individual practices to student out-
comes. By contrast, the framework in Singapore comprises what
teachers are as much as what they do in classrooms, and place
evaluative focus on more broadly defined dimensions (e.g. creative
teaching, or teaching that facilitates creative learning)—these
dimensions were developed inductively examining characteristics
and practices of teachers identified as exceptional by peers and
supervisors. The TFA framework balances emphasis on classroom
behaviors with consideration of psychological traits—interestingly,
it was developed inductively but through in-depth study of
teachers who made the largest gains in student achievement (Farr,
2010).
A key question facing developers and policymakers is whether
to develop a framework specific to the needs of the district or state,
or adopt (or adapt) an existing framework. Most frameworks
capture generalizable elements of quality instruction applicable to
any subject, but there are others thus far mostly used for
research purposes focusing on specific features of instruction in
particular subjects (e.g. the Mathematical Quality of Instruction,
MQI, observation instrument; Hill et al., 2008). In the United States,
commercially available models of instructional practice are
receiving widespread attention and are increasingly becoming
synonymous with the broader notion of a teaching framework (e.g.
Danielson). While these models can offer solid design, ease of use,
and comparability, locally developed or adapted frameworks may
offer an alternative that consider the local context and facilitate
involvement and buy-in from key local stakeholders.
If the decision is made to develop a framework locally, the next
set of questions involves the ideal combination of teacher
competencies, psychological traits, and observable classroom
behaviors to include. Developing the framework itself can then
involve a combination of inductive processes whereby high
performing teachers are first identified and used as the basis for
defining the dimensions of the framework, and theory- or
empirically-based approaches that rely on examination of the
literature or research studies relating teacher practices to desired
outcomes. However, developers should not underestimated the
number of empirical steps necessary for moving from a nascent
conceptual framework to a set of protocols and rubrics that can be
used for standardized classroom observation on a large scale. As
demonstrated by recent high-profile efforts, the development and
validation of such frameworks is a process that takes multiple
years and substantial human and financial resources (see Hill &
Grossman, 2013; Kane & Staiger, 2012).
6.2. Standardization and locus of control
Districts, states, and countries need to decide whether teacher
evaluation and development systems will be designed in a
standardized way based on analytical assessment, or as holistic
examination of instructional practice. Similarly, developers can
design a system to be either centrally or locally controlled. These
decisions typically involve not only technical considerations, but
are immersed in broader policy contexts and influenced by
resource availability, funding requirements, legal regulations, and
various other historical and social conditions and factors
surrounding the teaching force. Decisions regarding standardiza-
tion and locus of control are determined in no small part by the
level of professionalism and human resources available at local and
central levels. While these decisions may not be under direct
control of developers or even individual policymakers, it is
nevertheless important to maintain awareness of the variety of
designs as well as the trade-offs between them. Highly standard-
ized systems may gain centralized ability to set equitable standards
of teaching quality, but can also stifle creativity and capacity
building for locally driven professional development. Systems that
emphasize local control may win local buy-in and ownership but
risk losing control of comparable standards and process quality.
6.3. Reliability and validity
Attaining reliable and valid indicators of teacher practice based
on classroom observation can be challenging and costly, requiring
extensive training, and multiple visits to the classroom by multiple
observers. Observation systems face critical questions about how
many observers will be needed (for each teacher and for the system
overall), how much training will be necessary, and what costs are
to be incurred. The answers to these questions will naturally
present trade-offs in considering the resources available alongside
goals for accuracy and representativeness of the observation
results. Additional observations have direct cost implications, in
terms of training and substituting the time observers would have
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
spent on other activities. As Shavelson et al. (1986) pointed out,
developers need to assess the key relevant sources of error and cost
in classroom observation (e.g. observers, occasions, subject matter)
to understand their influence on the measures of classroom
processes derived. In this context, Generalizability theory (Sha-
velson & Webb, 1991) is reclaiming prominence as a flexible and
powerful tool suited to investigate the properties of measures of
teaching practice and the designs that maximize their reliability
and accuracy within existing organizational constraints (Hill et al.,
2012; Ho & Kane, 2013; Marcoulides, 1993; Newton, 2010;
Praetorius, Pauli, Reusser, Rakozcy, & Klieme, 2014).
It is also important to draw a distinction between evidence of
reliability and validity that generally supports the quality of the
information collected in a psychometric sense, and ongoing
mechanisms in place to ensure the quality of the observation
data collection process—these could include quality control checks,
checks of consistency or drift, and administrative indicators of
implementation among others. In our analyses we found a few
examples of the former, particularly in larger, more established
systems. On the other hand, we found very little evidence of the
latter, but this type of quality control mechanism should be seen as
a pre-requisite for any subsequent formal examination of
psychometric reliability and validity.
Finally, evidence on whether inferences about teaching quality
are in fact valid is yet more complex and costly to produce. Such
evidence relies on a comprehensive validation agenda that
includes different sources of evidence regarding observation
protocol content, factorial structure and relation to other relevant
variables (like student learning progress), to name only the most
crucial. Political commitment in terms of sufficient time and
financial resources to ensure valid inferences is necessary but
difficult to come by, particularly in the pilot stages of system
development. Undoubtedly, the higher the stakes of the decisions
to be taken, the more important validation evidence becomes, but
even formative uses should be based on sound diagnostic
information on teaching quality (American Educational Research
Association, American Psychological Association, & National
Council on Measurement in Education, 2014).
6.4. Appropriate uses of the information collected
Information collected through observation can serve summa-
tive and formative evaluation purposes and present policy
scenarios with different stakes associated for teachers and other
actors. In principle, formatively focused systems address how best
to provide feedback to teachers from the evidence collected
through observation in order to help them improve their practice.
In a summative system, the focus would be on the reliability and
precision of the scores and the most appropriate weight to give
observation scores in the overall assessment of teachers. We did
not identify any systems falling in this extreme group; all systems
incorporate some kind of formative component and aim to use
information to improve teaching—although they can vary consid-
erably in the extent of their formative emphasis, and the specific
ways in which formative mechanisms are implemented.
Using teacher appraisal data for both formative and summative
purposes requires a difficult balance, and a range of methodologi-
cal and policy considerations; summative decisions may lead to
defensiveness of those receiving lower results and thus limit
teachers’ openness to consider evaluation feedback for their
learning and development. While combining formative and
summative uses within the same appraisal system may be the
logical default from a policy perspective, whether and how these
parallel goals can be successfully achieved in practice remains an
open question (Herman & Baker, 2009). Observations targeting
teachers in training are typically formative in nature, but
27
considerations about objectivity and methodological rigor may
need to be incorporated for uses involving certification or tenure.
With in-service teachers, observation is increasingly tied to
external high stakes decisions related to promotion, referral to
development, or monetary incentives; but in-service teacher
observation can also be used within collegial systems for low-
stakes self-initiated professional development purposes.
For formative purposes the focus and nature of the feedback
provided depends on the one hand on the level of detail in the
teaching framework, and on the other hand on the degree of
standardization in the observation system itself. Systems also use
different formats for providing feedback; written reports allow less
detail and interaction than post-observation conferences or other
in-person mechanisms. Ultimately, the utility and consequences of
the feedback given and received will depend on the access to, and
quality of, support mechanisms and opportunities for professional
development in order to address diagnosed weaknesses.
Finally, the appropriate role to give the information collected
through observation within a broader performance appraisal
system also presents complexities that should be considered
carefully. All systems in our sample reflect formally or
informally- a growing consensus in the literature that calls for
combining multiple measures in order to present a more complete
picture of teacher performance. However, this is often interpreted
narrowly to suggest that these measures should be combined to
form a more robust weighted indicator of effectiveness (Mihaly
et al., 2013). In reality, however, there are a number of alternative
models for combining measures. The question is thus not what
weight to give observation scores to create a combined measure to
evaluate teacher effectiveness, but how to use multiple measures
in combination to more effectively appraise teaching and teachers.
Developers and policymakers need to consider the advantages and
disadvantages of compensatory models that rely on the creation of
summary weighted indices and those that consider the informa-
tion separately through decision rules (e.g. conjunctive, disjunctive
models; see Mehrens, 1990).
7. Conclusions
The growing prominence of classroom observation across a
wide variety of teacher evaluation and development systems calls
for much more systematic discussion of critical issues and
decisions related to what is observed? by whom? when? and
how? and equally important, how the information is used for
formative and summative purposes. Our study is ultimately
intended to start a conversation among education researchers
and policymakers around the theoretical and practical common-
alities and distinctions among observation systems, and the
assumptions, goals, and factors that influence how these systems
are designed and implemented on the ground. The trend towards
more comprehensive appraisal of teacher and teaching perfor-
mance continues to gain strength in regional and national systems
worldwide, and with it the prominence of classroom observation.
It is worth noting that a large-scale and highly standardized
approach to classroom observation is far from the norm among
traditionally high performing countries. There is little empirical
evidence to suggest that large-scale standardized observation, and
high-stakes teacher evaluation in general, constitutes a good
vehicle for countries to exert improvements and achieve educa-
tional success (Feuer, 2012), while at the same time there is little
dispute that high quality teaching is the main factor modifiable by
educational policy that predicts students’ educational outcomes
(Nye, Konstantopoulos, & Hedges, 2004). New policy efforts
involving classroom observation are under way among others in
Norway in the form of a national effort aimed at training school
principals as classroom observers; Australia, where national and
28 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
regional systems of observation and evaluation are being devel-
oped; and Mexico, where new legislation established a national
teacher evaluation and incentive system. Adopting a comparative
perspective based on concrete analytic dimensions like the ones
discussed in this paper will continue to be valuable for informing
research and policy debates about the appropriate uses of
classroom observation for assessing and improving teaching in
classrooms around the world.
Appendix A. Supplementary data
Supplementary data associated with this article can be found,
in the online version, at http://dx.doi.org/10.1016/j.
stueduc.2016.03.002.
References
Abelmann, C., & Elmore, R. F. (1999). When accountability knocks, will anyone answer?
CPRE research report No. RR-042. Philadelphia, PA: University of Pennsylvania,
Consortium for Policy Research in Education.
American Educational Research Association, American Psychological Association, &
National Council on Measurement in Education (2014). Standards for educational
and psychological testing. Washington, D.C: American Educational Research
Association.
Australian Institute for Teaching, & School Leadership (AITSL) (2016). National
partnership agreement on improving teacher quality. Australian Institute for
Teaching and School Leadership (AITSL). http://www.aitsl.edu.au/docs/default-
source/default-document-library/
national_partnership_on_improving_teacher_quality.
Australian Institute for Teaching and School Leadership (AITSL) (2012). Australian
teacher performance and development framework. education services Australia,
standing council on school education and early childhood (SCSEEC). Australia:
Carlton South.
Baker, E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L.,
(2010). Problems with the use of student test scores to evaluate teachers Vol. 278,
Washington, D.C: Economic Policy Institute. http://www.epi.org/publication/
bp278/.
Barnett, J. H., Wills, K. C., Hudgens, T. M., & Alexander, J. L. (2015). TAP research
summary spring 2015: examining the evidence and impact of TAP: the system for
teacher and student advancement. National Institute for Excellence in Teaching.
http://www.niet.org/assets/Publications/tapresearch-summary-0615.pdf.
Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012). An
argument approach to observation protocol validity. Educational Assessment, 17
(2-3), 62–87.
Bell, C., Qi, Y., Croft, A., Leusner, D., McCaffrey, D., Gitomer, D., et al. (2014). Improving
observational score quality: challenges in observer thinking. In T. J. Kane, K. A.
Kerr, & R. C. Pianta (Eds.), Designing teacher evaluation systems: new guidance
from the measures of effective teaching project (pp. 50–97).San Francisco, CA:
Jossey-Bass.
Braun, H. I. (2005). Using student progress to evaluate teachers: a primer on value-
added models. Princeton, NJ: Educational Testing Service.
Brookhart, S. M. (2011). Educational assessment knowledge and skills for teachers.
educational measurement. Issues and Practice, 30(1), 3–12.
Brophy, J., & Good, T. L. (1986). Teacher behavior and student achievement. In M.
Wittrock (Ed.), Handbook of research on teaching (pp. 328–375).New York:
Macmillan.
Bruns, B., & Luque, J. (2014). Great teachers: how to raise student achievement in Latin
America and the Caribbean. Washington, D.C: The World Bank. http://dx.doi.org/
10.1596/978-1-4648-0151-8.
Campbell, R. J., Kyriakides, L., Muijs, R. D., & Robinson, W. (2004). Assessing teacher
effectiveness: a differentiated model. London: Routledge Falmer.
Casabianca, J. M., McCaffrey, D. M., Gitomer, D. H., Bell, C. A., Hamre, B., & Pianta, R.
(2013). Effect of observation mode on measures of secondary mathematics
teaching. Educational and Psychological Measurement, 73(5), 757–783.
Chait, R. (2010). Removing chronically ineffective teachers: barriers and opportunities.
Washington, DC: Center for American Progress.
City, E. A., Elmore, R. F., Fiarman, S., & Teitel, L. (2009). Instructional rounds in
education: a network approach to improving teaching and learning. Cambridge,
MA: Harvard Education Press.
Danielson, C. (1996). Enhancing professional practice: a framework for teaching.
Association for Supervision and Curriculum Development (ASCD).
Danielson, C. (2011). Enhancing professional practice: a framework for teaching, 3rd
ed. Association for Supervision and Curriculum Development (ASCD).
Darling-Hammond, L. (2013). Getting teacher evaluation right: what really matters for
effectiveness and improvement. New York: Teachers College Press.
Ellett, C. D., & Garland, J. (1987). Teacher evaluation practices in our largest school
districts: are they measuring up to ‘state-of-the-art' systems? Journal of
Personnel Evaluation in Education, 1(1), 69–92.
Elmore, R. (2008). Leadership as the practice of improvement. Improving School
Leadership, 2, 37–67.
Engel, L. C., Williams, J., & Feuer, M. J. (2012). The global context of practice and
preaching: do high-Scoring countries practice what US discourse preaches? working
paper 2–3. Washington, DC: George Washington University.
Erlich, O., & Shavelson, R. J. (1978). The search for correlations between measures of
teacher behavior and student achievement: measurement Problem,
conceptualization Problem, or both? Journal of Educational Measurement, 15(2),
77–89.
Farr, S. (2010). Teaching as leadership: the highly effective teacher’s guide to closing the
achievement gap. San Francisco, CA: Jossey-Bass.
Feuer, M. (2012). Validity issues in international large-scale assessments: ‘Truth’
and ‘Consequences’. Paper presented at the teachers College/Educational testing
service conference on educational assessment, accountability and equity.
Fullan, M. G. (1992). Successful school improvement: the implementation perspective
and beyond. Bristol, PA: Open University Press.
Gates, B. (2013). Teachers need real feedback. TED conferences, LLC . http://www.ted.
com/talks/bill_gates_teachers_need_real_feedback.html.
Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness: a
research synthesis. Washington, DC: National Comprehensive Center for Teacher
Quality.
Hakel, M. D., Koenig, J. A., & Elliott, S. W. (2008). Assessing accomplished teaching:
advanced-level certification programs. Washington, DC: National Academy Press.
Halverson, R., Kelley, C., & Kimball, S. (2004). Implementing teacher evaluation
systems: how principals make sense of complex artifacts to shape local
instructional practice. In W. K. Hoy, & C. G. Miskel (Eds.), Research and theory in
educational administration: (Vol. 3. pp. Greenwich, CT: Information Age Press.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational
Research, 77(1), 81–112.
Herman, J., & Baker, E. (2009). Assessment policy: making sense of the babel. In G.
Sykes, B. Schneider, & D. Plank (Eds.), Handbook of education policy research (pp.
176–190).New York: Routledge.
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., et al.
(2008). Mathematical knowledge for teaching and the mathematical quality of
instruction: an exploratory study. Cognition and Instruction, 26(4), 430–511.
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not
enough: teacher observation systems and a case for the generalizability study.
Educational Researcher, 41(2), 56–64.
Hill, H. C., & Grossman, P. (2013). Learning from teacher observations: challenges
and opportunities posed by new teacher evaluation systems. Harvard
Educational Review, 83(2), 371–384.
Ho, A. D., & Kane, T. J. (2013). The reliability of classroom observations by school
personnel. Seattle, WA: The Bill & Melinda Gates Foundation.
Ingvarson, L., & Rowe, K. (2008). Conceptualising and evaluating teacher quality:
substantive and methodological issues. Australian Journal of Education, 52(1), 5–
35.
Jacob, B. A., & Lefgren, L. (2005). Principals as agents: subjective performance
measurement in education. Working Paper #11463. Cambridge, MA: National
Bureau of Economic Research.
Joint Committee on Standards for Educational Evaluation (2009). The personnel
evaluation standards, 2nd ed. Thousand Oaks: Corwin press.
Kane, M. T. (2006). Validation, In R. L. Brennan (Ed.), Educational measurement (pp.
17–64).4th ed. Washington, DC: The National Council on Measurement in
Education & The American Council on Education.
Kane, T., & Staiger, D. (2012). Gathering feedback for teaching. Seattle, WA: The Bill
and Melinda Gates Foundation.
Kennedy, M. (2010). Teacher assessment and the quest for teacher quality. San
Francisco, CA: Jossey-Bass.
Kimball, S. M. (2002). Analysis of feedback, enabling conditions and fairness
perceptions of teachers in three school districts with new standards-based
evaluation systems. Journal of Personnel Evaluation in Education, 16(4), 241–268.
Koppich, J. (2009). Toledo: Peer Assistance and Review. Retrieved from: http://
www.smhc-cpre.org/resources/.
Law 19.961 [Ministry of Education]. Sobre Evaluación Docente [About Teacher
Evaluation]. Diario Oficial de la República de Chile. Santiago, Chile.A ugust 14th,
2004.
Leahy, C. (2012). Teacher evaluator training: ensuring quality classroom observers.
Education Commission of the States. http://www.ecs.org/clearinghouse/01/01/
14/10114.pdf.
Levitt, R., Janta, B., & Wegrich, K. (2008). Accountability of teachers: literature review.
Prepared for the general teaching council england. Santa Monica, CA: RAND
Corporation.
Loup, K. S., Garland, J. S., Ellet, C. D., & Rugutt, J. K. (1996). Ten years later: findings
from a replication of a study of teacher evaluation practices in our 100 largest
school districts. Journal of Personnel Evaluation in Education, 10, 203–226.
La evaluación docente en Chile [teacher evaluation in Chile]. In J. Manzi, R. González,
& Y. Sun (Eds.). Santiago, Chile: Facultad de Ciencias Sociales, Escuela de
Psicología, Pontificia Universidad Católica de Chile.
Marcoulides, G. A. (1993). Maximizing power in generalizability studies under
budget constraints. Journal of Educational Statistics, 18(2), 197–206.
Martinez, J. F. (2013). Combining multiple measures of teacher practice and
performance: technical and conceptual considerations for teacher evaluation.
Pensamiento Educativo Latinoamericano, 50(1), 4–20.
McKay, S., & Silva, E. (2015). Improving observer training: the trends and challenges.
issue Brief. Stanford, CA: Carnegie Foundation for the Advancement of Teaching.
http://www.carnegiefoundation.org/resources/publications/improving-
observer-training-the-trends-and-challenges/.
 F. Martinez et al. / Studies in Educational Evaluation 49 (2016) 15–29
Mehrens, W. A. (1990). Combining evaluation data from multiple sources. In J.
Millman, & L. Darling-Hammond (Eds.), The new handbook of teacher evaluation:
assessing elementary and secondary school teachers (pp. 322–334).Newbury Park,
CA: Sage.
Medley, D., & Mitzel, H. (1963). Measuring classroom behavior by systematic
observation. In N. Gage (Ed.), Handbook of research on teaching (pp. 247–328).
Chicago: Rand McNally.
Mihaly, K., McCaffrey, D. F., Staiger, D. O., & Lockwood, J. R. (2013). A composite
estimator of effective teaching. Seattle, WA: The Bill & Melinda Gates Foundation.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: an expanded
sourcebook. Beverly Hills, CA: Sage Publications.
Milanowski, A. (2004). The relationship between teacher performance evaluation
scores and student achievement: evidence from Cincinnati. Peabody Journal of
Education, 79(4), 33–53.
Millman, J., & Darling-Hammond, L. (1990). The new handbook of teacher evaluation:
assessing elementary and secondary school teachers. Thousand Oaks, CA: Corwin
press, Inc..
Ministry of Education (2004). El Marco para la Buena Enseñanza. Santiago, Chile:
Ministry of Education.
Ministry of Education, Culture, Sports, Science, & Technology (2016). Japan country
background report. Ministry of Education, Culture, Sports, Science and
Technology. http://www.oecd.org/edu/school/29163349.pdf.
Mourshed, M., Chijioke, C., & Barber, M. (2010). How the world’s most improved school
systems keep getting better. London: McKinsey.
Müller, S., Pietsch, M., & Bos, W. (2011). Schulinspektion in Deutschland: Eine
Zwischenbilanz aus empirischer Sicht [School inspection in Germany: A summary
based on empirical evidence.]. Münster: Waxmann.
Newton, X. A. (2010). Developing indicators of classroom practice to evaluate the
impact of district mathematics reform initiative: a generalizability analysis.
Studies in Educational Evaluation, 36(1), 1–13.
Nusche, D., et al. (2011). OECD reviews of evaluation and assessment in education:
Norway 2011. OECD reviews of evaluation and assessment in education. Paris,
France: OECD Publishing. http://dx.doi.org/10.1787/9789264117006-en.
Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects?
Educational Evaluation and Policy Analysis, 26, 237–257.
OECD (2009). Creating effective teaching and learning environments: first results from
TALIS. Paris, France: OECD Publishing.
OECD (2013a). Teachers for the 21st century: using evaluation to improve teaching.
Paris, France: OECD Publishing.
OECD (2013b). Synergies for better learning: an international perspective on evaluation
and assessment. OECD reviews of evaluation and assessment in education. Paris,
France: OECD Publishing.
Patton, M. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA:
Sage.
Peterson, K. (2000). Teacher evaluation: a comprehensive guide to new directions and
practices, 2nd ed. Thousand Oaks, CA: Corwin press.
Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2007). Classroom assessment scoring
system—CLASS. Baltimore. MD: Brookes.
Praetorius, A., Pauli, C., Reusser, K., Rakozcy, K., & Klieme, E. (2014). One lesson is all
you need? Stability of Instructional Quality across Lessons. Learning and
Instruction 31, 2–12. http://dx.doi.org/10.1016/j.learninstruc.2013.12.002.
Protheroe, N. (2002). Improving instruction through teacher observation. Principal,
82(1) 48–51.
29
Putnam, R., & Borko, H. (2000). What do new views of knowledge and thinking have
to say about research on teacher learning? Educational Researcher, 29(1), 4–15.
Reform Support Network (2012). Evaluations of teacher effectiveness: state
requirements for classroom observations. U.S. Department of Education. https://
rtt.grads360.org/services/PDCService.svc/GetPDCDocumentFile?fileId=2421.
Richards, J. C. (1991). Towards reflective teaching. The Teacher Trainer., 5(3), 4–8.
Santiago, P., & Benavides, F. (2009). Teacher evaluation: a conceptual framework and
examples of country practices. OECD-Mexico workshop ‘Towards a teacher
evaluation framework in Mexico: international practices, criteria and mechanisms’.
Santiago, P., Benavides, F., Danielson, C., Goe, L., & Nusche, D. (2013). Teacher
evaluation in Chile 2013. OECD reviews of evaluation and assessment in education.
Paris, France: OECD Publishing. http://dx.doi.org/10.1787/9789264172616-en.
Schleicher, A. (2011). Building a high-Quality teaching profession: lessons from
around the world. Educational Studies 1, 74–92. http://dx.doi.org/10.1787/
9789264113046-en.
Sclafani, S., & Lim, E. (2008). Rethinking human capital in education: Singapore as a
model for teacher development. aspen institute education and society program.
Washington, D.C: Aspen Institute. http://files.eric.ed.gov/fulltext/ED512422.
pdf.
Seifert, C. F., Yukl, G., & McDonald, R. A. (2003). Effects of multisource feedback and a
feedback facilitator on the influence behavior of managers toward subordinates.
Journal of Applied Psychology, 88(3), 561–569.
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: a primer. Thousand
Oaks, CA: Sage.
Shavelson, R. J., Webb, N. M., & Burstein, L. (1986). Measurement of teaching, In M.
Wittrock (Ed.), Handbook of research on teaching (pp. 50–91).3rd ed. New York,
NY: Macmillan.
Stallings, J. (1977). Learning to look: a handbook on classroom observation and teaching
models. Belmont, CA: Wadsworth Publishing Company.
Tatto, M. T., Schwille, J., Ingvarson, L., Rowley, G., & Peck, R. (2012). Policy, practice,
and readiness to teach primary and secondary mathematics in 17 countries:
findings from the IEA teacher education and development study in mathematics
(TEDS-M). Amsterdam: IEA.
Taut, S., & Sun, Y. (2014). The development and implementation of a national,
standards-based, multi-method teacher performance assessment system in
Chile. Education Policy Analysis Archives, 22(71), 1–30. http://dx.doi.org/
10.14507/epaa.v22n71.2014.
Taut, S., Santelices, V., Araya, C., & Manzi, J. (2010). Theory underlying a national
teacher evaluation program. Evaluation and Program Planning, 33, 477–489.
http://dx.doi.org/10.1016/j.evalprogplan.2010.01.002.
Taut, S., Santelices, V., & Stecher, B. (2012). Validation of a national teacher
assessment and improvement system. Educational Assessment Journal, 17(4),
163–199. http://dx.doi.org/10.1080/10627197.2012.735913.
Taylor, E., & Tyler, J. (2012). The effect of evaluation on teacher performance.
American Economic Review, 102(7), 3628–3651.
U.S. Department of Education (2009). Race to the top program. Executive summary.
Department of Education. http://www2.ed.gov/programs/racetothetop/
executive-summary.pdf.
Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: our
national failure to acknowledge and act on differences in teacher effectiveness.
Brooklyn, NY: The New Teacher Project.