A mixed-methods analysis of achievement disparities in Guatemalan primary schools

1. Introduction

Often with the assistance of foreign aid and facilitated by the use of alternative models including
multigrade and community managed schools, many governments in poor developing countries
have engaged in campaigns to expand access to marginal isolated communities. In the wake of
rapid expansions, more challenging than building and staffing these schools has been promoting
education quality within them and addressing fundamental issues such as irregular student and
teacher attendance, limited class time dedicated to instruction in core subjects, and other factors
that limit students opportunity to learn (Abadzi, 2006, 2009; DeStefano et al., 2006; Gillies and
Quijada, 2008; Glewwe and Kremer, 2006).

Due to efforts to improve primary access and prohibit the collection of fees, most Guatemalan
rural students currently have access to primary school. While there are reasons to be concerned
about education quality in Guatemala across contexts, there is evidence that high proportions of
students in the poorest communities are failing to attain basic academic skills even after multiple
years of primary schooling as well as pronounced achievement disparities between groups and
regions (Alvarez and Pineda, 2006; Hall and Patrinos, 2006; McEwan and Trowbridge, 2007;
Pallais and Somerville, 2006).

This study uses a variety of data and methods to examine the roots of education quality
disparities in Guatemalan primary schools. The research focuses on the role of factors within the
education system, principally how schools are financed, managed, and supported and how and
why those functions vary across different contexts and across more and less disadvantaged
populations.

Findings from the research are presented in two sections. The first component uses multilevel
regression analysis to examine the magnitude of achievement disparities in Guatemala and to
measure the extent that such disparities can be attributed to different school types or to regional
and contextual differences (e.g. urban versus rural). The next component of the research
examines the structural and contextual roots of low achievement as well as disparities in
achievement by examining how the Guatemalan system of teacher management and support are
operating and how and why those operations appear to vary across contexts.

2. Background

During the post-civil war period in Guatemala there has been a rapid improvement in access to
primary education for rural populations. Participants in the Peace Accords came to a consensus
that basic education should be developed as a means of promoting a shared sense of values and
respect for diversity within a multicultural context and to support equity and economic
modernization (Anderson, 2001).
During the recent period of expansion policymakers in the Guatemalan Ministry of Education
(MINEDUC) have been confronted with the challenge of providing education access in
marginalized communities with limited existing infrastructure and resources. The MINEDUC
has used alternative school models, principally multigrade and community managed schools, to
facilitate rapid rural expansion and compensate for limited government capacity in poor rural
regions. Among the nontraditional school types in Guatemala are multigrade schools.

Multigrade schools are distinguished by having more than one student grade level per classroom,
ranging from two to six. To help to support learning in the heterogeneous context Guatemalan
multigrade schools receive specially designed instructional materials and teacher training
Cabrera et al., 2007; McEwan, 2008).

Along with multigrade schools, principle among the alternative models are the schools that were
created as part of thePrograma Nacional de Autogestion para el Desarrollo Educativo, or
PRONADE schools. These schools were created, supported, and in part, managed by community
members and received monitoring and support from contracted NGOs rather than the
Guatemalan Ministry of Education (MINEDUC).

The model appealed to policymakers at the time because PRONADE schools were faster and
cheaper to put into operation than traditional schools (Gershberg et al., 2009; Rojas et al., 2005).

The schools were also staffed by contracted teachers who were hired by local communities
(Rojas et al., 2005). Program administrators had greater flexibility in hiring teachers than in the
official system. Contracted teachers were also less expensive than teachers in the official system
because they did not receive the same benefits of traditional teachers and did not receive pay
increases based on experience. They also served on one-year contracts and could be replaced by
the local school management committee if they were found to be absent for more than three days
(Rojas et al., 2005).

3. Review of related research

The following provides a brief review of research examining how different factors internal to
education systems have been found to influence education quality in developing countries,
especially in contexts similar to Guatemala. The first component of the review focuses on the
role played by educational inputs. The second component examines how different alternative
education models that have been used to rapidly expand access to students in poor rural
communities might be contributing to educational disparities. More broadly, this component of
the review focuses on how variation in implementation of education has been found to influence
education quality and quality disparities.

3.1. The role of educational inputs

A large body of research set in more and less developed contexts has used the education
production function to examine the connection between different school inputs and
outputs, typically standardized test scores (Hanushek, 1996; Hanushek and Luque, 2003;
Card and Krueger, 1996; Fuller and Clarke, 1994; Glewwe, 2002; Glewwe and Kremer,
2006).. There is evidence that in poor countries, where education resources tend to be
limited and a large proportion of operating expenditures go to salaries, that basic
resources like text books and school facilities can have a large impact on education
quality (Glewwe and Kremer, 2006; Glewwe, 2002; Fuller and Clarke, 1994).

The problem of inadequate instructional time appears to be closely connected to the issue
of high rates of teacher absenteeism (Abadzi, 2006; DeStefano et al., 2006, 2007; Rogers
and Vegas, 2009). Teacher absenteeism has been found to be a particular problem in more
isolated rural schools (Rogers et al., 2004; Chaudhury et al., 2006). Along with
inadequate levels of monitoring another factor within the education system that appears
to impede teacher effectiveness in less developed contexts are inadequate levels of
guidance and support for teachers, especially teachers in rural schools (Carnoy et al.,
2007; Reimers, 2000; Rogers and Vegas, 2009).

Along with examining whether there is disadvantage associated with attending rural or
multigrade schools, this study also examines the association between student
achievement and school poverty and how different school models mediate the effects of
poverty.

3.2. The role of alternative school models

Multigrade and community managed schools are appealing to policymakers in less

developed contexts looking to expand access in rural communities as they can be
established more rapidly and cost less to create and maintain than traditional schools,
primarily because they are staffed by contracted teachers. Policymakers are confronted
with difficult tradeoffs, however, in their decision to construct multigrade schools
(McEwan, 2008). Placing schools in isolated rural communities and reducing the distance
that students have to travel to school is likely to improve access, especially for students
who live a great distance from the nearest primary school, who have limited income to
pay for transportation, or who face dangers in traveling to school. The direct costs of
building and maintaining multigrade schools are relatively low, especially, as is often the
case, if they are staffed by contracted teachers. On the other hand, the experience and
education levels of contracted teachers tend to be lower than in the official system in part
because of low rates of retention and higher demand for placement in more urban
contexts (A valos, 1999; Gershberg et al., 2009).

4. Data and methods

The research uses a mixed-methods sequential explanatory design drawing from
quantitative data and prior research to examine quality disparities and how much those
disparities appear to be attributable to factors within the education system and then case
study research, primary qualitative, to provide insight, context, and work to explain those
patterns (Ivankova et al., 2006; Johnson and Onwuegbuzie, 2004).

4.1. Multilevel regression analysis

The first component presents findings from a multilevel regression analysis of nationally
representative student achievement data. This component of the analysis focuses on
measuring the magnitude of achievement disparities between individual students, school
types, and regions and determining how much achievement disparities appear to be
explained at the different levels of analysis after controlling for student characteristics. It
examines how school types serving disadvantaged populations like multigrade schools
might be contributing to achievement disparities as well as whether there appear to be
systematic differences in education quality between regions in Guatemala. In addition, it
examines the extent that the composition of schools and regions appear to be influencing
achievement disparities. It is meant to serve as background for and inform the next
components of the analysis.

This component of the analysis uses first, third, and sixth grade reading and math test
score data. The data were collected by the Guatemalan Ministry of Education
(MINEDUC) during the 2006 2007 school year as part of their annual assessment of
school quality.

4.2. Case study analysis

The second component of the study draws on findings from the qualitative analysis as
well findings from other administrative data and previous research to explain the
observed patterns and demonstrate how much there appear to be systematic differences in
key quality related resources and practices between school serving different groups and
different school types. The analysis focuses on the MINEDUC as an organizational case
study, and within it, the administrative units charged with monitoring, supervising, and
supporting teachers. In part because of the limited availability of representative data, as
well as the complications involved in examining topics that are difficult to distinguish
from and are deeply embedded in complex contextual conditions this component of the
analysis uses a case study approach to examine how education quality is being supported
(Yin, 1993, p. 78).

The primary source of data for the second component of the analysis are interviews
conducted with school teachers, directors, supervisors, members of department level
administrative offices (direcciones departamentales).
5. Quantitative analysis of achievement disparities in Guatemala

This component of the analysis focuses on measuring differences in achievement levels

between alternative and traditional school models, after controlling, to the extent possible
with the available data, for systematic differences in student background characteristics.
The findings show that there is disadvantage associated with attending multigrade
schools and attending a high poverty school or living in a high poverty department even
after adjusting for student-level factors.

Table 2presents unadjusted and adjusted means of standardized test scores for sixth
graders for multigrade schools and PRONADE schools. As the test scores are
standardized, the figures represent differences in standard deviation units between
traditional and the alternative school models. The first two columns presents unadjusted
mean differences between traditional schools and the two models. The next two columns
presents the school coefficients (differences in mean standardized sixth grade test scores
between the non-traditional schools and traditional schools) adjusted for observable
student characteristics as well as whether a school is located in a rural rather than an
urban area. The fifth and sixth columns show coefficients for school type after adjusting
for the proportion of students in a school that are low SES (proportion of students that are
in the lowest quartile of household goods) and the last two columns adjust for school
composition (centered) as well as department composition.

Although it is difficult to rule out omitted variable bias, or to determine the counterfactual
what the achievement patterns of attending students would be if the model had not been
implemented even after controlling for student level differences, sixth grade students
attending multigrade schools are found to have significantly lower levels of achievement
than students in traditional schools. Findings are similar for first and third graders,
although the magnitudes of differences for first graders are lower and the differences are
not statically significant. Students in PRONADE schools also have lower test scores than
students in traditional schools after controlling for student level factors.

Table 3shows findings from models that include interactions between selected student
characteristics, whether students speak an indigenous language or are in the lowest
quartile of household goods, and the two alternative school type dummy variables.
Observing how the effect of being in a multigrade or PRONADE school vary among
indigenous and poor students, there appears to be some evidence that at least sixth grade
students speaking an indigenous first language attending PRONADE schools have
significantly lower test scores than Spanish native language speaking peers. Similar
negative and relatively strong coefficients are observed for the interaction between
PRONADE and the poverty indicator indicating that a student in the first quartile of
household goods for sixth grade students.

6. Case study analysis examining the roots of educational disadvantage

This component of the analysis explores the mediators of the relationships and patterns
identified in the previous section. It examines, for example, the sources of systematic
disadvantage associated with attending a multigrade school or going to a school in a rural
community. It also examines why, net of student level factors, students living in relatively
poor departments have lower levels of achievement and, within departments, why the
relative poverty levels of schools is also a significant negative predictor of achievement.
To do this, the research focuses on the two figures within the administrative system that
are most closely tied to supporting quality, school directors and supervisors.

6.1. School directors

School directors were found to play a limited role in promoting quality in Guatemalan
primary schools. They also frequently lack the capacity and authority to serve as effective
managers and address issues such as teacher attendance. There was some variation
observed in how directors performed their roles, however, and some signs that principals
can be strong authorities and positive sources of innovation in more favorable contexts.

According to interviewees, one of the key impediments to school directors acting as

leaders within their schools was that they did not receive a higher salary than teachers.
There was therefore, according to interviewees, little incentive for teachers to take on the
position. The director position was in fact commonly viewed as a burden and directors in
the rural context were not commonly respected as school leaders and supporters of
quality but as the teacher with the added responsibility of administrative duties.

Another impediment for rural directors and a probable contributor to inequality in

educational outcomes between urban and rural schools was that along with their
administrative duties, many directors were required to teach one or more grade levels.
This occurred in multigrade schools and in small schools with only one teacher per grade
level. Overall, school directors ability to promote education quality is limited, especially
in cases in which school directors were required to teach. As has been observed in other
contexts, principals act as middle managers, intermediaries between central policymakers
and service providers, who lack the authority and influence in the education system to
have a significant influence on the practices of teachers (Bidwell, 2001).

6.2. School supervisors

As was found to be the case with school directors, supervisors support of education
quality was found to be extremely limited, especially in poor rural schools. As has been
found to be the case in other analysis (Auditora Social de la Gran Camapanadela
Educacion 2006 cited in ICEFI 2007), supervisors were found to visit rural school
infrequently. The lack of visits to rural schools and other evidence that supervisors are not
supporting education quality in rural schools is explained by a combination of factors
including supervisors administrative workload, their perceived role as administrators,
and lack of transportation and other resources (Wise and Leal, 2007).

Another factor impeding supervisors ability to monitor quality, and again an issue that
appears to be a more severe impediment for rural supervisors was the number of schools
that they were assigned. Analysis of a nationally representative school survey shows a
great deal of variation in the number of schools assigned per supervisor.

The inequality in the distribution of supervisors again reflects a pattern of the schools
with the highest level of need receiving the least support. Supervisors with high numbers
of rural schools under their supervision were found to limit their functions to addressing
the most serious problems that were brought to their attention along with performing the
necessary administrative work. Many of the rural supervisors that were interviewed as a
part of the study were assigned more than 100 schools. Urban supervisors in the case
study districts, on the other hand, tended to have between 20 and 30 schools under their
supervision. Unlike the rural supervisors, they reported visiting schools regularly,
performing quality related activities like reviewing and giving feedback on teacher
planning and personally observing schools resource needs. It is important to note that
rural schools typically had fewer teachers per school so there may have been less
variation in the number of teachers assigned per supervisor. On the other hand, the fact
that rural schools are frequently located in more isolated areas that are difficult to access
increases the time and funding rural supervisors dedicate to transportation. Rural
supervisors discussed having inadequate means of transportation and inadequate funding
to support transportation. While many of the rural supervisors had motorcycles, they
stated that they lacked funds for maintaining the motorcycles or to pay for gasoline to get
to the rural schools.

7. Conclusion

The research suggests that there are relatively strong levels of disadvantage connected to
attending rural schools in poor communities and that this disadvantage can be linked to
both contextual and structural elements. Both the multigrade and PRONADE models
have been shown to be a low cost means to provide education to disperse rural
populations, but it is important to consider how quality can be better promoted in those
setting. Consistent with other research on non-traditional schools for rural populations
(McEwan, 2008), the findings suggest that there are important tradeoffs to consider in
making policy decisions around improving access for rural students and promoting
educational quality. The improved accountability that is facilitated by greater community
involvement in the PRONADE schools might not have been enough to compensate for
the fact that the schools were staffed by less experienced, qualified, and capable teachers
and the high levels of teacher turnover in those schools (Marshall, 2009).

The fact that workloads tend to be higher, logistical challenges more severe, and
bureaucratic institutions tend to be weakest in rural areas appeared to contribute to lower
levels of support for education quality in rural schools. The effectiveness of directors and
supervisors, especially director, also appears to be impeded by a lack of vision and
strategy among policymakers regarding how they could take a more active role in
promoting quality (Borden, 2002). Inadequate authority, capacity, and resources among
school directors and supervisors are important in explaining the weak links observed
between central policies and implementation, the lack of clarity around basic rules, and
low levels of accountability. There is a need to explore ways to ensure that directors and
supervisors have the skills, knowledge, resources, and incentives to play a stronger role in
supporting quality. The research also touches on the issue of local control and its inherent
strengths and limitations. Although it is clear that supervisors are confronted with a
number of obstacles impeding their ability to promote quality, many interviewees
identified the supervisors themselves and their lack of professionalism and commitment
as being among the central reasons that accountability was not functioning in primary
schools. Explanations for the pattern tended to range from characterizing the supervisors
as being irresponsible or too close to the people they were charged with monitoring,
politicization of the position (e.g. hiring of individuals based on their political affiliation
rather than merit), and inadequate accountability for the supervisors. On the other hand,
findings from this component of the analysis also provide a sense of some of the benefits
of local monitoring. While supervisors roles in promoting local accountability might be
impeded by their connection to those who they are supervising their knowledge of the
local conditions might assist them in addressing conflicts and challenges in their districts.
As other have noted, there are a number of challenges associated with local control in a
context in which local administration lacks administrative capacity and in which
bureaucratic institutions are weak (Bardhan and Mookherjee, 2005; Grindle, 2004; Goetz
and Jenkins, 2001). Being from the community in which they worked appeared at times
to serve both as a strength and a limitation to the effectiveness of supervisors. The finding
points to a need to balance locally based managers, who have the knowledge and
connections to respond to local challenges, with centralized impersonal accountability
mechanisms to identify irregularities and abuses in provision and management. The
recent creation of a centralized school monitoring program appears to be an important
and necessary step in creating such a system.