EU UNICEF WASH Project Evaluation Final Report

End of Project Evaluation
of project
Improving Access to Water, Sanitation and Hygiene (WASH)

in Rural School and Communities through Capacity Development
Timor-Leste
FINAL REPORT
Prepared by: Martin Rall

July 2015
Acknowledgements
Almost 200 people gave of their time and shared their knowledge, experience and insights to
make this evaluation possible from community members and their leaders to local government
officials, project partners and project staff to senior national government officials and
representatives of international organisations and for this we offer our sincere gratitude.
Special thanks are due to those who helped organise and facilitate the whole exercise, particularly
the field work, namely:
for their general guidance, support and facilitation in UNICEF:
Ramesh Bhusal: Chief of WASH Section
Toshiko Takahashi: Chief of Planning, M&E and Social Policy Section
Desiree Jongsma: Country Representative
Rene van Dongen: Deputy Country Representative
for setting up the field visits and accompanying the consultant:
Rodolfo Pereira: WASH Officer, UNICEF
Olga Corbafo: Consultant in Oecusse, UNICEF
Staff of the NGO partners ETADeP, HIM and AMAR
Staff of SAS in Viqueque
for assisting with translation in interviews in Dili and in the districts:
Rodolfo Pereira: WASH Officer, UNICEF
Olga Corbafo, Vicente Lopes, Indra Monemnasi and Eusebio Martins (UNICEF)
and for their reliable and professional logistical support: the administration staff and drivers of
UNICEF.
We would also like to acknowledge the quality assurance guidance and input provided by UNICEF
regional and headquarters staff, namely:
Chander Badloe: Regional WASH Advisor, UNICEF East Asia and Pacific Regional Office
Jeremie Toubkiss: WASH Evaluation Specialist, UNICEF HQ
Riccardo Polastro: Regional Evaluation Advisor, UNICEF East Asia and Pacific Regional
Office
Finally, special thanks are due to the following senior government partners for their political
support and contribution to the evaluation:
Mr. Joao Pereira Jeronimo, DG National Directorate Water & Sanitation Services, MoPW
Mr. Joao Pidade, Director National Basic Sanitation Services, MoPW.
Mr. Gustavo Da Cruz, Director National Water Supply Services, MoPW,
Mr. Martinus Nahak, Programme and SIB Officer, DNSA, SEASU, MoPW
Mrs. Tomasia De Sousa, Environmental Health Department, MoH
Mr Miguel GODINHO, Chief of Department, School Social Action, MoE
List of Acronyms, Terms and Abbreviations

Aldeia
Variously translated as community, village or sub-village or hamlet (see also

Bairro and Suco/Suku, below)
Bairro
Sub-aldeia, or neighbourhood
BeSI
Bee Saneamento no ljiene (Water, Sanitation and Hygiene in Tetum

Language)
BESIK
CAP
AusAID (DFAT)-funded Rural Water Supply and Sanitation Programme

Community Action Planning
DGAS
Directorate General for Water and Sanitation, Ministry of Public Works
DNSA
National Directorate for Water Services
CLTS
Community Lead Total Sanitation
DTO
District Technical Officer
EU
European Union
FGD
Focus group discussion
GMF
Grupu Maneja Fasilidade (Water Facility Management Group)
HH
Households
HWF
Hand washing facility
HWWS
JMP
Hand washing with soap

(WHO/UNICEF) Joint Monitoring Programme
KAP
Knowledge Attitudes and Practices
Logframe
Logical framework (of project)
O&M
OD
Operation and Maintenance

Open defecation
ODF
Open defecation free
MDG
Millennium Development Goal
NGO
PAKSI
Non Governmental Organisation
PSF
Planu Asaun Komunidade Saneamento no Ijiene (Community Action Plan

for Sanitation and Hygiene)
Promotor Saude Familiar (community-based health promoter)
SAS
District Water and Sanitation Service Ministry of Public Works
SDF
Sub-District Facilitator
SIBS
Suco/Suku
Water and Sanitation Information System

Lowest administrative unit of Timor-Leste. Normally translated as village,
comprising several sub-villages (aldeias).
TLDHS
UNEG
Timor-Leste Demographic and Health Survey

United Nations Evaluation Group
UNICEF
United Nations Childrens Fund
WASH
Water, Sanitation and Hygiene
WHO
World Health Organisation
ii
Table of Contents
Acknowledgements ................................................................................................................... i
List of Acronyms, Terms and Abbreviations .......................................................................... ii
Executive Summary ................................................................................................................ vi
1.
Introduction ....................................................................................................................... 1
2.
Description of Project ....................................................................................................... 1
3.
Purpose and Scope of the Evaluation ............................................................................. 6
4.
Evaluation Methodology ................................................................................................... 6

4.1
Evaluation Questions ................................................................................................... 7
4.1.1
5.
6.
M&E Indicators ...................................................................................................... 7
4.2
Sources of information and methods of data collection................................................. 8
4.3
Sampling ...................................................................................................................... 8
4.4
Endline Survey and KAP Study Report ...................................................................... 10
4.4.1
Methodology ....................................................................................................... 10
4.4.2
Sampling Design and Sample Size ..................................................................... 11
4.4.3
Field work............................................................................................................ 12
4.4.4
Findings .............................................................................................................. 12
4.5
Limitations to the Evaluation and Mitigation Approaches ............................................ 12
4.6
Ethical considerations ................................................................................................ 13
Activities Undertaken ...................................................................................................... 13

5.1
Phase 1: Preparation and desk review ....................................................................... 13
5.2
Phase 2: In-country field work .................................................................................... 13
Findings ........................................................................................................................... 15
6.1
Relevance .................................................................................................................. 15
6.1.1
Relevance for target groups ................................................................................ 15
6.1.2
Alignment with government priorities ................................................................... 15
6.1.3
Alignment with funders country strategies and objectives ................................... 15
6.1.4
Quality of project design ...................................................................................... 16
6.2
Efficiency.................................................................................................................... 16
6.2.1
Implementation against work plans ..................................................................... 17
6.2.2
Quality of project management............................................................................ 18
6.2.3
Quality of products .............................................................................................. 19
6.2.4
Cost effectiveness ............................................................................................... 19
6.2.5
Partnerships with implementing NGOs ................................................................ 20
6.2.6
Coordination with other programmes................................................................... 21
6.2.7
Risk management ............................................................................................... 21
6.3
Effectiveness .............................................................................................................. 21
6.3.1
Achievement of outputs and Results ................................................................... 21
6.3.2
Achievement of Specific Objective ...................................................................... 24

iii
6.3.3
Contribution to MDGs .......................................................................................... 25
6.3.4
Beneficiary perception of benefits from project .................................................... 26
6.3.5
Capacity building ................................................................................................. 26
6.3.6
Compliance with norms and standards ................................................................ 27
6.3.7
Risk management ............................................................................................... 27
6.4
6.4.1
Achievement of Overall Objectives ...................................................................... 28
6.4.2
Risk management ............................................................................................... 28
6.5
7.
8.
9.
Impact ........................................................................................................................ 28
Sustainability .............................................................................................................. 28
6.5.1
Community water supply systems ....................................................................... 29
6.5.2
Household sanitation ........................................................................................... 31
6.5.3
Sanitation and hygiene related knowledge and behaviour change ...................... 33
6.5.4
School WASH ..................................................................................................... 34
6.6
Equity ......................................................................................................................... 34
6.7
Implications for the UNICEF Country Programme ...................................................... 35
Summary of Conclusions ............................................................................................... 36

7.1
General ...................................................................................................................... 36
7.2
Community water supply ............................................................................................ 36
7.3
Household sanitation and hygiene.............................................................................. 37
7.4
School WASH ............................................................................................................ 37
Lessons learned .............................................................................................................. 38

8.1
General ...................................................................................................................... 38
8.2
Community water supply ............................................................................................ 39
8.3
Household sanitation and hygiene.............................................................................. 39
8.4
School WASH ............................................................................................................ 40
Recommendations .......................................................................................................... 41
Appendices
Appendix 1:
Terms of Reference of Evaluation
Appendix 2:
Logical Framework of Project
Appendix 3:
List of target aldeias and interventions in each
Appendix 4:
List of schools targeted for WASH interventions
Appendix 5:
Evaluation Questions and Data Collection Methodology
Appendix 6:
Work Programme
Appendix 7:
Documents Reviewed
Appendix 8:
List of meetings and persons interviewed
Appendix 9:
Data collection instruments
Appendix 10: Data base of information gathered (separate Excel file)

iv
Appendix 11: Curriculum Vitae of Evaluator

Appendix 12: Photographs (separate file)
List of Tables
No.
Title
Page
Project Summary Sheet
Project target communities by district/sub-district and NGO partners
Summary of Field Work Schedule and Itinerary
14
Summary of activities undertaken
14
Achievement of outputs of Result 2: water supply
23
Additional selected indicators of project outcomes
24
Functionality of water supply systems and GMFs visited during final evaluation
30
Trends in open defecation in target areas
32
Sustainability of hygiene related knowledge and behaviour change
34
List of Figures
No.
Title
Map of Timor-Leste showing target districts and sub-districts of project
Page
5
Executive Summary
Introduction
This is the final evaluation report of the project Improving Access to Water, Sanitation and
Hygiene (WASH) in Rural School and Communities through Capacity Development. Jointly
funded by the European Union (EU) and UNICEF and implemented by UNICEF, the project was
implemented in 10 sub-districts in 5 districts1 (Aileu, Ermera, Manatuto, Oecusse and Viqueque)
in Timor-Leste, over 45 months between June 2011 and March 2015. The project originally
targeted 55 communities (aldeias2) but by the time it was completed had carried out activities in
63 aldeias in the targeted sub-districts. It was implemented together with government ministries
(public works, health, education) and national NGO partners.
The purpose of the project was to assist communities in the target areas to access clean water
and improved sanitation facilities and improve their hygiene knowledge and practices, in order to
achieve better health outcomes for children and adults and enable children to participate better
in primary education. The project was designed with three main components (expected results),
namely: i) Institutional strengthening; ii) Community and schools water supply; and iii) Household
and schools sanitation and improvement in knowledge and attitudes relating to hygiene and
sanitation, using the CLTS (community-led total sanitation) approach.
The project was implemented in response to a number of challenges faced by Timor-Leste, in
spite of progress made since the restoration of independence in 2002. When it was designed, the
under-five mortality was 64 per 1000 live births vis a vis its MDG target of 96, and the primary net
enrolment rate (grade 1-6) was 93.6%, against the MDG target of 1003. Only 61% of the rural
population were using improved water sources and only 27% improved sanitation facilities4. Poor
access to water, sanitation and hygiene was one of the main causes of diarrheal diseases in
under-five children, also contributing to the very high rates of under-five malnutrition (58.1%
stunting and 18.6% wasting5). There were also major concerns regarding the WASH situation in
schools. About 54% of primary schools did not have access to improved water supply, 35% had
no latrines in the compound and only 12% had a regular supply of soap6, all indicating the lack of
a basic enabling environment for school children to practice safe hygiene behaviour.
The overall objectives of this evaluation were to strengthen UNICEFs accountability to its donors
and key stakeholders, including beneficiaries, and to learn from the experience of this intervention
to inform future WASH policy and programming. The evaluation had to assess compliance with
contractual obligations as well as adherence to UNICEFs equity focus and the potential
contribution of the project to the sector and to UNICEFs institutional learning agenda. The
intended audience of this report thus includes UNICEF, the EU, the government and NGO project
partners, government decision makers and other organisations supporting the sector.
The methodology applied in the evaluation was based on that defined in the terms of reference,
combining qualitative and quantitative methods to ensure that as full a range of data sources as
possible was consulted. These included documents produced by the project and information
provided by the UNICEF project team, three of the five partner NGOs, the implementing
government partners at national level, in four districts and eight sub-districts, beneficiaries in 10
aldeias in seven sub-districts, and four other international organisations supporting the sector.
The documents reviewed (24 in total, including 14 produced by the project) included the recent
endline KAP (knowledge, attitudes and practices) survey, which was a major source of data. Forty
1
Now called Administrative Posts and Municipalities: see page 1 for an explanation on nomenclature.
Variously translated as community, sub-village or hamlet
3 Demographic and Health Survey, Ministry of Health, 2009-10
4 Joint Monitoring Programme, WHO/UNICEF, MDG Progress on Sanitation and Drinking Water, 2010
5 Demographic and Health Survey, Ministry of Health, 2009-10
6 National Stocktake of Facilities, and Equipment in Basic Education Schools in Timor-Leste, Ministry of
Education, 2011
2
vi
semi-structured interviews were conducted with key informants as well as 13 focus group
discussions with GMFs (water committees). Field observation included 13 aldeias and 10 schools,
while survey/interviews were conducted with 37 beneficiary households in 10 aldeias.
The sampling plan was designed to allow for capturing a diversity of situations and points of views
rather than anecdotal evidence, hence the large sample of sub-districts visited (8 out of 10),
encompassing 13 aldeias (30% of those that benefited from all project components) and 10
schools (36%). Whenever possible, the various data sources or collection methods were
combined, analysed and compared in an objective manner, acknowledging possible bias and
other methodological limitations, in order to build a comprehensive, credible and nuanced picture
of the project.
Findings and Conclusions

General
The project was appropriately conceptualised and designed for the country context and stage of
development of the sector, was well planned and was mostly well executed. It made use of a nocost time extension to complete the school WASH component while using unspent funds to
surpass its targets for water and sanitation coverage and the number of beneficiaries, in the other
components, and to consolidate operation and maintenance arrangements.
The MDG target for increased water coverage was achieved, being raised from 45,5% to 90,3%
in the target communities, benefiting over 18.000 people (11% more than the original target). 84%
of the communities in which CLTS (community led total sanitation) was implemented were
declared ODF (open defecation free), thus providing a sanitary environment for over 20.000
people. By the end of the project just over 20% of households, on average, had reverted to OD
(Open defecation), but in the majority of aldeias the figure was less than 10%. Assuming the
households that were still using acceptable facilities in all five districts (87%) were all using
facilities that meet the JMP (WHO-UNICEF Joint Monitoring Programme) criteria for improved
sanitation, the MDG target of 55% was achieved in 70% of aldeias and in six of the ten subdistricts. Through the WASH interventions in 28 schools (22% more than planned), more than
5.000 children benefited from improved access to water, sanitation and hygiene information.
In relation to promoting equity, the project promoted greater representation of women on GMFs
(community water groups), with some success. Provision was made to improve the inclusion of
physically disabled persons through ensuring wheelchair accessibility to school latrines and water
system standpipes. In terms of the Overall Objectives of the project, improved school attendance
was achieved, with more impact noted for girls than boys.
In terms of the projects achievements in institutional strengthening and capacity building, it was
not able to clearly define the expected results up-front and used indicators that measure inputs
rather than outcomes. In practice the effect has to be evaluated in terms of improved performance.
In relation to the government partners, while good progress has been made in creating an
enabling policy environment, the project partners at district and sub-district levels are not yet able
to provide effective leadership without project support. There is inadequate funding and other
resources for DGAS (Water and Sanitation Directorate of Ministry of Public Works) and the
Ministry of Health to scale up water supply and CLTS programmes, and to ensure sustainability,
particularly of CLTS.
At community level the project made a significant difference to community capacity to operate
and maintain their water systems, with good prospects for sustainability, although government
capacity to provide the necessary support is questionable.
In comparison with similar interventions elsewhere, the project was cost effective. The average
cost per beneficiary was approximately 114 Euros. The proportion of project implementation costs
and overheads (UNICEFs total costs) to total expenditure was around 22%, also a cost effective
ratio.
vii
In general the approach of implementing in partnership with local NGOs who in turn partnered
with target communities organised by them, achieved positive results in this project. However, in
terms of the quality of the relationship between the NGOs and UNICEF as partners, the NGOs
perceive the relationship to be purely one of sub-contractors in which they are not genuinely
involved in joint decision making. It seems, however, that this is more likely the result of the
inevitably unequal power relations and their reluctance to express concerns more forthrightly.
Community water supply
The project was very effective in this component and amply exceeded its output targets in all
aspects. In addition, 24 point sources (protected springs and shallow wells), not originally
anticipated, were built where gravity-fed piped systems were not feasible. The 43 GMFs formed
also received additional refresher training in O&M (operation and maintenance) after completion
of their systems, and GMFs were formed per aldeia for some of the point sources. The objective
of improving water coverage was achieved, including for schools, if allowance is made for those
schools that had not been connected yet but would be within a few months.
The quality of the infrastructure built, of community participation, of documents produced and of
training provided was generally of a very high quality. An important exception was the water
system in Bahalara-Uain, Viqueque district, which had to be completed by SAS (District Water
and Sanitation Service, Ministry of Public Works) due to the poor performance of the partner NGO
which was unable to resolve the unusually difficult social and institutional problems at community
level. Unless an appropriately funded and facilitated rehabilitation intervention is undertaken on
this system, it will not be sustainable.
The effectiveness of GMFs was found to be good in most cases, with very specific problems as
the causes of dysfunctionality, namely poor leadership. Almost all the GMFs visited showed a
high level of ownership of their systems and adequate knowledge of their roles and
responsibilities. The problems noted should be able to be resolved by Sub-district Facilitators
(SDFs) but it is clear that in general they are under-funded and/or inadequately supervised in
most districts, constituting the weak link in the GMF system. This is an area that needs ongoing
attention and support.
Household sanitation and hygiene
The high level of success in achieving ODF certification, mentioned above, indicates an excellent
level of effectiveness of this element of the CLTS approach and bodes well for the future of the
strategy in Timor-Leste. There was also a significant spin-off effect in terms of an increase in the
number of permanent latrines, across all districts. At baseline 33% of households were using
these and by the end of the project the figure was 45%. Families built permanent latrines (pour
flush, in almost all cases) without any material incentive from the project, due to the messages
received and to the improved access to water.
With regard to the sustainability of latrine usage, however, it was found to be very variable.
Reversion to OD was between 42% and 53% in 30% of aldeias declared ODF (including all of
those in Manatuto), but 10% or less in 62% of them (all of which are situated in Aileu and Ermera).
This weakness will not be resolved until the health system is able to provide ongoing, frequent
follow-up at household level to sustain behaviour changes.
In Viqueque none of the target communities achieved ODF status. Several possible explanations
have been suggested, including the impact of other programmes in the district offering subsidies.
However, it was concluded that this was not a significant factor. In other districts the project had
to contend with the same challenge and managed to do so effectively. It would seem that the
weakness of the NGO was the main problem, as noted above.
viii
The evaluation confirmed the importance of the work being done to finalise a policy on incentives
to assist the most vulnerable, to be applied once communities achieve ODF status.
School WASH
Due to the good quality of design and construction and the relative newness of the facilities, their
level of functionality was high, the main problem being pending water connections or dysfunctional
water systems.
The project appears to have done everything possible to ensure maximum sustainability of the
facilities, to compensate as much as possible for the generally acknowledged shortcomings of
the Ministry of Educations facility maintenance system. The endline survey found that all but one
school visited had a PTA with specific WASH responsibilities and formal relationship with the
GMF of the water system supplying the school. The final evaluation also found that most GMFs
also take responsibility for maintenance of the water facilities at the schools.
Lessons learned
The model of institutional arrangements that was used to implement the project was based on a
partnership with government that involved sharing the responsibilities for implementation of
activities between UNICEF (and its NGO partners) and different government departments, both
at national and district levels, while UNICEF retained the responsibility for accountability to
donors. This model is appropriate and necessary to acknowledge the responsibility of government
for delivery of basic social services and to help build capacity to do so.
The model of implementing with local NGOs was also validated as one that is replicable generally
in the sector. Their comparative advantage in managing community based construction of
services that have to be managed by communities afterwards, was reflected in the high level of
ownership and effectiveness of GMFs. The NGO model is also cost effective, in relation to
commercial contractors. This is especially the case for very small jobs in remote locations which
contractors are reluctant to take on at an affordable price.
The CLTS approach worked well in this project in most districts and in all but two sub-districts,
thanks to a large extent to the intensive, repeated training of trainers and facilitators. Much has
been and continues to be learned by UNICEF and other stakeholders in the sector about applying
the methodology, the key challenge being sustainability, i.e. reversion to OD. The challenge of
sustainability will not be solved unless and until the health system is strengthened to provide
ongoing monitoring and promotion at household level. For this much more funding is needed.
Funding is also needed to scale up CLTS.
The main obstacle to moving up the sanitation ladder and enhancing sustainability is the cost of
materials and transport. The work being done currently to agree on an incentive policy to be
applied for the most vulnerable needs to be finalised and the policy applied.
Recommendations
For UNICEF and its future projects:
1.
2.
3.
While the implementation model used in the project of partnership between UNICEF and
government is sound, it could be further strengthened with a clearer definition of
governments responsibilities and tasks, especially those not funded by the project,
documented in joint work plans, with effective joint monitoring and evaluation.
In order to strengthen and enhance the sustainability of the generally effective model of
implementing in partnership with local NGOs, more responsibility should be assumed for
enhancing the organisational capacity of the NGOs.
Refresher training of NGO, UNICEF and SAS technical staff is needed on monitoring of
construction to ensure attention to detail for sustainability, with the use of checklists.
ix
For government:
4.
5.
6.
7.
8.
DGAS, Health, and Education: Accelerate the process of taking over more responsibilities as
a service provider and funder, with the donor as a supporting partner, increasing budgets,
especially for scaling up CLTS and supporting the O&M of water and sanitation in both
communities and schools. The challenge of sustainability of CLTS will not be solved unless
and until the health system is strengthened to provide ongoing monitoring and promotion at
household level.
DGAS and Health: Undertake a review of experience to date in implementing CLTS to obtain
greater understanding of factors at play in the successes and shortcomings.
DGAS: Include in the CAP (Community Action Planning) process a post-construction
mentorship period to enable SDFs to monitor GMF performance, provide further on-the-job
training and solve problems until the end of the project (with donor support in the case of
donor funded projects).
DGAS: Lobby for finalisation and approval of i) the latrine construction incentive policy for the
most vulnerable once a community has been declared ODF; and ii) the BeSI7 Committee
Guidelines and respective operating budget.
Education: Approve the School WASH Guidelines and include in the training package a
module on technical maintenance of facilities, prioritising the setting up of a functional system
that includes GMFs. DGAS: Include as a standard responsibility of GMFs the maintenance
of school water infrastructure.
Bee Saneamento no ljiene (Water, Sanitation and Hygiene in Tetum Language)
1.
Introduction
This is the final evaluation report of the project Improving Access to Water, Sanitation and
Hygiene (WASH) in Rural School and Communities through Capacity Development. Jointly
funded by the European Union (EU) and UNICEF and implemented by UNICEF, the project ended
in March 2015, and in terms of the funding agreement requires an end-of-project evaluation to be
carried out after completion. Consequently the services of a consultant with the requisite
competencies and experience were procured by UNICEF. A single consultant was considered to
be sufficient, rather than a team with more members, due to the fact that an endline survey and
knowledge, attitudes and practices (KAP) study had already been carried out a few months before
the end of the project, as a key source of information for the evaluation.
This report has been written by the consultant in accordance with the terms of reference of the
evaluation (see Appendix 1). After presenting some brief background information (sections 1, 2
and 3), the report describes the methodology employed, with the respective tools (section 4), and
the activities undertaken (section 5). The detailed findings, including those of the endline survey
and KAP study, are then presented and analysed (section 6), and the key conclusions described
(section 7). Finally, the lessons learned and recommendations are presented (sections 8 and 9
respectively).
Note on terminology
The government of Timor-Leste has recently made some changes to the terms applied to the
different administrative divisions in the country. In relation to those and the related terms that are
used in this report, the following are the changes that have been made:
Previous term/name
New term/name
District
Municipality (Municipio)
Sub-district
Administrative Post (Posto Administrativo)
District Administrator
Municipal Manager (Gestor Municipal)
Sub-district Administrator
Post Administrator (Administrador de Posto)
Sub-district Facilitator (SDF)
Administrative Post Facilitator (FAP Facilitador

Posto Administrativo)
In order to maintain consistency between the terms used in this report and those used in all other
project documents to date, to facilitate comprehension, the previous terminology will be used.
2.
Description of Project
Country context
Timor-Leste is the second youngest nation in the world. When its independence was restored in
2002, the country was in a state of ruins with shattered infrastructure and a stagnant economy,
and it has undergone numerous political, security and development upheavals since then.
However, steady progress has been made in peace and democracy with two successful
parliamentary elections in 2008 and 2012. The population of Timor-Leste is predominantly young,
with nearly half of the population under 18 years. Along with peace and democracy, the situation
of children and women has gradually improved, but a number of challenges persist.
When the project was conceived, the rate of under-five child mortality was 64 per 1,000 live births,
while more than one in ten Timorese children were acutely malnourished and almost 50 per cent
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
suffered from chronic malnutrition8. Access to an improved water source was found to be 69% in
20089, up from 52% in 2000, but with a significant disparity between urban areas (86%) and rural
areas (63%). The same JMP data showed that only 50% of Timorese had access to improved
sanitation facilities (up from 32% in 2000), with 76% of urban dwellers having access, compared
with only 40% of rural dwellers. The rate of open defecation in rural areas was 52% in 2008
(unchanged since 2000). Poor access to water, sanitation and hygiene was one of the main
causes of diseases in children, also contributing to the very high rates of malnutrition.
In the targeted rural areas in the five project districts, the major constraints faced by the children,
women and other vulnerable groups in regard to water, sanitation and hygiene can be
summarised as follows: 1) Low access to improved water sources and low capacity to manage
water facilities; 2) Low sanitation coverage and low access to information regarding sanitation
options; 3) Widespread use of unsafe hygiene practices and unsanitary conditions in schools and
communities, and; 4) Low capacity of WASH authorities, health professionals, teachers and
family health promoters.
It was found in a 2009 survey of two rural districts that 59% of schools had fully or partly
functioning water systems and 55% had fully or partly functioning sanitation systems10. The same
survey found that only 30% of community water systems were functioning one year after
completion. Other surveys also found very high failure rates within a few years of installation, with
contributing factors found to be: management failure on the part of communities, lack of spare
parts and skills and poor initial design. The national government had very little capacity to support
rural water supply and sanitation, with each of the 13 districts having only one rural water and
sanitation official assigned. Decentralisation of government services was very low, with no legal
frameworks in place to allow ministries to devolve control of resources below the national level.
Project Intervention Logic (see Logframe in Appendix 2 for details)
In order to address the issues just described, the project focused on achieving the short term
outcomes of assisting communities to access clean water and improved sanitation facilities and
on improving their hygiene knowledge and practices. In terms of the MDGs, the purpose was to
reduce the proportion of people without sustainable access to safe drinking water and basic
sanitation to work toward the achievement of Millennium Development Goal (MDG) 7.
The overall objectives of the project, in terms of the intended long term outcomes and impact,
were defined as:
Children, women and men in target areas will achieve better health outcomes
Children in target areas are better able to participate in primary education
The Project Purpose, or Specific Objective, was defined as:

National and district WASH personnel and community groups are better able to sustain
equitable access to, and use of, improved drinking water and basic sanitation, and
adoption of improved hygiene practices.
The project was organised into three components, with the respective Expected Results defined
as follows:
1. District Authorities, particularly the Water and Sanitation and Hygiene (BeSI) Committees
are better able to lead and support WASH activities in the five project districts, supported
by national authorities, policies and guidelines.
2. Target aldeais (sub-villages) and primary schools have improved water sources and are
able to manage and maintain their water systems.
8
Demographic and Health Survey, Ministry of Health, 2009-2010

Joint Monitoring Programme, WHO/UNICEF, MDG Progress on Sanitation and Drinking Water, 2010
10 Rural Water Supply and Sanitation Program survey of districts of Aileu and Lautem, 2009
9
3. Target aldeais and primary schools have improved sanitation and demonstrate
measureable improvement in knowledge and attitude relating to hygiene and sanitation.
The strategy for achieving Result 3 at community level was through the use of the CLTS
(community-led total sanitation) approach to eliminate open defecation (OD) without providing
any materials subsidy for latrine construction.
Based on the logframe, the specific Theory of Change guiding the evaluation can be
conceptualised as follows:
IF district authorities, particularly the Water, Sanitation and Hygiene
(BeSI) Committees are better able to lead and support WASH activities in
the five project districts, supported by national authorities, policies and
guidelines; and target aldeais (sub-villages) and primary schools have
improved water sources and are able to manage and maintain their water
systems; and target aldeais and primary schools have improved sanitation
and demonstrate measureable improvement in knowledge and attitude
relating to hygiene and sanitation;
THEN national and district WASH personnel and community groups will
be better able to sustain equitable access to, and use of, improved
drinking water and basic sanitation, and adoption of improved hygiene
practices.
Targeted Project Beneficiaries:

a.
b.
c.
d.
16,400 community members in 34 communities with access to improved water sources

4,600 school children provided water and sanitation facilities
34 communities that are declared open defecation free (ODF)
117,000 population in the sub-districts reached with hygiene promotion.
The project (see Table 1, below, for a summary of key project data) was implemented in 10 subdistricts in 5 districts (two in each) over 45 months between June 2011 and March 2015. The
original project duration was planned for 36 months but received a nine months extension in June
2014.
Table 1: Project Summary Sheet

Project Name
Improving Access to Water, Sanitation and Hygiene (WASH) in Rural

Schools and Communities through Capacity Development
EU Reference No.
FED 2011/266-9801
Project Budget
2.500.000 (100% spent)
Funders
European Union 1.500.000 + UNICEF 1.000.000
Start & End Dates
21 June 2011 to 20 March 2015 (45 months)
Relevant MDGs
MDG 2: Achieve universal primary education

MDG 4: Reduce child mortality
MDG 7: Ensure environment sustainability (Target 7C: Halve the
proportion of people without access to safe drinking water and basic
sanitation)
10 sub-districts (SDs) in 5 Districts:
Geographic Focus
Area
Liquidoe and Remexio SDs in Aileu District

Ermera and Railaco SDs in Ermera District
Laclo and Laclubar SDs in Manatuto District
Nitibe and Oesilo SDs in Oecusse District
Viqueque and Lacluta SDs in Viqueque District
Project Target Areas and Partners

By the time it was completed the project had carried out activities in 63 aldeias in the planned 5
districts and 10 sub-districts. The project was implemented in collaboration with government
ministries (public works, health, education) and national NGO partners selected by UNICEF in
each of the five districts. Table 2, below, contains a list of the districts and sub-districts, together
with the respective NGO partners. In addition to these NGOs, UNICEF also partnered with TimorLeste Media Development Centre (TLMDC) to assist with project communications and with two
others in small initiatives to pilot sanitation marketing.
Table 2: Project target communities by district/sub-district and NGO partners

District
Sub-District
National NGO Project Partner
Aileu
Liquidoe, Remexio
AMAR
Ermera
Ermera, Railaco
HIM
Manatuto
Lacio, Laclubar
NATILES
Oecusse
Nitibe, Oesilo
ETADeP
Viqueque
Lacluta, Viqueque
CPT
The location of the target districts and sub-districts is shown in the map in Figure 1, below.
Figure 1: Map of Timor-Leste showing target districts and sub-districts of project
A list of the targeted aldeias in which the community water supply and sanitation activities of the
project were implemented, including changes since the start of the project, is included in Appendix
3. Appendix 4 contains a list of the schools that benefited from WASH interventions.
3.
Purpose and Scope of the Evaluation
The overall objectives of the evaluation (see also Appendix 1), in line with standard practice, were
to strengthen UNICEFs accountability to its donors and key stakeholders, including beneficiaries,
and to learn from the experience of this intervention to inform future WASH policy and
programming. The evaluation thus needed to both assess compliance with contractual obligations
as well as adherence to UNICEFs equity focus and the potential contribution of the project to the
sector and to UNICEFs institutional learning agenda. An additional objective was to provide
strategic comment to UNICEF Timor-Leste in terms of its focus areas of support to relevant
ministries in WASH, as proposed under the current Country Programme (2015-2019).
To achieve these objectives, the evaluation was required to:
a. assess the relevance, effectiveness, efficiency, sustainability and impact of the project;
b. assess the extent to which the project contributed to broader development results at the
sub-national level, and the lessons learnt that will allow the replication and scaling up of
the interventions; and
c. draw operational recommendations and lessons learned for further improvement and
enhancement of relevant sector policies, plans and strategies through analyses of the
factors contributing to the success or failure of the project.
The detailed evaluation questions are described and discussed in the next section.
In terms of the geographical scope of the evaluation, it was necessary to take into account the
broad spread of the project across 10 sub-districts. This meant that, although ideally all sites
should be visited, the budget and thus time limitations made it necessary to select a practically
feasible sample of sub districts and communities within them to visit, that would still be
representative. The criteria for selection are described in the next section.
4.
Evaluation Methodology
The methodology applied in this evaluation was based on that defined in the terms of reference,
which is standard as per current good practice. Only minor modifications were introduced, namely:
adjustments to the distribution of time allocated in Phase 2 for desk work and interviews
in Dili, and work in the sub-districts (from 18 and 13 days respectively, to 16 and 15);
as a result, an increase in the number of sub-districts that were visited, from the specified
minimum of 3 districts and at least one sub-district in each, to 8 out of the total of 10; and
reduction in the time allocated to Phase 3 (analysis and reporting), from 20 to 16 days.
The methodology combined qualitative methods (structured and semi-structured interviews and
focus group discussions) with quantitative methods, namely gathering data from routine reports,
reports on interim external monitoring conducted by the EU, analyses and reports produced by
other interventions and, fundamentally, the data obtained by the baseline and endline surveys
and KAP studies.
The fact that an endline exercise was conducted virtually at the end of the project allowed this
final evaluation to focus more on the aspects of the project not covered in detail in that exercise,
i.e. to concentrate more on sustainability and lessons learned and less on detailed surveys to
assess KAPs at household level. This was the main motivation for increasing the sample size of
sub-districts to be visited (and thus of key role players and stakeholders to be interviewed), which
resulted in less time spent in each aldeia compared to the endline survey and KAP study.
The methodology employed is described in the sections that follow in terms of the following key
aspects:
4.1
evaluation questions and M&E indicators;

sources of information and methods of data collection;
sampling;
endline survey and KAP study; and
other considerations.
Evaluation Questions
The evaluation questions included in the terms of reference were expanded by the consultant in
his technical proposal, and then some further additions made, in agreement with the UNICEF
country office, after the initial desk review of documents provided before arrival in country, and
the final list documented in the Inception Report. In particular, the recommendations for this final
evaluation made in the Endline Survey and KAP Study Report were taken into consideration, of
which the following are highlighted:
a. Some field work needs to be done to visually capture the various types of latrines households
have in the project area to better understand the Timor-Leste context in relationship to JMP
latrine definitions.
This was done.
b. Consider further exploring what other benefits have been experienced by households as a
result of increased water access beyond having increased improved drinking water sources.
Specific questions in this regard were included in the household interview checklist (see
Appendix 9H).
c. Drinking water at schools was reported to be boiled, however this was not verified through
observation. Also 70% of children were reported to bring drinking water to school with them
to drink. Suggest that both of these be verified/further assessed during the final evaluation.
As explained in section 5, it was not possible to follow up on this issue due to all but one of
the schools being closed on the day/time of the visit, due to unforeseen circumstances beyond
UNICEFs control.
d. If water quality testing has not been done, it is recommended that such testing be undertaken
during the final evaluation.
Due to the fact that there is only one laboratory able to do this, in Dili, it was decided that it
would not be feasible to include water quality testing on a large enough scale to be meaningful.
However, this an important issue and is included as a recommendation.
e. Consider collecting random household point of use water samples from water storage
containers pre/post boiling to assess water quality and the effectiveness of water treatment
methods used by the majority of households.
As explained above, this was not feasible.
The expanded evaluation questions are listed in Appendix 5.
4.1.1
M&E Indicators
Besides the evaluation questions to be answered, the project established a Monitoring and
Evaluation (M&E) plan with specific indicators to measure its impact at completion, based on the
project logical framework (logframe). The latter is attached in Appendix 2. Both the baseline and
endline studies served to inform the projects M&E plan and measurement of results and
established indicators, and the results are incorporated into this evaluation.
The studies did not fully measure the results for the indicators for Result 1 (Local authorities
are better able to lead and support WASH activities ), although sub-district facilitators (SDFs)
and water facility management groups (GMFs) were interviewed to assess their capacity
development. The results for the indicators will be assessed in this evaluation, namely:
All SDFs in target areas have received ToT on CAP processes and are supporting GMFs
2 national WASH training modules/guidelines developed and implemented.
Due to the fact that the results of most of the indicators had already been measured by the endline
survey, and that this evaluation by its nature as a final evaluation is mainly focused on assessing
sustainability and identifying lessons learnt, it was not guided only by the indicators in the
logframe. In the case of household sanitation in particular, other issues were explored such as
whether villages declared ODF still maintain that status, to what extent households have
subsequently moved up the sanitation ladder (i.e. upgraded to more durable facilities or ones
providing a higher level of service), and what inspired them to do so, as well the role being played
at village level by local sanitation entrepreneurs. These issues are reflected in the corresponding
evaluation questions in Appendix 5.
4.2
Sources of information and methods of data collection
The details of sources of information and methods of data collection used, for each evaluation
question, are listed in Appendix 5. The actual number of each is reported in section 5. In summary,
the sources were the following:
a.
Secondary sources, i.e. documents produced by the project and others (see Appendix 7
for the list of those reviewed). These included reports of two EU-commissioned ROM
(results-oriented monitoring) reviews in 2012 and 2013.
b.
Semi-structured interviews with key informants, including UNICEF staff, NGO partners,
national and district government partners, SDFs, school coordinators and other donor
funded programmes and international NGOs active in sector. Pre-prepared checklists
were used to facilitate consistency and comparison see Appendix 9.
c.
Focus group discussions (FGDs) with GMFs (which also involved community leaders and
ordinary community members in most cases), based on pre-prepared guidelines (see
Appendix 9). No other FGDs at community level were held as this area was covered by
the baseline and endline KAP studies; as described in section 5, it was not possible to
meet with groups of school children as intended.
d.
Observation of WASH infrastructure and other aspects (including open defecation and
general cleanliness) at village level.
e.
Random sample surveys of households at all but two of the villages visited, using transect
walks, observation of latrines and other barriers against disease and brief discussions with
heads of households (except in two villages where few were at home), based on checklists
(see Appendix 9H).
All the information gathered was collated in a summarised form in a data base in Excel format
(see Appendix 10), organised as follows:
Sheet 1: Notes to form the basis for answers to evaluation questions, by question and
source of information (both as an aide to the consultant, and to assist the reader
to understand the sources of information for the observations and comments
made in this report);
Sheet 2: Observations on water supply systems and summaries of FGDs with GMFs;
Sheet 3: Observations of latrines and homesteads and summary of interviews with
householders;
Sheet 4: Observations of school WASH facilities at schools and summary of interviews
with Coordinators of two of the schools.
4.3
Sampling
The terms of reference specified that at least 3 districts should be visited. However, as noted
above, it was possible to organise the itinerary in such a way as to include visits to four districts
(only Manatuto could not be included, due to distance to the targeted aldeias and time constraints)
and both sub-districts in each, totalling 8 sub-districts.
The criteria used for deciding on the districts to be visited included the Endline Survey and KAP
Study Report recommendations for this final evaluation, already mentioned above, which include
the following with regard to districts and sub districts to be visited:
a. Explore what factors contributed to the differences among the targeted districts/subdistricts with respect to sanitation (especially how CLTS was implemented) and water
access. Selected districts/sub-districts should include both high performing and less
performing areas.
This was done; Viqueque was included as a less performing district in spite of the
relatively long travel distances (virtually a full day each way).
b. Consider gathering data in the communities that were dropped from the project post
baseline, such as in Viqueque and other unserved areas to provide data on non
intervention communities for comparison purposes.
In order to draw any valid conclusions for comparison it would have been necessary to
include a reasonable number of non intervention communities (say at least 3) and to
spend enough time in them to understand their dynamics. However, it would have been
difficult to gain meaningful access since the project was not active in these communities,
and there were time limitations. Another factor taken into account was the fact that the
project was not designed as a random control trial. It was thus decided not to try to
implement this recommendation.
With the above in mind, it was essential to include Viqueque district and Lacluta sub-district in
particular, as less performing and Liquido in Aileu as high performing, in terms of the
achievement of water system implementation and ODF status targets. In Viqueque both subdistricts were included in the sample.
In the other districts the initial targets were achieved or even exceeded, in terms of both water
systems and ODF status, but according to the Endline Survey, in some sub-districts, such as
Railaco in Ermera and Nitibe in Oecusse, there seems to be a fairly high incidence of reversion
to OD which needed to be analysed more closely.
In summary, the minimum list of the sub-districts to be visited, and the respective selection criteria,
were as follows:
Sub-district
Criterion for inclusion
1. Lacluta
Try to understand reasons for low level of achievement of ODF status.
2. Viqueque
Also a less performing area.
3. Liquido
High performing area maintaining high level of ODF status
4. Railaco, Nitibe
Originally high performing areas with more reversion to OD
As noted above, it was possible to include all the above sub-districts, including in Oecusse, as
well as the second sub-district in each district.
The main criterion for selection of aldeias to be visited in each sub-district was to include
examples of both those that had performed well and those that had performed relatively poorly.
Where there was a choice of two or more aldeias that qualified under either criterion, the issue of
practical access was considered. Aldeias that would require more than two hours travel by vehicle
from the sub-district capital were excluded. Aldeias with schools targeted by the project were also
prioritised.
In terms of sample sizes, this means that the following was scheduled:
18 aldeias to be visited, i.e. over one third of the total number of target aldeias;
7 SDFs to be interviewed, i.e. 70% of those with whom the project partnered;
FGDs with 13 GMFs (30% of all those formed and trained); and
6 schools to be visited and inspected, representing 22% of the total of 28 supported.
In terms of the household interviews, it was planned to select them as a random cluster sample
(see Appendix 9H) with the proviso that an adult male or adult female currently residing in the
house be present. This was not possible in all aldeias, as explained in the next chapter.
4.4
Endline Survey and KAP Study Report
The knowledge, attitudes and practices (KAP) endline study was undertaken at the end of the
project to assist in measuring the outcomes and impact of the project interventions against
baseline data collected in 2011. The data was intended to inform this final project evaluation as
well as future WASH interventions in Timor-Leste.
The endline study included a household and school survey, with 601 households with at least one
child age five and under, and 22 public school surveys11. The household surveys were carried out
in 50 of the same 55 aldeias that were surveyed in the baseline survey (five from Laline Suco in
Viqueque were excluded), as well as four which were added after the baseline survey, a total of
54. In addition focus group discussions were held with 12 GMFs and individual interviews with
four SDFs.
The results captured through the comparison of endline to baseline data are attributed to the
projects interventions in sanitation in all 54 aldeais and in the 43 aldeais where water
infrastructure projects were conducted. Changes in water supply in the 11 aldeias where UNICEF
was not working are attributed to the government supported District Development Programme
(PDD) and the NGO Triangle in Oecusse. No other organisations, or other WASH efforts, are
known by UNICEF to have been implemented in the targeted areas during the project period. As
the surveying was only done in the targeted project aldeais, the results are only representative of
the targeted communities and not necessarily to the districts and sub-districts in which they reside.
4.4.1
Methodology
The study was conducted with the use of the following methods:
Briefing by UNICEF Chief of WASH

Desktop review of UNICEFs Year 3 progress report to EU (June 2013 to June 2014) and
other project materials
Household surveys in all five districts in targeted aldeais
Survey of targeted primary schools in all five districts
Focus Group Discussion with GMFs
Individual interviews with SDFs.
Two survey instruments were developed for the baseline KAP a Household Survey and a School
Survey. These tools served as the foundation for the endline. Several questions were omitted
from the endline household survey that were not seen as relevant to measuring results and
additional questions were added to gather input on the perception of the level of open defecation
in the respondents communities and CLTS process. Questions and associated variables were
derived as much as possible from the 2009-10 Timor-Leste Demographic Health Survey (TLDHS)
and from other international recognized sources, such as the Millennium Development Goals
11
Endline was planned in all 23 schools where Baseline was done but one school was not accessible at
the time of end-line survey, so 22 schools were surveyed.
10
(MDGs), and UNICEF documents, e.g. UNICEFs WASH in Schools Monitoring Package, to allow
for standardisation of data and to enable comparisons wherever possible.
In addition to household demographic information, the household survey included 41 possible
questions on the following subject areas:
Drinking water access and treatment

Sanitation access and latrine conditions and solid waste disposal practices
Incidence of child under five diarrhoea and associated knowledge and practices
Hand washing knowledge and practices
Drinking water storage and safe-handling practices
Household decision makers related to sanitation
Perception of latrine coverage and CLTS-focused questions.
All but one survey question was designed to be open ended to elicit respondents knowledge
without prompting, along with observational questions to assess access, conditions, and
practices. The surveys were administrated by enumerators hired through a national NGO (Belun)
to assist the consultant with conducting the endline.
The school survey was targeted to primary school principals or head teachers in the target
communities. The survey included 27 questions in addition to basic descriptive information. The
survey was administered by the enumerators through an interview and was designed to illicit as
much information as possible about WASH conditions, knowledge and practices at the school.
The survey included questions on the following areas:
Student and teacher composition, including gender

Student attendance by gender
Access to water for drinking and to support hygiene/sanitation practices
Water treatment practices
Access to sanitation facilities and condition and maintenance of facilities
Solid waste disposal practices
Access to hand washing facilities and hygiene education promotion.
The focus group discussions with GMFs and questionnaires for SDFs were implemented by the
enumerator team leaders. These questions explored the roles and tasks of the GMFs as well as
their overall functionality and perception of OD in their community.
All survey instruments were field tested and modifications to the study instruments were made
following the field tests.
4.4.2
Sampling Design and Sample Size
The same sampling methodology used for the baseline was used for the endline study. Sample
size for the surveys was calculated for a margin of error of 4%, with a confidence level of 95%,
based on a 2010 census population of 22,798 from the 54 targeted aldeais, with a 50% response
distribution. This calculation resulted in a sample target 601 households to be surveyed by the
enumerators in the field. The 601 households were selected proportionately among the targeted
communities. There were some changes in the targeted aldeais from baseline to endline, however
interventions took place in the same districts/sub-districts.
The survey used a random cluster sampling method to determine what houses would be selected
for an interview. All houses had to have at least one child age five or under living in the home to
be selected. Respondents were requested to be either an adult male or adult female currently
residing in the house, while a head of household was desired for the interview, they were not
required for the survey. For the endline enumerators were asked to prioritise interviewing women
if both a male and female adult were available.
11
4.4.3
Field work
The survey team consisted of 12 enumerators, broken into three teams, each lead by a team
leader. Training was conducted with the enumerators to prepare them for piloting/field test and
implementing the survey instruments. Field work was done over 10 days in December 2014.
4.4.4
Findings
The findings of the Endline Study and KAP Survey are reported in section 6 in the sub-sections
relevant to each theme and issue.
4.5
Limitations to the Evaluation and Mitigation Approaches
As noted in the terms of reference, the project was not designed on the basis of a gender
differentiated analysis of the needs of beneficiaries, nor did it define specific outcomes aimed at
greater gender equality. Thus, although this report comments on gender and human rights
aspects, this limitation was taken into account, while following the UNEG guidance on how to
integrate human rights (HR) and gender equality (GE) considerations in evaluations. In examining
the HR & GE aspects, the consultant was guided by international and regional conventions and
national policies and strategies, ensuring key informants included both genders, and from human
rights aspects, both duty bearers (government, service providers) and rights holder (beneficiaries)
were interviewed.
As mentioned above in relation to the evaluation questions, comparisons between intervention
and non-intervention communities were not made because the project design did not employ a
randomised control approach. However, this is not considered a serious limitation; such a design
is usually only employed for research projects and this project did not have research as a
significant, explicit objective.
In terms of the quantitative data collected, especially regarding issues such as numbers of outputs
and outcomes achieved, budget spent, etc, the evaluator relied mostly on data provided by
UNICEF. However, this is normal and, in general, the reliance on self-reporting is not considered
to have been excessive and thus a limitation. Other factors such as bias in respondents answers
could be managed due to the considerable prior experience of the consultant, for example, being
aware that beneficiaries tend to try and anticipate in their answers what the interviewer wants to
hear, that local leaders dont want to appear ungrateful so avoid any criticism of the project and
that implementer s tend to want to show only the most positive examples. Where only limited or
no triangulation of some data was possible, this is highlighted.
In terms of the practical aspects of carrying out the evaluation, the most significant one, which
was however foreseen, was the non-availability of some project partners. For example, the
partner NGOs responsible for Viqueque and Manatuto districts, where least success was
achieved with CLTS, were not available as they are no longer collaborating with UNICEF and did
not part on very cordial terms. Their input would have been very valuable to gain greater insight
into the challenges faced in these districts. In particular, the views of CPT could have provided
greater clarity regarding the situation encountered in Bahalara-uain in Viqueque (described in
detail in section 6). However, in recognition of this limitation, special care was taken to solicit the
opinions of more role players than normal concerning the reasons for the problems in Bahalarauain and in Viqueque District in general.
Other practical challenges that placed some limitations on the achievement of the work plan and
thus on the scope of the evaluation were:
heavy rain the night before the planned trip to Ahic in Lacluta sub-district, which prevented
the consultant from reaching the community;
12
delays in starting the FGD with the GMF in Bahalara-uain (due to their disorganisation),
and subsequent late arrival at the school after it was closed, thus not being able to
interview anyone there or inspect the interior of facilities;
all schools being closed in Ermera District during the week scheduled for the field visits
there, due to teacher training, hence only one interview and interior of facilities inspected
only at this school;
all schools being closed in Aileu District during the week of the visit due to a religious
festival, hence no interviews and no interiors of facilities inspected;
because of the festival activities in Remexio sub-district of Aileu, few GMF members and
no other community members were available for the FGDs.
It was thus not possible to obtain as much information as desired about the sustainability of the
school interventions in the areas of training and awareness raising. However, these aspects were
adequately covered by the endline study (conducted only a few months before the final
evaluation) which assessed retention of messages and perception of quality of training, amongst
other factors.
4.6
Ethical considerations
No difficulties were encountered in adhering to UNICEFs ethical standards, as per the UNEG
Ethical Guidelines for Evaluation. All confidential information obtained by any means was treated
in confidence. Personal, confidential and sensitive information was not discussed with, or
disclosed to, unauthorised persons, knowingly or unknowingly. The interviewees and participants
of focus groups were treated with objectivity and impartiality. In relation to the interviews with
household members and inspection of their latrines, the consultant insisted on excluding
community leaders from accompanying him, to avoid possible bias in choosing or avoiding
particular homes (such as those known to not have latrines), or unconscious intimidation to
provide the correct answers, and requested permission to visit the latrines.
5.
Activities Undertaken
The detailed overall work programme and travel itinerary for all three phases of the evaluation is
included in Appendix 6. The following is a summary of the activities undertaken.
5.1
Phase 1: Preparation and desk review
This phase took place from the signing of the contract (18/03/2015) to arrival in Timor-Leste
(14/04/2015) and included the following:
briefing of the consultant by UNICEF via Skype call;

review of project documents;
further development of evaluation questions on the basis of documents reviewed;
proposals for criteria for deciding which sub-districts to visit; and
drafting of Phase 1 Report documenting all the above.
The documents reviewed in this phase, together with others reviewed in-country and afterwards,
are listed in Appendix 6.
5.2
Phase 2: In-country field work
This phase started on 15/04/2015 and ended with the departure of the consultant from TimorLeste on 28/05/2015. The key activities undertaken are summarised below in Table 3.
Table 3: Summary of Field Work Schedule and Itinerary
13
Week
Location
Main Activities
15-17 April
Dili
Briefings (UNICEF, Technical Committee), planning
20-24 April
Dili
Planning, organising, meetings with partners, interviews with

stakeholders and partner ministries.
Submission of Inception Report
27-30 April
Viqueque
Visit to 2 sub-districts and 1 aldeia in 2 days (+ 2 travel days).

Public holiday on 1st.
4-8 May
Ermera
Dili
Visits to 4 aldeias in both sub-districts over 4 days.

Stakeholder interviews and desk work in Dili on 7th
11-15 May
Oecusse
Dili
Visits to 4 aldeias in both sub-districts in 2 days (+ 2 travel days).

Desk work and interview in Dili on 15th.
18-22 May
Aileu
Dili
Visit to 4 aldeias in both sub-districts over 3 days.

Public holiday on 20th.
Attend National WASH Forum meeting in Dili on 21st.
25-28 May
Dili
Debriefings.
Departure
In terms of numbers of activities, these are summarised in Table 4, as follows:

Table 4: Summary of activities undertaken
Quantity
Activity
Description
Document
reviews
Project reports and evaluations, financial audits, other

Documents produced as outputs of project
UNICEF and EU programmatic documents & guidelines
Sector documents: policies, guidelines, manuals, etc
Reports from other programmes
9
4
3
4
4
Interviews
Project partners & UNICEF staff at national level

Other sector stakeholders at national level
Project partners in 4 Districts and 8 Sub-districts
Households in 10 aldeias in 7 Sub-districts
School Coordinators in Oecusse and Ermera
(no PTAs, no teachers, no pupils)
10
6
22
37
2
Focus group
discussions
GMFs with local leaders and community members in 7 sub-districts
13
Inspections
Water systems in 7 sub-districts

School toilet blocks with HWFs in 5 sub-districts
Household latrines in 10 aldeias in 7 sub-districts
13
10
43
The list of meetings with stakeholders and key informants interviewed can be found in Appendix
8.
14
6.
Findings
In this section the information obtained by the final evaluation and the endline survey is presented
and analysed in relation to the evaluation questions (as listed in Appendix 5), as the basis for the
subsequent discussion of conclusions, lessons learnt and recommendations.
6.1
Relevance
6.1.1
Relevance for target groups
The project and all its components remained highly relevant throughout, as confirmed by all target
groups, project partners and key sector role players in the interviews and FGDs conducted by the
consultant. At household level the priority accorded to improved access to potable water remains,
as is to be expected, extremely high. Less obvious, but no less important, is the increased
importance of sanitation and hygiene for the target communities, as expressed by them, thanks
to the projects efforts to raise awareness in this regard.
The importance of the school WASH interventions was also confirmed, the most frequent
comment (from all stakeholders) being that much more of the same is needed.
The different capacity building interventions remain highly relevant and necessary for all the
categories of beneficiaries and in this regard as well, additional needs were identified.
6.1.2
Alignment with government priorities
The project was fully aligned with and supportive of government policies and priorities as defined
in the Strategic Development Plan (SDP 2011-30) and specific WASH sector policies, and as
confirmed by key stakeholders interviewed. WASH in general, and water supply in particular, are
given high priority in the SDP, both as part of the health strategy and for improved nutrition as
part of rural development, being described as a vitally important element in the economic and
social development of Timor-Leste. The project design and strategies were fully in line with
government policies for the sector.
The project also contributed to developing policies such as the Basic Sanitation Policy (approved),
the National Water Supply Policy, National Water Resource Policy and Law, WASH in School
Guidelines, BeSI Committee Terms of Reference, and others. Although the latter have not been
approved yet, they are already being used in the sector. Practical support to sector development
included training of CLTS trainers, in cooperation with other donor programmes. UNICEF is
acknowledged as one of the lead organisations in the sector and a highly respected partner of
government.
6.1.3
Alignment with funders country strategies and objectives
The project was aligned with UNICEFs Country Programme strategy and objectives, WASH
being a key component of the Child Health and Nutrition component, where it is intended that
UNICEF will continue to support WASH services in rural communities, focusing on scaling-up of
CLTS in priority districts through partnership with the Ministries of Health and Public Works, NGOs
and development partners. It is also part of the Quality Education component, where the
commitment is to support the achievement of the Governments goal of ensuring that all public
schools have access to clean water and sanitation facilities by 2020, particularly in priority
districts.
The project was also aligned to EUs Country Strategy and general development objectives which
consider WASH as key enabling factors in improving nutrition, especially of children under five,
and health in general. Malnutrition (and stunting in particular) is a significant challenge in TimorLeste.
15
6.1.4
Quality of project design
The project was generally well designed with a clear intervention logic, based on the standard
components of a WASH intervention, namely water, sanitation and hygiene (including for
schools), and one for institutional capacity building. These Additional positive elements were the
inclusion of sanitation and hygiene in the same Result (a more appropriate arrangement when
the CLTS approach is used as in this case), and of policy advocacy activities in the institutional
capacity building Result, as well as implementation through local NGOs.
These components, together with the necessary activities to achieve them and the short, medium
and long term objectives are adequately summarised in the logical framework (LF) and describe
well the theory of change underpinning the projects design, as described in section 2. The LF
was mostly well designed, with all the elements necessary for it to use as the basis for planning
and monitoring of activities and outcomes. Some improvements to indicators were made as a
result of recommendations by the EUs ROM (results-oriented monitoring) reviews.
The assumptions were relevant and well formulated at the appropriate levels. Some of them have
not held completely and have affected the project. This is discussed under Risk management
for efficiency, effectiveness and impact, in the sections that follow.
The projects institutional design and capacity building strategy was generally appropriate, with
maximum involvement of local institutions and government, from GMFs and PTAs (parent teacher
associations) at community level, to local NGOs for implementation and local level government
to provide training and to support O&M. A dedicated capacity building component (Result 1) was
included to overcome weaknesses in all these institutions, but may not have been enough to
ensure sustainability; this is discussed further in section 6.5.
The project design placed significant emphasis on sustainability, recognising that it will depend
on government capacity to support community-based water management structures (GMFs) and
ongoing sanitation and hygiene promotion at household level. Additional training for GMFs was
included in the standard methodology.
The budget was well constructed and mostly accurate, although possibly too conservative, since
the project managed to surpass most targets (including construction of facilities) by some margin,
without exceeding the budget.
In terms of the inclusion of cross-cutting issues, there was no clear gender strategy, although
some elements were included during the course of implementation. Environmental issues were
also not specifically highlighted in the project proposal, or included as indicators in the logframe,
although in practice several aspects were included, such as protection of water sources. This
included protecting and planting trees in the upper reaches of streams, bio-fencing of intakes and
creating drain diversions for protecting water sources from contamination. All these measures are
included in the GMF training curriculum (CAP 2) and were applied in many cases.
One example of the absence of a clear policy was the lack of erosion protection guidelines for
construction; none of the reservoirs, standpipes or buried pipelines included storm water diversion
works to prevent erosion, of which a significant amount was observed. The technologies
employed were appropriate and the community water supply and school water and sanitation
facility designs used were based on governments standard guidelines, a very positive feature of
the sector in Timor-Leste.
6.2
Efficiency
The efficiency with which the project was implemented was high in most aspects; exceptions are
noted below. Planning, budgeting, monitoring and management of financial, human and other
resources were generally good and all appropriate management tools were used. All activities
16
were carried out, in many cases in excess of what was originally planned, due to some extent to
the nine month no-cost extension. The budget was fully spent, although there are some issues
around possible ineligible expenditure to be resolved. The quality of products was high.
6.2.1
Implementation against work plans
Although the project required a nine month time extension, equivalent to an additional 25% of
time, it also exceeded most targets by about the same proportion. Early delays were largely
corrected by the end of the third year, and all targets were achieved by then, except in the case
of school WASH facilities. The general delays were due to a number of factors which are not
uncommon and have to be expected in most projects, such as occasional procurement delays
due to having to import most materials and equipment, and having to redesign some water
schemes due to factors such as being denied access to water sources by land owners. The most
significant delays were caused by the national election in 2012 and a longer than average rainy
season and flooding in 2013 which limited access to communities.
The delays in the schools component were such that by the end of the third year only 17 of the
planned target of 24 schools had been completed and the tender for the remaining schools had
not been launched yet. Amongst the causes of slow progress was the use of some private
construction companies that were unable to complete works in the stipulated time due to lack of
financial capacity to manage the required cash flow; the competitive bidding procedures were not
always effective in eliminating these financially weak companies. In some case companies lacked
qualified technicians who could understand the latrine design, resulting in errors requiring time
consuming rectifications and rebuilding. However, the advantages of supporting the still fragile
local private sector (as per government policy) seem to outweigh those of contracting larger
foreign firms.
The project team was generally effective at planning ahead to get materials on site before the
rainy seasons, and the rate of expenditure was generally in line with the time elapsed. There were
also minimal delays with processing funding tranches from the EU.
The particular delays in Viqueque and Manatuto appear to have been mainly due to the poor
performance of the two implementing NGOs, according to all sources consulted (the NGOs
themselves could not be consulted as one no longer exists and the other did not return calls). In
spite of considerable additional supervisory support provided by UNICEF and SAS (district office
of DGAS) they were not able to perform as required and their contracts were not renewed in 2014.
In Viqueque the main impact was felt in the CLTS programme, but also with the multi-aldeia water
system in Bahalara-Uain suco, which had to be completed by SAS, with less than ideal results.
The NGO was unable to resolve the unusually difficult social and institutional challenges at
community level (which remain unresolved) and it was unfortunate that it was not replaced earlier
as all efforts to assist it had not produced the desired results.
have been suggested, including cultural differences between this and other districts, the weak
social facilitation skills of the NGO and the negative impact of other programmes offering materials
subsidies to households to build latrines. It was not possible in the time available for this
evaluation to undertake a detailed enough investigation to reach definite conclusions concerning
the latter hypothesis, nor did the endline survey shed any light on the matter. However, from all
the responses obtained from multiple sources including a small sample of households themselves
in Bahalara-Uain, community leaders, the SDF, the GMF, the PSF (community-based health
promoter) who conducted some of the promotion work for the project, project staff and partners,
the consultant concludes that the impact of subsidies was not a significant factor. It would seem
that the inability of the NGO to resolve the social and institutional problems was the main reason.
17
In other districts (especially in Oecusse) the project had to contend with similar potentially
negative impacts of other programmes offering subsidies and managed to do so effectively.
However, other programmes, such as BESIK, report this as a significant challenge.
6.2.2
Quality of project management
The quality of project management in general was good. Regular (annual) planning and review
exercises were conducted with the NGO and government partners and the plans formalised in
annual agreements with the NGOs. They received funding tranches at quarterly intervals after
reporting on activities undertaken and accounting for the previous tranche.
The monitoring and quality control system was well designed and seems to have been
implemented effectively. An M&E plan was established, based on the logframe indicators, and
these were improved on the recommendation of the EU ROM reviews. A baseline survey was
conducted to establish baseline values for the indicators. Regular monitoring and reporting was
ensured. Quality assurance was ensured by means of technical supervision by UNICEF staff and
consultants and oversight visits from the UNICEF Regional Office.
Financial administration and accounting was done by both UNICEF Timor-Leste and the five NGO
partners, with final certification of financial statements being done by UNICEF Head Office (in
New York). The NGOs received their funding tranches on an imprest basis, i.e. having to provide
acquittals (and returning unspent funds) before receiving new tranches. The procedures were
defined in the agreements signed between each NGO and UNICEF. Some difficulties were
encountered with delays in finalising acquittals when the delay exceeded six months as UNICEFs
system cannot maintain advances as such for longer than this, requiring the process of approving
the advance to be re-initiated. In addition, the NGOs reported that they had to pre-finance their
activities for several months at the beginning of every year, and for several weeks and sometimes
months, every quarter, due to the slowness of the procedures. As with some of the other
challenges cited by the NGOs, there is no consensus as to the allocation of responsibility between
them and UNICEF, and it was not within the scope of this evaluation to conduct a detailed audit
of every delayed quarterly tranche to arrive at incontestable conclusions.
Procurement of equipment and materials was UNICEF's responsibility and the system seems to
have been relatively efficient, within the constraints of the thorough procedures of a large
organisation like UNICEF. There were also challenges with importation of some materials which
cannot be sourced locally.
Several queries were raised by the international auditing firm in its report on the expenditure
verification they conducted in 2014 on the first three years expenditure, and as a result they
considered some 122.000 (23% of the value of the sample of transactions verified) to be
ineligible. However, 41% of this amount refers to salary costs apportioned to UNICEF staff
according to the ratios defined in the budget, without time sheets to confirm that the time was
actually spent on the project. This is a standard practice in EU funded projects and it is thus
difficult to understand why it is being queried in this case. A further 45% of the expenditure
considered ineligible refers to sand and other local materials delivered without respective delivery
notes, for facilities that were verified as having being built to specification. This lack of
documentation does not seem to justify the corresponding expenditure being considered as
ineligible.
By the end of the project the budget had been fully spent. In addition, it is possible that what
UNICEF spent from its own core funding increased its total co-funding to more than the
contractual 40%. For example, UNICEF continues to provide follow-up at its own expense to the
few school sanitation contracts that have not been fully completed (e.g. two water connections
still pending) even after the end of the project. This is one of the major advantages to the EU of
co-funding with large organisations which do not rely only on project funds to ensure the smooth
implementation of projects.
18
6.2.3
Quality of products
training provided is judged as generally of a very high quality. In most cases, however, these
could not be assessed against norms and standards previously defined by the project, as
suggested by the evaluation questions.
Infrastructure was the exception and was built according to the standard government design
guidelines. Detailed comments of the facilities inspected are contained in the data base in
Appendix 10; these were found to be in almost all cases of solid, durable construction. Some
areas of detail could be improved, however, to reduce the burden of maintenance, as noted in
Appendix 10. Some of these are shown in the photographs in Appendix 12, namely:
Lack of overflow drainage and control on water tanks at schools, with the result that most
of those inspected were overflowing, causing muddy and unhygienic surroundings. The
inlet valves were already dysfunctional due to tampering by children because they are not
buried in protected boxes.
School toilet blocks with lightweight drain inspection box covers that are susceptible to
tampering by children and filling of the boxes with stones and/or refuse.
Lack of storm water diversion works to protect building, pipelines and standpipes from
erosion, thus increasing the risk of cracking and other damage.
Poorly built waste water drainage systems on otherwise well-built standpipe aprons, that
are already separating from the aprons.
Poorly finished standpipe aprons with plaster screed already peeling off.
The quality of community participation, documents and training was (as is normal) not specified
up-front in terms of standard norms. The quality of participation and training has to be assessed
against the results achieved. Documents are assessed against the benchmarks of others
analysed by the evaluator elsewhere. This is discussed further below.
The results achieved in eliminating open defecation are a good indication of the quality of
facilitation of the CLTS process, at least in three of the five districts, as is the mostly good
performance of GMFs as an indicator of the quality of facilitation of the CAP (Community Action
Planning) process. GMF performance can also be assessed by the mostly very efficient process
of construction of water systems with a high level of community participation.
The exceptions in Viqueque and Manatuto, due apparently to the poor performance of the NGOs,
only serve to confirm the good quality of facilitation in the other districts. In this context it is
necessary to emphasise the poor quality of the water system installed in Bahalara-Uain, the
details of which are noted in Appendix 10. Unless an appropriately funded and facilitated
rehabilitation intervention is undertaken on this system it will not be sustainable and will thus not
provide the service to the community (around 2.000 people) it was intended to, and will be an
ongoing social and political challenge for government.
The quality of documents produced by the project and by government with project support or input
is also very good, notably the WASH in Schools Guidelines, the BeSI Committee Guidelines, and
the SDFs Manual and various training/awareness materials. They are all relevant and appropriate
to local conditions and to the target audience, comprehensive, gender and human rights sensitive
and professionally laid out.
6.2.4
Cost effectiveness
In comparison with similar interventions elsewhere, the project was cost effective. The consultant
has considerable experience with rural WASH projects and has also been provided with
comparative costs by EU operations managers. The average cost per person in the target
communities that have improved their access to water and sanitation (total project expenditure
19
divided by total number of beneficiaries) was approximately 114 Euros. The proportion of project
implementation costs and overheads (UNICEFs total costs) to total expenditure was around 22%,
also a cost effective ratio. In the case of some organisations, including international NGOs, this
ratio can be as high as 50%, with most at around 25-30%.
Also in terms of full lifecycle costs, the use of low maintenance technologies, together with solid,
durable designs, for water systems and school WASH facilities, although more expensive in terms
of the initial investment, are more cost effective in the long run and thus more sustainable. Some
relatively minor exceptions (notably the relatively less durable water tanks at schools) are noted
in Appendix 10.
One area where it was not possible to fully adhere to the principle of building long lasting
structures was in the CLTS programme where many of the latrines built have not lasted more
than a year or two. This is the main challenge of the CLTS approach and is discussed in more
detail under sustainability.
6.2.5
Partnerships with implementing NGOs
with target communities organised by them, was vindicated in this project. In three of the five
districts all of the advantages of the approach and none of the potential disadvantages were
confirmed. The advantages include a high level of ownership of the process and of the
responsibility for O&M, particularly of the water systems, as well as a generally thorough
understanding of how each system works, on the part of the GMFs. In addition, the combination
of culturally sensitive NGOs and proactive community leadership ensured that, in most cases, the
complex and potentially disruptive issues of ownership of and access to water sources and land,
right of way for pipelines and siting of facilities, etc were negotiated without undue delays.
The problems encountered in Viqueque, and to a lesser extent in Manatuto (where the NGO was
technically weak), have already been mentioned. In Viqueque the NGO should have been
replaced earlier. While it is understood that UNICEF wished to first exhaust all avenues to enable
the NGO to improve its performance, three years is considered too long to wait, since the losers
in this case were the community (over 1.000 families) who should be prioritised over the NGO.
The solution adopted in Bahalara-Uain of transferring the responsibility for implementation to SAS
for the last 15 months was not ideal and the results are not up to standard.
Experience with the alternative model, using private sector building contractors supervised by
government officials, has not been as positive, at least according to many of the GMFs and most
of the INGOs interviewed. In the case of school WASH facilities, for example, communities
complained of lack of communication and information about progress, lack of consultation and
delays in procurement of materials and payment of local labour. The quality of construction is not
necessarily better than that achieved by NGOs, and the costs are higher. The same level of
community participation and thus ownership, so vital for sustainability, is not achieved.
In the three districts where there were no problems with NGO performance and it was possible to
interview them, their assessment of the implementation process and the quality of project
management provided by UNICEF was mostly very positive. They acknowledged the support and
training received and the fluent communication and coordination. Some problems experienced
with funding and administration of the imprest system have been mentioned.
However, in terms of the quality of their relationship with UNICEF as partners, the assessment is
less positive. They perceive the relationship to be purely one of sub-contractors in which they are
not genuinely involved in joint decision making, although it was clear that the UNICEF team went
to considerable lengths to do just that. It would seem that this perception is more likely the result
of the inevitably unequal power relations and their reluctance to express concerns more
forthrightly. They also feel the need for support in organisational strengthening, such as provided
20
by a number of INGOs to their local NGO partners. This could include support for developing
organisational development strategies and even assistance in diversifying their funding sources,
as well as basic training in financial management, accounting, project management, etc.
6.2.6
Coordination with other programmes
Through UNICEFs very active participation in sector forums and coordination structures, the
activities of the project were effectively coordinated with those of other programmes at both
national and district levels. This was ensured right from the design phase of the project, during
which the identification of target sub-districts was made in such a way as to avoid overlap with
other donor-supported programmes. Support for development of policies and guidelines was also
effectively coordinated through participation in sector forums and regular bilateral communication
and coordination. Effective synergies were achieved with the BESIK programme in particular.
6.2.7
Risk management
Most of the risks associated with external factors preventing the project from achieving its planned
outputs, having implemented the planned activities and delivered the required products, were
anticipated in the logframe. However, in this regard the most significant external risk (not included
in the logframe) was the inability to influence government decision-making time frames. Even
though UNICEF has more influence and leverage with government than most cooperation
partners in the sector, it was not able to significantly influence these processes. One of the
outcomes is that the BeSI Committee Guidelines have still not been approved and it is clear that
without them and the accompanying formalisation of the Committees, their functioning is not likely
to improve.
Other risks, such as bad weather, procurement delays, etc appear to have been competently
managed.
6.3
Effectiveness
In general the level of effectiveness of the project was high. The targets for both water and
sanitation were exceeded by up to 25% and the (input) indicators for capacity building were
achieved. However, targets for capacity building in terms of the levels to be achieved, were not
defined; some progress was made, especially at community level, but less so within government,
as is to be expected in the context of a developing country emerging from a long period of conflict
and a stagnant economy at the time of restoration of independence.
6.3.1
Achievement of outputs and Results
The achievements of the project in terms of delivering the planned outputs and results were as
follows:
Result 1: Local authorities, particularly District WASH Committees (BeSI), are better able
to lead and support WASH activities in 5 districts, supported by national
authorities, policies and guidelines.
This component was focused on building the capacity of the project partners, particularly at district
level, through activities such as training of BeSI Committee members and SDFs, providing
technical and logistical support to the development of national WASH policies (e.g. WASH in
schools) and procedures, improving WASH information management through technical and
training support to the DNSA and Ministry of Health to improve their information systems and
conducting KAP studies.
The indicators for this Result, and respective levels of achievement were as follows:
i.
2 national WASH training modules/guidelines developed and implemented in target districts
21
UNICEF actively participated in the CAP review process initiated by DNSA with the support of
BESIK, which led to the finalisation of the improved SDF Manual. In addition, the project
developed a training module to provide refresher training to GMFs on O&M, to complement the
last module, as recommended by the second EU ROM review.
ii.
All SDFs in target areas have received ToT on Community Action Plan (CAP) processes
and are supporting GMFs
The training was carried out as planned for all the SDFs and their assessment of its usefulness
was very positive. According to the NGOs, the performance of the SDFs improved. The improved
SDF manual was especially appreciated by all those interviewed. To the limited extent that it was
possible for the consultant to evaluate the capacity of the SDFs, it seems that in general they
have the required skills, experience and motivation.
Regarding the level of support being provided by the SDFs to GMFs, achievements are very
variable from one district to another but it was not possible to obtain precise, quantified
information. SDFs report relatively regular and frequent supervisory visits in most districts
(Oecusse being an exception, due to unfilled posts), but most GMFs interviewed had not seen
their SDF for almost a year. On the other hand SDFs cite inadequate budgets from SAS to achieve
their work plans for travel to communities, whereas their SAS supervisors insist that they are
adequately funded.
However, it is clear that in general the SDFs are under-funded and/or inadequately supervised in
most districts for the GMF system to be fully sustainable. This is an area that needs ongoing
attention and support.
In conclusion, the results as defined by the indicators were achieved. However, they only measure
partially to what extent local authorities are better able to lead and support WASH activities.
The assessment of this evaluation, based on all the different sources consulted, is that while good
progress has been made in creating an enabling policy environment at national level, the project
partners at district and sub-district levels are not yet able to provide leadership effectively without
project support. The BeSI Committees are not functioning as intended yet, their terms of reference
have not been approved yet and there is inadequate funding for SAS and the Ministry of Health
to provide the leadership needed to scale up water supply and CLTS programmes, and fund
maintenance support.
Result 2: Target sub-villages and primary schools have improved water sources and are
able to manage and maintain their water systems.
The original targets for water supply included building 34 water systems through a participatory,
community-based process (the so-called CAP-Community Action Plan process), culminating in
the establishment of a community managed O&M system based on GMFs for all 34 systems. At
the same time 23 schools were to benefit from improved water supply and sanitation facilities.
The achievements are summarised in Table 5, below.
22
Table 5: Achievement of outputs of Result 2: water supply

Planned
target
Total
achieved
Above
planned
(%)
Participatory community action plan (CAP)
34
44
+ 26%
Number of community piped water supply systems
34
44
+ 26%
Number of protected springs or shallow wells
24
Formation & construction management training of GMFs
34
44
+ 26%
Post construction management training of GMFs
34
44
+ 26%
Number of schools with WASH facilities
23
28
+ 22%
Key Activity / Result
The figures show clearly that the project was very effective in this component and amply exceeded
its output targets in all aspects. In addition, 24 point sources (protected springs and shallow wells),
not originally anticipated, were built where gravity-fed piped systems were not feasible. The 44
GMFs formed also received additional refresher training in O&M after completion of their systems,
and GMFs were formed per aldeia for some of the point sources (not reflected in reports).
The indicator for this Result was: 90% of the targeted 34 GMFs and estimated 23 school PTAs
are maintaining their systems at project completion. This was intended as a measure of the
existence of functioning O&M systems rather than as a measure of their sustainability. However,
the exact meaning of functioning O&M systems was not specified. No comprehensive data is
available to assess this indicator: the project did not monitor system performance after
completion, and the data from DNSAs SIBS information system does not provide an accurate
reflection of system functioning. The endline survey included 12 systems and 22 schools, whilst
the final evaluation visited 13 systems and 10 schools.
Based on the observations of the endline survey and final evaluation, it can be confirmed that all
water systems had a GMF in place and all but a few of them were functioning reasonably to very
well, with enough funds collected for regular maintenance, uninterrupted supply and only dripping
taps (see Appendix 10 for details). All of the schools reported having some sort of maintenance
arrangement and a number of them have an arrangement with the local GMF to assist with
maintenance. 21 of the 22 schools surveyed in the endline survey had PTAs with WASH
responsibilities, although this did not include repairs. The effectiveness of these maintenance
arrangements is discussed further under sustainability.
Result 3: Target sub-villages and primary schools have improved sanitation and
demonstrate measurable improvement in knowledge and attitude relating to
improved hygiene and sanitation.
The original indicators and targets for this Result, and achievements, were:
i.
80% of target villages certified by MoH as ODF by project end (30% by mid-project)
The original number of target villages was 55. During the course of implementation, some were
excluded, mainly in Viqueque, and others added. The final number in which CLTS activities took
place was 63 (see Appendix 3 for the full list). ODF certification was achieved in 53 of these (with
a total population of over 20.000 people), which represents 84%, thus surpassing the target.
Although relatively high rates of subsequent reversion to OD were recorded in a few sub-districts
(discussed further under sustainability), this is an excellent level of effectiveness and bodes well
for the future of the CLTS approach in Timor-Leste.
23
ii.
70% of children can recall three critical hygiene messages
For reasons not explained in the report, the endline survey and KAP study did not attempt to
measure this indicator. The only hygiene related behaviour amongst children that was included
was the practice of children reported to wash their hands after using a latrine as reported by the
adult respondents in the household survey. This was found to have increased from 11% at
baseline to 46.4% at endline. This increase could be a result of handwashing campaigns at
schools but this is not known and it should be borne in mind that it is adults report of their
childrens practice.
Other aspects
Besides the indicators of the logframe, the baseline and endline surveys also measured a number
of other changes in knowledge, attitudes and practices that the project was designed to effect.
These are summarised in Table 6, below. Particularly noteworthy was the very significant increase
in knowledge amongst primary care givers of children of the causes of diarrhoea in children and
the most effective methods of prevention and treatment. On the other hand, it is clear that more
work is needed to increase the practice of hand washing after defecating.
The relatively high level of open defecation reported, in spite of all communities having achieved
ODF status, except in Viqueque, is a result of households ceasing to use the latrines built during
the CLTS process. This is discussed further under sustainability.
Table 6: Additional selected indicators of project outcomes

Indicator
Incidence of open defecation

Reported washing of hands prior to eating
Reported washing of hands after defecating
Special place set aside with soap & water for HWWS
unsafe water
unsafe food
Knowledge of causes of diarrhoea:
contact with
faeces
cook food well
Knowledge of
wash hands with soap
prevention methods:
safe disposal of childrens
faeces
Knowledge of treatment methods: provide oral rehyd. salts
6.3.2
Baseline
Endline
(2011)
(2014)
58%
4,4% of adults
27%
99% of adults
96% of
children
26% of adults
34%
58%
55%
58%
17%
88%
93%
31%
13%
6%
8%
52%
70%
41%
11%
56%
Achievement of Specific Objective
The Specific Objective of the project was: National and district WASH personnel and community
groups are better able to sustain equitable access to, and use of, improved drinking water and
basic sanitation, and adoption of improved hygiene practices. However, the indicators do not
directly measure the ability to sustain equitable access although they would seem to be a better
definition of what the project was trying to achieve. They and the degree to which they were
achieved were as follows:
24
i.
100% of target population in 34 sub-villages and 23 schools have access to improved

water source
The endline survey (with a sample size of about 13%) found that 90,3% of all households in the
43 aldeias visited where water systems were installed had access to improved water sources.
The average time needed to collect water was reduced from 52 minutes to 19 minutes. The report
does not shed any light on why some households do not have access, but it seems from an
analysis of the raw survey data that the overwhelming majority of those without access were
residents of aldeias whose water systems were not functioning at the time. In some aldeias visited
during the final evaluation, examples were reported of households who were excluded from
access due to technical impediments such as distance from the system or elevation of
homesteads above that of the source. Only one of the 13 systems visited (Bahalara-Uain in
Viqueque) was not functioning; as already noted, this system has very particular characteristics
and is not a typical example.
In the case of the schools, the final evaluation found that only seven of the 10 schools visited had
a functioning water supply; two of those without had not been connected yet and one (in BahalaraUain) was connected to a dysfunctional system. The endline survey found only one of the 22
schools visited didnt have water and 16 of the 21 with a source reported that it was very reliable.
Data for targeted schools not visited are not available.
In terms of numbers of people, the target number of beneficiaries of improved community water
supply was 16.400 whereas 18.132 gained access. The school WASH intervention benefited
5.450 against the target of 4.600.
In conclusion, the objective of improving water coverage was not achieved at the time the project
ended, but will be within a few months when those schools that had not been connected yet will
be. Perhaps the target of 100% coverage was too ambitious as there are always likely to be some
households excluded or a few systems temporarily out of order on any particular day.
ii.
4 of 5 targeted District BeSI Committees are meeting quarterly
This objective was not achieved. All project partners who are members of the committees in the
five districts were interviewed and confirmed that they meet only sporadically, and for specific
purposes such as to certify a community ODF, and only when project financial support is
available. In Ermera more progress has been made than in the other districts. Most respondents
reported that bilateral coordination between BeSI Committee members has improved.
One of the bottlenecks of this target is the absence of nationally approved BeSI Committee
Guidelines. UNICEF in coordination with WASH partners supported the drafting of BeSI
Guidelines. The document went through a number of national level consultation processes before
the final draft. This was submitted to the Secretary of State of Water, Sanitation and Urbanization
in Dec 2013 but it has not been approved yet. However government has demonstrated its
commitment to improve inter-ministry coordination by endorsing and implementing the Draft BeSI
Guidelines in their 2014 High Level Ministerial commitment for Sanitation and Water for All (SWA).
iii.
20% improvement in number of children that wash hands with soap or other hand washing
agent before eating and after defecating
The endline survey found that the proportion of children reported by their mothers to habitually
wash their hands before eating had increased from 82,0% at baseline to 95,8%, an increase of
17%. In the case of handwashing after defecating, there was a reported increase of 422% (from
11,0% to 46,4%). However, self-reporting and reporting from mother are unreliable indicators for
actual handwashing practices because of a number of obvious biases. Self-reporting has been
documented to overestimate actual practice by a factor of 3 to 10.
6.3.3
Contribution to MDGs
25
The project, as with any WASH intervention with a schools component, was expected to
contribute to the following MDGs:
MDG 2: Achieve universal primary education
MDG 4: Reduce child mortality
MDG 7: Ensure environment sustainability (Target 7C: Halve the proportion of people
without access to safe drinking water and basic sanitation)
The contribution made to the first one (MDG 2) was not measured, i.e. attendance rather than
enrolment was measured, and this is reported under Impact. The contribution to MDG 4 was not
measured. The contribution to the last one (MDG 7) was measured by the baseline and endline
surveys, with the following result:
Access to safe drinking water: Increased in aldeias where systems were built from 45,5% to
90,3% - this exceeds the MDG target for Timor-Leste in these communities.
Access to basic sanitation: The increase was from 32,9% to 44,9%, which falls short of the MDG
target. Basic sanitation was defined in the same way as improved sanitation is by the WHOUNICEF Joint Monitoring Programme (JMP), but its interpretation in the field by many of the
enumerators was incorrect. Many latrines with pit covers/slabs made with logs and earth were
incorrectly classified as not having a cover or without a slab (and hence open pit) and thus as
unimproved.
Open defecation was reduced from 58% to zero in 53 of the 63 target communities, but by the
time of the endline survey, which was up to three years later in some cases, the practice of OD
had been taken up again by 20% of households in aldeias previously declared ODF. This figure
varied from 3% to 53% from one sub-district to another (see table 8 in section 6.5.2). The resulting
figure of 73% of households using latrines is probably a more accurate reflection of sanitation
coverage achieved than the 45% registered by using the criterion of improved sanitation. This
imprecision around the definition of basic or improved, i.e. adequate sanitation was the main
shortcoming of the baseline and endline surveys.
6.3.4
Beneficiary perception of benefits from project
The almost universal answer given by community members to the question of what the main
benefit was that they had obtained from the project, was that they now had water close to the
home. In second place the reliability of the water supply was cited by most and only in some
cases improved quality was mentioned.
In relation to increased proximity, many also linked this to the possibility of building a good quality
toilet, in other words that they had only seriously considered building a toilet after the water system
was built.
A number of households are using water from the new systems to irrigate vegetable patches in
their back yards, others to wash coffee beans.
6.3.5
Capacity building
It is notoriously difficult to measure the effectiveness of capacity building, or even of a specific

training course, as so many other factors influence the performance of a trainee after being
trained. This project was not able to clearly define up-front what it intended to achieve in this
regard and used indicators that measure inputs rather than outcomes. This has already been
touched on under Result 1, above.
In practice the impact of the projects capacity building efforts has to be evaluated in terms of
improved performance, which is however not specified in easily measurable terms. From what
could be observed and inferred during this final evaluation, the following was noted:
26
CLTS:
The intensive and repeated training and on-the-job capacity building activities
undertaken to improve the skills of CLTS facilitators, including with the support of Kamal
Kar (the initiator and leading international expert on the approach), had the desired
impact when measured by the results. Other organisations that co-sponsored and
benefited from the training commented very favourably as well.
GMFs:
The positive impact of the post-project refresher training in O&M on the performance
of GMFs was commented on by all project partners. The new SDF Manual, produced
by DNSA with BESIK support and input from UNICEF and other agencies, is a key
enabler in this process.
SDFs:
While the SDFs consider that they have benefited substantially from the training they
received on the new SDF Manual and enhanced CAP process, and are performing
better as a result, their effectiveness seems to be constrained by factors other than
gaps in their knowledge and facilitation skills.
SAS:
Besides improving coordination through regular BeSI Committee meetings, the

required level of performance is not specified or measured consistently. At least all SAS
Directors consider that the project has helped them to carry out their tasks more
effectively. In particular, they have gained knowledge and experience in CLTS.
Schools: The School WASH Guidelines define what teachers are expected to be able to do, but
this is not being monitored at the level of detail required for their performance to be
assessed.
6.3.6
Compliance with norms and standards
The development of norms and standards in Timor-Leste is still at an early stage; in the rural
WASH sector they are limited to standard government design and process guidelines and
standards, which is more than exists in most developing countries, and is one of the very positive
features of the sector, as already noted. The project complied fully with all of these. As a result,
for example, the quality, functionality and durability of the facilities built is very good. Some
examples were photographed and are contained in Appendix 12. The use of the SDF Manual to
ensure a standard approach to implementing community-based water systems, and the
improvement of the manual during the course of the project, also impacted very positively on the
social and institutional development outcomes.
Of significance is the degree of involvement of government officials in all phases of the processes,
from approval of designs and guidelines to training, supervision and quality control.
6.3.7
Risk management
The risk of external factors preventing the project from achieving its Purpose or Specific
Objectives, having achieved the planned outputs, was anticipated in the logframe. The following
assumptions were made in this regard:
i.
ii.
iii.
iv.
v.
vi.
SAS and other authorities remain committed to local ownership

SAS continues to develop capacity at the district level
Sources of conflict over water ownership and/or resources can be managed by dialogue.
SAS maintains commitment to provide follow on support to completed water systems.
MoH and MoE remain collaborative
Local leaders are supportive
The assumptions held and no other factors prevented beneficiaries from accessing the services
provided. With regard to sustainability of access, however, inadequate SAS capacity (assumption
ii above) remains a challenge, as discussed extensively elsewhere. Reversion to OD is also an
issue which depends on government capacity.
27
6.4
Impact
The degree of achievement of the anticipated impact was variable, in terms of the achievement
of the indicators for the Overall Objectives, with the second being achieved and the first not,
although a positive trend was found. The beneficiaries did not comment of these aspects when
mentioning the benefits received from the project, rather they focussed on improved access to
services.
6.4.1
Achievement of Overall Objectives
The Overall Objectives of the project, the respective indicators and the achievements (as
measured by the Endline Survey and KAP Study) were:
a) Children, women and men in target areas achieve better health outcomes
Target:
Reduction of 20% in the number of mothers reporting incidence of

diarrhoea in children under five years age during the last two weeks
preceding the survey.
Achieved:
Reduction of 5% (not statistically significant).
b) Children in target areas are better able to participate in primary education

Target: -
5% increase in attendance rates at target schools
Achieved:
6% and 16% increases in attendance rates of boys and girls respectively.
The Endline Survey was not able to enquire into other factors which might have mitigated against
a greater reduction in the incidence of diarrhoea in spite of the improvement in access to WASH,
which included knowledge regarding the causes and most effective treatment of the condition.
None were obvious. Based on replies received by the evaluator to questions (to a small sample
of householders) regarding what messages they had received from the project, he concludes that
it is possible that the greater awareness created amongst mothers about diarrhoea in children led
them to take more notice of its frequency and thus were better able to recall and report it in the
Endline, compared to the Baseline. This needs to be verified, however.
The increased school attendance rate, especially in the case of girls, is noteworthy. However, it
is not known if any other factors, such as the considerable improvement in the economic
circumstances of the majority of households (as noted in the Endline Survey), also had an impact.
6.4.2
Risk management
No external factors prevented the project from achieving the desired health impact, having
achieved the planned outcomes of enabling access to WASH. The risks that were foreseen were
the following:
i.
ii.
Political conditions remain relatively stable

DNSAS receives sufficient budgetary support at sub-national level from government and
other donors
MoE and MoH continue to grow human resources and skills
iii.
The second and third are related more to sustainability of benefits and are a challenge which
the project took into account in its interactions with government partners.
6.5
Sustainability
An assessment of the sustainability of the benefits delivered by the project involves considering
a number of inter-related factors, the most important of which are:
sustainability/permanence of institutional arrangements
28
enabling policy environment

sense of ownership of institutions tasked with operation and maintenance (O&M)
capacity of institutions tasked with O&M
cost, sources of funding and affordability of O&M and replacement
other enabling factors (e.g. environmental sustainability, gender, human rights).
Due to the very different challenges and enabling factors that exist for the different components
of the project community water supply, household sanitation, hygiene behaviour change and
school WASH the findings of the evaluation in relation to sustainability will be presented per
component, as follows.
6.5.1
Community water supply systems
The institutional arrangements put in place to manage the community water systems built by the
project are the standard arrangements used in Timor-Leste, as stipulated by law, namely the
Decree-Law No. 4/2004 on Water Supply for Public Consumption. This requires a communitybased management group (GMF) to be established to participate in the design of the system,
support its construction and take charge of its O&M afterwards. Local government is responsible
for facilitating and supporting the process, mainly through sub-district facilitators (SDFs) who
report to the District SAS office. Clear procedures have been established for the whole process,
called the Community Action Plan (CAP) process, which are laid out in the SDF Manual. GMFs
are required to be formally registered with SAS, although this has not happened yet anywhere in
the country.
The institutional arrangements are thus solid and permanent. Their sustainability at community
level depends however on government capacity to support them effectively. The project invested
substantially in training and on-the-job capacity building of SDFs and SAS staff in general and the
feedback from trainees was positive. However, the overall assessment of this evaluation is that
the work of the SDFs is under-funded and in most districts inadequately supervised. Most GMFs
feel that they are not adequately supported. The prospects for improvement seem to be good,
however, as considerable work is being done by DNSA, with support from the BESIK programme
in particular, to strengthen the whole system. The report of a recent (September 2013) study of
GMFs undertaken by BESIK for DNSA gives a good indication of the direction that policy and
strategy development work is taking.
The relationship between GMFs and aldeia and suco leadership is mostly very good, which is an
important enabling factor for enhancing the authority of GMFs in their communities (especially
important for tariff collection) and providing support to solve non-technical problems. In all the
FGDs with GMFs and community members, local leaders were present and no obvious tensions
were noted, on the contrary, relations seemed sound. The only two exceptions were in Bahalarauain (already mentioned) where the main problem is precisely the dysfunctional village and suco
leadership, and Asumata (see Table 7, below).
Both the endline survey and the final evaluation (particularly the latter see Table 7, below, and
Appendix 10) found a high level of effectiveness amongst GMFs, with very specific problems as
the causes of dysfunctionality, namely poor leadership in most cases. These problems should be
able to be resolved by SDFs with more resources and supervision.
29
Table 7: Functionality of water supply systems and GMFs visited during final evaluation
Identification
No.
District
Subdistrict
Description
Aldeia/
system
Type
*
Functionality
No.
Level
fami(%)
lies
Fund
(USD)
Most
senior
woman
Ermera
Ermera
Poeana
GP
145
100
435
Treasurer
Ermera
Ermera
Huitaso
GP
57
100
209
Treasurer
Ermera
Ermera
Saramata
GP
80
100
30
Treasurer
Ermera
Railaco
Leborema
GP
67
100
35
Treasurer
Viqueque
Viqueque
BahalaraUain
GP
450
Treasurer
Oecusse
Oesilo
Webaha
GP
102
100
600
member
Oecusse
Nitibe
Baocnana
GP
115
100
123
Treasurer
Oecusse
Nitibe
Bona 1&2
GP
73
100
270
Treasurer
Oecusse
Nitibe
Bona 3
wells
175
100
Treasurer
10
Aileu
Remexio
Samalete 1
GP
27
100
60
Treasurer
11
Aileu
Remexio
Samalete 2
GP
56
100
400
Treasurer
12
Aileu
Remexio
Lerolissa
GP
102
100
Treasurer
13
Aileu
Liquidoe
Asumata
GP
73
100
Treasurer
Comments
Over 2 years old,

no problems
100% collection
rate
Started collecting
recently
Started collecting
recently
No water. GMF
dysfunctional
100% collection
rate
Started collecting
recently
100% collection
rate
No O&M costs, so
no collection
Over 3 years old,
no problems
Over 3 years old,
no problems
Over 3 years old,
no problems
GMF
dysfunctional
* Notes: GP = gravity-fed piped system
Almost all the GMFs visited showed a high level of ownership of their systems and adequate
knowledge of their roles and responsibilities. Most of the systems are functioning well, with
adequate funds for maintenance having been collected by GMFs in 11 of the 13 visited during the
final evaluation and 9 of 12 by the endline survey.
One of the issues raised by many of the GMFs when reflecting on the support received from the
SDFs was that of on-the-job mentorship. They found the post-construction refresher training very
helpful, but in most cases had not been visited by their SDF since then. This is closely linked to
the fact that most of the SDFs virtually only travel to visit those GMFs set up by donor funded
projects if the project covers the costs, whereas this project did not envisage any projectsponsored post-construction monitoring visits. This would suggest that a post-construction O&M
mentorship phase would have been very useful to further strengthen GMFs and might have
avoided some of the problems that beset Bahalara-Uain as well as getting the GMF of Asumata
(the last one in Table 7) to function.
O&M of the systems is facilitated by appropriate technologies, sound designs and good quality
construction in almost all cases, Bahalara-Uain in Viqueque being a notable exception. The fact
that all systems are either gravity-fed piped schemes or wells using a bucket on a rope, or
protected springs, i.e. there is no mechanised pumping, ensures that the O&M costs are very
affordable. The most common tariffs are 25c or 50c per family per month and even though most
GMFs are not able to achieve a 100% collection rate, the amount collected is sufficient to cover
maintenance costs. None of the GMFs interviewed had spent more than one months tariff on
repairs, some of them after 3 years. GMFs are trained to accumulate enough funds to cover a
major repair, none having been required on any system so far.
30
However, replacement costs are not factored into the tariff as the understanding is that this is
governments responsibility, together with O&M support limited to soft issues.
All GMFs have some women members and in almost all cases the treasurer is a woman, but no
cases of female coordinators were found.
One of the secondary issues that the final evaluation sought to understand was water user attitude
to and willingness to pay for higher levels of service, i.e. individual connections. This was
prompted by international experience which shows that providing the level of service that users
really want and are willing to pay for results in better rates of cost recovery and thus more
sustainable systems. It was found, first of all, that virtually all households consider the improved
proximity of water to be the main benefit of the project. Secondly, many households, if not the
majority, at least in some aldeias, have invested in plastic hosepipes, some of them over 100m
long, to have water virtually on tap in their yards. Thirdly, it was found that almost all households
aspire to owning a permanent pour flush toilet, for which having water close by is necessary.
It could be concluded, therefore, that the issue of individual permanent connections is high on the
agenda of most households and GMFs. However, of the 13 systems visited, only one (in Ermera)
has established rules for individual connections, which include a US$25 connection fee, and have
made three connections. Two others have established rules and procedures, but have not made
any connections yet. The others have taken a conservative approach due to their systems not
having enough water to guarantee adequate supply for everyone with individual connections, or
because they prefer the hosepipe system which allows them to maintain visual control over how
much water each household consumes.
In conclusion, it would seem that there is as yet little demand for permanent, individual yard
connections.
6.5.2
Household sanitation
Maintaining open defecation free (ODF) status

Although the main objective of the project in regard to the household sanitation intervention was
to support communities to eliminate open defecation, whilst also achieving total sanitation,
UNICEF and the government partners were fully aware that the main challenge of the CLTS
approach is to achieve sustainability, i.e. to avoid reversion to open defecation (OD) after project
completion. To this end considerable effort was directed at trying to support the health system to
provide the kind of ongoing community level health promotion work needed to sustain the attitude
and behaviour changes effected by the project.
However, the results of the endline survey showed that there was considerable reversion to OD
(as reported by households interviewed and by the absence of any kind of toilet facility), as
summarised in Table 8, below.
The figures in Table 8 are not easy to interpret. While the results for Viqueque are consistent with
the limited progress made there and the fact that no aldeias achieved ODF status, it is difficult to
understand what happened in Manatuto, where all target aldeias were certified ODF in 2012 and
two years later the majority of aldeias (the five in Laclubar) had reverted to the previous situation.
This begs the question of whether the ODF certification process was thorough enough.
In Ermera and Aileu reversion rates were less than 10%, which represents a high level of
sustainability, whereas sustainability in Oecusse was variable. Although the final evaluation
included a much smaller sample of household visits than the endline survey (43 in 10 aldeias vs
601 in 54), it observed a much lower rate of OD (3 out of 37) overall.
31
Table 8: Trends in open defecation in target areas

District
Aileu
Ermera
Oecusse
Manatuto
Viqueque
No. of
aldeias
Baseline (2011)
% OD*
Year ODF
certified**
Endline (2014)
% OD*
Remexio
14
23
2010-12
10
Liquidoe
73
2013-14
Ermera
13
66
2011-12
Railaco
88
2013-14
Nitibe
81
2012-13
42
Oesilo
19
2013
15
Laclubar
53
2012
53
Laclo
82
2012
52
Viqueque
64
none
53
Lacluta
87
none
88
63
58
Sub-district
Total / average
27
Notes: * The figures for no facility/bush/field/pig pen/river have been used, excluding those for pit latrine without slab/open pit (for
reasons explained in section 6.4, above).
** These are the years in which the majority of aldeias were certified; a few were certified in subsequent years in some subdistricts.
Experience elsewhere in the world has shown an association between performance in CLTS,
both in achieving ODF status and in sustaining it, and the quality of facilitation (besides other
sustainability factors). The fact that a different NGO was responsible in each district could explain
the differences, and problems in other areas of performance would bear this out in Viqueque and
Manatuto, but this does not seem to be the explanation for the differences between Ermera and
Aileu on the one hand, and Oecusse on the other. In Oecusse the consultant gained a very
favourable impression of the quality of social facilitation and involvement of GMFs in sanitation
promotion, besides their excellent performance in managing their water systems.
It is clear that there were other factors at play, besides the quality of facilitation, that need to be
studied in more depth than was possible either in the endline survey or in the final evaluation.
Anecdotal evidence gathered by the consultant includes comments by some households that they
built latrines because they were told that their aldeia would only get a water system if it achieved
ODF status. While this could be considered as an intelligent incentive strategy, it could also be
construed as a mild form of blackmail which does not involve a change in consciousness
regarding the negative impacts of OD, and thus offers little guarantee that the owner will invest in
maintenance.
Comments were also heard regarding some NGO staff actually building or helping to build latrines
for the last few households that were holding up the achievement of ODF status in a particular
aldeia. This was driven by the pressure to achieve targets for ODF aldeias each year. This does
not necessarily mean that the households were not adequately motivated, but it might help to
explain why reversion to OD takes place.
By far the most commonly expressed opinion regarding the reasons for reversion to OD was that
it was related to affordability; the poorest households could not afford to maintain their latrines.
However, what the consultant was able to observe did not corroborate this entirely; many poorly
maintained latrines belong to families with much better constructed and maintained homes and
other outbuildings.
32
The main conclusion from these findings is that sustainability of the CLTS intervention was
variable but generally poor and that this will not be resolved until the health system develops
enough to provide ongoing, frequent follow-up at household level to sustain attitude and behaviour
changes. The medium to long term prospects appear to be good as considerable effort is being
placed on developing and implementing the Ministry of Healths new Comprehensive Service
Package for Primary Health Care which includes the kind of promotion activities required.
Movement up the sanitation ladder
One of the most effective ways to enhance the sustainability of CLTS interventions is to
encourage and/or enable and/or assist households to improve the quality or durability of their
latrines, in other words to move up the sanitation ladder. In this way maintenance requirements
are reduced and a better level of service provided, both of which encourage willingness to
continue to use and maintain the facility.
This was another of the aspects of the sanitation intervention that the final evaluation paid some
attention to, by including it in the FGDs with GMFs and household visits and interviews, and
analysing what the project had done to promote sanitation marketing.
Of the 43 homes visited, 16 had upgraded their latrines from basic dry pit latrines to pour flush
units of varying degrees of solidness and durability. Of these, however, eight had received
subsidies of about half the materials to do so (from other sources, including a government pilot
project). Thus, of the total only eight (19%) had upgraded with only their own resources. The
endline survey noted an increase from 33% at baseline to 45% in permanent latrines (virtually all
pour flush with lined pits and solid superstructures) which attests to both the promotion work of
the project and the improved access to water.
However, virtually every householder interviewed said the only barrier to upgrading is the cost,
including that of sand and transport. Many families who have very precariously constructed and
poorly maintained latrines said they would prefer to save for a permanent pour flush unit than
spend more money on maintaining their temporary one. The evaluation confirmed the
importance of the work being done to finalise a policy on incentives for the most vulnerable
households, to be applied once communities have been declared ODF.
The two small pilot sanitation marketing initiatives funded by the project were largely
unsuccessful, seemingly because inadequate research was done into what the demand was, and
into constraints such as the cost of transport to get components to each homestead. In one
community pour flush components made by one of the projects remain unsold almost three years
later, due to the cost of transport and the preference for components that have a smoother finish
(plastic or ceramic) which reduces odours.
6.5.3
Sanitation and hygiene related knowledge and behaviour change
The endline survey assessed the extent to which the project had been able to improve knowledge
and awareness of sanitation and hygiene related issues and influence behaviour. The most
important findings were presented in Table 6, in section 6.3.1, above. Bearing in mind that the
projects interventions had taken place up to three years before the endline survey was
conducted, in some communities, the results also reflect the extent to which the impact was
sustained. Some of the data presented in Table 6 is thus repeated in Table 9, below, to facilitate
discussion. The fourth and fifth items, being reported rather than observed behaviours, probably
also reflect knowledge more than behaviour, like the first three. All five indicate a high level of
retention of messages.
33
Table 9: Sustainability of hygiene related knowledge and behaviour change

Indicator
unsafe water
unsafe food
contact with faeces
cook food well
wash hands with soap
safe disposal of childrens faeces
Knowledge of causes of diarrhoea:

Knowledge of prevention
methods:
Knowledge of treatment methods: provide oral rehyd. salts

Reported washing of hands prior to eating
Reported washing of hands after defecating
Special place set aside with soap & water for HWWS
Baseline
Endline
(2011)
(2014)
55%
58%
17%
13%
6%
8%
88%
93%
31%
52%
70%
41%
11%
56%
4,4% of adults
99% of adults
96% of children
26% of adults
34%
58%
With regard to behaviour change, the last indicator reflects a somewhat disappointing result, but
behaviour change is always more difficult than raising awareness. It is possible, however, that the
enumerators in the endline survey did not consider water storage facilities and bathrooms in the
same room as pour flush latrines as constituting a specific place set aside. The final evaluation
also did not find many facilities only used for hand washing, but all those who had pour flush
latrines said they used the same water used for flushing, for washing hands.
6.5.4
School WASH
Since all of the school water and sanitation facilities are still fairly new those visited by the final
evaluation were between a few months and two years old it was not surprising to find them all
in good working order, with only minor problems of leaking or broken taps at handwashing
facilities or latrine water storage tanks. The consultant was not able to inspect interiors, however,
as already noted.
One of the water supplies installed by the project, but not visited, includes an electric pump to
raise the water from the community water supply which is at a lower elevation, and is reportedly
functioning well. The sustainability of this arrangement will need to be monitored more closely.
GMF of the system supplying the school. The final evaluation also found that most GMFs also
take responsibility for maintenance of the water facilities at the schools. However, the training
provided to teachers on hygiene promotion and keeping facilities clean did not include any related
to maintenance and repair of facilities.
The potential additional maintenance burden that may be caused by poor attention to details in
design and installation has been mentioned in section 6.2, above.
6.6
Equity
As noted in section 4.5 in relation to the methodology of the evaluation, care was taken to integrate
human rights and gender equality considerations in all aspects. As far as possible, within the
constraints of community convened meetings, the consultant tried to ensure that key informants
34
included both genders. From the human rights aspect, it was possible to ensure that both duty
bearers (government, service providers) and rights holder (beneficiaries) were interviewed.
As noted in the terms of reference of the evaluation, the project was not designed on the basis of
a gender differentiated analysis of the needs of beneficiaries, nor did it define specific outcomes
aimed at greater gender equality. For an intervention involving the provision of gender neutral
public services to whole communities this is not necessarily a serious shortcoming. During the
course of the project the CAP process guidelines, in the form of the SDF Manual, were updated
and improved by DNSA, with input from the project and other stakeholders, and these do include
a focus on gender and other equity-related issues.
In practice, the project did try and promote greater representation of women on GMFs, with some
success. As noted previously, all but one of the GMFs visited by the final evaluation had a female
treasurer and all had at least 2 and mostly 3 women members (out of a total of between 5 and 8
members), although none had a woman as coordinator/leader. An obvious benefit for women in
particular from any water supply intervention is the reduction in workload and time to fetch water;
the extensive use of plastic hosepipes from standpipes to homes is an additional advantage. The
greater improvement in school attendance of girls compared to boys (as reported above) was
noteworthy, although further analysis is required to understand all the influencing factors.
Provision was made to improve the inclusion of physically disabled persons through ensuring
wheelchair accessibility to school latrines and water system standpipes. Some GMFs reported
making special provision to site standpipes closer to the homes of disabled people. However, no
provision was made in school toilets for menstruating girls.
6.7
Implications for the UNICEF Country Programme
Based on the consultants analysis of sector documents, interviews with a wide range of
stakeholders and field observations, it can be concluded that UNICEFs WASH strategy, as
defined in the new Country Programme Document, is appropriate to the needs of the sector and
UNICEFs comparative advantages, and covers all aspects comprehensively.
In terms of priorities, it seems that the following could benefit from more attention than is being
given at present:
As part of component (c) of the general strategy (strengthening systems at institutional

level to enhance social services), more could be done to provide generic capacity
building and skills development in government, particularly in District SAS and the suco
level PHC system.
As part of component (d) of the general strategy (strengthening partnerships, particularly
those at community level, for improving use of and demand for quality services), more is
required to improve demand, for example by promoting registration of GMFs and the
formation of associations of GMFs, to create upward pressure on government to improve
its performance.
In general, as stipulated in the strategy, it is necessary to apply a gender lens more
rigorously in all interventions. There appears to exist considerable potential to make
more impact in this matter.
35
7.
Summary of Conclusions
7.1
General
The project was appropriately conceptualised and designed for the country context and stage of
development of the sector, was well planned and was mostly well executed. It made use of a nocost time extension to complete the school WASH component while at the same time using
unspent funds to surpass its targets for water and sanitation coverage and the number of
beneficiaries, in the other components, and to consolidate operation and maintenance
arrangements.
The MDG target for increased water coverage was achieved, being raised from 45,5% to 90,3%
in the target communities, benefiting over 18.000 people (11% more than the original target).
84% of the communities in which CLTS was implemented were declared ODF, thus providing a
sanitary environment for over 20.000 people. By the end of the project 20% of households, on
average, had reverted to OD, but in the majority of aldeias the figure was less than 10%. Assuming
the households that had not reverted (73%) were all using facilities that meet the JMP criteria for
improved sanitation, the MDG target of 55% was achieved in 70% of aldeias and in six of the
eight sub-districts.
Through the WASH interventions in 28 schools (22% more than planned), more than 5.000
children benefited from improved access to water, sanitation and hygiene information.
In terms of the projects impact on institutional strengthening and capacity building, it was not able
to clearly define up-front what it intended to achieve in this regard and used indicators that
measure inputs rather than outcomes. In practice the impact has to be evaluated in terms of
improved performance. In relation to the government partners, while good progress has been
made in creating an enabling policy environment at national level, the project partners at district
and sub-district levels are not yet able to provide effective leadership without project support.
There is inadequate funding and other resources for SAS and the Ministry of Health to provide
the leadership needed to scale up water supply and CLTS programmes, and to ensure
sustainability, particularly of CLTS.
At community level the project made a significant impact on community capacity to operate and
maintain their water systems, with good prospects for sustainability, although government
capacity to provide the necessary support is questionable.
In comparison with similar interventions elsewhere, the project was cost effective. The average
cost per beneficiary was approximately 114 Euros. The proportion of project implementation costs
and overheads (UNICEFs total costs) to total expenditure was around 22%, also a cost effective
ratio.
with organised target communities, achieved positive results in this project. However, in terms of
the quality of the relationship between the NGOs and UNICEF as partners, the NGOs perceive
the relationship to be purely one of sub-contractors in which they are not genuinely involved in
joint decision making. It seems, however, that this is more likely the result of the inevitably unequal
power relations and their reluctance to express concerns more forthrightly.
7.2
The project was very effective in this component and amply exceeded its output targets in all
aspects. In addition, 24 point sources (protected springs and shallow wells), not originally
anticipated, were built where gravity-fed piped systems were not feasible. The 43 GMFs formed
also received additional refresher training in O&M after completion of their systems, and GMFs
36
were formed per aldeia for some of the point sources. The objective of improving water coverage
was achieved, including for schools, if allowance is made for those schools that had not been
connected yet but would be within a few months.
training provided was generally of a very high quality. An important exception was the water
system in Bahalara-Uain, Viqueque district, which had to be completed by DNSA-SAS due to the
poor performance of the partner NGO which was unable to resolve the unusually difficult social
and institutional problems at community level. Unless an appropriately funded and facilitated
rehabilitation intervention is undertaken on this system it will not be sustainable.
The effectiveness of GMFs was found to be good in most cases, with very specific problems as
the causes of dysfunctionality, namely poor leadership. Almost all the GMFs visited showed a
high level of ownership of their systems and adequate knowledge of their roles and
responsibilities. The problems noted should be able to be resolved by SDFs but it is clear that in
general they are under-funded and/or inadequately supervised in most districts, constituting the
weak link in the GMF system. This is an area that needs ongoing attention and support.
It would seem that there is as yet little demand for permanent, individual yard connections.
7.3
The high level of success in achieving ODF certification, mentioned above, indicates an excellent
level of effectiveness of this element of the CLTS approach and bodes well for the future of the
strategy in Timor-Leste.
With regard to the sustainability of latrine usage, however, it was found to be very variable.
Reversion to OD was between 42% and 53% in just under 30% of aldeias declared ODF
(including all of those in Manatuto), but 10% or less in 62% of them (all of which are situated in
Aileu and Ermera). This weakness will not be resolved until the health system is able to provide
ongoing, frequent follow-up at household level to sustain attitude and behaviour changes.
have been suggested, including the impact of other programmes in the district offering subsidies.
However, it was concluded that this was not a significant factor. In other districts the project had
to contend with the same challenge and managed to do so effectively. It would seem that the
weakness of the NGO in social facilitation and problem solving was the main problem.
The evaluation confirmed the importance of the work being done to finalise a policy on incentives
for the most vulnerable households, to be applied once communities have been declared ODF.
7.4
School WASH
Due to the good quality of design and construction and the relative newness of the facilities, their
level of functionality was high, the main problem being pending water connections or dysfunctional
water systems.
GMF of the system supplying the school. The final evaluation also found that most GMFs also
take responsibility for maintenance of the water facilities at the schools.
37
8.
Lessons learned
8.1
General
The model of institutional arrangements that was used to implement the project was based on a
partnership with government that involved sharing the responsibilities for implementation of
activities between UNICEF (and its NGO partners) and different government departments, both
at national and district levels, while UNICEF retained the responsibility for accountability to
donors. This model is appropriate and necessary to acknowledge the responsibility of government
for delivery of basic social services and to help build capacity to do so, but carries with it the risk
of UNICEF being accountable for some actions over which it does not have full control.
In general UNICEF seems to have managed this challenge well, but a number of lessons emerge
that could inform future projects, notably:
There is a tendency amongst some government officials to insist on more involvement

and a higher profile, but in practice to take a back seat arguing a lack of resources. This
was observed during the evaluation. A clearer definition of respective responsibilities,
documented in joint work plans, with effective monitoring and evaluation, could help
overcome this difficulty.
In the case of the poor performance of the NGO in Viqueque, the decision was taken not
to renew their contract and for SAS to complete the Bahalara-Uain water system, because
no alternative NGO could be found with the required social facilitation skills, local
knowledge and availability of staff, according to UNICEF. The result was a poor product
in terms of both quality of construction and social facilitation. It is likely that if UNICEF had
terminated the contract earlier, and involved other facilitators, even for a limited period, to
assist SAS, a better result could have been achieved. The lesson learned is that, although
implementing agencies have an obligation to try to assist NGOs to improve their
performance, there are limits (UNICEF tried for three years to improve the performance of
the NGO, to no avail) and the priority should be the wellbeing of the beneficiary community
(which, in this case, has not yet benefited from the project), rather than the NGO.
A number of key outputs of the project involved approval of policies and guidelines by
government at very senior level, beyond not only the control but also the influence of the
project. In future it would be advisable to define deliverables in terms of submission rather
than approval. The case of the BeSI Committee Guidelines is the one that has affected
the project most negatively, as it is also linked to formalising the committees and accessing
budgets for them to function effectively.
A related lesson is that training and other capacity building activities on their own have a limited
impact on the performance of government officials if the other factors that constrain performance
are not also addressed. In the case of SDFs and community or suco level health workers, for
example, the main constraints seem to be limited operating budgets and in some districts
inadequate supervision.
A positive lesson learned was the validation of the model of implementing with local NGOs. Their
comparative advantage in managing community based construction of services that have to be
managed by communities afterwards, was reflected in the high level of ownership and
effectiveness of GMFs. One obvious proviso is that the competency of the NGO, as with any
service provider, needs to be assured. The project did not manage this in Viqueque and Manatuto,
where project performance was poorest as a result, and took too long to terminate both contracts.
While it is acknowledged that there were limited alternatives available to UNICEF, or possibly
none, according to UNICEF, it remains a lesson for the future to improve the assessment of NGO
capacity up-front and on an ongoing basis.
38
The NGO model is also cost effective, in relation to commercial contractors. This is especially the
case for very small jobs in remote locations which contractors are reluctant to take on at affordable
cost.
One of the requirements for getting full value from using the NGO model, however, is to assume
some responsibility for enhancing the capacity of the NGO through general training and even
support for diversifying funding sources to improve their sustainability.
8.2
Since this component involved implementing a well-established model and process, the lessons
learned were not so much related to innovation as they were to validating approaches.
The project confirmed the value of employing standard design guidelines based on appropriate
technologies and solid, durable infrastructure, to minimise the complexity of operation and the
maintenance burden and thus enhance sustainability.
A number of approaches were validated, notably:
Implementing through local NGOs, as noted above, has important advantages.

Community management with GMFs, when done correctly, is very effective. Participation
and contribution by communities during design and construction enhances ownership,
willingness to fund O&M and skills for O&M.
The improved CAP process and SDF manual had a positive impact on the training and
performance of GMFs.
In particular, the post-construction refresher training provided to GMFs improved not only
their performance, but the skills and performance of the SDFs, in most cases.
Where GMFs are partially or totally dysfunctional they need external support to resolve the
situation, especially to solve non-technical problems, such as in Bahalara-Uain and Asumata
(Aileu). In principle the SDFs should be able to provide this support, but are clearly not doing so,
at least not in these two cases. Both situations could probably have been avoided if the project
had made budgetary provision for a post-construction mentorship period until the end of the
project, during which the NGO supported the SDF to monitor GMF performance and solve
problems.
The use of plastic hosepipes from tap stands in lieu of permanent private connections seems to
be an appropriate arrangement and should be encouraged as it reduces the workload of women
and girls in particular and facilitates greater use of water and resulting improved hygiene. The
practice appears to be well managed by GMFs, but it would be advisable to include in the training
of GMFs some guidance on regulations on the use of hosepipes, for example to prohibit
permanent connections and limiting use only to filling storage containers.
8.3
The CLTS approach worked well in this project in most districts and in all but two sub-districts,
thanks to a large extent to the intensive, repeated training of trainers and facilitators. Much has
been and continues to be learned by UNICEF and other stakeholders in the sector about applying
the methodology, the key challenge being sustainability, i.e. reversion to OD.
Amongst the lessons learned, some of them obvious, were the following:
The effectiveness of CLTS depends very heavily on the skills of the facilitators. Not
everyone can be a good facilitator as it depends on personality and sensitivity as well. In
general, NGO staff tend to be better suited than government officials or private sector
consultants.
39
The quality of facilitation needs to be monitored and retraining provided if necessary, or

ineffective staff replaced.
Facilitators also need to be able to adapt their messages and approaches in accordance
with local conditions. The standard shock and disgust triggering strategy may not work
in every community. Recruiting facilitators from the same area is essential.
The challenge of sustainability will not be solved unless and until the health system is
strengthened to provide ongoing monitoring and promotion at household level. For this much
more funding is needed. Funding is also needed to scale up CLTS at present very little is being
done by government outside of donor funded projects.
A key lesson learned was in relation to the design and implementation of the baseline and endline
surveys, in regard to the definition of improved sanitation. One of the categories used for
unimproved sanitation was pit latrine without slab. It is difficult to imagine what this definition is
supposed to mean, since a pit without a slab is not useable at all. One of the categories for
improved sanitation was pit latrine with slab (concrete, wood/bamboo). In practice, due to their
lack of experience, what many of the enumerators did was to define any latrine with a slab of
wood or bamboo covered by earth, but with the wood or bamboo not visible, as a pit latrine
without slab and thus as unimproved sanitation. As a result the number of households classified
as having adequate sanitation was some 28% lower than actual.
It should be recalled that the JMP definition of an acceptable pit latrine is as follows:
" a dry pit latrine whereby the pit is fully covered by a slab or platform that is
fitted either with a squatting hole or seat. The platform should be solid and can
be made of any type of material (concrete, logs with earth or mud, cement, etc.)
as long as it adequately covers the pit without exposing the pit content other
than through the squatting hole or seat."
No mention is made of pit lining or roof or squatting hole cover. All basic dry pit latrines observed
in the final evaluation met this definition, and presumably in the endline survey as well.
Another lesson learned was that the main obstacle to moving up the sanitation ladder and
enhancing sustainability is the cost of materials and transport. For many households the
prospects of ever being able to build a sustainable latrine are remote without some support in the
form of a subsidy. In particular, due to soil conditions in most areas of the country, unlined pits do
not last more than a year or two, in most cases. The work being done currently to agree on an
incentive policy to be applied for the most vulnerable once a community has been declared ODF,
needs to be finalised and the policy applied.
8.4
School WASH
The project confirmed once again the obvious advantages for sustainability of employing
appropriately simple technologies and standard designs, and investing in solid, durable
infrastructure.
The major delays experienced by the project in completing the school WASH interventions on
time would suggest that the use of commercial contractors is not necessarily the best solution in
every situation. A combination of commercial contractors where good ones were available locally,
and NGOs elsewhere, might have been a more effective strategy and would have provided a
useful comparison of the pros and cons of the different approaches.
Involving GMFs directly in the maintenance of school water infrastructure appears to be a valid
strategy to enhance its sustainability.
40
9.
Recommendations
The recommendations that follow are based on the main conclusions and lessons learnt, as
described above. They were finalised after a process of feedback and consultation that included
debriefing meetings with UNICEF senior management, EU Managers, some of the partners and
other stakeholders, besides various iterations of the final report.
For UNICEF:
1. While the implementation model used in the project of partnership between UNICEF and
government is sound, it could be further strengthened with a clearer definition of governments

responsibilities and tasks, especially those not funded by the project, documented in joint work
plans, with effective joint monitoring and evaluation.
2.
In order to strengthen and enhance the sustainability of the generally effective model of
implementing in partnership with local NGOs, the following should be considered:
More responsibility should be assumed for enhancing the organisational capacity of the
NGOs through general training (e.g. project and financial management, strategic planning)
and even support for diversifying funding sources to improve their sustainability. Use could
be made of the experience of INGOs in this approach.
Strengthen the spirit of partnership with NGOs through being more sensitive to their
relatively weaker position in the relationship and hesitance to express concerns,
especially around matters relating to non-negotiable contractual provisions.
3.
Do regular refresher training of NGO, UNICEF and SAS technical staff on monitoring of
construction to ensure attention to detail for sustainability, with the use of checklists.
4. In future baseline and endline surveys, ensure that the definition of improved sanitation is
correctly defined and thoroughly understood and applied by all enumerators.

5. In order to avoid making commitments over which a project has no control, deliverables such
as policies and guidelines that rely on high level government approval should be specified in
terms of submission for approval rather than actual approval.
For government:
6.
DGAS, Health, and Education: Accelerate the process of taking over more responsibilities as
a service provider and funder, with the donor as a supporting partner, increasing own
budgets, especially for scaling up CLTS and supporting the O&M of water and sanitation.
The challenge of sustainability of CLTS will not be solved unless and until the health system
is strengthened to provide ongoing monitoring and promotion at household level.
7.
DGAS and Health: Undertake a review of experience to date in implementing CLTS to obtain
greater understanding of factors at play in the successes and shortcomings: what motivates
people, how to improve community leadership, reasons for reversion to OD, potential for
involvement of local entrepreneurs, etc.
8.
DGAS and Health: Develop a mechanism, possibly through the BeSI Committees, to improve
formal coordination between Government and Implementing NGOs (both local and
international) to strengthen their technical capacity to adhere to government design and
process guidelines and policies, so that their relative strengths in community outreach and
facilitation can be optimised.
9.
DGAS: Include in the CAP process a post-construction mentorship period to enable SDFs to
monitor GMF performance, provide further on-the-job training and solve problems until the
end of the project (with donor support in the case of donor funded projects).
10. DGAS: Lobby for finalisation and approval of i) the latrine construction incentive policy for the
most vulnerable once a community has been declared ODF; and ii) the BeSI Committee
Guidelines and respective operating budget.
41
11. DGAS: Include as a standard responsibility of GMFs the maintenance of school water
infrastructure.
12. Education: Approve the School WASH Guidelines and include in the training package a
module on technical maintenance of facilities, prioritising the setting up of a functional system
that includes GMFs.
13. Education: In future school WASH infrastructure projects:
consider a combination of commercial contractors where good ones are available locally,
and NGOs elsewhere;
change the specification water tanks at schools to HDPE tanks to ensure greater
sustainability.
14. DGAS: Encourage the use of plastic hosepipes from tap stands where permanent private
connections are not feasible, and include in the training of GMFs some guidance on
regulating this practice, e.g. to prohibit permanent connections.
15. DNSA and Health: Decentralise water quality testing to facilitate regular monitoring of sources
susceptible to contamination.
16. Health: Include in the M&E system monitoring of moving up the sanitation ladder, i.e. of the
degree of permanence or level of service, of latrines.

For the EU
17. Clarify the interpretation of the auditors regarding the documentary evidence required to justify
apportionment of staff costs to projects.

18. Prioritise WASH sector funding for software issues related to meeting the policy and strategy
implementation gaps in rural water and sanitation and scaling up sanitation to complement
government efforts to address the high levels of under-five malnutrition.
42
APPENDICES
Appendix 1: Terms of Reference of Evaluation
Appendix 2: Logical Framework of Project
Appendix 3: List of target aldeias and interventions in each
Appendix 4: List of schools targeted for WASH interventions
Appendix 5: Evaluation Questions and Data Collection Methodology
Appendix 6: Work Programme
Appendix 7: Documents Reviewed
Appendix 8: List of meetings and persons interviewed
Appendix 9: Data collection instruments
Appendix 12: Photographs (separate file)
Appendix 1: Terms of Reference of Evaluation
CONSULTANCY NOTICE
UNICEF Timor-Leste is looking for a qualified international consultant to conduct evaluation of EUUNICEF jointly funded WASH project in Timor-Leste.
1. Background
Timor-Leste is the second youngest nation in the world. With its restoration of independence in 2002, TimorLeste was in a state of ruins with tattered infrastructure and a ceased economy. The country has faced
numerous political, security and development upheavals since then. However, the nation has made steady
progress in peace and democracy with two successful parliamentary elections in 2008 and 2012. The
demography of Timor-Leste is predominantly covered by young population with nearly half of the population
below 18 years . Along with the peace and democracy, the situation of children and women has gradually
improved in Timor-Leste. According to information from the Census 2010, the under-five mortality stands
at 64 per 1000 live births vis a vis its Millennium Development Goal (MDG) target of 96; and the primary
net enrolment rate (grade 1-6) is at 93.6, against the MDG target of 100.
However there are a number of on-going challenges. The most recent WHO-UNICEF Joint Monitoring
Programme (JMP) data shows an overall increase in the coverage of drinking water by 16 percentage
points from 2000 to 2012, while the progress in sanitation has been very slow in rural areas. According to
the JMP, 61% of the rural population use improved water sources compared to only 27% accessing
improved sanitation facilities. Poor access to water, sanitation and hygiene is one of the main causes for
diarrheal diseases in under five children, also contributing to the very high rates of under-five malnutrition
(50.2% stunting and 11% wasting). There are also major concerns regarding the WASH situation in schools.
About 54% of primary schools are without access to improved water supply and 35% have no latrines in
the compound and only 12% school have regular supply of soap which clearly indicates the lacks of basic
enabling environment for the school children to practice safe hygiene behaviors even though life skill
lessons related to sanitation and personal hygiene are covered in the classes.
Under the above circumstance The European Union (EU) and UNICEF Timor Leste agreed to implement
the program Improving Access to Water, Sanitation and Hygiene (WASH) in Rural School and
Communities through Capacity Development in June 2011. This Action targeted 55 sub-villages (aldeias)
in five districts (see the Annex 1 and 2 for more details) and has been implemented in collaboration with
national NGO partners and selected contractors by UNICEF and Government and the WASH sector line
ministries namely Ministry of Public Works, Ministry of Health, and Ministry of Education. This project,
covering 45 months (including 9 months no-cost extension), received funding support of 1.5 million from
EU and 1 million from UNICEF coming from different source including National Committees, DFAT and
Thematic Fund. The Project Intervention Logic is described as below.
2. Purposes of the evaluation
This evaluation will inform the results of the project based on five criteria (relevance, effectiveness,
efficiency, sustainability and impact) recommended by OECD-DAC. This type of external evaluation at the
end of the intervention is expected to contribute to both strengthening accountability of UNICEF for its donor
and key stakeholders including beneficiaries, and to learn from this experience to inform future WASH
projects. This end of the project evaluation was planned from the beginning as part of the EU funding
requirements. More specifically, the key objectives of the evaluation are:
To assess the relevance, effectiveness, efficiency, sustainability and impact of the UNICEF-EU
jointly supported WASH Action (for detail see section 3)
To assess the extent to which the project has contributed to broader development results at the
sub-national level, and lessons learnt that will allow the replication and scaling up of the
interventions.
To draw operational recommendations and lessons learned for further improvement and
enhancement of relevant sector policies, plans, strategies through analyses of the factors
contributing to the success or failure of the project.
To provide strategic guidance to UNICEF Timor-Leste in determining its focus areas of support to
relevant ministries in WASH under the current country programme (2015-2019) in addressing
critical child health issues
3. Evaluation criteria and questions

The evaluation criteria are mainly for five areas recommended by the OECD-DAC. Below are some
examples of the questions per criteria. International consultants for this evaluation are recommended to
elaborate the list of questions during the submission of their technical proposals, which will be used for the
selection process by UNICEF. The list of evaluation questions will be discussed and finalized with UNICEF
and the technical committee during the inception phase, before submission of the inception report. The final
list of questions will have to be realistic given the budget and timeframe previously proposed by the
evaluation team. The impact will be assessed mainly by using both qualitative and quantitative
methodologies by taking into account the perception of beneficiaries and stakeholders.
Relevance
Efficiency
Effectiveness
Sustainability
Impact
Did the intervention respond to the needs of the various target groups?
Was the logical framework coherent enough to achieve the results?
Did the operation support the vision of the government and was it in line with
existing policy?
Did the project contributed in national policy development and reviews?
Were resources utilized and managed in efficient manner?
How well did the operation co-ordinate with other, similar interventions (if any) for
synergy and in order to avoid overlaps?
Would it have been possible to achieve the same results at a lower cost?
Were the outputs delivered as planned and in a coherent manner?
What was the level of quality and compliance of activities to the norms and
standards previously established by the program, both for the construction and
social intermediation components?
What is the likelihood that the operation and maintenance and repair of the
installed facilities being financed at the local and sub-national level for continuity of
the services (access to clean water etc.) after the end of the project?
To what degree did the project consider the existing structure or resources to
enhance the sustainability after the end of the intervention?
What could the project have done different to improve sustainability?
What were the main impacts (positive/negative, expected/unexpected) as
perceived by the different actors and beneficiaries of the project?
Other areas of project specific concern

Was the project aligned with UNICEFs equity agenda in addressing the needs of
Equity
the target groups (i.e.to what extent the initiative reached different groups including
the most marginalized)?
Did the project contribute to equitable participation and benefits to various groups
(men, women, children and differently abled people)?
4. Scope of Evaluation
The evaluation will focus on the following:
Intervention, viz. the project on Improving Access to Water, Sanitation and Hygiene (WASH) in Rural
School and Communities through Capacity Development in the selected target communities.
Geographical scope: 5 districts (Aileu, Ermera, Manatuto, Oecusse and Viqueque ) in Timor-Leste
Implementation period of 45 months (including 9 months no-cost extension) from June 2011 to March
2015
5. Methodology
A mixed-method approach is required for this evaluation including the systematic use of qualitative (e.g,
structured interviews and focus groups) and quantitative (e.g. recent survey results, existing routine data)
methods. The overall methodology will be participatory and involve various social groups including the most
marginalized to capture their opinion. An initial proposal for a more detailed methodology is to be submitted
by the applicant (individual consultant) at the time of submission of the technical proposal which will be
used as a basis for proposal assessment by UNICEF. Afterwards, the contracted consultant will be
requested to develop a more holistic evaluation plan which must contain a work plan, a detailed description
of a specific methodological approach, a design for the evaluation with a list of questionnaires, and
information collection and analysis methods and tools including sampling plans, as necessary.
The below is general proposition to develop technical proposal for the applicant. There are mainly three
phases in this evaluation exercise as below.
i) Desk review and refinement of methodology
The list of references and documentations will be shared with the selected consultant. In relation to this
project, the baseline and end-line results of a Knowledge, Attitude and Practice (KAP) study (2011 and
2014) are available. These data need to be reviewed and utilized for the evaluation together with other
existing sources (DHS, Joint Monitoring Report, Project progress reports, etc.) as appropriate by
considering the disparities based on geography, gender and other key variables based on the availability
of disaggregated data. To ensure the participatory approach and refine the methodology based on the
context of the country, the consultant will have the opportunity to finalize the methodology during the incountry visit and meeting with key stakeholders. (For more detail, please see the part 6 below).
ii) Data collection, field work (site visit/interviews)
The systematic use of qualitative (e.g, structured interviews and focus groups) and quantitative (e.g. recent
survey results, existing routine data) methods are required to collect the additional evidence during the
country visit. The meeting with key stakeholders will be accompanied by UNICEF national staff during the
field work in Dili, and by the monitoring consultant (female, NOA level) hired by UNICEF during the site visit
outside of Dili based on the needs expressed by the consultant to facilitate the communication in local
language. Due to the nature of this end of the project evaluation with time and budget limitation, it is highly
recommended to identify two to three districts among five for the project site visit and data collection, focus
group discussion in the field based on the desk review analysis. The respondents and methods/tools
(interview, questionnaires, and focus group) need to be strategically selected based on the evaluation
questions.
iii) Analysis and Reporting
The final stage of analysis and reporting will be conducted remotely (Please see section 6 below for more
detail). The results of analysis need to systematically respond to the evaluation questions, and the report
will be written in a reader-friendly manner. Before finalization of the evaluation report, UNICEF will organize
one day workshop, gathering project team, stakeholders, beneficiaries and the evaluation team, to discuss
together recommendations and action plan drawn from the evaluation. This workshop would help ensure
recommendations are appropriate and owned by the project team and stakeholders, and will be facilitated
by UNICEF WASH section, with the international consultant joining through Skype.
6. Limitation of Evaluation:
Gender and Human Rights aspects need to be considered during the evaluation to draw relevant
recommendations as a formative evaluation. However, the project was not designed / planned based on a
gender differentiated analysis of the needs of beneficiaries and does not contain proper outcomes aiming
at greater gender equality in the areas of intervention. Even if project addresses practical aspects of gender
issues by improving access to water which is an activity mainly led by women, it's not sufficient to be
considered as having a gender sensitive approach. For the OECD Gender Marker is rated "0" because they
only consider practical aspects of gender issues without a real consideration of changes/developments on
the strategic aspects of gender issues.
While baseline /end-line KAP surveys were conducted, the data will not provide seasonal variance for some
of the key indicators including diarrhea incidence as this was not a longitudinal study. However, efforts were
made to conduct the end-line and baseline survey during the same season (Nov to Dec).
7. Activities, Task, deliverables and timeframe
The assignment is for 60 working days over the period of 15 March 2015 to 15 June 2015. The consultant
will provide a detailed timetable in its technical proposal, specifying the distribution of tasks and duration to
complete each task. The proposed sequencing in the table below is an indicative proposal which could be
improved in the technical offer. The right column indicates an estimated duration for the activity.
Activity
In
country
support
Remote
support
Duration
(working
days)
Phase 1: Desk Review From Distance ( Total number of days: 9 )

1. Participate in a briefing session on the assignment with technical
committee (through Skype call)
2. Review and analyse the relevant documents, reports, materials
3. Finalization of the questions and sub-questions of the evaluation
Phase 2 : Country Visits and Consultations ( Total number of days: 31 )

4. Identification of the evaluation methodology, information/data
collection method for each evaluation question, sampling for
interviews and field visits, and development of data collection, and
data analysis plan
5. Development of detailed planning for the evaluation with support

of WASH section for logistical arrangement (field visit)
6. Draft inception report and submission of inception report
7. Data collection in Dili (interviews & additional document review)
8. Data collection in the areas of intervention. At least 3 districts (one

sub-district/district) to be visited.
13
9. Debriefing meeting in Dili
Phase 3 : Analysis and Reporting ( Total number of days: 20)

10. Processing and analysis of the collected data, and drafting of the
interim report
10
11. Presentation of the interim report to the technical committee for

feedback (written and oral presentation teleconference)
12. Completing the interim report by incorporating feedback from the

technical committee and an action plan proposal prioritized for
UNICEF and other stakeholders.
14. Submission of the Final report and a PowerPoint presentation

incorporating feedback
15. A six page summary for publication in the newsletter or website.
13. Submission of the interim report to the technical committee. The

consultant will present report through a video-conference call for
validation (consultant will be connected through Skype)
Total duration = 60 days

8. End Product/ deliverables
a.
b.
c.
d.
Inception report (activity 1- 6)

PowerPoint debriefing meeting after completion of field activities (activity 7 9)
Interim report (activity 10 13)
Final Report in English and PowerPoint presentation (at the latest two week after the draft version):
Soft copy (1 electronic version with PDF and word/excel) (activity 14)
e. Database of collected / cleaned data ( Excel file preferable)
f. Max 6 page summary for publication in WASH newsletters, websites etc (activity 15)
The final evaluation report should not exceed more than 60 pages (without the annexes) and will include
at least the following:
Executive Summary
Brief description of the program, its context , financial arrangements , areas of intervention,
timing, implementation modalities and actors
Objectives , methodology, timing of evaluation and challenges / limitations of the analysis
Results in terms of relevance , efficiency, effectiveness , impact, sustainability
Analysis , including reflection on gender, human rights
Lessons learned , challenges , conclusions, recommendations , action plan
Annexes including list of the data with maximum disaggregation
The percentage of total remuneration for key deliverable
Activities
Submission of inception report
Timeline
Upon completion of deliverable (a); activity 1 - 6
in section 6.
Payment
20%
Submission of the interim report

Submission of Final evaluation report
and Power Point presentation with
data base of collected data
Upon completion of deliverables (b) and (c);

activity 7 13 in section 6.
Upon completion of deliverables (d), (e) and (f);
activity 14 - 15 in section 6.
30 %
50 %
9. Management
The Chief of Planning, M&E and Social Policy will supervise the consultant, with technical and logistical
support provided by the Chief of WASH of UNICEF Timor-Leste. The UNICEF East Asia and the Pacific
Regional Office (EAPRO), particularly the EAPRO Regional Adviser WASH and the EAPRO Evaluation
Officer, will provide oversight and technical support. For the quality assurance of the key evaluation
exercise, draft inception and final evaluation reports will be shared with the mentioned regional advisors
and with the WASH evaluation specialist in the UNICEF Evaluation Office, to obtain their technical input
and clearance before finalization of the inception and final Evaluation report.
10. Qualifications and experience required:
Training and experience in the field related to water, sanitation and hygiene with academic
qualifications (at least a Masters degree) in civil engineering, mechanical engineering, sanitary
engineering, social sciences or combination of it.
Credible international expert with at least 8 years of professional experience in planning,
implementation, management, monitoring and evaluation of integrated water, sanitation and hygiene
programmes in developing countries.
Proven experience with similar programme evaluations in the WASH sector in developing countries,
preferably for the fields of WASH in school, hygiene behavior change approaches including CLTS
(Community-Led Total Sanitation), and community interventions.
Proven experience for EU project evaluations is an asset.
Ability to work independently.
Good communication and report writing skills in English.
Commitments to deliver the final products in line with the set TOR within the agreed timeline.
The consultant should adhere to UNICEF Evaluation Policy; to UNEGs ethical guidelines for UN
evaluations; to UNEGs code of conduct for evaluation and to UNICEF Reporting Standards which are
enclosed in the advertisement notice.
11. Condition of Work
The consultant will be provided a working desk in the UNICEF office while in Dili. The consultant will use
his /her own laptop for the assignment.
UNICEF will facilitate the arrangement of meetings and workshops with partners as required and will
provide logistic and financial support as necessary
The consultant is expected to undertake field trips to the selected project locations in the target districts
with transportation provided by UNICEF.
Interested candidates are kindly requested to submit the following information:
Letter of interest
Technical Proposal
Proposed budget and fees
CV/P-11 form ( UN Personal History Form)
Please send the submission to dilihr@unicef.org
Closing date for receipt of proposals: COB 3 March 2015.
Only short-listed candidates will be contacted.
UNICEF IS A SMOKE-FREE ENVIRONMENT
Appendix 2: LOGICAL FRAMEWORK OF PROJECT

Objective
Intervention logic
Objectively verifiable indicators

of achievement
Sources and means of

verification
Overall
Objectives
Children, women and men in

target areas achieve better health
outcomes
i) % of mothers reporting incidence of

diarrhoea in children under five years
age during the last two weeks
preceding the survey (targeted
reduction 20%)
District health data on

prevalence of WASH related
diseases in target areas,
End-line KAP Survey
Children in target areas are better

able to participate in primary
education
ii) 5% increase in attendance rates,

above population increase, across
target schools
MoE attendance data for

target schools and census
data
National and district WASH

personnel and community groups
are better able to sustain
equitable access to, and use of,
improved drinking water and basic
sanitation, and adoption of
improved hygiene practices
i) 100% of target population in 30 subvillages and 23 schools have access

to improved water source
SAS monitoring reports

District WASH Comm.
Minutes
Baseline and endline
surveys
MoH data and reports
Mid-term and final
evaluations
Political conditions remain

relatively stable
DNSAS receives sufficient
budgetary support at subnational level from government
and other donors
MoE and MoH continue to grow
human resources and skills
ER 1: Local authorities,
particularly District WASH
Committees (BESI), are better
able to lead and support WASH
activities in 5 districts, supported
by national authorities, policies
and guidelines
i) All (100%) Sub-district Facilitators in

target areas have received ToT on
Community Action Plan (CAP)
processes and are supporting Water
Supply Management Groups (GMF).
SAS monitoring reports and

SDF assessments
Mid-term and final
evaluations
Documented training
modules and guidelines
SAS and other authorities

remain committed to local
ownership
SAS continues to develop
capacity at the district level
Specific
Objective
Expected
Results
ii) 4 of 5 targeted District WASH

Committees are meeting quarterly
iii) 20% improvement in number of
children that wash hands with soap
or other hand washing agent before
eating and after defecating
ii) 2 national WASH training

modules/guidelines developed and
implemented in target districts
Assumptions
ER 2: Target sub-villages and

primary schools have improved
water sources and are able to
manage and maintain their water
systems
iii) 90% of the targeted 34 Water

Supply Management Groups (GMF)
and estimated 20 school PTAs are
maintaining their systems at project
completion
WSMG and PTA records

surveys
Monitoring reports
Sources of conflict over water

ownership and/or resources can
be managed by dialogue.
SAS maintains commitment to
provide follow on support to
completed water systems.
ER 3: Target sub-villages and

primary schools have improved
sanitation and demonstrate
measurable improvement in
knowledge and attitude relating to
improved hygiene and sanitation
iv) 70% of children can recall three

critical hygiene messages

surveys
MoH records
SAS monitoring reports
MoH and MoE remain

collaborative
Local leaders are supportive
v) 80% of target villages certified by

MoH as ODF by project end (30% by
mid-project)
Appendix 3: List of target aldeias and interventions in each

List of Water Supply and Sanitation Communities - EU WASH Project
SN
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Sub District
Aileu
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Liquidoe
Liquidoe
Liquidoe
Suco
Suco
Tulataqeuo
Tulataqeuo
Tulataqeuo
Tulataqeuo
Acumau
Acumau
Acumau
Fahisoi
Fahisoi
Fahisoi
Faturasa
Faturasa
Faturasa
Faturasa
Acubilitoho
Acubilitoho
Aldeia
Families
Total
Population
Water
Supply
Samalete (2 bairos)
Roluly
Aikurus
Dasilelo
Lerolissa
Fatumenario
Aimerahun
Berliurai
Mautoba
Diruhati
Berelico
Fakulaun
Caitasu
Raimerhei
Hautoho
Acumata
Aitoi
178
81
95
78
102
124
200
76
100
49
63
71
39
66
44
73
25
752
451
211
360
594
897
1265
573
600
353
317
171
279
275
215
338
150
Completed
Completed
no action
no action
Completed
no action
no action
Completed
Completed
Completed
Completed
Completed
Completed
no action
Completed
Completed
Completed
Completed Type of WS
System
2011 2 - WSS
2012 2 - WSS
2012 1 WSS
Oct-13
Oct-13
Feb-15
2013
2013
2013
1
1
1
1
1
2
WSS
WSS
WSS
WSS +1PS
WSS
PS
Feb-15 1 WSS
2013 1 WSS
Dec-14 1 WSS
12 Aldeias
Sanitation
Year of
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
2010
2010
2010
2010
2013 Dec
2012
2012
2012
2013 Dec
2014 Jan
2011
2011
2011
2011
2014 Jan
2013
2014
13 W Systems
Viqueque
18 Viqueque
Bahalara Uain
Welaco
95
467
Completed
Sep-14
No Progress
19 Viqueque
Bahalara Uain
87
427
Completed
Sep-14
20
21
22
23
24
25
26
27
Viqueque
Viqueque
Viqueque
Viqueque
Viqueque
Viqueque
Lacluta
Lacluta
Ermera
Bahalara
Bahalara
Bahalara
Bahalara
Bahalara
Bahalara
Ahic
Ahic
Aidac
Aisahe
Wetalitua
Caninuc
Cailoi/Aidak
Derok Oan
Camelitur
Cai Ua
Halimean/Wecec
30
33
142
60
80
32
37
73
161
168
699
261
429
147
193
381
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Sep-14
Sep-14
Sep-14
Sep-14
Dec-13
Feb-15
Jun-13
Jun-13
No Progress
1 WSS; 2 PS
(Point Source) No Progress
No Progress
in Cailoi
No Progress
No Progress
1 WSS
No Progress
1 WSS
No Progress
1 WSS
No Progress
1 WSS
No Progress
28
29
30
31
32
33
34
25
36
37
38
39
40
41
42
43
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Railaco
Railaco
Railaco
Manatuto
Laclubar
Laclubar
Laclubar
Laclubar
Laclubar
Laclo
Laclo
Laclo
Laclo
Oecusse
Nitibe
Nitibe
Nitibe
Nitibe
Nitibe
Nitibe
Nitibe
Oesilo
Oesilo
Oesilo
Oesilo
Humboe
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Samalete
Samalete
Samalete
Poeana
Huitaso
Hamric Metin
Tatoli
Tajaquina
Coracao De Jesus
Indau Lulic
Saramata
Moris Mesac
Culau Haburas
Lihmo
Railacan
Sinai
Aiurlala
Eraulo
Leborema
161
36
35
95
61
120
11
80
75
33
60
41
19
102
71
77
953
175
231
457
315
652
63
464
437
192
353
320
105
670
452
483
Completed
Completed
no action
Completed
Completed
Completed
Completed
Completed
Completed
Completed
no action
Completed
no action
Completed
Completed
Completed
Funar
Funar
Funar
Funar
Orlalan
Hohorai
Hohorai
Hohorai
Hohorai
Macucurian
Bamatac
Laoudo
Fahilihun
Turilalan
Anicolaun
Sabumata
Miri Huhun
Heatu Ermera
18
52
110
52
86
52
51
66
23
393
253
407
247
845
232
247
142
868
Completed
Completed
Completed
Completed
Completed
Completed
Completed
no action
no action
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Total Completed
Total target
Additional achieved
Uain
Uain
Uain
Uain
Uain
Uain
10 Aldeias
5 W Systems
Feb-13 1 WSS
Dec-13 1 WSS
Dec-13
Dec-13
Dec-13
Dec-13
Dec-13
Dec-13
Feb-14
Nov-14 1 WSS
Nov-14 1 PS
Nov-14 1 WSS
Water supply
Water supply
Water supply
Total completed between June 2014 - March 2015
35
40
33
71
15
69
165
54
13
73
80
0
121
205
175
330
35
245
756
254
35
294
436
0
3 602
3100
502
1 178
19 269
16400
2 869
6 690
Final coverage for CLTS: 63 Aldeias, 4,368 famlies and 23,976 people
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
1
1
1
1
1
1
1
WSS
PS
PS
PS
WSS
WSS
WSS
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
2012
2012
2012
2012
2014
2012
2012
2012
2012
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
2012
2013
2013
2012
2013
2012
2014
2013
2013
2013
2014
4 W System
2012
2014
2014
2012
2012
2013
2015
2013
2013
2013
Dec-14
1 PS
1 WSS
14 PS - well
1 WSS
1 WSS
1 WSS
1 WSS
1 WSS
1 WSS
2 WSS
1 WSS
10 Aldeias
10 W Systems
53
34
19
16
44
34
10
10
aldeia
Aldeias
Aldeias
aldeia
2011
Jul-05
2011
2011
2011
2012
2012
2012
2012
2012
2012
2012
2012
2014
2013
2013
12 W Systems
2013
2013
2013
2013
Dec-14
2013
2013
7 Aldeias
Oamna/Fatubena
Oamna/ Bona (1&2)
Oamna /Bona 3
Fatunababo
Lamasi Foho
Manan/Bitila
Bocnana
Webaha (Toapenu)
Webaha (bieli)
Hoineno
Buqui /Havana
WSS
WSS
WSS
WSS
WSS
WSS
WSS
Dec-12 1 WSS
13 Aldeias
Usi-Taco
Usti-Taco
Usti-Taco
Usi-Taco
Bene-Ufe
Bene-Ufe
Bene-Ufe
Bobometo
Bobometo
Bobometo
Usitacae
1
1
1
1
1
1
1
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
WS, 24 PS
WS
WS & 24
WSS, 3 PS
53 ODF aldeias
34 Aldeia
19 Aldeia
36,99 F/
3 Aldeia
20643 P
ODF: 53 Aldeias, 3,699 famlies and 20,643 people
Appendix 4: List of schools targeted for WASH interventions

No. School nam e
EMIS
District
CODE
Aldeia
EB F N3
Tulataqueo
49
Aileu
Samalete
EB F Buburnaro
41
Aileu
EB Manucasa
Aileu
EB Faturasa
105
EB F Mautoba
EB 1.2 Slau Lara
EB Fahisoi
EB F Aiurlalan/
Samalete 2
EB F Samalete
No of Access to Access to
School im proved im proved
Children w ater
sanitation
Hand
No. of No of
WashToilet disable
ing
cabins cabins
Facility
6
2
4
Toilet
Com plete
Hygiene
prom otion
2013
conducted
2014
conducted
96
Completed
Completed
Samalete
87
Completed
Completed
Hautoho
164
Completed
Completed
Aileu
Faculau
210
Completed
Completed
30
Aileu
Mautoba
86
Completed
Completed
102
Aileu
Bereliurai
213
Complete
Completed
494
Aileu
Tatilisame
213
Completed
Completed
44
Ermera
Aiura Lalan
116
Complete
Completed
2014
conducted
Ermera
Samalete
156
Complete
Completed
2014
conducted
Mar-15 Completed
2013
conducted
2013
conducted
2014
conducted
Mar-15 Completed
10 EB 1.2 Aitura
203
Ermera
Aitura
213
Completed
Completed
2013
conducted
11 EB F Domingos
Martins Bassar
12 EBF Cocoa
30
Ermera
Saramata
96
Completed
Completed
2014
conducted
5301 Ermera
Cocoa
130
Completed
Completed
2015
conducted
13 EBC Fatuquero
243
Ermera
Fatuquero
622
Completed
Completed
2015
conducted
14 EB F Novel da Paz
96 Hituria
70
Ermera
Hatali
138
Completed
Completed
2013
conducted
1187 Ermera
544
Completed
Completed
2013
conducted
16 EB Hatu Ermera
599
113
Completed
Completed
17 EB Rubae
5523 Manatuto Anicolaun
139
Completed
No Plan
18 EB 08 Funar
606
Manatuto Laoudo
212
Completed
Completed
19 EBF Orlalan
601
Manatuto Torilalan
307
Completed
Completed
20 EB F Fatuquenfua
1239 Oecusse Fatuna-Babo
111
Completed
Completed
21 EB F Lamasi
1259 Oecusse Lomose
38
Completed
Completed
22 EB F Bitila
1260 Oecusse Bitila
93
Completed
Completed
23 EB 1.2 Fatubena
859
Oecusse O-Amna
306
Completed
Completed
24 EB Faefnome
1070 Oecusse Havana
228
Completed
Completed
25 EBF Heubenais
1162 Oecusse Saden
15 EB 1 Nunusakari
Nunutali/
Sosoher
Manatuto Hatu-ermera
2014
conducted
Exting
conducted
2013
conducted
2015
conducted
2011
conducted
2011
conducted
2011
conducted
2013
conducted
2015
conducted
92
Completed
Completed
2015
conducted
26 EB F Bona
Oecusse Bona
114
Completed
Completed
2013
conducted
27 EB F Cailaque
Viqueque Babo
144
Completed
Completed
2014
conducted
28 EBC Bahalara-Uain
Viqueque Welaco
517
Complete
Completed
Rehab
2014
conducted
5498
Water - 28 Latrine- 27
110
30
64
Completed by 20 March 2015
28
Appendix 5: Evaluation Questions and Data Collection Methodology

Evaluation Criteria and Questions
Information to be gathered
Sources of information
Data collection tools
- socio economic situation, coverage of

basic services
- beneficiary communities perception
of priority needs
- sector statistics, evidence of

consultation in design
- document review
2.) Did the operation support the vision of the

government and was it in line with existing
policy and priorities?
- sector context: legislation, policies,

development plans, priorities, causes
of low WASH coverage
- sector documents, govt plans and

strategies, etc
- reading s
3.) Did the project support UNICEFs Country

Programme strategy and objectives?
- CP strategy for WASH sector
- UNICEF CP Document
- document review
4.) Did the project support the EUs Country

Strategy and general development objectives?
- priority given to WASH in EUs CSP
- EU/GoT Country Strategy Paper
- document review
5.) Was the logical framework coherent enough

to achieve the results and was it technically
adequate?
- quality of Logframe: coherence,

correct identification of causal
linkages, quality of indicators, etc
- revised logframe
- document review
6.) Was the institutional design and capacity

building strategy appropriate and adequately
resourced?
- selection of partners and role of govt,

their capacity, strategy for building
capacity, budget for this
- project proposal & logframe

- project budget
- reports on capacity of partners
- key informants (govt at all levels,
donors, GMFs)
- document review
- semi-structured interviews
based on pre-prepared
checklists
7.) Were cross-cutting elements such as

appropriate technologies, environmental
protection and gender incorporated into the
design?
- details of intervention strategy and

methodology
- project funding proposal /

contribution agreement and
Logframe
- document review
8.) Did the design include a sustainability

strategy?
- existence of explicit or implicit

sustainability strategy
- project proposal & logframe
- document review
Relevance and Quality of Design

1.) Did the intervention respond to the priority
needs of the various target groups?
- key informant stakeholders at all

levels, especially beneficiaries
checklists
- other project documents
Efficiency of implementation
9.) How well did UNICEF train the implementing

NGOs, monitor their performance and take
corrective steps when necessary?
- content and quality of training, how its

impact was measured
- frequency of reporting and site visits
- measures to correct shortcomings
- training materials, reports

- key informants: implementing NGO
partners, local govt, local leaders,
GMFs, UNICEF
- document review
checklists
10.) How well did the operation co-ordinate with

other, similar interventions (if any) for synergy
and in order to avoid overlaps?
- project coordination mechanisms,

including between districts and with
other projects/programmes
- reports
- minutes of WASH forum meetings
- other programmes
- document review
11.) How well did the project cope with

situations where different subsidy policies
applied by other projects negatively affected
beneficiaries willingness to participate without
receiving materials subsidies?
- challenges encountered with

conflicting subsidy approaches
- perceptions of beneficiaries
- strategies to overcome challenges
- reports
- key informants: donors providing
subsidies (e.g. CVTL), local govt,
local leaders, beneficiaries, NGOs,
UNICEF
- document review
- household visits
12.) Would it have been possible to achieve the

same results at a lower cost?
- expenditure incurred, unit costs for

key inputs (infrastructure,
implementation, overheads)
- comparative costs of other similar
programmes
- budget, financial reports

- experience of evaluator
- document review
13.) Were resources utilized and managed in an

efficient manner, as per work plans and
budgets?
- expenditure against budget and

activity schedule
- quality of financial procedures and
controls
- budget, financial reports

- expenditure verification reports
- activity reports
- key informants: NGOs, UNICEF,
SDFs, GMFs, govt partners, school
coordinators
- document review
- focus group discussions
with GMFs
with other key informants
14.) Were activities carried out as planned or

corrections made timeously to introduce
modifications to solve problems?
- activities planned
- activities undertaken, % completion
- monitoring activities and corrective
measures
- activity reports
coordinators
- document review
with GMFs
15.) Was the M&E Plan well designed and

implemented (including specific quality
assurance activities)?
- details of plan: reporting schedule

and responsibilities, meetings, followup, monitoring of results and impact
indicators
- M&E Plan
- project monitoring reports
- document review
with GMFs
- attendance of WASH
forum meeting
- evidence of implementation

coordinators
- quality of products
- evidence of definition of quality
standards
- inspection of infrastructure
- analysis of documents produced
- Baseline and Endline Surveys and
KAP Studies
GMFs, govt partners
- notes of interviews and

observations
- document review
with GMFs
17.) Were the outputs and outcomes/results

delivered as planned and in a coherent
manner?
- contribution of activities to outcomes

- % achievement of outcomes
- quality of outcomes
- document review
- observation
18.) What was the level of quality and

compliance of outputs and outcomes to the
norms and standards previously established by
the programme, both for the construction and
social intermediation components?
- evidence of definition of quality

standards for outcomes
- quality of outcomes
- project proposal and LF indicators

- project M&E reports
coordinators
- inspection of infrastructure
KAP Studies
19.) Were the outcomes of capacity building at

all levels pre-defined and measured (including
development of CLTS facilitation skills), and
were they achieved?
- evidence of definition of outcomes of

capacity building
- evidence of measuring of impact of
capacity building activities
- project proposal and LF indicators

- project M&E reports
SDFs, GMFs, govt, school heads
- reports by Kamal Kar
- document review
with GMFs
- observation
20.) Did the project achieve its Purpose/Specific

Objective?
- achievement of indicators of
Purpose/Specific Objective

KAP Studies
- key informants
- MoH and SAS reports
- document review
with GMFs
16.) Were the activities and products delivered

of acceptable quality, both for the construction
and social intermediation components?
Effectiveness
- document review
with GMFs
- observation
Sustainability
21.) To what degree did the projects

sustainability strategy achieve its objectives? In
particular, what is the likelihood that the O&M
and repair of the installed facilities being
financed at the local and sub-national level for
continuity of the services (access to clean water
etc.) after the end of the project?
- quality of sustainability strategy

- effectiveness of implementation of
sustainability strategy
- views on likelihood of achieving
sustainability:
- enabling policy and institutional
environment
- ownership & participation
- capacity developed
- financial viability
- sustainability of water supply
schemes, particularly tariff collection
and maintenance
- sustainability of latrines built and
maintenance of ODF status
- sustainability of school WASH
activities
- project documents and reports,

especially the Endline Survey Report
SDFs, GMFs, national and district
govt partners, other similar projects
- observations and interviews at
household level
- analysis of documents
with GMFs, children (if
possible)
- own assessment based
on experience
- sample surveys of
households at each village
visited, using transect
walks, observation of
latrines and other barriers
against disease and brief
discussions with heads of
households, based on
questionnaires
22.) How able is government at different levels

to provide continuity to the activities, especially
CLTS?
- capacity and resources available to

SDFs
- capacity of MoH, especially subdistrict enviro. health staff and comm.
health workers (PSFs)

coordinators, other donors
- Endline Survey
- document review
with GMFs
23.) To what extent have villages that were

declared ODF remained as such?
- current use of latrines and evidence

of OD
- Endline Survey
- key informants: SDFs, GMFs, local
govt partners
- observations and interviews at
household level
- interviews with key
informants
- sample surveys of
24.) How effective has the sanitation marketing

strategy been in enabling the emergence of
local entrepreneurs to support moving up the
sanitation ladder?
- evidence of existence of
entrepreneurs operating independently

SDFs, GMFs, govt partners
- households
- interviews with key

informants
- sample surveys of
25.) To what extent have household who built

latrines subsequently invested further in
improving them?
- evidence of recent upgrades of

latrines
- households
- observation
- sample surveys of
26.) What could the project have done different

to improve sustainability?
- based on analysis of weaknesses in

sustainability, identifications of
possible solutions

SDFs, GMFs, govt partners, other
donors
- consultants own experience

with GMFs
27.) What were the main impacts

(positive/negative, expected/unexpected) as
perceived by the different actors and
beneficiaries of the project? Include
impacts/benefits not directly linked to improved
sources of drinking water.
- the main benefits and impacts in

terms of improved living conditions that
are perceived by beneficiaries and
other actors
- other unplanned or indirect impacts/
benefits
- households
coordinators
- sample surveys of
with GMFs
- observation
28.) Did the project manage the risks of impact

of external factors adequately?
- existence of risk management

strategy
- effectiveness of strategy
- project reports
govt partners
- document review
29.) Did the project achieve its Overall

Objective?
- whether the overall goal was

achieved, in terms of the defined
indicators for the Overall Objective, i.e.
better health outcomes and school
attendance
- Endline Survey and KAP study

- statistics on WASH coverage
- document review
30.) To what extent did the project contribute to

the MDGs?
- link between indicators at Specific

Objective and Overall Objective levels
with MDGs

- statistics on WASH coverage
- document review
- whether equity considerations were

part of criteria for selection of target
communities
- achievement of project in reaching
the most marginalised
- project proposal
- other project reports
govt partners
- document review
Impact
Equity
31.) Was the project aligned with UNICEFs
equity agenda in addressing the needs of the
target groups (i.e. to what extent the initiative
reached different groups including the most
marginalised)?
32.) Did the project contribute to equitable

participation and benefits to various groups
(men, women, children and differently abled
people)?
- who benefited directly and indirectly

from the project, whether intentionally
or not
- who participated and in what way:
decision making, contributing work or
money, receiving a service, etc
- project reports, including Endline

Survey and KAP study
coordinators
- households
- document review
with GMFs
- sample surveys of
Lessons learnt for replication and sector approaches

33.) Which approaches, techniques,
methodologies, tools, technologies, etc were
shown to be particularly effective for achieving
project results, or which should no longer be
used? Focus particularly on CLTS.
Which of the above should be promoted as
examples of good practice to government,
UNICEF and the EU?
- what worked well, what not

- successes and challenges in water
supply
- successes and challenges in CLTS
- other successes and challenges
- good practice that has potential for
replication
- project reports, including Endline

Survey and KAP study
coordinators, other donors
- households
- consultants own experience
- document review
with GMFs
- sample surveys of
34.) What locally used latrine designs should be

promoted in order to ensure that as many
latrines as possible qualify as improved in
terms of the JMP definition?
- as per questions
- tipification of locally used latrine
designs
- project reports and other project

documents, such as case studies
- all key informants interviewed
during the evaluation
- observation
- interviews
- observation
- documentation
- key informants: national govt
partners, other donors
- interviews
Strategic guidance for the UNICEF Country Programme

35.) Identify areas or aspects of the strategy
which should be modified/reinforced or
changed, including an assessment of the
approach to gender mainstreaming.
- gain detailed understanding of the

Country Programme
- obtain views of govt and other
stakeholders on needs and priorities
- understand role of other donors
- analysis of gender approach in the
project and its relation to the overall
WASH strategy
Appendix 6: Work Programme
Date
Location
Time
Activity
April
15
Dili, UNICEF
am
Desk work, admin
3pm
Briefing meeting
16
Dili, UNICEF
am
Planning, arranging meetings, arranging

field trips, drafting Inception Report
4pm
Meeting of Technical Committee
17
Dili, UNICEF
am
Planning, etc
3pm
Meet UNICEF Rep and Dep Rep
18
Dili
19
Dili
20
Dili, UNICEF
21
22
Dili
Dili, UNICEF
Notes
Desk work (Activities 4-6 of ToRs)

Rest
8am
Planning, etc
10am
Meet with NGOs
Briefing and planning
2.30pm
Meet with UNICEF WASH team
Briefing and planning
am
Admin, drafting Inception Report
3pm
Interview Ministry of Health
9am
Meet with UNICEF Education Section
Briefing and interview
Draft Inception Report
23
24
Dili
Dili
3pm
UNDSS briefing
5 pm
Submit draft Inception Report
8.30am
Meet UNICEF Health & Nutrition Section
10.30am
Interview EU
3.30pm
Interview PLAN
8am
Interview BESIK
9am
Interview ETADeP Director
11.15am
Interview WaterAid
2.30pm
Interview CVTL
3.30pm
Interview DGSA
25
Dili
26
Dili
27
Dili
8am
Travel to Viqueque
Viqueque
pm
Confirm meetings for next day
28
Viqueque
8am
Meet with district & SD authorities, SDFs
2pm
Visit Bahalara-Uain Suco, Welaco aldeia
29
Lacluta
8am
Travel to Lacluta, meet SD authorities
11am
Visit to Ahic Suco aborted due to rain
Lacluta
8am
Travel back to Viqueque
Viqueque
9am
Interview Health, GMF Coord Bah-Uain
10.30am
30
Cannot go to Oecusse
Rest / processing interview data

Rest
Dili
Travel back to Dili

PUBLIC HOLIDAY / processing data
Dili
Meeting with Chief: WASH
Dili
Dili
8 am
Travel to Gleno
Ermera
10.00am
Meet with district authorities, NGO
2pm
Visit Poeana aldeia, meet Suco leader
Overnight in Viqueque
Overnight in Viqueque
Overnight in Lacluta
Second draft of Inception Report
Overnight in Gleno
Date
5
6
7
Location
Time
Activity
Ermera
9am
Meet SDF
11am
Visit Huitaso aldeia
pm
Cont. visit to Huitaso aldeia
8am
Visit Saramata aldeia & school
pm
Cont. visit to Saramata aldeia
8am
Visit Fatuquero and Cocoa schools
Ermera
Ermera
Notes
Overnight in Gleno
Overnight in Gleno
10am
Travel back to Dili
Dili
pm
Meetings with UNICEF WASH
Overnight in Dili
Dili
8 am
Travel to Railaco, interview NGO on way
11am
Visit Leborema aldeia & school
5pm
Travel back to Dili
Dili
Rest / processing field data
10
Dili
Rest
11
Dili
8am
Travel to Pante Macassar by road
12
P. Macassar
8 am
Meet with district authorities, SDFs
Oesilo
1pm
Visit Webaha aldeia
Nitibe
8am
Visit Baocnana aldeia
11am
Visit Oamna aldeia, B. Bona 3
12pm
Visit Oamna aldeia, B. Bona 1&2
Overnight in P. Macassar
8am
Meet with NGO
Debriefing and interview
9am
Meet with SDF
10am
Travel back to Dili
9am
Meet UNICEF WASH, Processing data
4pm
Meet Penny Dutton re CLTS
13
14
15
P. Macassar
Dili
16
Dili
Rest / processing field data
17
Dili
Rest
18
Dili
8am
Travel to Aileu
Aileu
10am
Meetings with district authorities & NGO
Remexio
2pm
Travel to Remexio, visit Lerolia aldeia
5.30pm
Travel back to Dili
Dili
7.30am
Travel to Remexio
Laulara
8am
Interview District Health
Remexio
10am
Visit Samalete 2 aldeia + schools
1pm
Visit Samalete 2 aldeia
2.30pm
FGD with both GMFs in town
3.30pm
Travel back to Dili

PUBLIC HOLIDAY / processing data
8.30am
Interview UNICEF Health Section
10am
Attend meeting of National WASH Forum
Made presentation
pm
Processing data, prep of debriefing
Dili
8am
Travel to Liquidoe
Liquidoe
9.30am
Meet SD authorities
10am
Visit Health Centre, interview Meet SDF
12pm
Visit Aumata aldeia
3pm
Travel back to Dili
19
20
Dili
21
Dili
22
Date
Location
Time
Activity
23
Dili
Prepare debriefing
24
Dili
Rest
25
Dili
26
Dili
27
28
By 15
June
Dili
Notes
Prepare debriefing
10.30am
Debrief UNICEF senior management
pm
Prepare general debriefing
10am
Debrief EU
3pm
General Debriefing meeting
Dili
Consultant departs
Home
Deliver First Draft of Final Report to

Technical Committee
Appendix 7: Documents Reviewed

1.
UNICEF-Adapted UNEG Evaluation Reports Standards
2.
UNICEF Country Programme Document 2015-2019
3.
Timor-Leste European Union Country Strategy Paper and National Indicative

Programme for the period 2008-13.
4.
Project Proposal to EU, with budget and Logical Framework
5.
Revised project Logical Framework and monitoring matrix.
6.
Annual Progress Report to EU 2013
7.
Annual Progress Report to EU 2014
8.
Final lists of communities that benefited from water supply and sanitation interventions,
and WASH in schools, as at 20 March 2015. UNICEF.
9.
Community-Led Sanitation and Water in Humboe, Ermera, Timor-Leste: An Appreciative

Inquiry. UNICEF
10.
EU ROM reports 2012 and 2013.
11.
Expenditure Verification Report for Years 1, 2 and 3. Auditors Moore Stephens LLP.
February 2015.
12.
Baseline Study Report
13.
Endline Study and KAP Survey Report
14.
An Assessment of Latrine Use in Timor-Leste. Study undertaken for UNICEF by Belun:

Empowering Communities Together, 2013.
15.
Timor-Leste National Basic Sanitation Policy: 11 January 2012.
16.
BeSI Committee draft Terms of Reference, 2013.
17.
Water, Sanitation and Hygiene (WASH) in Schools: Guidelines for Timor-Leste
18.
Sub-District Facilitator Manual. DNSA, October 2014.
19.
Comprehensive Service Package for Primary Health Care. Ministry of Health (including
Guidelines for Domiciliary Visits; and Domiciliary Visit Register: Village Health Profile),
National Directorate of Public Health. 2015.
20.
Decree-Law No. 4/2004 of 11 February 2004 on Water Supply For Public Consumption
21.
GMF Study: What is the situation of GMFs in Timor-Leste? DNSA-BESIK, December

2013.
22.
Willetts, J. (2012) A service delivery approach for rural water supply in Timor-Leste:
Institutional options and strategy. BESIK, March 2012.
23.
Dwan, P. (2014) Look Back Study: An impact evaluation of Cruz Vermelha de TimorLestes water and sanitation program. Prepared by FH Designs for Australian Red
Cross, December 2014.
24.
Grumbley, A. (2011) Community financing of rural water supplies in Timor-Leste. Paper

presented at 35th WEDC International Conference.
Appendix 8: List of persons interviewed and/or who provided information or

comment in meetings
No. Entity
Position
Name
Date
UNICEF
Chief: WASH Section
Ramesh Bhusal
UNICEF
Chief: Planning, M&E, SP
Toshiko Takahashi
UNICEF
Country Representative
Desiree Jongsma
UNICEF
Deputy Country Representative
Rene van Dongen
UNICEF
WASH Officer
Rodolfo Pereira
UNICEF
Chief: Health Section
Hemlal Sharma
21 May
UNICEF
Chief: Education Section
Takaho Fukami
22 April
UNICEF
Consultant in Oecusse
Olga Corbafo
12-14 May
European Union
Manager: Finance & Contracts
Charles Dubay
23 April
10
European Union
Programme Officer
Johanes Don Bosco
23/04, 27/05
11
European Union
Deputy Head : Finance & Contracts
Nick Heilegger
27 May
12
Ministry of Health
Chief: DSAVS
Tomasia de Sousa
16, 21 April
13
DGAS
Programme Manager
Martinus Nahak
16, 21 April
14
DGAS
Director General
Joo Pereira Jernino
24 April
15
DGAS
Director: DNSB
Joo de Piedade
24 April
16
DGAS
Director: DNSA
Gustavo da Cruz
24 April
17
AMAR
Director
Antonio Silva
20/04, 18/05
18
AMAR
Coordenador
Ronaldo Soares
20/04, 18/05
19
HIM
Director
Gilberto
20/04, 08/05
20
HIM
Coordenador Tecnico
Serafim Pereira
20/04, 08/05
21
ETADeP
Director
Victor Carvalho
20, 24 April
22
ETADeP
Assistant Coordinator
Pedro dos Santos
24/04,14/05
23
ETADeP
CLTS Facilitator
Augusto Varela
14 May
24
PLAN
Country Director
Terence McCaughan
23 April
25
PLAN
Senior WASH Manager
John McGown
23 April
26
BESIK
Behaviour Change Comm. Advisor
Heather Moran
24 April
27
WaterAid
Country Representative
Alex Grumbley
24 April
28
CVTL
Head of Health Department
Joo Pinto Soares
24 April
29
CVTL
WASH Advisor
Stuart Bryan
24 April
30
Indep consultant
Social Marketing & CLTSSpecialist
Penny Dutton
15 May
31
Viqueque District
District Administrator
Jos da Sousa
28 April
32
Viqueque District
Director: SAS
Marito da Costa
28 April
33
Viqueque District
SDF for Viqueque town
Ilfonso Fatima
28 April
34
Viqueque District
Director: Education
Emidio Amaral
28 April
35
Viqueque District
Director: Health
Apolinario Soares
28 April
Numerous
briefing &
follow-up
meetings and
interviews
36
Bahalara-Uain
Representative of Chefe Suco
Claudio Pinto
28 April
37
Bahalara-Uain
Coordinator of GMF, nurse at PS
Evaristo Saviana
30 April
38
Bahalara-Uain
7 ordinary members of GMF
39
Lacluta Sub-district
Administrator
Rui da Costa
29 April
40
SDF
Rogerio Pires
29 April
41
Director of Health Centre
Benedito Soares
29 April
42
Ermera District
Director: SAS
Thomas da Silva
4 May
43
Ermera District
SAS: 4 SDFs
44
Ermera District
Health: Head Envir. Health Dept
Jari Alves
4 May
45
Ermera District
Education: Head School Feeding
Paulo Amaral
4 May
46
Poeana
Chefe Suco
Pedro Exposito
4 May
47
Poeana
Suco official
Paulino Lemos
4 May
48
Poeana
Chefe Aldeia
Toms Martins
4 May
49
Poeana
GMF Coordinator
Jacob de Carvailho
4 May
50
Poeana
GMF Treasurer
Amelia Xavier
4 May
51
Poeana
4 GMF members
52
Huitaso
Chefe Suco
Antnio Santos
5 May
53
Huitaso
Chefe Aldeia
Fernando Soares
5 May
54
Huitaso
6 Chefes of other aldeias in suco
55
Huitaso
GMF Coordinator and PSF
Abilio Salsinha
5 May
56
Huitaso
GMF Treasurer
Georginha Soares
5 May
57
Huitaso
2 GMF members and 2 others
58
Saramata
Chefe Aldeia
Fernando Soares
6 May
59
Saramata
Chefe GMF
Claudio Soares
6 May
60
Saramata
Treasurer
Teresa Martins
6 May
61
Saramata
62
Fatuquero
School Coordinator
Cornelio Nascimento
7 May
63
Leborema
Chefe Aldeia
Armando da Costa
8 May
64
Leborema
Chefe GMF
Antonio Honoratus
8 May
65
Leborema
Treasurer
Maria Aveira
8 May
66
Leborema
67
Oecusse District
Rep of Administrator (DPRD)
Domingos Maniques
12 May
68
Oecusse District
Director: SAS
Gil de Sousa
12 May
69
Oecusse District
Director: Education
Domingos Ramos
12 May
70
Oecusse District
Head Dept Health Education
Gastao Poto
12 May
71
Oecusse District
SAS: SDF
Jose Ili
12 May
72
Webaha
Chefe Aldeia and PSF
Domingos Bobo
12 May
73
Webaha
Chefe Bairo and Treasurer of GMF
Agustinho Nono
12 May
74
Webaha
Chefe GMF
Luis Bobo
12 May
28 April
4 May
4 May
5 May
5 May
6 May
8 May
75
Webaha
12 May
76
Baocnana
Chefe GMF
Julio Calo
13 May
77
Baocnana
Secretary GMF
Jose Aloi
13 May
78
Baocnana
Treasurer GMF
Carmelita dos Santos
13 May
79
Baocnana
8 GMF members
80
Bona
Coordinator of school
Marcelo Tamelab
13 May
81
Bona 1&2
Chefe Aldeia
Serafin Bacs
13 May
82
Bona 1&2
Chefe Bairo
Domingos Calo
13 May
83
Bona 1&2
Chefe GMF
Marcos Tefu
13 May
84
Bona 1&2
Treasurer GMF
Anjela Calo
13 May
85
Bona 1&2
8 GMF members
13 May
86
Bona 3
Chefe, Deputy & Secretary of GMF
13 May
87
Aileu District
Secretary of Administration
Jacinto Mendosa
18 May
88
Aileu District
Director: Education
Isac Sarmento
18 May
89
Aileu District
Director: SAS
Martinho Kamoes
18 May
90
Aileu District
Chefe: Environmental Health
Amaro Belo
18 May
91
Aileu District
SAS: SDF
Cesar de Andrade
18 May
92
Remexio SD
Administrator
Carlos Araujo
18 May
93
Tulataqueo Suco
Chefe Suco
Adolfo Mau
18 May
94
Lerolisa
Tecnico of GMF
Armando da Silva
18 May
95
Lerolisa
Tecnico of GMF
Francisco Benevides
18 May
96
Samalete 1
Chefe GMF
Manuel da Costa
19 May
97
Samalete 1
Assistant Chefe GMF
Albano Dorrego
19 May
98
Samalete 2
Chefe GMF
Martino de Andrade
19 May
99
Samalete 2
Treasurer of GMF
Santina de Jesus
19 May
100
Liquidoe SD
Coordinator of PNDS
Mouzinho Santiago
22 May
101
Liquidoe SD
Chefe of Health Centre
Mariano de Jesus
22 May
102
Liquidoe SD
2 doctors at Health Centre
Barreto, Rangel
22 May
103
Asumata
Tecnico of GMF
Celestino da Costa
22 May
104
Asumata
Health Promoter of GMF
Rossa da Costa
22 May
105
Asumata
7 community members
13 May
22 May
Appendix 9: Data collection instruments
Appendix 9A:
Interview Checklist: Partner Ministries

1. Role and responsibilities of Ministry and Department
2. Role of Dept in WASH in general
3. Role of Dept in this project
4. Nature and quality of partnership:
consultation by UNICEF in planning and prioritisation (including CPD)
resources provided by UNICEF
role of Dept in implementation (contracting, decision making, reporting, etc)
5. Opinion of quality of UNICEFs performance in implementing this project
Quality of infrastructure and compliance with national standards
What does UNICEF do best?
What is UNICEF not so good at?
6. Challenges in WASH
O&M of water
Implementing CLTS and sustaining ODF status
Impact on effectiveness of CLTS of organisations providing subsidies
7. Strategy of government to ensure sustainability

8. Capacity to ensure sustainability
9. Recommendations for UNICEF:
future WASH interventions
WASH in Country Programme
Appendix 9B:
Interview Checklist: Partner NGOs
1. Mission and work of NGO in general

2. Role of NGO in WASH in general
3. Role and responsibilities of NGO in this project (including planning, decision making,
reporting, etc)
4. Main successes and challenges, lessons learnt
5. Description of implementation approach and methodology, including strategy for targeting and
including vulnerable groups
6. Nature and quality of partnership:
consultation by UNICEF in planning and prioritisation
resources provided by UNICEF and contract modalities
capacity building
partner or contractor?
Quality of financial management, procurement, general project management
8. Effectiveness of UNICEFs approach to capacity building of NGO

9. Challenges in WASH sector, especially O&M of water and CLTS
10. What was the sustainability strategy of the project?
11. Govt and community capacity to ensure sustainability
general
Appendix 9C:
Interview Checklist: District Authorities (SAS, Health, Education)

1. Role and responsibilities of each Servio, with special reference to WASH
2. Existence and achievements of District BeSI Committee (if exists, ask to see minutes)
3. Role of each Servio in this project
4. Nature and quality of partnership with UNICEF:
consultation by UNICEF in planning and prioritisation
how were vulnerable groups targeted and reached?
resources provided by UNICEF
joint implementation?
Was the project implemented as planned?
What was the quality of products and outcomes?
6. Challenges in WASH
7. Strategy of government to ensure sustainability
O&M of water systems
Sanitation / CLTS
O&M of school infrastructure
8. Capacity to ensure sustainability (including capacity to support SDFs and GMFs)
Describe budget and other resources
Monitoring system? (ask to see SAS monitoring reports and SDF assessments)
in general
Appendix 9D:
Interview Checklist: SDFs

1. Role and responsibilities, main tasks
2. Role in this project, including start date
3. Resources provided by project
4. Nature of relationship with UNICEF / NGO (direct or through SAS?)
5. Opinion of quality of NGO / UNICEFs performance in implementing this project
Was the project implemented as planned? Is every targeted aldeia ODF? Is there water
for everyone in each targeted aldeia?
What went well? What was the biggest success of the project?
What was the most difficult aspect?
6. Describe strategy to ensure sustainability
O&M of water systems
Sanitation / CLTS
O&M of school infrastructure
Inclusion of vulnerable groups
7. Capacity to ensure sustainability (including capacity to support GMFs)
Describe budget and other resources
Monitoring system?
8. Recommendations for UNICEF (What should be done differently next time?):
in general
Appendix 9E:
Interview Checklist: Donors / INGOs

1. Mission and work of Organisation in general, and in TL
2. Role in WASH and role in sector coordination structures
3. Relationship with UNICEF
4. Main successes and challenges
5. Description of implementation approach and methodology
6. Challenges in WASH sector, especially O&M of water and CLTS
7. Government and community capacity to ensure sustainability
8. Opinion of quality of UNICEFs work:
best way to contribute to WASH sector
to improve future WASH projects
general
Appendix 9F:
Interview Checklist: School Coordinator / Director

Interview (normally informal, while inspecting schools WASH facilities)
1. Position and responsibilities
2. Description of activities undertaken with project support (with approximate dates), and role
of Coordinator / Director
Water supply
Sanitation
Promotion of hygiene behaviour change (role of pupils, their availability to chat)
PTA?
3. Description of O&M arrangements and source(s) of funding, relationship with GMF,

frequency of meetings
4. Relationship with NGO and UNICEF
5. Main successes and challenges
Inspection of WASH facilities
6. Water supply
Reliability (ask)
Last repair, its cost and who paid
State of maintenance and cleanliness
7. Toilets
Ratio of boy and girl pupils per toilet cubicle/stall and whether separated
Whether designed for disabled, menstruating girls, back-up when water fails
State of maintenance and cleanliness, who cleans
Last repair (if any), its cost and who paid
8. (If possible) FGD with children involved in H&H activities
Description of activities, frequency, numbers involved
What did they learn?
What were they able to take home and put into practice there?
Appendix 9G:
FGD Checklist: GMFs

FGD:
1. Name of GMF and number of members, gender, with roles
2. Responsibilities and tasks, frequency of meetings
3. Changes since establishment, process by which these were made, were new members
trained and by whom
4. Description of water system, number of aldeias served, how well it is working
5. If there is a school, what is relationship with PTA
6. Describe sanitation / latrine coverage, OD
7. Have any households moved up the sanitation ladder? Are there any sanitation
entrepreneurs active in the area?
8. Main successes and challenges since project ended
9. Main benefits of water system, latrines, etc (explore indirect benefits as well)
10. Describe tariff collection, show records, state balance in fund
11. Show tools and spare parts (if any)
12. When was last breakdown / repair, describe, what did it cost
13. When last did SDF visit, what support does s/he provide
Inspection:
Inspect water system and note state of maintenance and cleanliness, etc
Appendix 9H: Household Visit and Interview Checklist

Depending on time available (15-20 minutes per homestead), the consultant (accompanied by a UNICEF
staff member acting as a translator) will visit up to 10 homesteads per village, chosen randomly in a transect
walk, approximately every fifth home along route chosen, but only those where head of household or other
adult is present. He will carry out an inspection while chatting to the head of household or other adult family
member. He will also observe in public spaces whether any OD is visible, and the general state of
cleanliness.
Household Visit and Interview Checklist
No.
District: SD: .. Suco: ... Aldeia: .... Date: .........

Name: Status: .... Age: No. in hh: .
Observation:
Type and quality of homestead and buildings:
Description of latrine:
State of use, O&M, cleanliness:
Upgrades?
HWF & soap nearby?
Use?
Sanitary state of yard:

Water storage:
Interview checklist - Sanitation:
When latrine was built and why:
If not used, why not:
If built after project started, what message(s) made most impact in decision to build latrine?
Why do some people not use latrines?
If any upgrades made on toilet, why, how much did they cost
How & frequency of HWWS:
Messages received from project:
Any other questions prompted by observations:
Interview checklist - Water:
Where do they get water from (different sources?) and where did they before?
What benefits are perceived as a result? Any benefits not related to health?
How reliable is water supply, do they pay tariff, how good is GMF, etc

Sheet 1: Data Obtained per Evaluation Question
Data Obtained per Evaluation Question
Source of Information
Evaluation Question
Inspection of water
infrastructure
FGDs with GMFs,

leaders, community
Inspection of latrines &

h/hold interviews
Inspection of school
infrastructure
Interviews with
school staff
District government
partners
SDFs
National government
partners
UNICEF staff
NGO partners
Other CPs and INGOs

active in sector
Analysis of documents
Own comparative
experience
Relevance and Quality of Design of Project

1
Did the intervention respond to the

priority needs of the various target
groups?
Did the operation support the vision of

the government and was it in line with
existing policy and priorities?
Did the project support UNICEFs
Country Programme strategy and
objectives?
Did the project support the EUs
Country Strategy and general
development objectives?
Was the logical framework coherent
enough to achieve the results and was
it technically adequate?
Was the institutional design and

capacity building strategy appropriate
and adequately resourced?
Were cross-cutting elements such as

appropriate technologies,
environmental protection and gender
incorporated into the design?
Did the design include a sustainability
strategy?
Confirmed by all GMFs

and community leaders
interviewed
Confirmed by all households

interviewed
Confirmed by 2
principals interviewed
Confirmed. Needed
more generic training.
Confirmed, including
Result 1.
Need more support for

school WASH
Confirmed by all 3.
Especially school
WASH gets little other
donor support
As per govt's Strategic Development Plan
and WASH sector policies
In line with UNICEF CPD
Consistent with EU CSP
Generally coherent and appropriate as

implementation tool. Some formulation
inadequacies. ROM recommendations to
improve these and indicators were
incorporated into revised version.
Sound design of
community
management
arrangements. Good CB
strategy, notably post
construction refresher
training.
Technologies
appropriate. No erosion
protection.
Guidelines for inclusion

of 30% women on
GMFs adhered to in
most cases
Insufficient to assess
whether needs
adequately addressed
Technologies appropriate.
Good design and CB

Good CB strategy
strategy, except more
generic training needed.
Good design, including

implementation by
NGOs, which is
supported by most
officials. Good CB
Good design and CB

Technologies
appropriate. No erosion
protection.
Sound, as per project proposal, training

plans and reports, training materials (e.g.
SDF Manual)
Appropriate approach to
govt CB, acknowledging
that it is a slow process
Gender not mainstreamed in proposal.

Rights of children promoted.
There was potential for

achieving more in terms
of women's
empowerment.
Strong emphasis on sustainability and

adequately included in all components.
Efficiency
9
Were resources utilized and managed

in an efficient manner, as per work
plans and budgets?
10
Were activities carried out as planned

or corrections made timeously to
introduce modifications to solve
problems?
11
Was the M&E Plan well designed and

implemented (including specific quality
assurance activities)?
12
Were the activities and products

Mostly very good quality Mostly good quality
delivered of acceptable quality, both for infrastructure and
facilitation of CAP
the construction and social
community
process, and efficient
intermediation components?
participation, except in construction phase.
Bahalara-Uain. Details
in Sheet 2.
How well did UNICEF train the
Good quality of
implementing NGOs, monitor their
products reflects good
performance and take corrective steps supervision. Problems
when necessary?
in Manatuto and
Viqueque, steps were
taken.
13
Problems in BahalaraUain due to weakness

of CPT, which misled
with false data in
reports.
Positive assessment of
project implementation
efficiency.
When delay in aquittal

of an advance exceeds
6 months whole
process has to start
anew. This happened a
number of times and
caused delays for
NGOs.
Complain of delays in
processing of tranches
by UNICEF and
problems with financial
system for advances.
Accurate budgeting, expenditure close to

budgeted, other resources also used as
planned. Likely overexpenditure of UNICEF
co-funding (+/- 5%), pending final audit.
Issues raised in Expenditure Verification
report.
Problems in BahalaraUain due to weakness

of CPT, which misled
with false data in
reports. Not removed
soon enough.
Work plans and reports indicate

substantial initial delays that were largely
overcome (except school WASH). Time
extension used to spend budget savings,
exceed targets and improve sustainability.
Well designed. Original indicators
improved. Baseline established. Monitoring
and reporting regular. QA via technical
supervision, audits, Regional oversight, EU
ROM reviews. Main recommendations of
latter implemented.
Generally high quality of

CLTS facilitation and
transmission of messages,
but exceptions noted. See
also Endline Report.
Mostly very good quality Positive assessment of

infrastructure. However, quality of products
was not possible to
inspect interiors in most
schools. Details in
Sheet 4.
Variable quality of CLTS

facilitation.
14
How well did the operation co-ordinate

with other, similar interventions (if any)
for synergy and in order to avoid
overlaps?
15
How well did the project cope with

situations where different subsidy
policies applied by other projects
negatively affected beneficiaries
willingness to participate without
receiving materials subsidies?
Did not appear to be an issue

at community level. But there
was limited time available for
in-depth analysis in
Viqueque, and Manatuto not
visited.
16
Would it have been possible to achieve Solid, durable designs

the same results at a lower cost?
and materials used will
ensure lower full
lifecycle costs.
Zero materials subsidy

approach makes CLTS more
cost effective than other
approaches.
Solid, durable designs

and materials used
(except tanks) will
ensure lower full
lifecycle costs.
High level of achievement of

ODF status in target villages
in 3 of 5 districts.
High level of compliance

with standard designs
and construction quality
norms. 3 schools with
pupil:toilet ratios above
norm. Problems at level
of construction detail
noted in Sheet 4.
quality of products
quality of products
quality of products
performance, except in
Viqueque. However not
true that there was no
progress in sanitation
there; some aldeias
almost achieved ODF
status.
Weak NGO in
Viqueque.
performance
Coordinated with govt to

avoid overlaps in same
SD with other projects.
quality of products
Detailed explanations
given of CB provided
and efforts to correct
poor performance.
Debatable whether
strategy in Viqueque
was best option.
Positive assessment by UNICEF could do more

3 best performing NGOs to strengthen NGOs
of UNICEF CB support
beyond purely projectand supervision.
related performance
Regular communication requirements.
and coordination. Would
have appreciated more
generic CB.
Coordinated with
Detailed information
national and district govt provided, indicating
to avoid geographic
effective cordination.
overlaps. Active in
WASH Forum to
improve synergies and
policy/thematic
coordination.
None of 4 districts
visited raised this as a
significant issue
Was not possible to

interview NGOs that
worked in problem
districts (Viqueque,
Manatuto). Others did
not consider this as a
problem, even though all
3 districts had subsidy
projects in other SDs.
No negative coments on
cost effectiveness.
No negative coments on
cost effectiveness.
effectiveness
effectiveness
Compliance also
monitored by district
SAS - positive
comments.
Compliance also
monitored by DNSA and
MinEd - no negative
comments.
Reports, ROM reviews, Endline Study, etc Generally high quality

generally very positive, but noting variability products and
in quality of social facilitation, also
processes.
technical in Manatuto.
Good quality project documents produced.
Good support to govt in policy work and
good
qualityin
docs
(e.g.
School reports,
WASH).
As
detailed
annual
progress
performance appears to have been good.
Possible exception of Viqueque and
strategy of handing over to SAS.
UNICEF's role in
coordination of sector
Detailed information provided in annual

progress reports, indicating effective
coordination.
BESIK in particular has

had negative
experiences with
subsidy programmes
impacting negatively on
CLTS. Others also see
need to harmonse
approaches.
No documented evidence of impact of

conflicting approaches.
Likely that other factors

were more influential,
such as poor quality
facilitation by NGOs.
No negative coments on No negative coments on Expenditure and progress reports show

Good ratios compared
cost effectiveness.
cost effectiveness.
overall cost at 114 / beneficiary and 22% to other projects in
UNICEF overheads - both reflect good cost Africa.
effectiveness.
Effectiveness
17
Were the outputs and

outcomes/results delivered as planned
and in a coherent manner?
18
What was the level of quality and

compliance of outputs and outcomes
to the norms and standards previously
established by the programme, both for
the construction and social
intermediation components?
19
Were the outcomes of capacity

building at all levels pre-defined and
measured (including development of
CLTS facilitation skills), and were they
achieved?
20
Did the project achieve its

Purpose/Specific Objective?
High level of compliance GMF effectiveness

with standard designs
noted in Sheet 2.
and construction quality
norms. Problems at
level of detail noted in
Sheet 2. Water quality
only tested once, no
ongoing monitoring.
GMF roles, functions

and tasks are defined in
SDF Manual for CAP
process. Outcomes
measured in terms of
effectiveness of
functioning.
From the point of view of From the point of view of
beneficiaries, the
beneficiaries, the project
project achieved its
achieved its purpose.
purpose.
Only 2 interviewed,
Outcomes for govt staff
unable to provide
not clearly defined,
quantitative assessment except for indicator of
of training outcomes.
BeSI Cttees meeting.
Effectiveness of training
on CAP process
measured in terms of
project and GMF
effectiveness.
Excellent effectiveness. Details of

achievement of Results in progress reports
and end-of-project monitoring
spreadsheets.
Design guidelines seen.
Reports include stats on pupils per school.
Otherwise compliance not specifically
mentioned.
Complemented
information in reports,
particularly regarding
CB of govt officials.
Outcomes of teacher training defined in

School WASH Guidelines, that for SDFs in
Manual. For other govt officials not defined
in measurable terms. CLTS facilitation
skills assessed in terms of results
achieved.
From the point of view of

beneficiaries, the
project achieved its
purpose.
Not all targets achieved. For indicators of

SO/Purpose, Endline showed:
1) 90% of people in 44 aldeias and 96% of
28 (i.e. 26) schools have access to
improved water sources vs target of 100%
in 34 sub-villages and 23 schools.
2) None of district WASH Cttees are
meeting quarterly vs target of 4.
3) 17% and 422% improvement
respectively in number of children who
wash hands with soap before eating and
after defecating, vs target of 20%.
Sustainability
21
To what degree did the projects

sustainability strategy achieve its
objectives? In particular, what is the
likelihood that the O&M and repair of
the installed facilities being financed at
the local and sub-national level for
continuity of the services (access to
clean water etc.) after the end of the
project?
22
How able is government at different

levels to provide continuity to the
activities, especially CLTS?
23
To what extent have villages that were

declared ODF remained as such?
24
How effective has the sanitation

marketing strategy been in enabling
the emergence of local entrepreneurs
to support moving up the sanitation
ladder?
25
To what extent have households who

built latrines subsequently invested
further in improving them?
26
What could the project have done

different to improve sustainability?
Most facilities well

maintained (12 of 13)
and sufficient funds
being collected (11 of
13).
Most GMFs (11 of 13)

Sustainability requires followfunctioning well or
up from MoH staff which is
adequately. SDF
not happening at the level
support still inadequate required.
in most sub-districts.
Most GMFs also take
responsibility for school
water infrastructure
maintenance.
Most facilities still very

Confirmed no
new and fully functional. maintenance training
Special attention
provided.
required to monitor
sustainability of electric
pump for water supply
to Urbadan primary
school in aldeia
Hautoho in Aileu.
SDF support still

As above
inadequate in most subdistricts.
MoEd confirms little

Gap between what
capacity to support
SDFs say they do to
school maintenance.
support GMFs and what
SAS has limited budget GMFs say they receive.
and staff constraints in
some districts for
adequate functioning of
SDFs.
Good sustainability at
Education Section
level of policy support,
confirms weakness of
but inadequate budget
school maintenance
support for O&M. Health system.
system for PHC is
being strengthened, but
needs more budget.
Schools weak and get

None of partners at
Most consider have
No reflections
little support from MoEd required capacity level, capacity to perform as
specifically around
Health weakest, unable required, but need more sustainability
to support CLTS
budget.
implementation, even
through NGOs, due to
lack of staff and budget.
Small sample, but shows

good sustainability (see
Sheet 3)
No evidence found of local
entrepreneurs. One example
seen of abandoned
components of project
initiative.
UNICEF could do more

to support NGO
sustainability
Confirm weaknesses in Confirm weaknesses in

govt capacity, but
govt capacity
significant progress
being made. Much more
needed for CLTS to take
off as required. Only +/50% of school WASH
facilities functional.
Include postInclude post-ODF monitoring

construction mentorship phase to support health staff
phase to support SDFs to do follow-up of those
to support GMFs.
reverting to OD.
Promote use of O&M
fund for micro credit.
Endline Report shows good sustainability

of GMFs and reasonable of latrines (with
great variation between districts).
While govt is still weak,

progress in developing
systems to support
sustainability is good
and prospects are good
for medium term,
including in health
system.
Confirm weaknesses in
govt capacity and need
for much more funding
and resources to scale
up CLTS and sustain
ODF.
Documents describing sector challenges

and evaluations of performance of different
programmes all highlight weaknesses in
govt capacity and need to match policy
commitments with resources.
As above
Endline Report shows good sustainability

of latrines in some districts, but with great
variation between districts.
Little info in reports.
Confirm failure of
strategy of trying to
create businesses from
scratch and not
carefully identifying
bottlenecks first.
All communities
Limited number have moved
requested financial
up ladder; nearly all
support to improve
permanent structures were
latrines and make them first version built. Main
"permanent". Need for
constraint is finances.
targeted subsidy policy.
Improve construction
finishes (see Sheet 2).
Sustainability strategy
supported by all
BESIK reports policy

work on smart
subsidies is at
advanced stage. Likely
to make big impact in
stimulating latrine
improvement.
Improve construction
finishes (see Sheet 4).
Provide O&M training to
schools to set up
system.
Felt need for training of

a more generic nature.
Provide copies of asbuilt drawings and
BoQs to SAS.
Improving of latrines not monitored by

project or govt. Endline Study did not
enquire into this aspect. UNICEF has
provided input into govt subsidy policy.
Felt need for training of

a more generic nature
Good summary of many recommendations UNICEF could do more

emerging from this evaluation can be found to support NGO
in DNSA-BESIK study of GMFs in 2013.
sustainability
Support with
motorcycles, office
equipment; training in
CLTS, CAP process.
Reports include testimonies from

beneficiaries.
Impact
27
What were the main impacts (positive/

negative, expected/ unexpected) as
perceived by the different actors and
beneficiaries of the project? Include
impacts/benefits not directly linked to
improved sources of drinking water.
Proximity and reliability

of water. Then quality
and health benefits.
Proximity and reliability of

water. Then quality and
health benefits. Proximity
also means for many being
able to build a "decent" (pour
flush) toilet. Many welcomed
hygiene messages and
encouragement to build
latrine. Some use water for
vegetable gardening.
Proximity and reliability

of water. Then quality
and health benefits.
Funding support, staff

training.
Support for training in

CAP process and to
facilitate process in
communities.
Support in development
of policies, strategies,
guidelines
28
Did the project manage the risks of

impact of external factors adequately?
29
Did the project achieve its Overall

Objective?
By indicators of OO (Endline Study):

1) 5% reduction in diarrhoea in children
under five vs target of 20%
2) 6% and 16% increases in attendance
rates of boys and girls vs target of 5%
Also: reduced water collection time, less
burden on women, improved school
maintenance arrangements, validation and
consolidation of approaches
30
To what extent did the project

contribute to the MDGs?
MDG 7 (from Endline Study):

Increase in access to water from 45,5% to
90,3% - exceeds MDG target.
Access to sanitation: OD reduced from
58% to 27%. Problems with stats for
"improved" sanitation make comparison
difficult.
Risks well managed but

limited success with
approval of policies and
guidelines (School
WASH, BeSI Cttees)
Time delays for

approvals by
government is generally
acknowledged
challenge
Risk monitoring documented, also efforts

to lobby govt to approve policies
Equity
31
Was the project aligned with UNICEFs The only households

equity agenda in addressing the needs excluded were those
of the target groups (i.e. to what extent that are located too high
the initiative reached different groups
above sources.
including the most marginalised)?
32
Did the project contribute to equitable

participation and benefits to various
groups (men, women, children and
differently abled people)?
Wheelchair access at
standpipes included in
design.
All households were able

(some with community
support) to build a latrine, but
poorest could only manage
flimsy, temporary structures.
Efforts made to include
women in GMFs, with
some success. CAP
process includes equity
focus.
Toilets designed for

wheelchair access. No
menstrual facilities.
Children at centre of
school hygiene
promotion activities.
All policy documents and guidelines make

adequate provision for equitable benefits for
all groups.
Project did not have gender mainstreaming
strategy.
Lessons learnt
33
Which approaches, techniques,

Employing appropriate
methodologies, tools, technologies,
technologies and
etc were shown to be particularly
standard designs, and
effective for achieving project results, or investing in solid,
which should no longer be used?
durable infrastructure
Focus particularly on CLTS.
enhances sustainability.
Which of the above should be
promoted as examples of good
practice to government, UNICEF and
the EU?
34
What locally used latrine designs

should be promoted in order to ensure
that as many latrines as possible
qualify as improved in terms of the
JMP definition?
Community management with GMFs can be

effective. Participation
and contribution by
community during
design and construction
enhances ownership.
Post construction
refresher training vital,
also mentorship phase
with project support to
SDFs and GMFs. For
sustainability, external
support is essential;
more resources needed
for SAS/SDFs.
CLTS worked well in this

Where GMFs take
project (thanks to large
responsibility to support
extent to intensive training of schools, maintenance
facilitators), sustainability
is improved.
being the weakness. The
health system requires
strengthening, both to
enhance sustainability and to
scale up CLTS. Targeted
subsidies needed to enable
poorest to build durable
latrines.
Strong support for

CLTS.
Model of implementing
with local NGOs is
appropriate.
Most basic dry pit units

(made only of local materials)
did not have roofs, but could
qualify as acceptable. Only 1
example with roof (no. 30 on
Sheet 3). Several examples
of PF-I found, e.g. nos 40-43.
Model of implementing
with local NGOs is
appropriate. More
support required to
strenghten NGOs to
enhance their
sustainability.
Substantial progress
made already in
developing subsidy
policy to be applied
once aldeias declared
ODF.
Construction of school WASH facilities

was component with most delays. Also,
poor communication between contractors
and community. For faster implementation
and greater community ownership for
O&M, might be better to use NGOs via
restricted tender for non-profits.
2013 DNSA-BESIK study of GMFs
contains several of the lessons learned
from this evaluation.
JMP definition of acceptable pit latrine with

slab: "Is a dry pit latrine whereby the pit is
fully covered by a slab or platform that is
fitted either with a sqatting hole or seat.
The platform should be solid and can be
made of any type of material (concrete,
logs with earth or mud, cement, etc.) as
long as it adequately covers the pit without
exposing the pit content other than through
the sqatting hole or seat."
UNICEF Country Programme

35
Identify areas or aspects of the

strategy which should be
modified/reinforced or changed,
including an assessment of the
approach to gender mainstreaming.
Empower GMFs to
demand service from
SAS-SDFs
Strengthen efforts to
build generic capacity in
govt (e.g. SAS and
PHC system)
Strengthen efforts to
build generic capacity in
govt (e.g. SAS and
PHC system)
Apply gender lens

more rigorously in
interventions. Potential
for making more impact
appears to exist.
Sheet 2: Inspection of Water Supply Infrastructure and Assessment of GMF

Summary of Viqueue District
1) Due to road conditions, it was not possible to visit a second system as planned. However, it has reportedly been functioning well since completion 2 years ago (according to the SDF and some residents met on the road to the village before we had to turn around due to the road conditions).
2) No refresher training in O&M was reportedly done for any of the GMFs as the contract with the NGO had ended by the time this was implemented in the project.
3) The problems with the Bahalara-Uain system seem to require a higher and more intense level of intervention than has been attempted so far.
Ermera
Ermera
Humboe
Poeana
Std gravity
piped
2013
(Feb)
145
15
Fully functional
20%
$435 collected 2013,
2014
Being a coffee
growing area, nearly
all households pay
tariff annually ($6)
after harvest (JuneAugust).
Ermera
Ermera
Estado
Huitaso
Std gravity
piped
2013
(Dec)
57
Fully functional
100%
The general quality of the infrastructure observed (reservoir, 5 standpipes) is good, however the Good turnout at FGD (suco and aldeia leadership, full GMF, reps of 2 other neighbouring GMFs, several users) but only 1 woman.
$209 collected 2014 same examples of poor finishing as for Poeana were noted.
Well informed, organised and trained GMF. Sense of ownership high. 3 of 8 members are women (only one, Treasurer, present).
Pay tariff ($6)
Same applies to other 2 GMFs represented. Have considered other uses of O&M funds collected, e.g. micro credit; Saramata has
annually
already started to lend (at 10% per month). Not enough water to allow private connections - need to control consumption. No
disabled in village. Seem well informed on sanitation situation, request help to build permanent (PF) latrines. SDF has visited. PSFs
don't pay much attention to WASH.
Not all GMFs in suco manage to collect tariffs - while most collect from the majority (some from all), 2 have not managed to collect
anything (not project GMFs).
Ermera
Ermera
Estado
Saramata
Std gravity
piped
2013
(Dec)
39
Fully functional
56%
$130 collected 2014
Pay tariff ($6)
annually
The general quality of the infrastructure observed (reservoir, 5 standpipes) is good, however
most of the same examples of poor finishing as for Poeana were noted. But no erosion
observed yet and waste water drains are well built and functioning. Some standpipes have poor
screed finish on apron, so already flaking. Indicates variable skills amongst masons (and
inadequate guidance?). Too much water for reservoir overflow to cope so some taps left open.
Not very high turnout at FGD (Chefe Aldeia, 4 of 6 GMF, 3 users). Well informed, organised and trained GMF. Sense of ownership
high. 2 of 6 members are women (only one, Treasurer, present). Have started using O&M fund for micro credit. Do not allow private
connections, only hoses from taps - need to control consumption. No disabled in village. Will only assist school with maintenance if
they can pay.
Seem well informed on sanitation situation, request help to build permanent (PF) latrines (only 40% of families have PF latrines).
SDF has visited. PSFs don't pay much attention to WASH.
Ermera
Railaco
Samalete
Leborema
Std gravity
piped
2014
(Oct)
67
Fully functional
10%
$35 collected this
year, rest will be after
harvest
most of the same examples of poor finishing as for Poeana were noted. Erosion at reservoir has
already exposed pipe. Poor quality taps used, all already broken, not replaced yet.
Only 3 standpipes installed for 67 households, so heavier use than most systems (hence taps
broken so soon?). GMF insisted on few because amount of water available is limited and needs
to be carefully controlled; fewer taps make this easier. Maximum distances are still reasonable
as housing density is high.
Reasonable turnout at FGD considering it was market day (Chefe Aldeia, 5 of 6 GMF, 4 users). Well informed, organised and trained
GMF. Sense of ownership high. 3 of 6 members are women (all present: Treasurer, operator, environmental health promoter). Not
enough water to allow private connections - need to control consumption. No disabled in village.
Seem well informed on sanitation situation, request help to build permanent (PF) latrines (only 2 families have). SDF has visited.
PSF is the Treasurer of GMF, promotes sanitation.
Take responsibility for maintenance of water to school and plumbing to toilets. No APP (PTA) members on GMF.
there are a few examples of poor finishing and attention to the details that are important for
facilitating maintenance and ensuring greater sustainability, notably:
- no erosion prevention works at reservoir or at standpipes, with resulting evidence of erosion
already
- no gravel surround for standpipes or reservoir to facilitate hygienic conditions (however, the
extensive use of hosepipes from standpipes to houses means that there is not much movement
of people around the standpipes)
- in some cases wheelchair ramps at standpipes are nor positioned to be usable
- most taps were dripping
The source is an open stream which becomes dirty after rain, so a filter has been included in
the design. However, to keep costs down, a design which uses palm fibre as one of its filter
materials has been used, which requires up to weekly cleaning and/or replacement. This may
be a weakness as such frequent and labour intensive maintenance may not be sustainable. It
also requires paying some compensation (at least a meal) to the operators and community
members who do the work.
Summary of Ermera District
All systems fully functional with effective GMFs. Good prospects for sustainability. Suggests high quality of implementation process and supervision. High quality products and outcomes.
Oecusse Oesilo
Bobometo
Webaha
(Toapenu)
Std gravity
piped
2013
102
10
Fully functional
100%
$600 collected 2013
and 2014
Pay tariff ($3)
annually
Oecusse Nitibe
Usi-Taco
Bona 1&2
Std gravity
piped
2013
73
11
Fully functional,
some water
quality and
quantity
problems
100%
$270 collected
Pay tariff (25c)
monthly
Oecusse Nitibe
Usi-Taco
Bona 3
Hand dug
wells
2013
175
Oecusse Nitibe
Bene-Ufe
Citrana,
Bairro
Baocnana
Std gravity
piped
2014
(Nov)
115
32
Fully functional
56%
$123 collected
Pay tariff (50c)
monthly, started Jan
2015
?
The quality of the infrastructure observed (reservoir, 1 standpipe) is good. No erosion observed.
Pay tariff (50c)
No fences remain but no sign of animals around tapstands. Tapstands have 2 wastewater
monthly, started Jan drains, working well, no wastewater nearby. Hosepipes from taps to some houses. No major
2013
dripping at taps.
Wheelchair ramps at standpipes, but GMF doesn't know what they are for.
Summary of Oecusse District
High quality of construction. Also good attention to detail. Gravel surrounds and fences at
reservoir and tapstands, waste water chanelled to soakaways where fruit trees or sugar cane
are planted. One example of pipe exposed due to erosion near reservoir. No major dripping at
taps. Some dry because water is rationed with limited opening hours.
No wheelchair ramps at standpipes.
Due to distances and lack of vehicle accessibility, the reservoir was not inspected, only 2
tapstands. High quality of construction. Also good attention to detail. Gravel surrounds and
fences at tapstands, but wastewater not chanelled away. However, the extensive use of
hosepipes from standpipes to houses means that there is not much movement of people
around the standpipes. No major dripping at taps. No wheelchair ramps at standpipes.
Due to no other source being available, one with less water than would be ideal, and high
dissolved salts content, has been used. However, the former constraint is only felt at the end of
the dry season and the latter is within safety limits while the taste of the water is acceptable.
However it does cause serious maintenance problems as deposits form in the pipes, especially
in the flat areas, such as at the school. No affordable solution appears to be available.
14 wells Fully functional No maintenance fund One well inspected. Good quality construction. GMF reports all wells always have enough
(25c collected per water (3-15 families per well). Pulley not lubricated and has started wearing. Otherwise well
family for meals when maintained.
cleaning is done)
Good turnout at FGD considering it was market day (Chefe Aldeia, Chefe Bairro, 4 of 6 GMF, 10 users). Well informed, organised
and trained GMF. Sense of ownership high. 2 of 6 members are women but not office bearers (both away at market). Not enough
water to allow private connections - need to control consumption. Have to ration through limited opening hours. No disabled. Pay
operators "incentives" ($5/month) when O&M more intensive, e.g. annual cleaning around spring, etc.
Fully involved in sanitation promotion - first case encountered of sanitation seemingly genuinely community led. Organise
community solidarity to help poorest build latrines. Request help to build permanent (PF) latrines (only 20 families have). SDF has
not visited (only 2 in district currently). Chefe Aldeia is PSF, promotes sanitation.
In spite of the visit being on market day, the whole GMF and local leadership were present (Chefe Aldeia, Chefe Bairro, 6 GMF + 3
tap stewards, 5 users). There are 3 women on the GMF, the Treasurer and two sanitation promoters. Well informed, organised and
trained GMF. Sense of ownership high. Excellent level of tariff collection. Are trained for all operations and have made necessary
repairs and preventive maintenance such as replacing palm fibre in filter at source. Operators get paid when this work and repairs are
carried out. Take responsbility for water at school and maintenance of plumbing there.
Fully involved in sanitation promotion - another case of sanitation seemingly genuinely community led. Apparently all homes have at
least temporary latrines, HWF and dish racks. Request help to build permanent (PF) latrines.
GMF is responsible for all 14 wells. 3 of the 5 members were present, plus Chefe Bairro and 7 others. Treasurer (only woman) not
present. Very satisfied with wells. Have considered installing a pump with electric motor now that electricity is being brought to the
village.
Report that all households have latrines. Need financial help to upgrade to permanent (PF) structures.
This system was not on the original project list and was implemented when additional (non-EU) In spite of the visit coinciding with a major religious ceremony in the village that day, 8 of the 14 GMF members (all the office
funding was secured. Thus the EU contribution was only used to co-fund the implementation
bearers) and local leadership were present (Chefe Bairro, 8 GMF, 5 users). Consider large GMF necessary due to size and
costs (including training), not materials. However, this was only known after the visit and the
complexity of system. Have tapstand leaders for each for cleaning and tariff collection. There are 3 women on the GMF - Treasurer,
system is thus included in this report. The sign on the reservoir includes the EU flag.
one of the operators and the sanitation promoter. Well informed, organised and trained GMF. Sense of ownership high. Are trained
It is a relatively large system supplying a semi-urban settlement that includes numerous public for all operations and have made necessary repairs and preventive maintenance such as replacing palm fibre in filter at source.
buildings and shops. The design effectively and creatively incorporated some components (e.g. Operators get paid when this work and repairs are carried out. The GMF takes responsbility for water for government and other
reservoirs) of previous systems that had ceased to function.
public buildings, such as schools (where they do maintenance of plumbing) and the clinic, as well as churches and commercial
The quality of the infrastructure observed (reservoir, tank, 2 standpipes) is good. Also good
establishments. Payments of tariffs by government and civil servants is poor. GMF is supported by SAS to reinforce their authority.
attention to detail. Gravel surrounds and fences at tapstands, but wastewater not chanelled
Sanitation promotion appears to be genuinely community led and the coordinator of the sanitation promoters (woman) also gets an
away. However, the extensive use of hosepipes from standpipes to houses means that there is incentive to encourage families to move up the sanitation ladder. The local PSF is a member of the GMF.
not much movement of people around the standpipes. No major dripping at taps. No wheelchair The SDF has not visited since the inauguration 5 months ago.
ramps at standpipes.
Very impressive results achieved, both in water and sanitation. Sanitation seems genuinely community led, due to the quality of CLTS facilitation. Due to lack of capacity in SAS, more training of the GMF was done by the NGO in this district than in others, which may have been a positive factor, although long term
sustainability (once NGO support is withdrawn) may suffer due to this lack of capacity.
10
Aileu
Remexio
Asumau
Lerolisa
Std gravity
piped
2012
102
Fully functional
11
Aileu
Remexio
Tulataqueo
Samalete 2 Std gravity

piped
2012
(Jan)
56
Fully functional
50%
$400 in fund
Pay tariff (50c)
monthly, started 2013
The quality of the infrastructure observed (reservoir, 3 standpipes) is good. No erosion observed
at tank or standpipes, but main line down from tank has been exposed in several places due to
erosion (no erosion prevention works were done). No fences remain but no sign of animals
around tapstands. Tapstands have 2 wastewater drains, working well, no wastewater nearby.
Numerous hosepipes from taps to houses. No major dripping at taps.
Wheelchair ramps at standpipes.
12
Aileu
Remexio
Tulataqueo
Samalete 1 Std gravity

piped
2012
27
Fully functional
25%
+/-$60 in fund
Pay tariff (50c)
monthly, started 2013
The quality of the infrastructure observed (reservoir, 3 standpipes) is good. No erosion observed.
No fences remain but no sign of animals around tapstands. Tapstands have 2 wastewater
drains, working well, no wastewater nearby. One of them has plaster screed on apron partially
peeled off. Hosepipes from taps to some houses. No major dripping at taps. Pipe accidentally
cut by one homeowner when laying cable last year has not been properly repaired yet.
Wheelchair ramps at standpipes, one reportedly used.
13
Aileu
Liquidoe
Acubilitoho
Asumata
2014
(July)
40
Fully functional
zero
Monthly tariff (25c)
not collected yet
Summary of Aileu District
Good turnout at FGD (suco and aldeia leadership, full GMF, some users). Well informed, organised and trained GMF. Sense of
ownership high. 2 of 6 members are women (Treasurer and one operator).
Have considered other uses of O&M funds collected, e.g. micro credit. Also have rules for private connections and 3 have been
made. Make provision to help disabled.
Seem well informed on sanitation situation, request help to build permanent (PF) latrines. SDF has visited. PSFs don't pay much
attention to WASH.
GMF takes responsibility for maintenance of water supply and plumbing at school toilet facilities.
Std gravity
piped
Due to major religious festival in Remexio during week of visit, it was difficult to contact GMF members. Only two (operators) were
interviewed, hence no financial data obtained. Report 7 members, 3 of them women (Treasurer, health promoter, one of 3 operators).
Few repairs needed over nearly 3 years of operation - only taps at one tapstand. All other taps still originals, not even washers
changed. So tariff collection is not seen as very important. No maintenance needed at source as it is a protected spring.
Health promoter only focuses on tapstand hygiene.
Due to major religious festival in Remexio during week of visit, it was difficult to contact GMF members. Only two (Coord +
Treasurer) were interviewed. GMF has 12 members, 4 women (treasurer, 3 tap stewards). Meets every 3 months, but not all attend
regularly, some are tiring (same since 2012), no elections held as aren't confident anyone will want to join GMF due to unpaid nature
of work. Received refresher training 2014, and toolkit. No problems with system, just taps.
GMF does maintenance at schools, which pay monthly tariff (also have blue recording book).
Health promoter is also PSF, does latrine promotion work.
No reporting system to SAS. No visit from SDF since training last year.
Due to major religious festival in Remexio during week of visit, it was difficult to contact GMF members. Only two (Coord +
Assistant) were interviewed, hence only sketchy financial data obtained. Chefe Aldeia keeps the money. GMF has 10 members, 2
women (treasurer, health promoter), same team since 2012. Received refresher training 2014, and toolkit. No problems with system,
just taps. Have to clean filter every 3 months. One person in aldeia uses wheelchair and collects water.
GMF does maintenance at schools, which pay monthly tariff (also have blue recording book).
Health promoter is also PSF, does latrine promotion work.
No reporting system to SAS. No visit from SDF since training last year.
The quality of the infrastructure observed (reservoir, 3 standpipes) is variable; 1 standpipe is
GMF has 8 members, 3 of them women (treasurer, health promoter, operator). Have been functioning for one year. Received
incorrectly sloped (hence waste water doesn't drain as intended) and has peeling screed finish. refresher training 2014, and toolkit. Not functioning adequately and have not started collecting tariff yet. No maintenance involving
Erosion observed along main line from tank and pipes exposed. Standpipe fences in disrepair
expenditure has been required yet. Some households expressed dissatisfaction with the Coordinator. The only poorly maintained
but no sign of animals around tapstands. No major dripping at taps.
tapstand is that under his responsibility. He, his deputy and the treasurer were away on private business. Another 3 members were
Wheelchair ramps at standpipes.
at an official meeting elsewhere. Only 2 members were present in the FGD, with 8 community members. No disabled in aldeia.
No reporting system to SAS. No visit from SDF since training last year (changed jobs shortly after and has not been replaced).
Generally good quality construction and maintenance, with mostly well functioning GMFs. Exception is Asumata which requires an intervention from SAS.
Sheet 3: Inspection of latrines and interviews with owners

Inspection of Latrines and Interviews with Owners
Identification of aldeia
No.
District
Sub-district Suco
Aldeia
Apparent
income
level *
Type of latrine*
In use?
When
built
Influenced by
project?
Trigger to build
Barrier(s)
Approximate
First or
Cost (if known) to moving
upgrade
up ladder
USD
upgrade Subsidy +/-$100,
total cost not
upgrade
upgrade known (+/-$400?)
approx $500
upgrade
without subsidy.
n/a
n/a
n/a
n/a
General
hygiene
Benefits from water
Water
tariff
paid?
yes
yes
yes
yes
good
good
good
good
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
proximity, grows
vegetables
proximity
proximity (from old
system - not
connected to new)
yes
no
Project messages
recalled
HW
facility
nearby?
HWWS?
none
none
none
none
bathroom
bathroom
bathroom
bathroom
say
say
say
say
Bahalara-Uain
Viqueque Viqueque
1
Bahalara-Uain
Viqueque Viqueque
2
Bahalara-Uain
Viqueque Viqueque
3
Bahalara-Uain
Viqueque Viqueque
4
Summary of Welaco aldeia
Subsidy
no
2009
yes
PF-P
Above ave.
Welaco
Subsidy
no
2009
yes
PF-P
Above ave.
Welaco
Subsidy
no
2009
yes
PF-P
Above ave.
Welaco
Neighbours
no
2015
yes
PF-P inside
Average
Welaco
No OD observed, reported as practiced only away from homes in fields. All subsidised latrines in use. One built
Summary of Viqueque District
Only one aldeia visited. From interviews, observations, comments from stakeholders, seems impact of subsidies is not necessarily insurmountable barrier to CLTS.
Ermera
Ermera
Humboe
Poeana
Average
PF-I
yes
2012
no
n/a
none
bathroom
say yes
good
6
7
Ermera
Ermera
Ermera
Ermera
Humboe
Humboe
Poeana
Poeana
Below ave.
Average
PF-B
only pit dug
yes
no, OD
2013
n/a
yes
yes
cost
cost
most
unwilling to talk
bathroom
no
say yes
no
good
good
importance of latrines bathroom
Summary of Poeana aldeia
$700
replace
Replace previous one that
collapsed
$50
first
project messages
n/a
first
Project messages. Little space
to build so wants to save for PF
to avoid smell near house.
Limited time available. One case of OD but fully aware of need for latrine and has started. Variable reported impact of hygiene messages.
Ermera
Ermera
Estado
Huitaso
Average
PF-B
yes
2014
yes
unspecified project messages
first
$100
cost
say yes
good
Ermera
Ermera
Estado
Huitaso
Average
PF-I
yes
2014
no
proximity of water for PF
upgrade
$400
n/a
none
bathroom
say yes
good
10
Ermera
Ermera
Estado
Huitaso
Below ave.
PF-B
yes
2014
yes
proximity of water to upgrade

from first (2012), previously OD
upgrade
$150
cost
none
bathroom
say yes
poor
11
Ermera
Ermera
Estado
Huitaso
yes,
proximity so use more,
grow vegetables, wash pays
annually
clothes at home
yes,
proximity, quality
annual
proximity, quality
yes,
annual
yes,
annual
13
Ermera
Ermera
Estado
proximity
poor
yes
bathroom
none
cost
local materials
first
unspecified project messages
yes
2013
yes
basic dry pit, no
roof, hole cover
No OD observed or reported. Proximity of water that makes PF viable is main motivation. Clearly this is what most consider the minimum acceptable standard, refered to as "permanent"; dry pit considered "temporary".
proximity, quality
good
bathroom say yes
all (leader)
n/a
upgrade $300 + same for
proximity of water to upgrade
no
2014
yes
PF-P
Saramata Above ave.
bathroom
from first (2010), previously OD
proximity, quality
good
bathroom say yes
all (leader)
n/a
$600
upgrade
always had
no
2010
yes
PF-P
high
Saramata
14
Ermera
Ermera
Estado
Saramata
Above ave.
2015
no
Replace full dry pit built in 1999.
15
Ermera
Ermera
Estado
Saramata
Below ave.
2014
yes
safety for women
16
Ermera
Ermera
Estado
Saramata
Summary of Huitaso aldeia

Estado
Ermera
12 Ermera
Summary of Saramata aldeia

Samalete
Railaco
17 Ermera
Average
yes
basic dry pit, no
roof, no hole cover
basic dry pit, no yes(+ by
roof, no hole cover 2 other
families)
yes
Below ave. None, use no.15's
19
Ermera
Railaco
Samalete
20
Ermera
Railaco
Samalete
Leborema
Poor
21
Ermera
Railaco
Samalete
Leborema
Poor
Usi-Taco
29
Oecusse Nitibe
Usi-Taco
30
Oecusse Nitibe
Usi-Taco
no
say yes
fair
none (did not attend

meetings)
no
say yes
poor (flies,
dirty yard)
average
yes
yes
all (leader)
$300
upgrade
Replace original dry unit built
no
2014
yes
PF-P
Average
Webaha
average
yes
yes
all (leader)
n/a
$200
upgrade
with subsidy in 2006, with PF
no
2014
yes
PF-P
Above ave.
Webaha
poor
say yes
no
most
cost
local materials
first
Dignity, safety at night
yes
2012
yes
PF-B
Average
Webaha
poor
say yes
no
all (leader)
n/a
$200
upgrade
Replace original dry unit
no
2015
yes
PF-I
Below ave.
Webaha
average
say yes
no
did not participate
cost
not known
upgrade
As for 22 and 23
no
2003
no, OD
PF-B (full)
Very poor
Webaha
Almost all latrines seen (and all inspected) are PF. Only case of OD seems to be temporary until funds available to replace old latrine. Most already used latrines before project. Very good recollection of project
good
yes (tippy- not soap
latrines, HWWS
cost
local materials
first
no (prior) Right thing to do
2012
yes
PF-B
Bona 1&2 Below ave.
tap)
average
say yes
no, at
latrines, HWWS, OD
cost
local materials
first
Right thing to do - message
no
2009
yes
PF-B offset
Bona 1&2 Below ave.
kitchen
is disgusting
from Indonesian time.
poor
not soap
no
latrines, HWWS,
cost
local materials
first
Project messages
yes
2013
no, OD
PF-B (collapsed)
Poor
Bona 1&2
dish rack
good
yes (tippy- apparently
latrines, HWWS,
cost
local materials
first
Project messages (including
yes
2013
yes
basic dry pit, well
Poor
Bona 1&2
tap)
flies cause disease
"OD is illegal")
built (+roof) & maintained, hole cover
Summary of Bona aldeia

Summary of Oecusse District
Also mostly PF and many already had before project. Good recollection of messages.
Best transmission of messages of all districts. Only OD observed seems temporary. Good GMF role in leading sanitation promotion.
Acumau
Remexio
Aileu
Summary of Lerolisa aldeia
Tulataqueo
Remexio
31 Aileu
Tulataqueo
Remexio
32 Aileu
Lerolisa
Due to schedule slippage all day long there was no time to visit homes (departure from the village was at 17:45).
?
?
?
upgrade
No-one at home
?
2014
yes
PF-P
Samalete 2 below ave.
with water close,
cost, new
local materials
first
Right thing to do, can't send
no
2006
use 31's
basic dry pit,
poor
Samalete 2
must build
house first
visitors to OD.
collapsed 2013
?
?
?
?
No-one at home
?
?
yes
basic dry pit, no
Samalete 2 average
roof, hole cover,
needs repairs
none
cost
local materials
Replace first one built 2012, no replaceno
2014
yes
Samalete 2 below ave. basic dry pit, no
ment
reason given
roof, no hole cover,
in good condition
none
cost
local materials
first
no reason given
no
2009
yes
basic dry pit, no
poor
Samalete 2
roof, hole cover,
needs repairs
?
?
n/a
n/a
No-one at home
no
n/a
n/a
none
Samalete 2 average
OD?
Due to major event in Remexio, most houses had no-one at home. Only 1 out of 6 possibly OD. Mostly basic dry pit collapsed or in need of repairs. Relatively poor aldeia.
?
?
?
?
No-one at home
?
?
yes
PF-B
Samalete 1 above ave.
33
Aileu
Remexio
Tulataqueo
34
Aileu
Remexio
Tulataqueo
35
Aileu
Remexio
Tulataqueo
36
Aileu
Remexio
Tulataqueo
Summary of Samalete 2 aldeia

Tulataqueo
Remexio
37 Aileu
average
38
Aileu
Remexio
Tulataqueo
Samalete 1
39
Aileu
Remexio
Tulataqueo
Samalete 1 above ave.
Summary of Samalete 1 aldeia

Acubilitoho
Liquidoe
40 Aileu
41
Aileu
Liquidoe
Acubilitoho
42
Aileu
Liquidoe
Acubilitoho
43
Aileu
Liquidoe
Acubilitoho
Summary of Asumata aldeia
Summary of Aileu District

*
PF now viable,
proximity, wash coffee
proximity
no (too
poor)
some
Samalete
Oecusse Nitibe
none
cost
proximity
say yes poor (flies,
no
cost
local materials
first
so as not to contaminate fields
no
2004
yes
basic dry pit
dirty yard)
meetings)
(doubtful)
poor state, flies
no queues
say yes poor (flies,
no
none
cost
local materials
first
aldeia now more populated,
yes
2012
yes
basic dry pit
dirty yard)
need to keep clean
poor state, flies
Generally much poorer aldeia. Less PF latrines. Most dry pit latrines in need of repairs. Most already used latrines before project. Poor state of cleanliness in general. Little apparent impact of hygiene/general sanitation messages.
CLTS has been effective. Little reversion to OD. But also little maintenance of "temporary" latrines. Less investment in PF than in other districts.
Railaco
28
cost
local materials
yes,
annual
Ermera
Bobometo
22 Oecusse Oesilo
Bobometo
23 Oecusse Oesilo
Bobometo
24 Oecusse Oesilo
Bobometo
25 Oecusse Oesilo
Bobometo
26 Oecusse Oesilo
Summary of Webaha aldeia
Usi-Taco
27 Oecusse Nitibe
local materials
first
yes,
annual
yes,
annual
yes,
annual
yes,
annual
proximity
poor
say yes
no
cost
n/a
n/a
n/a
no
n/a
meetings)
No OD observed or reported, although 2 familes (one interviewed) use neighbour's latrine. Most already used latrines before project. Generally cleanliness not up to standard. One of two dry pits in poor condition and not repaired.
proximity
good
yes
yes
all (leader)
n/a
$500
replaced previous PF built 2006 replace
no
2012
yes
PF-P
Average
Leborema
fired local bricks
say yes poor (flies, proximity, no queues
no
OD is bad
cost
local materials
first
OD is bad
yes
2014
yes
Leborema Below ave. basic dry pit, no
dirty yard)
roof, no cover
no closer than before
fair
say yes
yes
mentioned all (from
cost
upgrade Can't recall, +/proximity of water to upgrade
no
2009
yes (+
PF-I
Leborema Below ave.
but more reliable
different project)
$150?
from first, previously OD
neighbour)
18
Summary of Leborema aldeia

Summary of Ermera District
replace
n/a
PF-I, apparently
good condition
PF-I
no, use
37's
yes
No-one at home
n/a
No-one at home
n/a
proximity
proximity
proximity
n/a
proximity
hygiene messages.
Ease of use, still use
well in yard.
Ease of use, still use
well in yard.
proximity
proximity, cooler than
well water
yes,
annual
yes,
annual
most,
annual
paid up
paid up
paid up
n/a
paid up
paid up
paid up
partially
paid up
paid up
average
?
proximity, can bathe
children more daily
?
?
paid to
March
?
average
proximity
paid to
Feb
say yes
below
average
proximity
only $6
so far
no
average
Standpipe
nearby
Standpipe
nearby
bathroom
average
average
good
?
say yes
good
average
no
no
say yes
no
bathroom
bathroom
Due to major event in Remexio, most houses had no-one at home. No OD observed. All PF. Unclear why one (PF-I) is not being used. GMF says because too far from water, although standpipe is only 50m away.
no
proximity, quality,
good
bucket in say yes
OD is disgusting,
n/a
upgrade
yes
2014
yes
below ave.
Asumata
quantity
latrine
HWWS, childrens'
subsidy (cement,
faeces are bad
PF-I
zinc sheets,
no
proximity
good
OD is bad, HWWS, bucket in say yes
n/a
upgrade
yes
2014
yes
below ave. (some cement on
Asumata
insufficient sand)
latrine
bury refuse, security
(tho' pit
floor, bamboo
+ local materials
project messages, subsidy
of latrines for women
walls, zinc roof, collapsed)
(including bought
offset unlined pit
no
proximity (hose)
good
hose from say yes
can't remember
n/a
upgrade
yes
2014
yes
average
Asumata
sand) + own
with cement cover)
standpipe
labour
no
proximity, quantity
good
bucket in say yes
can't remember
n/a
upgrade
yes
2014
yes
poor
Asumata
latrine
Most households benefited from a materials subsidy and have built intermediate PF latrines. The subsidy was from SAS as part of a pilot project to support the most vulnerable. The project started in 2013 with an NGO promoting latrines
without subsidies, and most households built basic dry pit latrines. Then the materails arrived, intended only for the very poor ("vulnerable"), but due to the lack of clear criteria, the community demanded equal treatment for all and it was left to
the Chefe Suco to manage distribution, so in this village almost eveyone received the subsidy. However, very few actually did anyhting about improving their latrines until the EU-UNICEF project stated CLTS (May 2014) and then all those who
had received the subsidy built improved pour flush latrines.
No OD observed (2 cases of using neighbour's latrine). Interesting unintentional combination of subsidy and CLTS in Asumata. Clear link between % of permanent latrines and wealth of village.
Classifications
Apparent income level
Refers to perceived overall standard inferred from home and outbuildings observed in all aldeias visited. Higher than most of Sub-Saharan Africa, comparable with poorer
Central American countries.
Type of latrine
Basic = Simple pit latrine made entirely of local materials, unlined pit, no cement, often without roof.
PF-B = Basic pour flush directly over partially lined pit with concrete cover, ceramic or plastic pan, floor at least partially cemented, no tank, superstructure entirely of
local materials, with roof. Usually not used as bathroom.
PF-I = Intermediate pour flush with offset, lined or unlined pit with concrete cover, ceramic or plastic pan, floor at least partially cemented, concrete tank,
superstructure mostly of local materials, zinc roof. Usually also used as bathroom.
PF-P = Permanent pour flush. As with the intermediate version, but with fully concrete floor and concrete block or brick walls. Nearly all also used as bathroom.
Sheet 4: Inspection of schools

Inspection of Schools
School Identification
SubSuco
Aldeia
district
Viqueque Viqueque Bahalara- Welaco
Uain
No. District
1
School
Name
BahalaraUain
Intervention
Rehabilitation of 2 (non-standard)
toilet blocks and connection to
new water system by NGO.
Training by NGO.
Pupils Water
No.
Observations
per availstalls
shift able?
2015
517
no 5 (1D)* School closed by time of visit so unable to inspect interior or interview anyone. No water as system
is down. Figures for no. of pupils and stalls different from reported. Ratio of pupils per stall far
exceeds standard. See below for comments that apply to all units built.
Completed
Ermera
Ermera
Humboe
Nunutali
Nunusakari New block, connection to new

water system by contractor
(pending).
Training by NGO.
2015
544
no
3** Ermera
Ermera
Estado
Culau
Haburas
Estado
New block, connection to new
(EMIS 139) water system by NGO.
Training by NGO.
2014
yes
Saramata
Domingos
2014
96
yes
1 SB School closed at time of visit (teacher training in District) so unable to inspect interior or interview
anyone. Tank overflowing (stop valve disfunctional and no float valve), access to toilets impossible
without stepping in mud. Waste water from HWF des not drain to spetic tank - blocked?
See below for comments that apply to all units built.
Fatuquero New blocks, connection to

existing (old) water system by
contractor.
Training by NGO.
Cocoa
New block, connection to existing
(old) water system by contractor.
Training by NGO.
2015
340
no
2015
130
yes
2 SB School partially functioning for teacher training, Coordenador interviewed. No water (existing system
(1D) disfunctional), HIM to install new system this year. Training did not include any on maintenance of
facilities or establishing a maintenance system with responsibilities and links to GMF or SAS.
anyone. List says project made connection. Inlet to tank goes through overflow pipe, overflow runs
down riser pipe (which is loose and susceptible to damage), forming pools and muddy area. Access
hole of tank on lid is for rainwater pipe, has no lid.
anyone. Not on UNICEF list. Sound of leaking taps inside (probably preventing tank from overflowing
as water was flowing into it, but shortening life of septic tank?). No vent pipe for septic tank.
Ermera
Ermera
Estado
Ermera
Railaco
Fatuquero Fatuquero
Ermera
Railaco
Railako
Leten
Ermera
Railaco
Samalete Leborema
Samalete
New blocks, connection to new

water system by NGO.
Training by NGO.
2014
156
yes
(1D) anyone. List says 1 block. Not funded by EU? Dripping taps, not replaced yet by UNICEF.
Oecusse Nitibe
Usi-Taco
Bona

Training by NGO.
2013
114
yes
1 SB Coordinator interviewed, children too young. Described training provided to teachers and promotional
activities. Tippy-taps apparently in use still in front of school. Good relationship established with
GMF for maintenance. Children trained on HWWS before school lunch, but were observed later to
not wash hands before having their lunch, although the HWF was functional.
Taps at HWF leaking, one not functional, no funds to replace, GMF requires payment. Taps not
replaced by UNICEF. Stop valve to tank broken, no float valve.
Aileu
Remexio Tulatakeu Samalete 2 Buburnaro New blocks, connection to new

Training by NGO.
2014
87
yes
5 (1D) School closed at time of visit (religious holiday in SD) so unable to inspect interior or interview
anyone. Blocks in good condition, no taps dripping, tank inlet valve (ball type) functional, being used
to control water to tank. Severe erosion around building from rainwater.
Remexio Tulatakeu Samalete 2 Tulatakeu
2013
96
yes
2 SB School closed at time of visit (religious holiday in SD) so unable to inspect interior or interview
anyone. In good condition, taps not dripping (not originals, but not installed by UNICEF).
10 Aileu
Cocoa
Bona
New block, connection to new

Training by NGO.
anyone. Report says 2 blocks, completed 2013. Ratio of pupils per stall far exceeds standard (even
with 2 old toilets included). Water not connected from tank to toilets by contractor yet (contract with
NGO had ended by the time construction was completed). Not functioning yet. Some exposed
polypipe. No signage. No vent pipe for septic tank.
See below
for comments
that (teacher
apply to training
all unitsinbuilt.
1 SB School
closed
at time of visit
District) so unable to inspect interior or interview

Training by NGO.
** N.B. School no. 3 was not funded by this project but was visited anyway as it was
beside the road and the same design and methodology was used.
* SB = Standard block = 3 stalls, one each for boys and girls, and one for disabled/wheelchair
access, plus hand washing facility. D = stall for wheelchair access.
General observations and recommendations valid for all toilet blocks inspected:
1.) Good design - functional, rugged, good quality of materials. Exceptions:
a) no facilities for menstruating girls;
b) concrete or even plastic tanks instead of galvanised ones would be longer lasting.
2.) Good quality of construction.
3.) All but 1 block had signage (M, F, disabled).
4.) Lack of attention to details that make maintenance easier and less costly:
a.) No erosion works (e.g. simple diversion channels/humps) to prevent erosion from rainwater around building. Erosion
was observed in a number of cases.
b.) Covers for some inspection boxes too light, not countersunk, susceptible to vandalism.
c.) Not all taps had been replaced with good quality ones as per UNICEF programme.
d.) Overflow from tank not piped away to avoid pools of water around tank. Many tanks observed were overflowing.
e.) Water into tank without ball valve to close off, and stop valve at waist height and vulnerable to damage, instead of being under
ground with inspection box and lockable lid.

(P11 with some administrative data removed)
1. Family name
RALL
First name
Martin
Middle name
Giles
2. Date Day Mo.

Yr.
3. Place of birth
4. Nationality (ies) at birth
of Birth
01
02
53
Durban, South Africa
South African
22. KNOWLEDGE OF LANGUAGES. What is your mother tongue? English
READ
WRITE
Not
Easily
Maiden name, if any
5. Present nationality (ies)

South African
SPEAK
Not
Easily
Not
Easily
OTHER LANGUAGES
Easily
Easily
Easily
Portuguese
X
X
X
Spanish
X
X
X
24. EDUCATIONAL. Give full details - N.B. Please give exact titles of degrees in original language.
A. UNIVERSITY OR EQUIVALENT
Please do not translate or equate to other degrees.
ATTENDED FROM/TO
DEGREES and ACADEMIC
NAME, PLACE AND COUNTRY
Mo./Year
Mo./Year
DISTINCTIONS OBTAINED
University of Stellenbosch, South Africa
1972
1973
First 2 years of B.Sc.Eng (Agric)
University of Natal, South Africa
University of Natal, South Africa
1974
1977
1976
1978
B.Sc.Eng (Agric) (Cum Laude)

M.Sc.Eng (Agric)
6. Sex
Male
UNDERSTAND
Not
Easily
Easily
X
X
MAIN COURSE OF STUDY

Mixture of civil, mechanical
and hydraulic engineering
Agricultural mechanization
Tractor fuel consumption
25. LIST PROFESSIONAL SOCIETIES AND ACTIVITIES IN CIVIC, PUBLIC OR INTERNATIONAL AFFAIRS
South African Institute of Agricultural Engineers
Water Institute of Southern Africa
26. LIST ANY SIGNIFICANT PUBLICATIONS YOU HAVE WRITTEN (do not attach)
Rall, M. & de Jager, J, 1997. Greater Expectations: Technology choices for community managed water supply in the new South Africa. Water
Lines, Vol 15, No 3, Jan 1997.
Rall, M, 1997. Technical choices and operation and maintenance on the Mvula Trust groundwater supply projects. Borehole Water Journal,
Vol 35, 1997.
Rall, M. 2000. "Towards More Successful Rural Water Supply Schemes - The Mvula Trusts Experience Guides Policy Changes". Engineering
News, April 2000.
Rall, M. 2005. Mvula Trust en Sudafrica: un colaborador del gobierno independiente. Medio Ambiente y Urbanizacion, Ano 21, Nov 2005:
Removiendo Bareras: Agua y Saneamiento Para Todos (IIED-AL).
Rall, M. 2002. The Mvula Trust in South Africa: An Independent Partner to the Government. Blue Gold Series, Field Note 14, Water &
Sanitation Program (WSP-AF), World Bank.
Rall, M. 2008. Manual de Formao dos Grupos de gua e Saneamento (GAS). Direco Nacional de guas, Angola.
Rall, M. 2008. Manual de Formao de Mobilizadores Sociais. Direco Nacional de guas, Angola.
27
.
EMPLOYMENT RECORD:
A. PRESENT POST (LAST POST, IF NOT PRESENTLY IN EMPLOYMENT)
SALARIES PER ANNUM
STARTING
FINAL
TO
FROM
MONTH/YEAR
MONTH/YEAR
Feb 2007
Current
NAME OF EMPLOYER:
See below for each assignment
ADDRESS OF EMPLOYER:
N/a
EXACT TITLE OF YOUR POST:
Independent consultant
N/a
TYPE OF BUSINESS:
Consulting in WASH, M&E, family sector agriculture

NAME OF SUPERVISOR:

NO. AND KIND OF EMPLOYEES
SUPERVISED BY YOU:
N/a
REASON FOR LEAVING:
N/a
DESCRIPTION OF YOUR DUTIES

Position
Description
Date
Location
Company & reference

person
October
2014
Mozambique
WaterAid UK
Papa Diouf
PapaDiouf@wateraid.org
Team
Leader
Led a team of 5 to evaluate WaterAids Mozambique Country

Programme (CP), assessing at strategic level the relevance and
effectiveness of the CP; and at operational level the strengths and
weaknesses in implementation, capacities, partnerships and practice
policy linkages, making recommendations to improve performance and
fine tune the CP for the next period.
Feb &
Jun
2014
Home
based
EU Mozambique
P. Calistri. piergiorgio.
calistri@eeas.europa.eu
Consultant
Assessment of project funding proposals of call for proposals for

improving peri urban and rural livelihoods through TVET, and making
recommendations for improving logframes and indicators.
April
2014
Angola
DW Angola: Allan Cain

allan.cain@gmail.com
Consultant
Trainer
Developing training materials and conducting training for government

and NGO water supply officials on community-based O&M of water
supplies and promotion of sanitation and hygiene.
Mar
2007 to
date
Latin
America,
Africa,
Asia
EPTISA (LA): Fabrizio

Mencarelli
mencarelli@eptisa.be
EPTISA (Africa): Frans
Geilfus
fgeilfus@eptisa.com
Senior
International
Expert
Rapid evaluations of 69 projects funded by the European Commission

(8 times as Mission leader), using the Results Oriented Monitoring
(ROM) approach. Projects included urban and rural water supply &
sanitation (25 projects, including Timor Leste, Mozambique, Angola,
Namibia, Malawi, Ethiopia and Somalia), irrigation, integrated agroforestry systems, eco-friendly farming systems in the family sector and
institutional arrangements for the preservation of rainforest biodiversity.
Water projects monitored:
2013
Namibia: Support to Namibia Water Supply and Sanitation Sector

Programme (NAWASA). Sector budget support programme including
all WASH sector activities. Value 18m.
Timor-Leste: Creating a healthy environment for children in rural
communities. Standard rural WASH activities and capacity building,
including in schools. Value 2,5m.
Timor-Leste: Improving access to water, sanitation and hygiene in rural
schools and communities through capacity development. Standard
rural WASH activities and capacity building. Value 2,5m.
Mozambique: Capacity Building in Sustainable Water Infrastructure
Management in Mozambique Northern Region through a FIPAG-VEI
Partnership. Strengthening of FIPAG NRO. Value 1,2m.
Angola: Improving access to water, sanitation and hygiene services in
poor peri-urban and rural communities. Standard WASH activities,
improved governance. Value 2,5m.
Angola: Water Loss Reduction Programme in Public Water Supply
Systems. Introduction of improved tools and systems, capacity building
to reduce UFW in Huambo and Kuito. Value 1,2m.
2012
Ethiopia: Small Town Water and Sanitation Project. Rehabilitation and

augmentation of water and sanitation systems in 15 towns, institutional
strengthening. Value 36,2m.
Namibia: Support to Namibia Water Supply and Sanitation Sector
Programme (NAWASA). Sector budget support programme including
all WASH sector activities. Value 18m.
Angola: Institutional Support to the Development of Water Supply and
Sanitation Sectors in Angola. Training, capacity building, water quality
control, promotion of sanitation. Value 17,7m.
2011:
Mozambique: Maputo Water Supply Project. Augmentation of the citys

water supply, from treatment works to bulk mains to distribution network
and connections. Value 95m.
Malawi: Malawi Peri-Urban Water and Sanitation Project. Support to
Blantyre and Lilongwe Water Boards, to increase water and sanitation
coverage and operational efficiency. Value 30,8m.
Angola: Institutional Support to the Development of Water Supply and
Sanitation Sectors in Angola. Training, capacity building, water quality
control, promotion of sanitation. Value 17,7m.
Angola: Ex-post. Water Supply and Sanitation for the Suburban Areas
of Luanda. Augmentation of water supply network, construction of
latrines, institutional strengthening. Value 22,0m.
2010:
Ethiopia: Accelerating Progress Towards the MDGs on Water and

Sanitation In Ethiopia. Water systems, sanitation, hygiene education,
institutional strengthening. Value 21,6m.
Ecuador: Strengthening of local water and sanitation role players within
an IWRM framework. W&S infrastructure, community and institutional
capacity building in O&M and IWRM. Value 1,0m.
El Salvador: Participatory hygiene and sanitation education to achieve
behaviour changes. Community managed water and sanitation
infrastructure, higiene education. Value 1,3m.
Mozambique: Beira Sanitation Project. Rehabilitation of the existing
sanitation system and extension of the sewerage and drainage network.
Value 62,6m.
Mozambique: Water Supply and Sanitation Project in Cabo Delgado.
Community managed O&M, water and sanitation infrastructure,
improved hygiene, improved WSS coordination. Value 2,2m.
Mozambique: Improvement of the Livelihood of Rural and Peri-Urban

Poor through Increased Access to Safe and Sustainable Water and
Sanitation and Hygiene Facilities. Value 5,6m.
Ecuador: Ex-post. Development of the Paute River Catchment.
Strengthening of catchment management authority, rehabilitation of
catchment resources. Value 18,9m.
2009
Nicaragua: Ex-post. Esteli-Ocotal Integrated project. Rehabilitation and

augmentation of water supply and sewage systems in the two cities.
Value 18,9m.
Somalia: Somalia Water & Land Information System III. Systems and
methodologies for monitoring water and land resources, government
capacity building, institutional development. Value 3,5m.
2008
Bolivia: Ex-post. Rehabilitation of the water supply and sewage

networks of the city of Santa Cruz. Rehab and augmentation of
networks, capacity building of utility. Value 20m.
2007
Cape Verde: Project of Water Supply and Eco Sanitation in Maio

Island. Desalination plant, network augmentation, toilets, hygiene
education, irrigation. Value 0,6m.
2006
Nambia: Rural Town Sewerage Scheme Luderitz. Network,

Treatment Works augmentation. Value 3,5m.
Oct/Nov
2011,
Jul/Aug
2012
Bolivia,
Guatemala
worldwide
BPD (Building
Partnerships for
Development in Watsan)
Tracey Keatman
traceyk@bpdws.org
Consultant
Evaluation of two projects of the Water and Development Alliance

Programme (an initiative of USAID and Coca-Cola) involving publicbusiness-community-NGO alliances for water resources management.
Case studies of 9 water resources management projects worldwide in
the Water Futures Partnership Programme (SABMiller, WWF, GIZ).
Feb
2012
Mozambique
ABIODES: N. Mahalambe
nmahalambe@gmail.com
Consultant
Contracted by Solidaridad Southern Africa to provide support to the

Mozambican NGO ABIODES to develop recommendations to
strengthen its governance, management and administrative systems
MarchMay
2010
Angola
UNDP / SIWI
UNDP: Amaya Olivares
amaya.olivares@undp.org
Consultant
Design and management of baseline study and development of M&E

system for MDG-F rural water supply and sanitation programme, based
on CLTS for sanitation and hand pump and small scale reticulated
technologies for water, including small towns.
Jul-Aug
2011
Angola
DW Angola: Allan Cain

allan.cain@gmail.com
Consultant
Trainer
Writing training materials and running 9 day training course for

municipal and DW water supply officials on municipal planning in water
and sanitation.
Writing training manuals and giving respective courses to government
water supply officials on community-based O&M of rural water supplies
based on hand pump and small scale reticulated technologies. Advising
on O&M and M&E strategies.
Nigeria
UNICEF: Bisi Agberemi

bagberemi@unicef.org
Consultant
Support for development of national sanitation programme

implementation guidelines, based on recently adopted policy and
strategy. Included drafting and facilitating consultations with
stakeholders.
Aug
2010,
Mar
2011
Peru
Consultant
Oct
2009
Paraguay
BPD (Building
Partnerships for
Development in Watsan)
Tracey Keatman
traceyk@bpdws.org
Case studies of multi-stakeholder partnerships in Lima and Asuncin,

as part of regional project: Assessing multi-stakeholder partnerships in
the water and sanitation sector within the context of urban policies in
Latin America and the Caribbean (LAC). Also as resource person in
regional workshop in Lima, and member of Reference Group for overall
project.
Nov
08 to
Feb
09
Nov
2007 to
Feb 08
B. PREVIOUS POSTS (IN REVERSE ORDER)

TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
MONTH/YEAR
October 2000
Feb 2007
NAME OF EMPLOYER:
The Mvula Trust
PO Box 32351, Braamfontein 2017, South Africa
FINAL
Executive Director (CEO)

TYPE OF BUSINESS:
Water and sanitation NGO

NAME OF SUPERVISOR:
Board of Trustees
SUPERVISED BY YOU: 60-110
REASON FOR LEAVING:
Emigration
Overall management of organisation of 60-110 staff and annual turnover of up to R 140m. Strategic leadership, stakeholder liaison,
fundraising, monitoring and evaluation, quality control, human resource management. Also project management of selected strategically
important contracts with key government and donor clients. Writing of articles for publication.
TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
FINAL
MONTH/YEAR
January 1996
Sept 2000
NAME OF EMPLOYER:
The Mvula Trust
Projects and Policy Director

TYPE OF BUSINESS:

NAME OF SUPERVISOR:
Piers Cross, Horst Kleinschmidt

SUPERVISED BY YOU: 12-20
REASON FOR LEAVING
Promotion
Management of regions, support in terms of policy guidance, capacity building, project and programme management systems, implementation
procedures, budget control, financial performance. Leadership and management of Policy Unit dedicated to co-ordinating and leading policy
development and advocacy work of organisation. Networking, partnership development, writing.
TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
FINAL
MONTH/YEAR
January 1995
Dec 1995
NAME OF EMPLOYER:
The Mvula Trust
Water Engineer
TYPE OF BUSINESS:

NAME OF SUPERVISOR:
Johann Froneman
SUPERVISED BY YOU: None
REASON FOR LEAVING
Promotion
Reporting to Projects Director, providing technical expertise in water supply to Head Office and regional offices. Included appraisal of project
feasibility studies, development of project implementation procedures and guidance to regions on appropriate technologies.
FROM
MONTH/YEAR
TO
MONTH/YEAR
SALARIES PER ANNUM

STARTING
FINAL
July 1991
Dec 1994
NAME OF EMPLOYER:
GTZ
Unknown - no longer operating in this area.
Water Supply & Irrigation Programme Manager, and

Infrastructure Programme Co-ordinator
TYPE OF BUSINESS:
Rural Development programme

NAME OF SUPERVISOR:
Dr Uli Weyl
REASON FOR LEAVING
SUPERVISED BY YOU: 4
Return to South Africa
Working within provincial and district government departments, supported the planning, implementation, monitoring and evaluation of water
supply development projects in four districts. Managed the introduction of improved, hand-dug, shallow well technology, and promotion and
expansion of the programme from pilot areas into all four districts, based on community contributions, local artisans and decentralised
government support. Initiated and expanded demand responsive and community managed programme of low pressure, gravity fed, spring
supplied piped schemes.
TO
FROM
MONTH/YEAR
MONTH/YEAR
January 1990
June 1991
NAME OF EMPLOYER:
Citrinos de Manica
No longer exists.
SALARIES PER ANNUM

STARTING
FINAL
Chief Engineer
TYPE OF BUSINESS:
Commercial citrus parastatal

NAME OF SUPERVISOR:
Osias Manjate
REASON FOR LEAVING
Career move
Responsible for upgrading of irrigation systems for 600ha of citrus plantations. Conversion of four fruit processing facilities for palletised
transport. Liaison with South African financiers and technical support.
TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
FINAL
MONTH/YEAR
January 1986
Dec 1989
NAME OF EMPLOYER:
Uniao Geral das Cooperativas Agro-pecuarias de Maputo
No longer exists.
TO
FROM
MONTH/YEAR
MONTH/YEAR
April 1981
Dec 1985
NAME OF EMPLOYER:
Citrinos de Maputo
No longer exists.
Chief Engineer
TYPE OF BUSINESS:
Co-op support agency

NAME OF SUPERVISOR:
Prosperino Gallipoli
REASON FOR LEAVING
SUPERVISED BY YOU: 9
Career move
Reporting to CEO, responsible for Water Supply Department, providing potable and irrigation water to small farmer co-operatives in outskirts
of city of Maputo. Managed the design, installation, training of staff and maintenance of over 100 small, appropriate technology irrigation
schemes.
SALARIES PER ANNUM
STARTING
FINAL
Chief Engineer
TYPE OF BUSINESS:
Commercial citrus parastatal

NAME OF SUPERVISOR:
REASON FOR LEAVING

Company collapsed
Responsible for setting up preventative maintenance systems for farm machinery and equipment, and training staff, as well as rehabilitation of
irrigation systems after Cyclone Demoina.
Appendix 12: Photographs
1.) Bahalara-uain, Viqueque.

Example of poorly buried main distribution pipe
due to shallow trenches.
2.) Bahalara-uain, Viqueque.

Abandoned standpipe, no wheelchair ramp,
severe erosion around apron, poorly built spillage
drain, already separated from apron.
3.) Poeana, Ermera.

Well built and maintained standpipe, with
wheelchair ramp, but erosion already evident at
corner of ramp.
4.) Saramata, Ermera.

Well built and maintained standpipe, but poorly
made spillage drain already separating from
apron.
5.) Domingos school, Saramata, Ermera.

Good quality design and construction, but no
overflow control on tank (inlet valve already
broken) causing muddy access to toilets. Zinc
coated steel tank not durable.
6.) Cocoa school, Ermera.

Good quality design and construction, but inlet to
tank passing through overflow hole and not
secured at base. Surroundings muddy. Zinc
coated steel tank not durable.
7.) Bona 3, Oecusse.

Hand dug well with lining, roof pulley and rope,
well maintained and functioning well.
8.) Bona 1&2, Oecusse.

Well built and maintained basic pit latrine made
only of local materials, with cover for squat hole.
9.) Samalete 2, Aileu.

Main distribution pipe exposed by erosion due to
lack of storm water diversion works.
10.) Tulataqueo school, Samalete 2, Aileu.

Well designed and constructed facilities but septic
tank already exposed by erosion due to lack of
storm water diversion works.
11.) Nunusakari school, Ermera. Well designed

and built facilities. Waste water drain inspection
box cover too light and not countersunk so
susceptible to misuse by children and possible
blockage of drain.

EU UNICEF WASH Project Evaluation Final Report

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EU UNICEF WASH Project Evaluation Final Report

Uploaded by

Copyright:

Available Formats

End of Project Evaluation

Improving Access to Water, Sanitation and Hygiene (WASH)

Prepared by: Martin Rall

List of Acronyms, Terms and Abbreviations

Variously translated as community, village or sub-village or hamlet (see also

Bee Saneamento no ljiene (Water, Sanitation and Hygiene in Tetum

AusAID (DFAT)-funded Rural Water Supply and Sanitation Programme

Directorate General for Water and Sanitation, Ministry of Public Works

National Directorate for Water Services

Community Lead Total Sanitation

District Technical Officer

Focus group discussion

Grupu Maneja Fasilidade (Water Facility Management Group)

Hand washing facility

Hand washing with soap

Knowledge Attitudes and Practices

Logical framework (of project)

Operation and Maintenance

Open defecation free

Millennium Development Goal

Non Governmental Organisation

Planu Asaun Komunidade Saneamento no Ijiene (Community Action Plan

District Water and Sanitation Service Ministry of Public Works

Water and Sanitation Information System

Timor-Leste Demographic and Health Survey

United Nations Childrens Fund

Water, Sanitation and Hygiene

World Health Organisation

Description of Project ....................................................................................................... 1

Purpose and Scope of the Evaluation ............................................................................. 6

Evaluation Methodology ................................................................................................... 6

Evaluation Questions ................................................................................................... 7

M&E Indicators ...................................................................................................... 7

Sources of information and methods of data collection................................................. 8

Endline Survey and KAP Study Report ...................................................................... 10

Sampling Design and Sample Size ..................................................................... 11

Limitations to the Evaluation and Mitigation Approaches ............................................ 12

Ethical considerations ................................................................................................ 13

Activities Undertaken ...................................................................................................... 13

Phase 1: Preparation and desk review ....................................................................... 13

Phase 2: In-country field work .................................................................................... 13

Relevance for target groups ................................................................................ 15

Alignment with government priorities ................................................................... 15

Alignment with funders country strategies and objectives ................................... 15

Quality of project design ...................................................................................... 16

Implementation against work plans ..................................................................... 17

Quality of project management............................................................................ 18

Quality of products .............................................................................................. 19

Cost effectiveness ............................................................................................... 19

Partnerships with implementing NGOs ................................................................ 20

Coordination with other programmes................................................................... 21

Risk management ............................................................................................... 21

Achievement of outputs and Results ................................................................... 21

Achievement of Specific Objective ...................................................................... 24

Contribution to MDGs .......................................................................................... 25

Beneficiary perception of benefits from project .................................................... 26

Capacity building ................................................................................................. 26

Compliance with norms and standards ................................................................ 27

Risk management ............................................................................................... 27

Achievement of Overall Objectives ...................................................................... 28

Risk management ............................................................................................... 28

Community water supply systems ....................................................................... 29

Household sanitation ........................................................................................... 31

Sanitation and hygiene related knowledge and behaviour change ...................... 33