Strategic evaluations

There is increasing interest in how evaluation can contribute to the evidence base on what works (and what doesnt) in
aid. ODE is uniquely placed to assess performance across the Australian aid program and bring international best
practice to bear in identifying new and better ways of working.
ODEs evaluations are guided by relevant professional standards including the Australasian Evaluation Society's
guidelines for the ethical conduct of evaluations [PDF 496kb], and the DAC quality standards for development evaluation
[PDF 1.8mb].
ODE conducts evaluations and reviews to assess the effectiveness of Australian aid to country programs and priority
sectors, identify areas of good practice and highlight important lessons.
This page houses completed and published ODE evaluations and reviews. Visit the Current work page for evaluations
currently in progress.
2014 Quality Review of Aid Program Performance Reports
A desk based review of donor experiences in Sector-wide approaches in the health sector
Evaluation of the Australia-Vietnam country strategy 201015
A window of opportunity: Australian aid and child undernutrition
Research for better aid: an evaluation of DFAT's investments
Working in decentralised service systems: challenges and choices for the Australian aid program
Smart Economics: Evaluation of Australian aid support for womens economic empowerment
Australias response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report
Two reviews of Australian aid operational evaluations
Evaluation of Australian aid to Timor-Leste
Evidence reviews
2013 Quality Review of Aid Program Performance Reports
Lessons from Australian Aid report (2013)
Australian Volunteers for International Development (AVID) Evaluation Report (2013)
Thinking and Working Politically: An evaluation of policy dialogue in AusAID (2013)
Building on Local Strengths: Evaluation of Australian law and justice assistance (2012)
Responding to Crisis: Evaluation of the Australian aid programs contribution to the national HIV response in
Papua New Guinea, 20062010
From Seed to Scale-Up: Lessons learned from Australias rural development assistance
Working Beyond Government: Evaluation of AusAIDs engagement with civil society in developing countries

2014 Quality Review of Aid Program Performance Reports

The 2014 Quality Review of APPRs found that the 201314 APPRs are well written, informative descriptions of program
activities and progress towards objectives. They discuss successes and failures in a largely balanced and evidencebased manner and appear to be used by programs as performance management tools. A particular highlight in the
APPRs was strong content on gender equality.
The quality review identified a few issues facing some APPRs. Firstly, many do not clearly explain program strategy
though the introduction of Aid Investment Plans (due later in 2015) should assist with this. Secondly a substantial number
of program objectives, and/or indicators lack clarity. The review also compared the 201213 and 201314 APPRs and
found there has been considerable movement in some programs objectives over the two years, which needs to be
considered if APPRs are to be used to measure changes in performance over time.
Both this and the previous years quality review of APPRs demonstrates that the credibility of APPRs assessments of
program performance is enhanced when programs can draw on good quality performance assessment frameworks
(PAFs). As such, the 2014 Quality Review recommends that all programs that produce APPRs should develop PAFs as
part of their Aid Investment Plans.
The full 2014 Quality Review of APPRs can be found below:
2014 Quality Review of Aid Program Performance Reports (PDF 375kb)
2014 Quality Review of Aid Program Performance Reports (Word 219kb)

A desk based review of donor experiences in Sector-wide approaches in the

health sector

A desk based review of donor experiences in Sector-wide approaches in the health sector report presents the findings
from a desk-based review of evidence on the effectiveness of donor support for the health sector through the use of
sector-wide approaches (SWAps), sector budget support and government systems in six countries where the Australian
aid program has used such approaches and/or systems (Bangladesh, Cambodia, Nepal, Papua New Guinea, Samoa,
and Solomon Islands). The report is based on the findings of donors (including Australia) experience in supporting
sector-wide approaches in Asia and the Pacific. It does not evaluate the effectiveness of Australias broader interventions
in health system reform.
Read the review:
Sector wide approaches in the health sector: A desk-based review of donors experience in Asia and the Pacific (PDF
1.29mb)
Sector wide approaches in the health sector: A desk-based review of donors experience in Asia and the Pacific (Word
387kb)

Evaluation of the Australia-Vietnam country strategy 201015

ODEs evaluation of the Australia-Vietnam 2010 2015 country strategy assesses the effectiveness of the strategy and
the results achieved through Australian aid up to 2013. As the current strategy period comes to an end the evaluation
aims to help inform the development of a new Aid Investment Plan (a new process which replaces country strategies) to
guide the targeting and delivery of Australias ongoing aid program with Vietnam.
The evaluation found that the Australia-Vietnam country strategy 2010-15 was developed through a collaborative process
which resulted in a robust framework for the Australia-Vietnam aid relationship. The implementation of the Australian aid
program has also been strong, with good achievements in sectors such as rural water and sanitation, infrastructure and
scholarships. However, there is room for improvement in areas such as monitoring the outcomes of Australias aid to
Vietnam and integrating gender issues and the private sector into the aid program.
The evaluation made four recommendations to strengthen future Australia-Vietnam Aid Investment Plans and the delivery
of Australian aid to Vietnam. DFAT has accepted all four recommendations.
Read the evaluation report:
Evaluation of the Australia-Vietnam country strategy 201015 (PDF 0.99mb)
Evaluation of the Australia-Vietnam country strategy 201015 (Word 968kb)

A window of opportunity: Australian aid and child undernutrition

The Australian Governments 2014 development policy identifies early childhood nutrition as a critical driver of better
development outcomes. Undernutrition in children between conception and 2 years of age can result in irreversible
stunting (short stature), poor cognitive development and poor health. This ultimately affects educational attainment and
future productivity and earnings. Several countries in Australias immediate neighbourhood have stunting rates
comparable to those of Sub-Saharan Africa and South Asia. ODEs evaluation found that the Australian Government's
investments generally constitute good practice. A number of recommendations were made to improve the targeting of
child undernutrition (especially during the critical first 1000 days of a childs life) and the monitoring and reporting of
Australia's efforts.
Read the summary brief:
ODE brief: A window of opportunity: Australian aid and child undernutrition (PDF 540kb)
Read the evaluation report:
A window of opportunity: Australian aid and child undernutrition (PDF 1.06mb)
A window of opportunity: Australian aid and child undernutrition (Word 1.33mb)

Research for better aid: an evaluation of DFAT's investments

ODEs Research for better aid evaluation assesses the degree to which the investment by the Australian aid program into
research has been appropriate, effective and efficient, and provides recommendations for improving DFATs future
management of research investment. The evaluation highlights the need for robust knowledge management systems and
a strong culture of research use to be embedded in the department. It also finds that there are clear benefits in looking to
improve the investment in developing country research capacity, either directly or through partnerships with Australian
and international researchers. The full evaluation report and management response can be found below.
Research for better aid: an evaluation of DFAT's investments (PDF 1.81mb)
Research for better aid: an evaluation of DFAT's investments (Word 6.66mb)

Working in decentralised service systems: challenges and choices for the

Working in decentralised service systems: challenges and choices for the

Australian aid program
This is the full report of an ODE-commissioned evaluation (conducted December 2012April 2014) which examined
Australias support for service systems in decentralised contexts. The evaluation focussed on the health, education and
infrastructure (water, sanitation and roads) sectors. It examined programs and initiatives in Indonesia, Papua New
Guinea, the Philippines, Solomon Islands, Vietnam and Bangladesh, with the first three countries covered in more depth
through field work. The evaluation found Australian aid is beginning to respond to the challenges of supporting service
delivery in decentralised contexts, but noted the results are mixed and there is room for improvement. The evaluation
made four recommendations for improving country level analysis, program planning and design to better address
decentralisation, and to strengthen DFATs capacity to effectively manage and monitor service delivery programs in
decentralised contexts.
Working in decentralised service systems: challenges and choices for the Australian aid programFull report (PDF
3.02mb)
Working in decentralised service systems: challenges and choices for the Australian aid programFull report (Word
3.84mb)

Working in decentralised service systems: Short report on implications for aid managers
This is the ODE summary of the findings and recommendations of the evaluation Working in decentralised service
systems: challenges and choices for the Australian aid program (Office of Development Effectiveness, 2015). The short
report synthesises advice for program managers from the full evaluation report, drawing on Australias experience with
decentralised delivery of health, education and infrastructure (water, sanitation and roads) in Bangladesh, Indonesia,
Papua New Guinea, the Philippines, Solomon Islands and Vietnam. Three priorities are suggested for improving future
approaches: i) work to improve service systems rather than deliver services where this is feasible; ii) choose
implementation partners carefully, and consider unintended effects; iii) work towards consistent and coordinated
investments, policies and systems.
Working in decentralised service systems: Short report on implications for aid managers (PDF 193kb)
Working in decentralised service systems: Short report on implications for aid managers (Word 198kb)

Smart Economics: Evaluation of Australian aid support for womens economic

empowerment
ODE has released its evaluation of Australian aid support for womens economic empowerment. The evaluation
assesses the effectiveness of Australias policies and programming to promote womens economic development. It finds
that whilst Australias policies to promote womens economic empowerment are sound, implementation approaches are
generally weak. The evaluation makes four key recommendations to improve the effectiveness of Australian economic
programming.
Read the full evaluation report, management response, policy brief and related documents

Australias response to the Horn of Africa Humanitarian crisis, 2011 Evaluation

Report
ODE has released its evaluation of Australias response to the Horn of Africa Humanitarian crisis, 2011. The evaluation
aimed to assess the effectiveness of Australias response and identify how to improve Australian humanitarian assistance
to future slow-onset crises. The evaluation found that Australias response was in keeping with the magnitude of the
crisis. Australias leadership role and the speed and flexibility of Australias response were notable strengths. The full
evaluation report and management response can be found below.
Australias response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report (PDF 3.30mb)
Australias response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report (Word 9.70mb)
Read the summary briefs:
ODE Brief: Horn of Africa humanitarian crisis, 2011 (PDF 1.31mb)
ODE Brief: Gender dimension, Horn of Africa humanitarian crisis, 2011 (PDF 1.08mb)

Two reviews of Australian aid operational evaluations

The Government is committed to improving the effectiveness and efficiency of Australian aid, and has introduced a new
performance framework which will see funding at all levels of the aid program linked to progress against rigorous targets
and performance benchmarks. Independent evaluations are another important way of ensuring a strong focus on results
and value for money.
All significant Australian aid initiatives require independent evaluation at least once, under current DFAT policy. These

All significant Australian aid initiatives require independent evaluation at least once, under current DFAT policy. These
independent operational evaluations are commissioned and managed by DFAT areas with aid management
responsibilities. They are an essential part of Australias aid performance management and reporting system. The 87
operational evaluations completed in 2012 are the subject of these two ODE publications.
Quality of Australian aid operational evaluations assesses the quality of the evaluations, and considers underlying
factors influencing evaluation quality and utility.
The review confirms that the majority of operational evaluations (74 per cent) are credible, and provide robust evidence
about the performance of the Australian aid program.
The review identifies several areas for further improvement. It provides practical guidance for the DFAT areas that
commission operational evaluations, as well as specific recommendations to improve evaluation quality and utility.
The reviewed operational evaluations that have been published can be accessed via ODEs operational evaluations
page.
Read the summary brief:
ODE Brief: Quality of Australian aid operational evaluations (PDF 1mb)
ODE Brief: Quality of Australian aid operational evaluations (Word 841kb)
Read the full report and management response:
Quality of Australian aid operational evaluations (PDF 1.63mb)
Quality of Australian aid operational evaluations (Word 568kb)
Learning from Australian aid operational evaluations synthesises the findings of the 64 evaluations assessed as
credible and offering lessons of potential value to a wider audience.
This review addresses an identified gap in the dissemination of Australian aid evaluation findings since many of the
original evaluation reports can be hard to locate or readily digest. The reviews purpose is to inform and improve program
design and management within the Australian aid program, and to provide lessons of value to the broader development
community.
The report identifies nine key lessons for improving the effectiveness of the Australian aid program.
Read the full report:
Learning from Australian aid operational evaluations (PDF 1.27mb)
Learning from Australian aid operational evaluations (Word 468kb)

Evaluation of Australian aid to Timor-Leste

ODEs evaluation of Australian aid to Timor-Leste focused on the planning and implementation of aid to Timor-Leste from
2006 to 2013. It concluded that the country program has made steady but uneven progress in implementing several of
the country strategys key commitments, and that DFAT needs to build coherent, long-term and appropriately-resourced
programs that address a limited set of clear and compelling development objectives.
ODE also found that in countries affected by or transitioning from conflict, shorter-term, more flexible planning
approaches which focus on prioritising and sequencing aid efforts may be required. These should pay stronger attention
to resourcing requirements than in contexts where business-as-usual approaches may be adequate.
The evaluation made five recommendations aimed at strengthening the effectiveness of assistance to Timor-Leste, and
the effectiveness of DFATs country program aid more broadly. DFAT has accepted all the recommendations.
Read the evaluation report:
Evaluation of Australian aid to Timor-Leste report (PDF 5.59mb)
Evaluation of Australian aid to Timor-Leste report (Word 4.37mb)
Read the summary brief:
ODE Brief: Australian aid to Timor-Leste (PDF 1.12mb)

Evidence reviews (2014)

ODE has released three evidence reviews that will inform its current work on evaluating child nutrition, womens
leadership and economic empowerment, and teacher quality. These reviews gather and assess evidence that can be
used to inform decisions about the scope, focus, questions and methods of proposed evaluations.

Teacher quality: Evidence review (2014)

An evidence review was commissioned to gather and assess evidence and inform recommendations about appropriate
research methods for an evaluation of Australian aid for teacher development.
The review focuses on:
the meaning of teacher quality and its relationship to student outcomes
factors that influence teacher quality
international influence and engagement with teacher quality.
Teacher quality evidence review (PDF 1.01mb)
Teacher quality evidence review (Word 1.41mb)

Womens leadership: Evidence review

ODE has commissioned an evidence review of womens leadership. This topic is the second of the three evaluations
focused on gender equality and womens empowerment.
The review focuses on:
Definition of womens leadership: How does Australian aid define womens leadership (in policies, strategies or
programs)? How does this compare to other donor approaches and international literature?
Data availability and quality: Are there adequate data (in Australian aid and other donor contexts) to assess the
effectiveness of womens leadership programs on gender equality outcomes and broader development outcomes?
What is the quality of these data?
Leadership and empowerment links: Is there evidence to link womens economic empowerment and womens
leadership outcomes? What does existing evidence say and what is the quality of this evidence?
Womens leadership: Evidence review (PDF 1.32mb)
Womens leadership: Evidence review (Word 2.09mb)

Child nutrition
Addressing child nutrition: Evidence review (2014)
ODE has commissioned a review of the current evidence-based approaches to addressing child under-nutrition and how
they may apply to the Australian aid program. The review focuses on:
interventions to address child under-nutrition
contemporary policy thinking and approaches to nutrition
child nutrition data.
Addressing child nutrition: Evidence review (PDF 2.07mb)
Addressing child nutrition: Evidence review (Word 1.47mb)

2013 Quality Review of Aid Program Performance Reports

ODE assesses the quality of APPRs and other performance management instruments that DFAT uses to monitor and
evaluate the aid program.
The 2013 ODE APPR Quality Review confirms a gradual, clear improvement in the quality of APPRs since 2007-08 when
they were first introduced. It found that the reports generally provide frank, well-written explanations of performance and
serve their accountability role well.
APPRs have the potential to provide considerable guidance to senior managers on what constrains and enables good
performance in programs. They highlight three key drivers of good program performance:
1. Effective policy dialogue to enable stronger partnerships and improved country ownership of aid programs
2. Capacity-building focused on institutional support
3. Less fragmentation of programs.
To improve their clarity and credibility, ODEs Quality Review recommends that APPRs provide better explanations about
the rationale underlying program objectives, broaden the use of credible evidence (especially that gathered from
partners), and improve the utility of the reports as management tools. Whole of Government performance also needs to
be better assessed in APPRs.
The full Quality Review can be found below:
2013 Quality Review of Aid Program Performance Reports (PDF 668kb)
2013 Quality Review of Aid Program Performance Reports (Word 1.39mb)

Lessons from Australian Aid report

Lessons from Australian Aid report

The Office of Development Effectiveness (ODE) has released its 2013 Lessons from Australian Aid report. The first in an
annual series, this major report provides an independent perspective on the strengths and weaknesses of the Australian
aid program.
Lessons from Australian Aid synthesises findings from six recent strategic evaluations conducted by ODE. It also
analyses the findings from 2012-13 performance reporting undertaken by DFAT program managers, and quality-assures
these products. The report contains useful lessons for Department of Foreign Affairs & Trade (DFAT) aid managers,
grouped around three main themes:
Well-informed policy dialogue with Australias development partners helps create high-quality policy frameworks.
As a result, Australian aid reaches more of the poor.
Government isnt the only development partnerDFAT should keep building strong relationships with the private
sector and civil society as well, to leverage high-quality outcomes.
In low-capacity settings, building the capacity of institutions (rather than individuals) in ways that are tailored and
paced to reflect local realities holds most potential for success.
While these lessons are not new to international development, they continue to be relevant to the Australian aid program.
They also pose a challenge for all other major donors. Applying the reports lessons consistently across the aid program
should improve the effectiveness and impact of Australias development assistance.
The full evaluation report can be found below:
Lessons from Australian Aid2013 report on independent evaluation and quality assurance (PDF 527kb)
Lessons from Australian Aid2013 report on independent evaluation and quality assurance (Word 503kb)

Australian Volunteers for International Development (AVID) Evaluation Report

ODE has released its evaluation of the Australian Volunteers for International Development (AVID) program. The
evaluation aimed to enhance the effectiveness of the AVID program and the contribution that volunteers make to
development.
Launched in 2011, AVID was designed to unite a variety of volunteer programs into one major program. AVID is funded
and managed by DFAT, and is delivered by three core partners: Australian Volunteers International, Australian Red
Cross, and Austraining International.
Overall, the evaluation found that AVID is making an effective and highly-visible contribution to the Australian
Governments development and public diplomacy objectives. Volunteers contribute to developing the capacity of their
host organisations, develop people to people links and generate goodwill for domestic and foreign diplomacy.
However, the programs efficiency and effectiveness can be improved and the evaluation makes recommendations to this
effect including:
Further consolidating AVID into a single program through one volunteer stream
Improved branding of the program
Improved administrative efficiencies including through consolidation of the number of countries receiving
volunteers
Greater focus on developing the long-term capacity of host organisations
Developing a simplified and effective performance monitoring system to promote continuous improvement of the
AVID program.
Read the summary brief:
AVID summary brief (PDF 536kb)
AVID summary brief (Word 452kb)
The full evaluation report and management response can be found below:
Evaluation of the Australian Volunteers for International Development (AVID) program (PDF 1.74mb)
Evaluation of the Australian Volunteers for International Development (AVID) program (Word 3.08mb)

Thinking and Working Politically: An evaluation of policy dialogue in AusAID

The Office of Development Effectiveness (ODE) has released its evaluation, Thinking and Working Politically: An
evaluation of policy dialogue in AusAID.
The need for AusAID to identify, define and practice good policy dialogue was highlighted in ODEs 2009 Annual Review
of Development Effectiveness and the 2008 OECD Development Assistance Committee peer review of Australias aid
program.
The evaluation included a literature review of international best practice, case studies from Indonesia and Solomon
Islands, interviews with AusAID staff and a web-based staff survey.

Islands, interviews with AusAID staff and a web-based staff survey.

Broadly, the evaluation recommended that policy dialogue be better incorporated into the agencys practices by:
promoting a common understanding and providing senior direction on policy dialogue
embedding policy dialogue into aid management practices
ensuring policy dialogue is properly resourced
supporting the skills development of staff.
More about the evaluation of policy dialogue in AusAID

Building on Local Strengths: Evaluation of Australian law and justice assistance

Law and justice is an increasingly important area of international development assistance, but there is little consensus
among OECD donors on how exactly law and justice contributes to wider development processes. Australia has emerged
as a leader in this area, delivering law and justice assistance through a number of Australian Government agencies
including DFAT, the Australian Federal Police and the Attorney-General's Department. Law and justice assistance
comprises nearly 15 per cent of the bilateral aid program.
ODEs evaluation, Building on Local Strengths: Evaluation of Australian law and justice assistance, assessed the
relevance and effectiveness of Australias law and justice programs.
More about the evaluation of the Australian law and justice assistance

Responding to Crisis: Evaluation of the Australian aid programs contribution to

the national HIV response in Papua New Guinea, 20062010
The Australian aid program has played a prominent role in responding to HIV in Papua New Guinea (PNG) since 1995.
The Office of Development Effectiveness conducted a major independent evaluation into AusAIDs contribution to the
response. The evaluation focused on the current HIV program activities from 2006, led by the PNG-Australia HIV and
AIDS Program, in the broader historical context.
Responding to Crisis: Evaluation of the Australia aid programs contribution to the national HIV response in
Papua New Guinea finds that AusAID has been a major driver of the national HIV response in PNG. However, the
relevance and effectiveness of AusAIDs interventions has been mixed.
In light of the evolving nature of the epidemic and response, the evaluation recommendations that AusAID:
moves to an integrated health approach, with HIV activities managed as part of broader sexual and reproductive
health activities
steps back from a dominant role in shaping and implementing the response and intensifies support for PNG
champions of the HIV response
retains the leadership of the in-house senior technical expert but contracts out grant management and capacity
building functions.
AusAID has provided a management response to the evaluation.

From Seed to Scale-Up: Lessons learned from Australias rural development

assistance (2012)
ODE has conducted a review of Australias rural development assistance, to identify ways of maximising the benefits of
Australias growing investment in rural development. From Seed to Scale-Up: Lessons learned from Australias rural
development assistance reviewed 23 recent Australian activities across six countries in the Asia-Pacific. These
activities were implemented by AusAID and/or the Australian Centre for International Agricultural Research (ACIAR).
The review found that Australian aid has helped poor rural women and men access more value from new markets and
make more effective use of scarce natural resources. Australian aid has also helped them to accumulate assets so that
they can afford to send children to school, pay for health care and gain access to other essential services.
From Seed to Scale-Up has identified 12 principles that characterise efficient and effective aid investment in rural
development and makes recommendations to increase the capacity of Australia and its development partners to deliver
results in rural development A joint AusAID-ACIAR management response is included in the report.

Working Beyond Government: Evaluation of AusAIDs engagement with civil

society in developing countries (2012)
Civil society in developing countries can be a powerful agent for change, and is an important development partner in

Civil society in developing countries can be a powerful agent for change, and is an important development partner in
addition to partner governments and the private sector. ODE has conducted an evaluation of AusAID's engagement with
civil society, assessing how well AusAID is helping civil society in developing countries contribute to the development of
effective states.
Working Beyond Government: Evaluation of AusAIDs engagement with civil society in developing countries
recommends that Australia build on current strategic approaches for engaging with civil society, work with local systems
and partners, and apply good practice in the design of individual programs. The evaluation also found that choosing
appropriate intermediaries can help donors enhance sustainability, manage risk, reduce high transaction costs and take
small activities to scale.
Read the Working Beyond Government evaluation

Philippines Country Program and Strategy Evaluation (2012)

Country strategy and program evaluations are an important way for ODE to evaluate the performance of the Australian
aid program. In 2007, the Philippines country strategy was the first to be evaluated by ODE. This evaluation follows on
from that first one and provides a mid-term assessment of the AustraliaPhilippines Development Assistance Strategy for
200711.
Full report
Australian aid to the PhilippinesMid-term evaluation of the AustraliaPhilippines Development Assistance
Strategy 200711 [PDF 1.2mb]
Australian aid to the PhilippinesMid-term evaluation of the AustraliaPhilippines Development Assistance
Strategy 200711 [Word 5.3mb]
Supporting documentation for this evaluation
Gender Assessment of the Australian Philippines Aid Program [PDF 1.1mb]
Gender Assessment of the Australian Philippines Aid Program [Word 1.2mb]