Setting Up a Poverty Monitoring and Evaluation System:

Experience from Asia and Africa

by

Sylvain Lariviere, Ph.D. and Frederic Martin, Ph.D.

University Laval and IDEA International Institute

Preliminary Draft

Paper presented at the Regional Conference on

Poverty Reduction Strategies

4-6 December 2001

 2

Ha Noi, Viet Nam

1. PRELIMINARY CONSIDERATIONS

Let us start by outlining three points. First comes the recognition that data collection and analysis
are often misguided and useless because they are not guided by a sense of priority PME needs
and they are not fed into the decision-making process to become policy recommendations and
eventually budget allocations.

PME needs Data Data Policy Budget

assessment collection analysis recommendations allocation

The starting point for a PME system should be a rigorously defined Poverty Reduction Strategy.
This implies a strategic planning approach including:

1. Choosing a reduced number of strategic orientations of poverty reduction and setting

Specific Measurable Objectives (SMOs)1 such as the Millenium Summit International
Development Goals;
2. Identifying a set of priority actions, including policies and programs, to reach the target
SMOs of the strategic orientations at a given deadline;
3. Setting preliminary SMOs for priority actions, describing implementation strategies, and
costing
4. Finalizing the specification of SMOs for priority actions to ensure consistency between
available financial resources and the costs of reaching the SMOs.

Second, technical issues are intertwined with institutional issues: on one hand, the choice of
methodology depends partly on human resource capabilities and institutional relative strengths;
on the other hand, the choice of institutions and their respective roles in the PME system
depends on the technical characteristics of the proposed system.

Third, there is no unique PME system which is valid for all countries all the time. The starting
point is to define poverty (poverty criteria and poverty line) and related concepts of vulnerability

1
A Specific Measurable Objective is a target level at a given date on a cardinal scale for a given indicator.
 4

and inequality to reflect national and regional realities. Then the government in consultation with
other stakeholders (civil society, NGOs, development partners) should assess its needs in
terms of PME since various needs along with institutional and human capital characteristics will
lead to various PME objectives and designs.

2. PROPOSAL OF A GENERIC PME SYSTEM

Four types of needs have been expressed in many African and Asian countries regarding
poverty monitoring and impact evaluation:

1. The need to monitor the poverty situation, understand it, and identify the poorest areas;
2. The need to monitor the level of achievement of the Poverty Reduction Strategy (PRS) to
see if the work plan adopted is actually being implemented according to schedule and if
financial resources are allocated according to the set priorities;
3. The need to assess the impact of key policies and programs on poverty;
4. The need to assess the overall impact of the PRS.

These needs have to be assessed much more specifically and a consensus built around the
priority needs. However, in most cases, a good PME system should include three components
to address each of those needs (Figure 1):

• A Poverty Monitoring Component which focuses on poverty evolution (outcome

indicators);
• a PRS Monitoring Component which concentrates on budgetary (input) indicators and PRS
objectives achievement levels indicators (output);
• an Impact Assessment Component which, as indicated, focuses on impact indicators.2

These three components taken together make it possible to evaluate the global impact of the
PRS on poverty. Each component is discussed in the following sections.

2
The result-based management litterature distinguishes 5 categories of indicators which correspond to the
sequence of actions in any program or project : Inputs → Activities → Output → Outcome → Impact.
 5

FIGURE 1 : THE TECHNICAL COMPONENTS OF A GENERIC PME SYSTEM

POVERTY MONITORING COMPONENT

Population Well-Being related Poor Well-Being

Indicators System related Indicators System

• International and sub-regional • Poverty profile indicators

environment indicators
• Macroeconomic indicators
• Sectoral development indicators
• Local well-being indicators
• Vulnerability indicators
Source : Administrative services
• Poverty characterization indicators
• Poverty dynamics indicators
• Vulnerability indicators
• Inequality indicators
Source : Household survey system

INPUTS OUTPUTS OUTCOMES IMPACT

PRS Monitoring Poverty Impact Assessment

Component Component

Monitoring of Monitoring of • Impact of selected macroeconomic

budget allocations SMO’s achievement policies on poverty
for priority actions levels • Impact of selected sectoral policies on
poverty
• Impact of selected programs on poverty

Overall PRS Assessment

 6

Poverty Monitoring Component

This component may have several objectives:

• Keeping track of poverty trends. One may focus simply on income poverty criteria (using a
moneymetric criteria such as consumption expenditures) or include other dimensions of
human poverty as well (nutrition, health and education levels, other basic needs indicators,
etc.) and even human rights criteria. This will also help identify the poorest areas to focus
interventions and budget allocations;
• Understanding the endogenous and exogenous factors behind the poverty trends, i.e., why
do we observe a reduction or an aggravation of poverty over time? This introduces the
causal analysis of poverty;
• Keeping track of the dynamics of poverty over time, i.e. who is getting out of poverty, who
is getting into poverty, who is remaining in poverty? While the analysis of poverty trends
focus on the stocks of poverty, the analysis of poverty dynamics focuses on the flows in and
out of poverty;
• Understanding the endogenous and exogenous factors behind the dynamics of poverty, i.e.
the influence of shocks and types of capital endowment on poverty flows.

Indicators of the Poverty Monitoring Component fall into two categories: (i) population well-
being related indicators and (ii) poor’s well-being related indicators. The first category includes
indicators of the well-being of the whole population (e.g.: GDP per capita) and of factors
influencing this well-being level such as growth, rainfall, supply of social services, etc. They can
be sub-categorized as:

• International and sub-regional environment indicators

• Macroeconomic indicators
• Sectoral development indicators
• Local well-being indicators
• Vulnerability indicators.

Most of those indicators come from secondary sources and can be usually routinely collected
by public services.
 7

The second category includes indicators on the well-being of the poor and factors influencing it.
Often, these indicators are also calculated for the non-poor to serve as a basis for comparison.
These indicators can be grouped under the following sub-headings:

• Poverty profile indicators

• Poverty characterisation indicators
• Poverty dynamics indicators
• Vulnerability indicators
• Inequality indicators.

Most of the data needed to calculate these indicators require primary data collection. The most
efficient way to obtain those data is to set up a household survey system including various
surveys with qualitative and quantitative modules.

All these analyses must be disaggregated to be meaningful because poverty is unevenly

distributed. The analysis must be broken down from the national level towards subregional units.
These units may be administrative (province, district, commune) and agro-economic (urban/rural
milieus, agro-ecologically homogenous zones). The analysis must also be broken down by
gender since women and men do not have the same opportunities and constraints and poverty
levels in most societies.

Various poverty indicators might be monitored with different frequencies because they vary
more or less over time; some indicators need to be monitored quaterly, others yearly, every 2 or
3 years, etc.

Let us also note the possible use of proxies to real indicators which might be easier and cheaper
to monitor. An example of such a proxy might be the wage rate of unskilled wage labor or the
price of a staple food like rice.

The more disaggregated and frequent the monitoring, the higher the requirements in terms of
human and financial resources for the PME system. In a country characterised by significant
human capital limits, a stepwise approach is suggested whereby the PME system starts with the
monitoring of poverty trends with a limited number of poverty indicators and, as capacity
 8

building and experience increase human capital in the PME team, (i) more indicators are added
to the poverty trend modules, (ii) other modules are progressively added, and (iii) the level of
disaggregation is increased to lower administrative levels (province → district →commune).

After defining poverty, the second step in the elaboration of a PME system is to agree on a
priority set of core poverty, vulnerability and inequality (PVI) indicators.

PRS Monitoring Component

This component usually has two objectives :

• Monitoring the level of achievement of the Specific Measurable Objectives (SMO’s) of the
Poverty Reduction Strategy (PRS)3 and check whether the work plan adopted is actually
implemented according to schedule;
• monitoring the budget allocations made to reach the SMO’s by priority action of the PRS.
An adequate allocation of funds to the right use at the right time is a precondition to reach
SMO’s.

The indicators monitored therefore relate to the level of achievement of the PRS SMO’s at
various time horizons with a calculation of relative gaps and the level of budget allocation to
achieve the SMO’s of a given priority action. Setting up this component implies that SMO’s
have been specified and validated at the time of PRS formulation.

Poverty Impact Assesment Component

This component may have two objectives :

• Assessing whether a specific macroeconomic or sectoral policy or a specific program (e.g.,

a microfinance program) succeeded in reducing poverty ;

3
A Poverty Reduction Strategy usually involves a set of global poverty reduction objectives, a matrix of
poverty reduction oriented macroeconomic and sectoral policies, a set of strategic axes and priority actions,
a budget for financing those actions and a calendar of realisation. For the PRS to become operational and its
level of implementation monitored, one must transform the matrix of poverty oriented macroeconomic and
sectoral policies into a set of Specific Measurable Objectives (SMO’s), i.e., levels of key indicators related to
policies, strategic axes and priority activities at different time horizons.
 9

• assessing the level of efficiency of specific policies or programs, i.e., could we have
achieved higher poverty reduction at a reduced cost, faster or in a more sustainable way?

Impact assessment implies addressing a series of methodological challenges, including the

isolation of the impact of the policy or program under scrutiny on poverty from all the other
endogenous and exogenous factors that may affect the poverty outcome. The notion of impact
assessment also implies tracing the indirect effects on non target group well-being (e.g., children
benefiting from an income generation project for women) as well as the medium and long run
sustainable effects of the policy or program.

Ex post impact assessment is very useful to understand the past effects of policies and
programs and improve upon their future design. However, even more useful, is ex ante impact
assessment in which the impacts of a given policy or program are simulated, so that better policy
recommendations can be formulated. Ex ante impact assessment requires a clear and
structured analytical framework, a good understanding of the expected behavior of economic
agents, a minimum amount of valid and up to date data, and simple, but rigorous analytical tools
such as small empirical trade or budget models.

Selected Selected Selected

macroeconomic sectoral representative
policies policies programs
Ex post
evaluation
Ex ante
evaluation

When done seriously, impact evaluation is not easy, is costly and may require time4. At the same
time, it is most useful for policy analysis. Therefore it is suggested that this component be
gradually added to the PME with a careful selection of the policies and programs to be
evaluated.

This comment actually applies to the whole PME system. The suggested PME need assessment
exercise should be conducted early in the process of elaboration of this system and should not
end up being an endless shopping list of indicators in order to satisfy every user’s whims. In a

4
A serious ex post impact assessment may require from 1 to 2 years.
 10

country faced with strict human and financial capital constraints, priorities have to be established
and a gradual increase in depth and breadth of the PME system be considered as capacity
building is reinforced.

The overall impact of the PRS cannot be easily measured quantitatively since it is very hard, if
not impossible, to isolate the effect of an inters-sectoral strategy such as the PRS per se from all
other exogenous and endogenous variables that affect the population and the poor living
conditions. However, putting together and triangulating the results from the three PME system
components should provide a pretty good indication of the impact of the PRS and explain the
reasons behind the observed performance. Since there is a logical causal linkage between
inputs, outputs, outcomes and impact, for the PRS to have an impact implies in sequence that (i)
inputs have been allocated efficiently according to the set priorities (ii) the SMOs of priority
actions and strategic orientations have been achieved (iii) poverty and well-being indicators
indicate a better off situation with the PRS compared to a situation without the PRS and finally,
(iv) key policies and programs within the PRS have proven to reach the poor, change their
behavior in a sustainable way, and resulted in improved living conditions.

However, one should move beyond the quantitative impact measurment of the PRS on living
conditions and poverty to also take into account the impact of the process on two key
conditions of sustainable human development, i.e.:
(i) capacity building: designing a PRS, including a PME system, involves substantial formal
and learning by doing training to put the nationals in a position to master up to date
inter-sectoral and participatory strategic planning tools and qualitative and quantitative
data collection and analysis methods;
(ii) improved planning: The process should improve absorption capacity, more efficient
policies and program design and implementation, a closer articulation between civil
servants and civil society, and nationals being better equipped to negotiate with
development partners

3. PRELIMINARY LESSONS LEARNT FROM AFRICAN AND ASIAN

EXPERIENCES

Let us underline the very preliminary nature of lessons learnt since most countries are in the
process of designing their PRS and PME system.
 11

A rapid overall assessment of the PRS and PME processes indicates a number of challenges,
among which:

Challenges related the PRS, and especially the PRSP process:

• Often the time frame does not allow for a full participatory process and an indepth strategic
planing exercise;
• There is much generalities in a number of existing PRS without clear definitions of prority
actions, SMOs and costing of SMOs;
• There is often a lack of articulation between the costs of SMOs and available financial
resources;
• Designing a truly intersectoral PRS requires a change in the traditional way of designing
policies, programs and projects. Coordination is often the missing link among national
structures and among donors.

The implication for the PME system is that monitoring the PRS is very hard in such a context.

Challenges related the PME system:

• Routine administrative data are often: (i) not necessarily useful; (ii) collected on an irregular
basis; (iii) with very little quality control; (iv) not necessarily at the same level of geographic
desaggregation; (v) or at the same time, and (vi) poorly analyzed, especially in a policy
making perspective.

• Household surveys are still often biased toward large scale, expansive operations that
exceed national human, institutional and financial capacities and are therefore conducted on
an ad hoc basis. A lot of energy is still spent on data collection on large samples at the
detriment of analysis and policy recommendations.

• Apart from difficulties already mentioned at the PRS level, PRS monitoring is faced with the
difficulty of obtaining (i) disaggregated budgetary allocations to be able to establish a clear
link between financial resources and outputs (ii) the required data to measure the level of
achievement of SMOs on a timely and rigorous basis.
 12

• Impact assessment of key policies and programs requires addressing several challenges
among which: (i) methodological issues, in particular related to isolating the impact of the
specific policy or program under consideration, such as control group design; (ii) the time
frame required to obtain cardinal measurements of impacts; (iii) the cost of conducting a
serious impact assessment; (iv) the expertise required in qualitative and quantitative methods
and field experience in household analysis.

Above all, the PME greatest challenge is to build upon existing functional units and ensure
efficient co-ordination among national structures and development partners to ensure a timely
production of useful and a consensual analysis of results to be fed into the policymaking
process. This means that the PME requires the design and implementation of a Management
Information System (MIS) for greater efficiency in information flows.