Monitoring and Results

Measurement Manual

MRM Guide for PSD Programme





VEEDA Vranje and RDA Zlatibor
Version 1.3, March 2014




2 | P a g e

Contents
Abbreviations and Acronyms ...................................................................................................... 3
Executive Summary ................................................................................................................... 4
Section One: Introduction ........................................................................................................... 7
1.1. Standards ........................................................................................................................ 8
1.2. Purpose of MRM system ................................................................................................. 8
Section Two: The Results Measurement System ....................................................................... 9
2.1. Sector Analysis ................................................................................................................ 9
2.2. Intervention design .......................................................................................................... 9
2.3. Monitoring and measurement of intervention results .......................................................10
Section Three: Monitoring Documents ......................................................................................11
3.1. Sector Strategy ...............................................................................................................11
3.2. Intervention Guide ..........................................................................................................12
3.3. Impact assessment report ..............................................................................................17
Section Four: Methods, research practices and attribution ........................................................18
4.1. Definition of the PSD programmes target group .............................................................18
4.2. Impact indicators definition .............................................................................................18
4.3. Research practices .........................................................................................................20
4.4. Measurement and attribution methods means of verification .......................................21
4.5. Measuring indirect impact ...............................................................................................24
Section Five: Reporting .............................................................................................................25
Section Six: Roles and Responsibilities ....................................................................................25
Annex 1 Gender and youth ....................................................................................................27
Annex 2 Sustainability, systemic change and scale................................................................28
Annex 3 Tips on conducting good research ...........................................................................30
Annex 4 Business model example .........................................................................................31

3 | P a g e

Abbreviations and Acronyms

DCED Donor Committee for Enterprise Development
GoS Government of Serbia
GE Gender Equality
LED Local Economic Development
LSG Local Self-Government
MRM Monitoring and Results Measurement
M&E Monitoring and Evaluation
M4P Making Markets Work for the Poor
OECD-DAC Organisation for Economic Co-operation and Development-Development Assistance
Committee
PSD Private Sector Development programmes implemented by VEEDA and RDA Zlatibor
SECO Swiss State Secretariat for Economic Affairs
SCO Swiss Cooperation Office in Serbia
SDC Swiss Development Cooperation
VEEDA Regional Economic and Entrepreneurship Development Agency Pcinja District
RDA Zlatibor Regional Development Agency Zlatibor
WEE Womens Economic Empowerment









4 | P a g e

Executive Summary
This guide outlines how results measurement is done in Private Sector Development Programme. The
aim of the PSD programes MRM system reflected in this guide is to conform to existing international
standards, such as Donor Committee for Enterprise Development (DCED) Standard, and good practice
while remaining compatible with Serbian legislation and Serbias development context. This guide is
based on general MRM good practice, tailored to the M4P approach.
The objectives of PSDs results measurement are:
o To improve programme management.
o To generate credible information about PSDs results and lessons learned, to report to SDC.
o To generate credible information about PSDs results and lessons learned which can be used
to inform other interested parties.
o To be objective and transparent.

The nature of the market system (M4P) approach has significant influence on how PSD monitors and
measures results. Monitoring and results measurement should help project team ask the right questions
and point the where they should look for answers. To guide project team in their line of questioning and
support their search for answers, a number of tools can be used. These tools include results chains,
Adopt, Adapt, Expand and Respond matrix, etc. They are used trough routine process that support
knowledge accumulation and inform decision-making. These processes include internal strategy reviews,
notes and updates section in intervention strategy, etc.
When intervening, PSD will aim to stimulate sustainable additional employment and income for workers
(employees in the formal economy) in target municipalities. PSD will give particular attention to
stimulating jobs and income for women and youth. PSD defines youth as persons between 15 and 30
years old. Among workers, PSD will focus on those earning less than RSD 34,000 (about CHF 370).
PSD will also stimulate additional income for enterprises in target municipalities. Among those
enterprises, PSD will report as impact additional income generated only by micro, small and medium-
sized enterprises (MSMEs). PSD will not partner with enterprises to raise their profits in ways which have
little or no benefit for workers (e.g. replacing employees with machinery).
Serbian legislation affects how PSD define additional employment.
PSD do not count informal jobs in its baselines or its impact assessment. PSD will only count only
formally employed women and men (with labour / employment contract).
In agricultural sectors the programme counts only those women and men who have registered their
household (farm) and do business through a bank account (ie legal channels).

When reporting additional jobs to SDC, PSD will report on how many full-time equivalent jobs the
programme has created. PSD count as a full-time job in two ways, depending of the sector:

Industry/tourism full-time job definition:
Employed person is a person who is employed in accordance with the labour legislation and as such is
contributing to the mandatory social insurance. Here, a full-time job is defined as 240 working days per
year.


5 | P a g e

Agriculture full-time job definition:
In the case of agriculture, PSD counts income equivalent jobs instead of using 240 days per year as the
measure, with some additional conditions that should be fulfilled.

By income equivalent, we mean that a householder earns equal or more than the annual gross minimum
wage from new or more productive activities caused by PSDs intervention. Here, this will be counted as
one job. If a householder earns less than gross minimum wage, this will be calculated as a percentage of
a job.
To reflect PSDs particular emphasis on creating jobs and income for women and youth, PSD measures
additional income as follows:
- Total net additional income (CHF) for workers and enterprises.
- Within total net additional income, amount (CHF) generated by women.
- Within total net additional income, amount (CHF) generated by youth.
- Within total net additional income, amount (CHF) generated by workers and self-employed; amount
generated by SMEs.
- Women and youth who generate additional income as workers would be included. So would women and
youth who generate additional income as business owners.
- The number of jobs stimulated by the programme would also be disaggregated by gender, by youth, and
by employees/ self-employed.
Attribution refers to extent of change that can be claimed by a project/intervention out of total change
that takes place.

The methods which PSD uses to attribute changes to its interventions include:
o Creating results chains, to logically articulate how activities will lead to impact.
o Reviewing these results chains regularly, to check that the results reported at each level can
logically explain changes at the level above. If they cannot, PSD will commission extra research
to clarify attribution.
o Where applicable, measuring baselines.
o Where feasible, comparing beneficiaries with a control group of similar non-beneficiaries. When
doing this, PSD will take care to isolate the impact of differences between beneficiaries and non-
beneficiaries in the control group. This means looking out for differences (e.g. in attitudes
between people in treatment and control group) whilst testing surveys.
o Interviewing partners and beneficiaries to understand if the changes that they have made, which
appear to have resulted from PSDs facilitation, would have occurred anyway. Asking why
people change, in in-depth interviews, can provide information on sustainability too.
o For example, if PSD is promoting access to blueberry seedlings, PSD would ask new blueberry
farmers where they got their seedlings from. If there are other factors that contribute to the
change, PSD will conduct research which attempts to isolate changes created by PSDs
intervention. E.g. partners will be asked how much they would have changed without PSD
support. Where there is lots of uncertainty about what would have happened without PSDs
6 | P a g e

intervention, PSD will instead estimate how much its contribution influenced the change. Here,
PSD will only report its estimated contribution to changes as impact.
o Triangulate, e.g. complementing closed-question surveys with a few in-depth interviews. This
helps to avoid PSD hearing what we or they think we want to hear and not noticing hidden
influences, costs or benefits.

Estimating attribution will differ in a pilot phase and during scale up. During the pilot, PSD will validate the
different levels of changes in a results chain to ensure that change has occurred triggered by programme
activities. I.e. PSD will measure to what extend the change occurred due to the programme activities, and
keep asking why change happened at each step.

During the scale up, when there are other factors that are likely to be significantly affecting the change,
then the programme needs to layer another method on top of the above to assess attribution. A mix or
combination is often the best strategy to overcome disadvantages of each individual method.

PSD considers as direct impact the benefits to the target group created by PSDs support to partners (eg
cost sharing, training, mentoring etc). I.e. if PSD supports a processor and as a result 50 householders
start to cultivate herbs/fruit, these householders additional employment and income will be considered as
direct impact.
Indirect impact is defined as benefits to the target group and/or companies as the result of copying and
crowding in that takes place without project support.
Crowding in is the process through which interventions catalyse or bring in other players and functions in
the market system so that it works better for the poor.

New target enterprises or workers who benefit from this crowding in are referred to as copying; their
additional employment and income are considered indirect impact.
Examples of indicators at appropriate levels to monitor progress towards crowding in include:
Number of players who adopt new business model, without assistance from the project.
Number of farmers who adopt new cultivation practices, due to assistance from a processor who
has not received direct project support.
PSD will look for the changes in the market system and try to find evidence of copying and crowding in.
This involves visiting market players, both those directly supported by the programme, and those who are
not.
When interviewing partners, PSD will ask if partners are aware of other organisations which have
changed their practices due to PSD-supported changes. If so, PSD will follow up with these
organisations. PSD will informally ask them to verify where they got the idea to change from. Where
possible, PSD will try to measure what impact the crowding-in enterprises have on PSDs target group,
disaggregating as always by gender and youth.
PSD also measures indirect impact, when rural households spread an innovation between themselves.
For example, when PSD supports the cultivation of a new fruit variety, and beneficiaries give seedlings to
other farmers. Where this is likely, PSD includes a question in surveys and interviews with known
beneficiaries, asking if they have shared knowledge / inputs with other enterprises / rural households. If
they have, and if PSD was unaware of this, the PSD interviewer will ask for contact details of the copying
enterprises/ rural households. PSD will then interview these copiers to find out how much they have
benefited from the information.
7 | P a g e

As required by SDC, PSD reports on semi-annual basis with two types of reports: Operational report and
financial report. Both reports are presented at the same time so that SDC can compare results achieved
with costs, for each market.
The reports need to be comprehensive yet concise, they need to reflect on the expected information, and
most importantly the reports need to be accurate and up to date. Presenting the correct monitoring data
and current challenges is as important as underlining the progress and successes.

All MRM related responsibilities need careful planning, and oversight from the Team Leader. Hence,
within PSD, monitoring and results measurement is part of all everyones job. All technical staff job
descriptions and performance reviews include monitoring, results measurement and analysis of results.
Section One: Introduction
The purpose of this manual is to explain how Private Sector Development (PSD) programmes measure
results. Private Sector Development (PSD) programme is implemented by the Regional Development
Agency for Pcinja District (VEEDA) and RDA Zlatibor.
Private Sector Development (PSD) in South-West Serbia. RDA Zlatibor: The programme works in
tourism and traditional products, sectors which are anticipated to provide the best opportunity for growth,
employment and income generation in the programmes target region - 25 municipalities in Southwest
Serbia.
Private Sector Development (PSD) in South Serbia:
VEEDA: The project works in wood, finished wood products
(primarily furniture) and Non Timber Forest Products
(NTFPs) sectors. The programme has a national mandate
with a regional focus, and from September 2012 in addition
to South Serbia, covers Central, West and East Serbia,
where 70% of the targeted sectors production occurs. The
extended mandate covers 14 districts, instead of the initial
2 districts and targets 4 times the number of beneficiaries.
The two PSD programmes strive to further systematize
and develop their Monitoring and Results Measurement
(MRM) systems. This guide aims at supporting this effort of
bringing these MRM systems even closer to existing
international standards and good practice while remaining
compatible with Serbian legislation and Serbias
development context.
The programmes also aim to harmonise their MRM
systems, allowing for greater cross-learning and a degree
of comparability. Harmonisation is helped by the
similarities between the programmes: they share similar
project goals and the same implementation methodology
(M4P), donor and back-stopper. The two programmes also
jointly implement some interventions. This guide supports
the harmonisation of the programmes MRM systems,
With funding from the German,
Austrian and Norwegian
governments, GIZ Montenegro
used DCED guidance to
improve how it measured
results of a tourism
development programme. The
programme, Support to tourist
destinations in the central and
mountainous regions of
Montenegro, found that
adopting DCED guidance
enabled staff to measure more
systematically. The programme
used DCED guidance to
improve its results chains,
indicators and attribution. Staff
began to measure the
immediate effects of
programme activities, as well as
tourism businesses
employment and income. This
enabled the programme to
gather stronger evidence that
its activities contributed to
tourism businesses success.

8 | P a g e

serving as a common reference point.

This guide is based on general MRM good practice, tailored to the M4P approach.
This guide is not a static document. As the PSD programme evolves, so the measurement system may
need to evolve together with this guide.
This guide gives the general guidance in monitoring and measurement, and does not cover every
exception. If exceptions from general direction take place, PSD will deal with them on a case-by-case
basis.

1.1. Standards
The PSD programmes MRM system is based on international standards. It aims to be compliant with the
Donor Committee for Enterprise Development (DCED) Standard, which is designed specifically for private
sector development programmes. In doing so, PSDs measurement system should be capable of
producing information for evaluators, who often refer to OECD-DACs Evaluation Criteria.
1

Whilst aiming to conform to international standards, our MRM systems also reflect Serbian legislation.
2


1.2. Purpose of MRM system
Our MRM system has three main purposes: to provide accountability, transparency, and to enable
learning and based programme management decisions.
The objectives of PSDs results measurement are:
o To improve programme management. Providing managers with information to assess which
interventions are working, and which are not, and why. For this, PSD needs:
Results chains and indicators which are useful to track progress.
Regular tracking of these indicators, especially during pilots.
o To generate credible information about PSDs results and lessons learned, to report to SDC
o To generate credible information about PSDs results and lessons learned which can be used
to inform other interested parties. For example, Serbian government counterparts and other
development practitioners.
o To be objective and transparent. Maintaining partners and SDCs trust is vital to the success
of PSD. Some interventions may fail, either due to unforeseeable circumstances or wrong
decisions. PSD will rather learn from failure, and not try to hide such situations.



1
For example, both DCED and OECD-DAC emphasise the need for programme staff to be clear about
each interventions theory of change.Typically, programmes express this theory of change through their
Project Documents, results chains, and written intervention strategies. This is what PSD does. DCED and
OECD-DAC also see clear, specific, attributable indicators and targets as essential to monitoring progress
towards programme objectives. PSDs measurement system features both targets and indicators.
2
Serbian legislation affects how we define additional employment. See Section 5.2 for details.
9 | P a g e

Section Two: The Results Measurement System
This section summarises the main features of the PSD programmes measurement system. It briefly
describes the main roles played by different staff members. Due to the programmes size and the desire
for efficiency, the PSD programmes do not have separate departments for MRM, where MRM is seen as
an executive function. Intervention Managers are in charge of monitoring interventions with the support
from an MRM officer. Intensity of MRM officers support depends on real needs of Intervention Managers
and it could be described as precision consulting i.e. assistance in creating research methodology and
selecting appropriate market research methods.

2.1. Sector Analysis
When you decide whether to intervene in a sector, or where to intervene, gathering good evidence is
often a key to making the right decision. Gathering qualitative and quantitative information is therefore a
key part of sector analysis. Secondary information (eg studies written by people outside the programme)
can help staff to understand their target sectors, but PSD often needs to do conduct its own research to
fill gaps.
Here, MRM officers support Intervention Managers to design their research methodology, and may
participate in the research, and the processing of research findings, depending on needs and availability.
Particular role of MRM officer (who is not being 100% involved in research process) is to objectively and
independently support Team Leader in evaluating sector analysis. More on issues which sector analysis
has to address may be found in the section Sector strategy.
Figure 1: How PSD researches target sectors, and creates its sector strategies







2.2. Intervention design
Interventions address constraints to sustainable employment and income for PSDs target identified
through sector analysis. To decide which intervention ideas to pursue, PSD assesses each intervention
idea according to the following criteria:
Creating
method/plan of
sector analysis
Collecting
data on
sector
Analysing data and
determining supporting
fuctions and rules
Identifying
underlying
market
constraints
Analising
feasibility of
adressing market
constraints
Identifying gaps in
market research
Creating market
strategy
10 | P a g e

The number of additional jobs that the intervention is likely to create overall, and particularly for
women and youth.
The amount of additional income that the intervention is likely to create overall, and particularly
for women and youth.
How likely PSDs intervention is to create sustainable systemic change.
How much the intervention would contribute to womens economic empowerment.
Whether the intervention would support Serbias EU integration agenda.
In the early stage of intervention design, Intervention Manager must confirm that a proposed intervention
can bring about systemic change. It also is essential to determine if it is possible to monitor the change
and prove that it is result of project activities. The first stage is to create a results chain (for details on
result chain design techniques, see section 3.2). The results chain illustrates how you expect your
activities to lead to changes in the market system, and these changes to create employment and income
for the target group. Drawing a results chain helps you to visualise an intervention, and test how logically
it can be expected to lead to results.
Next, the Intervention Manager and MRM officer set indicators and propose how these indicators will be
measured. They also identify the main assumptions about how one change will lead to another, and make
plans to verify them. Setting indicators also helps staff to check that an intervention will lead to
measurable, attributable results for PSD. This is essential when deciding whether to pursue an
intervention. Next, data is collected data and baseline indicators collected.
Using this baseline data, an interventions expected results can then be projected. As the criteria above
show, two key criteria when deciding which interventions to pursue are additional employment and
income.
After PSD has decided to pursue an intervention, the responsible Intervention Manager may decide to
refine the results chain, or to alter it based on a change in intervention tactics. Indicators and projections
would then be revised accordingly.

2.3. Monitoring and measurement of intervention results
PSD has decided to pursue an intervention, and staff has
finished drafting the results chain and indicators, the next
stage is to plan data collection. For this, the Intervention
Manager and MRM Officer use the Measurement Plan (see
4.2). Data collection should be timed carefully. Firstly,
where possible, key results should be measured in time for
their inclusion in semi-annual reports to SDC.
3
Secondly,
monitoring and measurement should be regular, so that
findings can be used to improve project management.
Intervention Managers, with the Team Leader, should

3
For more information on Semi-annual Reports, see Section Five.
Cretaing
detailed result
chain
Defining key
indicators of
change
Collecting data Reports
Intervention
analysis
11 | P a g e

compare actual results to projections, in order to determine directions of further intervention
implementation.
Reporting on status and results of intervention implementation is provided through the documents Impact
assessment (see section on documents) and Semiannual report (see Section 5 on reporting).


Section Three: Monitoring Documents
In this section the main documents used in PSDs monitoring and results measurement are listed and
described
4
. The main documents are:
Sector Strategy: When deciding how and where to intervene, PSD follows an M4P diagnostic process.
The Sector Strategy is where PSD writes up its findings from the diagnostic process. The document
includes analysis of the sector, taking this forward into outlining which key constraints the programme
aims to address, with a clear rationale as to why they were chosen over others (i.e. feasibility &/or risk).
Intervention Guides: this document is the main document for monitoring and results measurement. It
includes a narrative description of the background to each intervention, the intervention strategies and
results chains with separate lines for direct impact (supported by the project) and indirect impact (the
result of crowding in or scaling up in another way, with or without project facilitation), monitoring plan and
documented projections on impact
Impact Assessment Report: this document outlines what the situation was before intervention, the results
achieved (qualitative and quantitative) by PSDs intervention, and how these results were measured. It is
drafted when PSD first measures impact and updated at the end of the monitoring period.

3.1. Sector Strategy
5

The Sector Strategy should answer five overarching questions:
1. What is the potential for work in this sector (feasibility and size of potential impact)?
2. What does the sector look like? (how does it work, who are the main players)
3. Whats wrong with it? Why doesnt it work effectively? What needs to change for things to improve?
4. How do we envisage the sector will function differently/better in the future?
5. What are the main things that PSD is going to do?
Sector strategy has two main purposes. First, strategy should give the big picture of the sector - its size,
demand and supply, players involved, their roles, the trends etc. It is mapping of the specific market
system, its dynamic and the position of target group. It should explain where problems lie and what is
preventing the solution. It is narrowing down to the specific market constrains.

4
The formats of the documents are available with Results Measurement specialist
5
Details on overall structure and specific content can be found in the Sector Strategy template available
with Intervention Managers and Results Measurement specialist

12 | P a g e

Secondly, drafting a Sector Strategy helps PSD to articulate its vision of how PSD can make the market
work better for the target group. The Sector Strategy thus forms the basis for intervention design.

3.2. Intervention Guide
The key document in the MRM system is the Intervention Guide (IG). Each intervention needs to have its
own separate IG. An IG is created when PSD starts to design intervention. Even if an intervention has
been put on hold, it should still have an IG. If the intervention is in the early stage of development, there
should be a results chain that should be tested. For best transparency and utility all IGs needs to be up
to date at all times and without any inconsistencies. The IG consists of following pages:
Cover page: lists the main information about the intervention: market in question, intervention area
(intervention title), general information (such as starting and closing dates of intervention, monitoring
dates, etc.), key indicators and additional information on scale, outreach, etc.
Strategy (sometimes called explicit strategy): summarises why the intervention has been chosen, and
how it is expected to achieve results. It includes relevant background information on the market
(answering the question why this market?). Under the Background/rationale section PSD outline the
market system constraint this intervention addresses (key indicators and information), and to what extent
this constraint prevents the market to perform better. Here PSD identifies the root causes for the market
failure that lead PSD to intervene. It should start by explain symptoms and how this symptoms are linked
to the root causes of underperformance. This is followed with the explanation of how PSD will tackle root
causes how target group will benefit as a result.
The section Activities under intervention gives the short description of main activity areas under this
intervention and the type and amount of support PSD expects to give.
Intervention strategy section is a written explanation of why and how one level of the results chain is
likely to lead to the next. This section allows PSD to tell the story of how each activity leads, via boxes in
the results chain, to impact. Here, the major assumptions that are being made should be outlined, as this
is not always clear from the results chain. If boxes in the results chain are gender sensitized, it should be
written down how the women will benefit from the changes. Where possible, estimation of how many
women will benefit, and by how much, should be noted
6
.
If another programme is working in the same geographical area with similar aims to PSD, this should also
be mentioned. It should be noted if this programme is likely to contribute towards the changes measured
by PSD. If PSD believes that other programmes working in the same sector and area do not contribute to
the changes which PSD is measuring in relation to the intervention, this should be explained.
Within the strategy section, in the Sustainability strategy box, it should be explained how PSD will
achieve sustainability, i.e. how the changes in the results chains will lead to lasting impact. For guidance
on what to write here, see Annex 2.
The Notes and updates section captures the latest changes/events in relation to the intervention. This is
updated after every strategy review.


6
More on how PSD will monitor and measure WEE in Annex 1

13 | P a g e

Results chain:
For each intervention a results chain is developed by an Intervention Manager. Results chain represent a
clarification of what programme is doing and what outcomes are expected.
Each results chain has these levels:
o Interventions (where PSD does an activity). This level includes activities that PSD is directly
involved in or has control over.
o System level: where PSDs partners, or others who follow them, change something, due to PSD
activities.
o Core market change (where PSDs target group change something due to PSD partners, or
others who follow PSDs partners). This level reflects how beneficiaries use new or improved
products or services, and benefits from these in terms of greater sales, yields, etc.
o Income and employment (where PSDs target group earn additional income or access newly-
created jobs, due to the changes in the core market described). The Income and employment
level, include boxes which state how much additional income will be created and how many
additional jobs will be created and for how many people. (Eg 1,000 blueberry farmers earn
CHF 1 million in additional income).

The diagram below shows the levels of a results chain define what results we expect to see at the levels
they are related to:


















Interventions (activities)
Market system level
Core market
Income and employment
What PSD does or support partner to do
Direct result from PSD activities
Transactions of
service/products to
beneficiaries



Crowding in
Other beneficiaries
influenced to seek
service/product
Benefits of using the service

Greater production/profit by beneficiaries
More income, employment generated

14 | P a g e


Results chains should clearly show why one level will lead to the next and credibly attribute changes at
the higher level to changes at the lower level. Attribution is built into the results chain so that it is clear
that the changes mentioned have a link between them
7
.
It should be clear which actor(s) are involved in each activity or change. For example, write Farmers earn
additional income and not just additional income.
There should be enough boxes to cover all changes that would happen between programme activities
and reaching programme goals. On the other hand, having too many boxes can make result chains
difficult to understand , so it make sense to merge or even to left out boxes which add little to PSD
attribution. Aim for no more than twenty boxes per results chain.
Results chain should have boxes and arrows which show how crowding in (system level) and copying
(core market) are expected to occur. Where pilot interventions will not cause wider market change without
supporting activities, another results chain should be drawn, to show how additional activities will trigger
wider market change.
Projections should be written in each box in italics. They should include the date by when the results are
expected to be achieved. Once results have been measured, the results chain should be updated with
actual results added to the relevant box, in bold, along with the date when the results were measured.
This will help Intervention managers to assess the difference between projected and achieved results and
give an early warning if the expected results are not occurring.
The partners business model (or service improvements, in the case of public sector partner) should be
clear from the results chain. If thats not possible, a diagram showing the business model should be
included in the strategy document
8
.
Measurement plan
Staff uses the Measurement Plan to direct data collection the process of gathering information on the
activities, their immediate results and changes in the higher levels that are effect of the system level
results. Each Measurement Plan is based on a results chain. It contains indicators for each box of the
results chain. For each indicator, it explains what type of information is needed to assess changes in the
indicator, and the source of this information (eg partners records, or a survey of farmers by PSD). When
completing the Measurement Plan, the Intervention Manager and MRM Officer also agree who will be
responsible for gathering information, for each study, and when the studies are needed. This information
is included in the plan.
Measurement Plans also prompt PSD to explore attribution. Measurement Plan has following columns:
Results Chain box list each box in the results chain, one below another. Use the whole text used in
the relevant box of the result chain. Like Results Chains, measurement plans list boxes at the Impact
level at the top of the Measurement Plan, and boxes at the Interventions level at the bottom.
Questions help staff to understand if the change is taking place as expected. When deciding what
to write in this column, it may be useful to ask yourself At this stage of the intervention, what do we
really need to know? Remember to include some questions about sustainability and womens
economic empowerment, where appropriate.

7
More on attribution in section 4.4.
8
See Annex 4

15 | P a g e

Indicators One or more indicators are set for each box in the results chain. Indicators should be
specific, feasibly measurable and capable of answering the questions described in the previous
column. Indicators should measure changes in ways that are attributable to PSDs intervention.
Each measurement plan should include indicators which measure sustainability. These include
qualitative indicators, for example measuring partners:
o Satisfaction with changes;
o Future plans (eg to scale up changes)
o Willingness and ability to continue changes without PSD support (eg signs of partners taking
responsibility for things which were at first funded or done by VEEDA).
9

These indicators are usually linked to the boxes at the market system level; usually the information
needed to assess them comes from partners. Intervention Managers should monitor these
sustainability indicators frequently (eg every six months), to detect and remedy any threats to
sustainability before its too late.
Projects often focus on impact indicators, such as income or employment. However, intermediate
indicators, such as behavior changes of service providers and/or beneficiaries use of new services,
are also critical to effective MRM. Monitoring changes in intermediate indicators enables PSD to
know if interventions are on the right track, i.e. if the designed and implemented intervention is likely
to result in expected impacts.
In some cases PSD may use proxy indicators
10
where this is more practical than measuring the
actual change itself. In this case, PSD should add a note under How explaining why a proxy is being
used and how it is a suitable proxy (eg why a certain indicator would be hard to measure).
How describes how data on indicators will be collected. The method of data collection depends on
the type of information that is required. As Section 5.4 explains, it can vary from observation to
interviews and surveys. Often, it is sensible to use more than one method to measure key indicators,
particularly at the Income and employment level. This improves the credibility of PSDs
measurement; different methods can compensate for each others weaknesses. For example,
research which explains why something changes can improve attribution, instead of PSD just
measuring how much something changes after PSD intervenes. Usually this means asking partners
why did you decide to change x?
Time shows when data will be collected. The expected dates of data collection will correspond to
the time when the desired change is expected to occur. For example, the satisfaction trainees a
partners training course can be measured after the training is complete. This box can have multiple
dates; often if it is useful to |monitor changes in an indicator at several stages during the intervention.
When an indicator is measured several times, the first entry in the When column shouldnt be
deleted, but new entries should be added (eg July 2012; July 2013; July 2014).
Who this can be MRM Officer or Intervention Manager. Sometimes, data collection will be
outsourced and this should be stated in the column.
What we have baseline PSD should decide which indicators needs to measure a baseline for and
to make a plan to measure the baselines before PSDs intervention would change the status of the
indicator. PSD should ensure that the When column reflects these plans.

9
For more examples of sustainability indicators see Annex 2
10
Proxy indicator: An indicator for which measurable change is clearly and reliably correlated with an
indicator of a change that the programme aims to achieve (but is more practical to measure). Source:
DCED (2013) DCED Standard Version VI, p. 17.

16 | P a g e

Measurement plan needs to include a section which summarises how attribution will be handled
11
by the
programme. It should contain explanation of which methods will be used for estimating attribution and
why that method has been chosen.
Risk analysis
Risk analysis is created when the intervention is being considered /designed. It helps Intervention
Managers to assess how risks can affect intervention and the impact of intervention. The chances for the
risk occurring needs to be estimated and there must be a plan to reduce negative effects of these risks.
Supporting calculations
As described in section 2.2, changes in key quantitative indicators should be projected, so that PSD can
estimate future programme impact. For example, blueberry growers earn CHF 1 million additional income
by August 2015. Projections for income, employment and outreach are essential. Each intervention plan
needs to include detailed calculations for the projections made in results chain. Calculations should be
based on existing information (for example, from the field or in consultation with the relevant experts). The
sources of information used in projections should be clearly listed. Sometimes assumptions must be
made that are based on expected project achievements, findings from the context and other performed
analytical documents. This should be also recorded in the calculation sheet. Results should be always
discounted by opportunity costs while taking into account local conditions. Changes in qualitative
indicators do not have to be predicted, but Intervention Managers should know when these indicators can
be expected to change.
Internal strategy reviews
Internal strategy reviews usually take place during backstopping missions. If this is not the case, PSD will
review its strategy at least once per year, scheduling an Annual Strategic Review meeting.
The findings of a portfolio review during a backstopping mission are reflected in the backstoppers aide
memoire, which contains observations, recommendations and advises on strategy and tactics. If needed,
results chains and measurement plans should be updated after backstopping missions.
Before a portfolio review or Annual Strategic review meeting, all Intervention Guides should be updated
by Intervention Managers. Results chains should reflect current intervention strategy. Projections and risk
analysis should be reviewed too. All IGs should be then sent to Results Measurement specialist, who will
send them to the back-stoppers. Before an annual review, the RM Specialist will also aggregate the
results so that the logframe, with the latest results, is ready in time for the annual review. RM specialist
should compare latest results with initial projections. If there are differences between projections and
latest results, RM specialist should put it in writing what was learned from monitoring. Intervention
Manager and Team Leader with the support from Results Measurement Specialist should then decide
what will be done and what changes are required.
The whole team should be involved in the Annual Strategy Review meeting, while Intervention Managers
are responsible to update results chains to reflect change in intervention strategy and as well as
projections that correlate to the results chain.
The Annual Strategic Review meeting report of what was discussed and agreed is usually written by the
programmes Administrator, and reviewed by the Team Leader.

11
More on attribution methods and means of verification in section 5.4.

17 | P a g e

All changes should be listed in the update sections of the Intervention Strategy. The old versions of result
chains should be kept
12
.
The only exception is when PSD is updating the results chain of an intervention which has finished. If
measurement shows that there was no crowding in or coping, why this did not happen should be
explained in Impact Assessment Report.
If an IG does not require updates after a review, this should be also noted in the Intervention Strategy
section, saying when the IG was reviewed and that no changes were considered necessary.
As well as during backstopping missions and annual reviews, PSD staff can also review strategy at other
times, if circumstances change or there are new findings on the market which make a new strategy worth
considering. PSDs interventions should be opportunistic and flexible in response to changes, in order to
maximize benefits for the target group. In this case, a one-to-one meeting should be held between
Intervention Manager and Team Leader, to decide on the necessary changes. All other steps (updating
the results chains and projections and recording these changes) are done as described above.

3.3. Impact assessment report
The Impact Assessment report is a document that describes changes facilitated by an intervention and
how these changes were measured. The Impact Assessment Report has two purposes: to store impact
information, and to describe the reliability of this information. Each intervention should have its own
Impact Assessment Report. It is an internal document, which can later be used to develop
communications material (reports, case studies, etc.)
An Impact Assessment Report is first written when PSD first measures impact (change in income and
employment). It should be updated when PSD completes the final impact assessment.
It is not necessary to describe why the intervention was chosen, what the market looks like, or how PSDs
activities were completed as this is documented in Strategy/explicit strategy document of the IG.
The Impact Assessment Report format contains the following sections:
Intervention Story summarizes textually what the programme did. Staff can copy the explanation given
in the IG/ Strategy paper.
Data Gathering Methodology describes the methodology used in the impact assessment. If the
methodology conforms with the good research practices described in section 5.3 this should be
highlighted. If PSD faces challenges which forced PSD to deviate from good practices, this should be also
described, and any effects that deviations might have on the accuracy of findings should be noted. If the
research methodology is different to what PSD originally planned, even if it still complies with good
practice, explanation of how and why changes were made is needed.
In the next section, Findings, the changes in indicators which measurement recorded are explained.
Every box mentioned at system level and at the core market level must be mentioned, explaining results
at each level. Where relevant, for indicators that feature in PMs logframe, three types of result should be
provided: direct results
13
, indirect results
14
and the total (combined direct and indirect results). Where

12
VEEDA keeps old versions of results chains in Public folder on server
13
Change by, or because of, market players assisted by PM.

18 | P a g e

relevant, results for male and female beneficiaries should be presented separately. Any evidence of
sustainability should be noted
15
.
Each Impact Assessment report ends with a Summary and Conclusions section which summarizes the
impact
16
achieved by the intervention (extrapolated from PSD research). Any learning that can contribute
to further work in this market or others should also be included under this section.
If the intervention was unable to achieve any impact, this section should explain why.
The report is considered as final after the final impact assessment when the report is updated with
evidence of sustainability and systemic change observed.


Section Four: Methods, research practices and attribution

4.1. Definition of the PSD programmes target group
In order to be in compliance with the Swiss strategy and at the same time with the local regulations, PSD
programmes accepted Serbian standards and definitions in terms of income and employment.
PSD will stimulate sustainable additional employment and income for workers (employees in the formal
economy) in target municipalities. PSD will give particular attention to stimulating jobs and income for
women and youth. PSD defines youth as persons between 15 and 30 years old. Among workers, PSD
will focus on those earning less than RSD 34,000 (about CHF 370)
17
.
PSD will also stimulate additional income for enterprises in target municipalities. Among those
enterprises, PSD will report as impact additional income generated only by micro, small and medium-
sized enterprises (MSMEs). PSD will not partner with enterprises to raise their profits in ways which have
little or no benefit for workers (eg replacing employees with machinery).

4.2. Impact indicators definition
In order to be compliant with Serbian legislation and at the same time with SDC priorities reflected in
Swiss Cooperation Strategy 2014-2017, PSD defined income and employment impact indicators as
follows below:


14
Indirect results are the results of crowding in. In other words, change by, or because of, market
players imitating innovations that PM helped to create, without PMs assistance.
15
More information on sustainability see in Annex 2
16
By impact we mean the changes in Impact level logframe indicators caused by the intervention. In
other words, the additional income, outreach and employment caused by the intervention for the whole
target group, specifically for women, and specifically for youth.
17
According to the Household Budget Survey done by Serbian government in 2010, households whose
consumption was below 8,544 dinars per consumer unit, were poor. The average family counts members,
which gives 8.544*4=34,175 RSD.

19 | P a g e

Jobs
PSD is aware that many donor programmes report jobs in the informal economy, both in their baselines
and as results. Serbian legislation views jobs in the informal economy as illegal, however. As PSD is
reporting in accordance with Serbia legislation, PSD will not count informal jobs in its baselines or its
impact assessment. PSD will only count only formally employed women and men (with labour /
employment contract).
In agricultural sectors the programme counts only those women and men who have registered their
household (farm) and do business through a bank account (ie legal channels).
In this context, workers who were previously in the informal economy and because of PSD interventions
gain formal employment are counted as additional jobs. The same applies for rural households. If a
previously unregistered household that was performing any type of agricultural activities becomes
registered, as a result of PSD activities, PSD counts this as a new job.

When reporting additional jobs to SDC, PSD will report on how many full-time equivalent jobs the
programme has created. PSD count as a full-time job in two ways, depending of the sector:

Industry/tourism full-time job definition:
According to Article 2 of the Rulebook
18
an employed person is a person who is employed in accordance
with the labour legislation and as such is contributing to the mandatory social insurance. Thus, in
determining the number of employees, the persons taken into account are:

- Employed for an indefinite period = 1 full-time job
- Employed for a definite time = % of full-time job
- Part-time employed = % of full-time job

Here, a full-time job is defined as 240 working days per year. Persons who were employed as apprentices
are also considered as employed persons.

Agriculture full-time job definition:
Agricultural production in Serbia is a seasonal activity. Farmers work long hours during peak season, and
have quiet periods in winter. Depending on the type of production and cycle duration, farmers in Serbia
are usually active for six months or more, but in all cases less than one year. Regardless of the
production type, all registration of the household lasts one year and is recognised as a whole year
activity. Also, social and pension contributions householders pay for the whole year. For these reasons, in
the case of agriculture, PSD counts income equivalent jobs instead of using 240 days per year as the
measure, with some additional conditions that should be fulfilled.

By income equivalent, we mean that a householder earns equal or more than the annual gross minimum
wage from new or more productive activities caused by PSDs intervention. Here, this will be counted as
one job. If a householder earns less than gross minimum wage, this will be calculated as a percentage of
a job. In other words:

1 income equivalent job: If total additional income for the production season / 12 months 300 CHF
19


18
"Official Gazette of RS", No. 33/10
19
As of January 2014, the gross minimum wage in Serbia was 28,582 RSD 300 CHF per month. If the
minimum gross wage changes, this formula will be adapted accordingly.

20 | P a g e


% of income equivalent job: If total additional income for the production season / 12 months < 300 CHF

Example:
Additional income for household from blueberry growing is 5,000 CHF.
5,000 CHF / 12 months = 416.66 CHF > 300 CHF
If the household is registered, blueberry growing in this case created 1 (one) additional job.

Additional income:
To reflect PSDs particular emphasis on creating jobs and income for women and youth, PSD measures
additional income as follows:
- Total net additional income (CHF) for workers and enterprises.
- Within total net additional income, amount (CHF) generated by women.
- Within total net additional income, amount (CHF) generated by youth.
- Within total net additional income, amount (CHF) generated by workers and self-employed; amount
generated by SMEs.
- Women and youth who generate additional income as workers would be included. So would women and
youth who generate additional income as business owners.
- The number of jobs stimulated by the programme would also be disaggregated by gender, by youth, and
by employees/ self-employed.

When PSD calculates formalized jobs as impact, the complete income earned from the new formal job will
be calculated as additional income. PSD will measure income from informal jobs to see if farmers are
earning more from PSD interventions, which will be one of the sustainability indicators.
When farmers replace old crops with more profitable ones, due to PSD interventions, PSD will first
consider if demand for the less productive crop is likely to be replaced by supply from outside the target
area. If so, PSD will estimate how much income farmers would have earned from the old crop, then
discount it from reported additional income. This is because the overall value of food production within
the target area will only have increased by the extra value of the new, more profitable crop.

4.3. Research practices
PSD collects data at different times during the monitoring period of the interventions. At a minimum,
system-level indicators are measured annually, while core market and impact indicators are measured at
the baseline, mid- and end-point of the programme. PSD will also conduct additional research two years
after the completion of the intervention. The timing of data collection also depends on the interventions
progress. During the pilot, smaller, more frequent monitoring is done, because during pilots the
uncertainty over whether something is working is highest. Regular monitoring will inform managers about
whether they need to change intervention strategy or tactics. Larger, less frequent, more precise studies

21 | P a g e

of impact are done during scale up. These studies focus on how much and how many people have
benefited overall, and specifically how many women and youth have benefited.
Initial frequent contact with beneficiaries, to determine if they benefit, is later during the uptake replaced
with an assessment focusing more on measuring the outreach of the intervention by working with the
service providers and thus estimating the outreach. This measuring scale and indirect outreach at the
uptake phase requires staff to shift from more traditional results assessment tools to a more creative and
investigative reporting mode. Hence there is a need for innovative ways to track the diffusion of new
behavior and business models through the value chain. Still, while assessing the benefits to target
beneficiaries can be less intensive in the scale up phase, it is still important to check through small
measurements that target enterprises continue to benefit to the expected extent. If they do not, this is a
signal to conduct more information gathering to determine why not. If they do, the project can continue to
focus on encouraging adoption of new behavior i.e. models.

PSD uses different methods to measure different indicators. Methods can range from observation, in-
depth interviews with key informants, comparison between a target and control group, to surveys and
analysis that require expertise from outside the PSD team.
The MRM officer should advise on the choice of method, taking into account importance of the study,
likely size of impact, cost of intervention, cost of measuring, desired confidence interval and level and
triangulation options.
20


4.4. Measurement and attribution methods means of verification
Attribution is defined by the Glossary of Key Terms developed by the DAC Network on Development
Evaluation as the ascription of a causal link between observed (or expected to be observed) changes and
a specific intervention
21
. In simple terms this means that attribution refers to extent of change that can be
claimed by a project/intervention out of total change that takes place.
For all reported impact, PSD should demonstrate attribution well enough to convince a reasonable but
sceptical observer.
22


The methods which PSD uses to attribute changes to its interventions include:
o Creating results chains, to logically articulate how activities will lead to impact.
o Reviewing these results chains regularly, to check that the results reported at each level can
logically explain changes at the level above. If they cannot, PSD will commission extra research
to clarify attribution.
o Where applicable, measuring baselines.
o Where feasible, comparing beneficiaries with a control group of similar non-beneficiaries. When
doing this, PSD will take care to isolate the impact of differences between beneficiaries and non-
beneficiaries in the control group. This means looking out for differences (eg in attitudes between
people in treatment and control group) whilst testing surveys.

20
More on how to conduct a good research in Annex 3
21
http://www.oecd.org/dac/2754804.pdf
22
Definition adapted from the glossary of the DCED Standard. The DCED Standard for Measuring
Achievements In Private Sector Development. Version V, 13 January 2010.

22 | P a g e

o Interviewing partners and beneficiaries to
understand if the changes that they have made,
which appear to have resulted from PSDs
facilitation, would have occurred anyway. Asking
why people change, in in-depth interviews, can
provide information on sustainability too.
o For example, if PSD is promoting access to
blueberry seedlings, PSD would ask new blueberry
farmers where they got their seedlings from. If there
are other factors that contribute to the change, PSD
will conduct research which attempts to isolate
changes created by PSDs intervention. E.g.
partners will be asked how much they would have
changed without PSD support. Where there is lots
of uncertainty about what would have happened
without PSDs intervention, PSD will instead
estimate how much its contribution influenced the
change. Here, PSD will only report its estimated
contribution to changes as impact.
o Triangulate, eg complementing closed-question
surveys with a few in-depth interviews. This helps to
avoid PSD hearing what we or they think we want
to hear and not noticing hidden influences, costs or
benefits.

Estimating attribution will differ in a pilot phase and during scale up. During the pilot, PSD will validate the
different levels of changes in a results chain to ensure that change has occurred triggered by programme
activities. I.e. PSD will measure to what extend the change occurred due to the programme activities, and
keep asking why change happened at each step.

During the scale up, when there are other factors that are likely to be significantly affecting the change,
then the programme needs to layer another method on top of the above to assess attribution.

The table below summarises some of the options that programme staff may use at each step in the
results chain; this table is not intended as a hierarchy as different circumstances will determine which
options are more appropriate. The options are not mutually exclusive and a mix or combination is often
the best strategy; of the MRM Officer and Theme Coordinator will discuss which means of verification will
be used for each step in the results chain.
23

Means of verification table:
Method Application Advantages Disadvantages
Opinions of key informants
and expert interviews
May be important when
the key change is driven
by one person (e.g.
Low cost May be influenced by
interviewer; likely to be
somewhat subjective.

23
Guidelines to the DCED Standard for Results Measurement: Estimating Attributable Changes,
Nabanita Sen, March 2013
The Rural Development
Programme for England ran
from 2007 to 2013, funded by
the European Union and the UK
Government. The programme
aimed to improve the
competitiveness of Englands
farming and forestry sectors,
the rural environment, quality of
life and to diversify the rural
economy. The programmes
results were measured with
means of verification which
PSD also uses: data on
programme activities, case
studies and surveys of
programme beneficiaries and
non-beneficiaries.
Measurement identified a
number of opportunities to
improve the programmes
delivery.


23 | P a g e

politician changing a
policy)
Comparison of treatment
and control group
(randomised samples)
When samples are large
enough - in measuring
changes attributable to
one step in the results
chain (probably not
feasible for the whole
model in one trial)
Held by statisticians to be
the most reliable way to
measure results (albeit
based mainly on
experiences with simple /
single treatments)
Difficult to design and
administer if the treatment
group is self-selecting (e.g.
buying a service). In that
case, a randomised sample
would need to be refused a
service they tried to
purchase
Quasi-experimental
design (difference of
difference - comparing
before and after for
treatment and control
groups)
Often appropriate for pilot
efforts and/or measuring
attributable changes for
one step in the results
chain
More approximate, in
acknowledging that the
control group is not an
exact control
Cheaper than randomised
controlled trials, but still
expensive. Careful design
and measurement needed
to ensure accuracy. Not
valid when the target group
is unique, as is often the
case with large urban
clusters, or when
interventions can influence
the control group as well as
the treatment group.
Participatory approaches
(focus groups etc.)
Where the change in
behaviour might have
been caused by different
factors
May be the only way to
show attribution in some
cases
May be subjective, open to
bias (e.g. high subsidies
may attract positive ratings,
even though not
sustainable)
Observation Where attribution is fairly
clear (e.g. resulting from
new technology)
Low cost May not be perceived as
convincing especially
where attribution is not
obvious
Regression Analysis Where a wide range of
data can be accurately
gathered

Can be reasonably
accurate if well
designed and executed

High level of skill needed;
Accuracy relies on
identifying and gathering
data on other significant
factors contributing to the
change
Extrapolation of attribution
proven in pilot or case
study
Where funds are not
available for large-scale
measurement
Low cost, relatively
convincing
Needs periodic verification
by other means (e.g.
through surveys or
additional case studies)
Trend analysis Where other, larger
trends are very significant
and trends can be
reasonably tracked and
estimated
Takes into account larger
economic and market
trends; relatively low cost
Risks assuming that the
identified and measured
trends are the only (or main)
ones applicable; best used,
therefore, in combination
with other methods
Case studies analyzing
behaviour and
performance changes at
each step of the results
chain
Where qualitative
understanding is needed,
in order to interpret
quantitative data
Low cost; can be a good
indication of attribution if
well designed and
executed
Many not represent the
universe of beneficiaries;
can be time consuming;
may be influenced by
interviewers

24 | P a g e

4.5. Measuring indirect impact
PSD considers as direct impact the benefits to the target group created by PSDs support to partners (eg
cost sharing, training, mentoring etc). I.e. if PSD supports a processor and as a result 50 householders
start to cultivate herbs/fruit, these householders additional employment and income will be considered as
direct impact.
Indirect impact is defined as benefits to the target group and/or companies as the result of copying and
crowding in that takes place without project support.
Crowding in is the process through which interventions catalyse or bring in other players and functions in
the market system so that it works better for the poor
24
.

Example of crowding in: PSD supports targeted enterprises to start providing a new service, by supplying
training. Other enterprises see that this training can be profitable, and start supplying it as well.

New target enterprises or workers who benefit from this crowding in are referred to as copying; their
additional employment and income are considered indirect impact.
Unassisted crowding in and copying are frequently slow; they require market players to understand very
clearly how a new innovation can benefit them, how they can adopt it, and be willing to risk investing in
the innovation. When designing an intervention, Intervention Managers should consider when unassisted
crowding in and copying can realistically be expected to occur. Then, set indicators at the appropriate
levels to monitor progress. Examples of such indicators include:
Number of players who adopt new business model, without assistance from the project.
Number of farmers who adopt new cultivation practices, due to assistance from a processor who
has not received direct project support.
While measuring direct impact is relatively straightforward, measuring indirect impact may be extremely
challenging.
PSD will look for the changes in the market system and try to find evidence of copying and crowding. This
involves visiting market players, both those directly supported by the programme, and those who are not.
When interviewing partners, PSD will ask if partners are aware of other organisations which have
changed their practices due to PSD-supported changes. If so, PSD will follow up with these
organisations. PSD will informally ask them to verify where they got the idea to change from. Where
possible, PSD will try to measure what impact the crowding-in enterprises have on PSDs target group,
disaggregating as always by gender and youth.
PSD also measures indirect impact, when rural households spread an innovation between themselves.
For example, when PSD supports the cultivation of a new fruit variety, and beneficiaries give seedlings to
other farmers. Where this is likely, PSD includes a question in surveys and interviews with known
beneficiaries, asking if they have shared knowledge/ inputs with other enterprises/ rural households. If
they have, and if PSD was unaware of this, the PSD interviewer will ask for contact details of the copying
enterprises/ rural households. PSD will then interview these copiers to find out how much they have
benefited from the information.

24
The Operational Guide for the M4P Approach, Springfield Centre

25 | P a g e

Documenting assumptions and calculations
Systemic change is normally expected to impact large number of people or businesses. Consequently, it
will generally be impossible to survey or measure all of them. Instead, assumptions are used to calculate
the impact, based on a fairly representative sample of data. All assumptions and calculations should be
clearly documented and explained. The RM Specialist will decide the sample size for each survey. The
sample size will primarily depend on the means of verification, how many beneficiaries the intervention is
thought to have reached, how much impact is expected to vary between beneficiaries and the overall,
expected size of impact. High-impact interventions will be measured more rigorously.

Section Five: Reporting
As required by SDC, PSD reports on semiannual basis with two types of reports: Operational report and
financial report. Both reports are presented at the same time so that SDC can compare results achieved
with costs, for each market. By results, we mean changes in each Impact-level logframe indicators, as per
the latest measurement. By costs, we mean how much PSD had spent in a market at the time when
results of that markets interventions were last measured.
The format of the reports is set by SDC
25
.
PSDs semiannual reports to SDC should note which other development programmes or government
initiatives are supporting PSDs target group, in the same target sectors as PSD, and outline how. The
description of other programmes or government initiatives should be presented in the Annex of the
Operational report. If other government or donor-funded initiatives have contributed to the results, PSD
writes clearly and honestly in this report how they have done so.
PSD will produce case studies for SDC and wider development community when interventions show
enough evidence of sustainability and systemic change.
On a more general note, the efforts on the MRM system are necessarily aimed to enable accountability
towards the donor through regular external reporting. The reports need to be comprehensive yet concise,
they need to reflect on the expected information, and most importantly the reports need to be accurate
and up to date. Presenting the correct monitoring data and current challenges is as important as
underlining the progress and successes.

Section Six: Roles and Responsibilities
Intervention managers are not accountable for how many interventions they complete, but for maximising
long-term results. Intervention managers therefore need to understand first-hand if the programme is
having the desired results. This will only happen if Intervention Managers are given enough time to be
involved in measurement. All MRM related responsibilities need careful planning, and oversight from the
Team Leader. Hence, within PSD, monitoring and results measurement is part of all everyones job. All

25
Format of reports are available from the RM specialist and Project Assistant.

26 | P a g e

technical staff job descriptions and performance reviews include monitoring, results measurement and
analysis of results.
The table below summarises roles and responsibilities of different PSD team members.

Task Who is responsible
Research to inform the selection and/or design of a new
intervention.
Intervention Managers with the support from MRM officer
Drafting the Intervention Guide when a new intervention
is being considered/ designed
Intervention Managers. Quality control and finalizing is
done by MRM officer
Gender inclusion in the intervention Intervention Managers with the support by Gender focal
point
Deciding on study methodology MRM officer with help from the Intervention Manager
Deciding when data needs to be gathered MRM Officer with the help from Intervention Managers
Deciding who will gather data MRM Officer
Drafting surveys/ questionnaires Intervention Managers with help from MRM officer
Checking surveys/ questionnaires MRM officer
Training enumerators MRM officer
Supervising enumerators during data collection MRM officer
Cleaning and analysing data MRM officer
Calling back respondents, where data is missing or
unclear
Intervention Managers if they originally conducted the
interview
Enumerators if they originally conducted the interview,
after receiving instructions from MRM officer
Using results in decision-making Intervention Managers with the help from MRM officer
Updating IG (including results chains, projections and
references to researches that underpin assumptions)
Intervention Managers, checked and finalized by MRM
officer
Training new staff on measurement MRM officer with the support from Intervention Managers
Aggregating results MRM officer
Reporting results to SDC MRM officer drafts the Operational report. Project
Assistant drafts financial report. Theme Leader gives
final approval of both reports
Writing case studies MRM officer with the support from Intervention
Managers. Final approval given by Team leader
Making any necessary changes to PSDs approach to
measurement
MRM Officer with the approval from Team Leader




27 | P a g e

Annex 1 Gender and youth
Promoting gender equality and empowering women is a priority for all agencies involved in development.
The importance of Womens Economic Empowerment (WEE): the economic advancement of women,
their access to opportunities and life chances, assets, services and support and the development of
decision making authority in public and private spheres, is widely recognized as being essential to
economic growth and poverty reduction worldwide.
As a result of the economic crisis, youth unemployment in Europe has grown, even in the most developed
countries. Youth unemployment is an important indicator of the future development of the economy and
social development.
The overall economic situation in Serbia hampers the creation of new job opportunities, especially for
young job-seekers and women. This is why PSD is putting particular focus on youth
26
and women when
stimulating systemic market development in selected sectors and value chains.
The PSD programmes want to know if youth and women in targeted regions are benefiting from the
changes in market systems stimulated by PSD. At minimum, monitoring of each intervention should
measure the following:
How many young people below 30 and women are benefiting as the result of intervention.
How many additional jobs are created as the result of intervention for youth and women
How much additional income youth and women generate as the result of intervention.
How many youth and women have access to innovations supported by the intervention.
For this to happen, PSD disaggregate all impact indicators by gender and youth and non-youth. PSD
disaggregate those indicators at lower levels of the results chain which track changes with the potential to
exclude women and youth. For example, if PSD facilitates changes which lead to the creation of a new
training course, PSD would measure womens participation in this training course.
As well as tracking and reporting on sex-disaggregated indicators, PSD also measures womens
economic empowerment in other ways. Where a PSD intervention can empower women, for example by
giving women greater access to technical knowledge from which they have historically been excluded,
PSD first gender-sensitizes the results chain. PSD adds boxes describing how its activities will stimulate
changes which the market works better for women. Next, PSD adds indicators, to assess how much
women are economically empowered by the intervention. Examples of the WEE indicators include:
Womens access to public decision making - # of women accessing public decision making fora/
# of decisions made/ WEE initiatives generated by public womens fora/ % increase in annual
budget allocated to Local Gender Committee , by the end of the programme/annually/during the
programme
Womens control over HH income - # of women exercising control over the use of HH income, by
the end of the programme/annually/during the programme
Womens use of knowledge & skills provided through programme interventions - # of women
using knowledge & skills/information provided through programme interventions to access
livelihood options/make choices regarding livelihood options/skills/training, by the end of the
programme/annually/during the programme

26
Persons between 15 and 30 years old are considered as youth in Serbia, National strategy for youth,
Republic of Serbia

28 | P a g e

Annex 2 Sustainability, systemic change and scale
Sustainability is essential to the PSD programme and it is considered from the early stages of the
intervention design.
Sustainability can be defined as the capability of the market system to ensure that relevant, differentiated
products and services continue to be offered beyond the period of intervention
27
.
Sustainability, systemic change and scale are closely related.
For M4P projects indications that systemic change is taking place include crowding in (e.g. service or
input providers starting services similar to those developed with the project support), copying (e.g.
enterprises copying improvements introduced in other companies with project support), sector growth
(businesses grow, new entrants to the market), changes along the value chain resulting from project
support at a certain level, and other indirect impact (e.g. the emergence of a related service in completely
different sectors)
28
.
Systemic change is often important to achieving scale. However, some level of scale can be achieved
without immediate systemic change, by providing direct assistance to replication for instance. In that
case, an M4P project will expect that replication would lead to systemic change as more market players
become aware of the advantages of the innovation others have adopted.
PSD monitors and measures each of the interventions progress towards sustainability and scale. For this
purpose, PSD uses the Adopt, Adapt, Expand and Respond matrix.
Elements of systemic change
29
:

Each intervention is different and requires a different approach to assessing changes, but each of them
needs to have relevant sustainability indicators. PSDs choice of sustainability indicators, linked to the

27
The Springfield Centre
28
Adam Kessler with Nabanita Sen, February 2013, Guidelines to the DCED Standard for Results
Measurement: Capturing Wider Changes in the System or Market
29
Resource: The Springfield Centre

29 | P a g e

results chain and included in the measurement plan, is informed by the concepts of Adopt, Adapt, Expand
and Respond.
In the table below are some examples of the types of sustainability indicators
30
which Intervention
Managers should include in their measurement plans. It is not necessary that each intervention has all
indicators listed in the table, but it is necessary to select those indicators that can best fit to the
intervention and give the answer to the question will the changes last after the intervention stops?
Examples of the sustainability indicators:
Stage Question Indicator
Adopt If programme support
stops, will partners
return to their
previous way of
working?
Benefits for the partner: change in partners costs and revenues due to
innovation; number of new customers/contract partner gain from the pilot;
Satisfaction level: partners satisfaction with pilot results; target group
satisfaction with new product/service
Partners investment/share of investment: partners share in the cost of pilot
(financial and non-financial)
Adapt If programme support
stops, will partners
upgrade the
innovation they have
adopted?
Independent investment/continuation of investment: partners investment in
innovation after the programme support ends; partners independent
investment in improvements; description of the improvements made by
partners
Target group continue to benefit from the innovation: number of people
benefiting from the innovation is increasing
Partners business model changed/adjusted due to the innovations:
description of the changes in partners business model or strategy (eg new
staff recruited and dedicated to work with innovation, job description of
current staff amended, changes in budget)
Expand If programe support
stops, will target
group depend on few
firms?
Original partners carry forward the innovation: eg roll out to new regions -
number of new regions; number of potential target group that can be reached
in new regions
Crowding in: number of market players that copy changes or improve due to
the changes; number and capacities of scale agents involved (players that
can influence others to adopt the innovation)
Competition or collaboration in the market: level of competition between
market players; assessment of the barriers that new adopters would face;
level of collaboration between partners; explanation and description of the
collaboration (eg joint ventures, joint actions toward institutions, advocacy)
Respond If programme support
stops, will partners be
able to respond to
challenges?
Resilience and flexibility: evidence on the system ability to resists to external
factors (shocks)
Responsiveness: evidence that players in core and supporting market
system are flexible to the changes (eg adjustment of the practices, revised
regulations)

30
This list of indicators is not final, indicators can be added with regard to the specific intervention

30 | P a g e

Annex 3 Tips on conducting good research
Research is a part of a broader results measurement system. High quality research will not show impact
by itself. It should be backed up by a clear results chain, relevant indicators, and a solid attribution
strategy. The starting point of the research is to ensure that intervention has a well-developed results
chain. This will show exactly what type of changes are expected, and so help frame the research
question.
This section will outline some of the main tips related to the research practices most commonly used by
PSD programme. Readers can find more information on practical tips in Practical Guidelines for
conducting research
31
.
The most common methods of data collection used by the PSD programme are in-depth interviews and
surveys.

Survey
Survey is the one of the most widely used modes of data collection. A survey collects data from a large
number of people, using a standardized set of questions.

Advantages: give quantified, reliable data on a wide range of issues e.g. opinions and needs of
stakeholders, the socio-economic situations of various groups, changes in income, perception, and more;
useful for answering quantitative questions, such as establishing changes in income, employment, or
growth rates of firms; useful to evaluate impact of the programme
Disadvantages: often resource intensive, less useful for uncovering perceptions of the project, sensitive
issues, or unanticipated benefits, may give a simplistic picture of the situation

Tips for conducting a good survey
32
:
o Keep it simple, clear, easy, and short.
o If there are similar surveys conducted by others, find them and review
o Make sure the questions are well worded
o Avoid double-barrelled or double negative questions.
o Use multiple items to measure abstract constructs.
o Do not use "leading" or "loaded" questions.
o To improve quality of the major surveys, pilot them first
o If the survey is conducted by external enumerators, ensure that they are they are properly
briefed. It is useful to conduct a mock interview session with them
o Look for opportunities to group together indicators and measure them together in one
survey/questionnaire.
Determining sample size
Sample sizes for surveys and interviews are decided by the RM specialist. In theory, the sample size
should be representative of the targeted population. In practice, time and fund limitations often mean that
samples are smaller, and a combination of qualitative and quantitative research methods are used to
partly compensate.

31
http://www.enterprise-development.org/page/download?id=2133
32
Source: DCED

31 | P a g e

The RM specialist will decide on sample size case-by-case.
In some cases PSD may use smaller samples because there is little variation between respondents. For
example, if after interviews with 20 respondents PSD receives the same answers (saturation), the RM
specialist may decide that that it is unnecessary to interview more respondents.
Also, where there will be less variation within control group, the control group can be smaller.
When choosing sample size, control and treatment group will be treated as separate populations.
Where it is necessary to conduct more robust surveys, and when it is feasible, MRM officer will decide to
increase sample size.

Annex 4 Business model example
33



33
Blueberry production