1

“ Rapid Appraisal Methodologies for Assessing Impact”

Notes edited by the consultant and the FAO Evaluation Service

A. General
The discussion focussed on Rapid Appraisal Methodologies (RAM) to be used (1) at
field level and (2) during a project evaluation mission.

The basic feature distinguishing RAMs from more "impressionistic" methodologies is

the use of "triangulation" in order to cross-validate all findings. Triangulation means
gathering data from (at least) 2 different sources - preferably using 2 different
techniques (e.g. key informant interviews, focus groups, observation) – for each
variable or issue being studied.

The following steps of a rapid appraisal were outlined. They are to be used in
approximately this chronological order or, if time and resources allow, concurrently:

1. Literature Review = from technical journals; case studies, demographic data and
general reference documents and maps; and locally available project reports.

2. Quantitative data gathering = from statistical documents (central bank, statistical

office) available locally and at national levels; project reports; analysis of existing
data.

3. Interviews with key informants from both an outsider’s and insider's angle of
vision:
• outsiders may include other donors, competing NGOs, other institutions;
• insiders can be national and local authorities overseeing the project;
project management and front line field and project staff.
During this phase, the objective is to identify underlying issues and develop a first
topic list to be used for field investigations.

4. Interviews with focus groups and key informants at regional, local and village/field
levels. At this point, Ms. Blumberg emphasized the need for:

• small (5 being the ideal number), homogenous focus groups. Homogeneity

within the group being interviewed means avoiding groups composed of
people with conflicting interests and/or resources within the community
(e.g. large landowners and landless families and, in most places, men and
women should be interviewed separately);

• the use of control groups1 selected from non-project’s clients but with
salient features similar to the project's clientele (focus groups);

1
Control groups can help ensure that the results being verified are truly the results of project activities and not
due to ongoing events or other externalities. They can also serve as the basis for comparison to measure effects,
impact, and changes in living/working conditions brought about by the project (see also para. 4 below).

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 2

• the use of outsider key informants such as staff at health centres,

credit/lending institutions and/or schools who are knowledgeable about the
community and possess both qualitative information about the local
situation and, ideally, lists of clients that can be used for eventual random
sampling (see mini-surveys below).

Homogeneity can be achieved by identifying different categories of focus groups

distributed according to: agro-ecological areas; gender; age groups; social,
religious and racial-ethnic variations within the community. Social, religious and
racial-ethnic variations tend to be linked and usually can be correlated with class
differences.

In the absence of marked social divisions involving caste, race-ethnicity and/or

religion, class differences could be initially identified at three levels: the
landless/semi-landless; those that possess a moderate level of resources (land,
livestock, shop, etc); and the elite - rich and/or village authorities (religious,
administrative, spiritual, traditional, etc).

5. Field Observation = used to verify what is actually being done by the participants
themselves (e.g. the level/quality of use of land conservation practices, use being
made of outputs); tangible measures of standard of living (such as type of
housing, household possessions); and, what has been achieved versus what is
being reported.

6. Feedback and verification of mission's findings back and throughout the pyramid
of stakeholders starting at village level. At this first level, findings and issues that
have emerged are shared at a village (large) group meeting to seek confirmation,
assent and ideas for potential use and/or reforms.

7. Feedback, verification and discussion of mission's findings at institutional level.

Discussions here focused on the use of multiple stakeholder brainstorming
sessions and inter-institutional workshops to:
• evaluate the validity, relevance and potential use of results with insiders
and outsiders; and,
• assist in future work-planning and detect administrative, or other, obstacles
to overcome;
• identify additional success criteria; and
• discern measures to ensure the sustainability and/or "institutionalization" of
results.
8. Finally, the possible use of mini-surveys, at the end of the investigative process,
was noted. These would be composed of a few well-defined/focused questions,
and should preferably involve random sampling and control groups. Mini-surveys
would be used as a way to:
• establish parametric data about the people in the project, especially those
with complex social settings where it is difficult to identify the right weight to
give to the various social components and focus group distribution;
• answer questions that remain open;
• quantify findings and/or provide quantitative information in support of these
findings.

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 3

N.B. Mini-surveys should not be used for general information gathering, such as
identifying broad socio-economic features of the project clients. Instead, mini-surveys
should be focused on specific issues and targeted on pertinent groups.

B. Rapid Appraisal Methodologies for Assessing Results

(for M&E at Village level)

1. At this level, results need to measured according to success criteria that come, at
least partially, from within the target groups. Therefore, very early in the project
cycle, village focus groups need to be selected to develop indicators of success
and the consequent M&E system.

These focus groups should help suggest the M&E data to be collected to follow-
up progress; and to help select new data to be included in the system as
activities develop. For this purpose, a system of semi-annual or annual focus
group meetings can be established to collect information/data for indicator
measurement (results and impact) and to detect problems.

N.B. Participants of the focus groups may change at each data gathering exercise, or
be maintained as panel members. The choice is to be made on a case-by-case basis
either to ensure continuity of the information gathered; or to guarantee the validity of
the findings by enlarging the number of people covered by the reporting system.
However, there are generally more drawbacks to the panel method in these rural
settings.

2. Control groups need to be utilized to serve as the basis for comparison from
which to measure changes brought about by the project. Control groups can be
derived from focus groups in a “snowball sample” by asking one or more focus
group participants to suggest persons similar to themselves but who are not
involved in the project.

This helps ensure similarity in wealth, education and other characteristics with the
focus groups. They can also be chosen at random from adjacent villages not
covered by the project if sampling lists are available. In either case, they can
provide comparative information on indicators such as:
• the spread (use being made) of results into non-project areas;
• unanticipated negative effects of the project not found in control
groups;
• the incidence of externalities in project results, e.g. a general
economic downturn or boom, or drought, etc. in the whole area,
affecting both project and non-project households.
3. Impact indicators to be gathered at ultimate target group/village level can estimate
three main impact areas:
• what has happened to the business/individual plot area?
• what has happened to the participant/stakeholder including
increases/decreases in empowerment and wellbeing?

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 4

• what has happened within the households including

increases/decreases in children’s (male and female) nutrition,
education and health; changes in the division of labour, time
constraints, etc.?
N.B. Men tend to spread increased income under their control over a wide range of
activities, whereas women with increased income under their control tend to spend it
in a much more focussed manner on:

a. improved family diet

b. improved education for children
c. improved healthcare.
4. The outsider helping to set up the results-oriented M&E system should be
accompanied to the field by at least two project staff members who will be in
charge of maintaining the system and, if at all possible, their boss, in order to
convert him/her into a stakeholder. During the focus group phase, several
intelligent, dynamic participants should be recruited and trained to help facilitate
subsequent waves of focus groups, as well as to join the staff of M&E people in
orally presenting the results of the latest focus groups to project management.

5. Data collection should be done regularly (e.g. by semester). If no M&E staff

person is present at the focus group, and the village facilitator is not totally
literate, a designated M&E village representative, who can be a child or youth of
the village, a school teacher, or other interested and literate volunteer, can take
notes on the focus group discussion.

N.B. Villagers should progressively become stakeholders in the M&E system.

• Interested and potential candidates for the data collection exercises should
be identified and trained (while on-the-job) at each field visit.

• Village reporters should work with project staff to convert findings and data
gathered into the more formal institutional M&E system.

• Data gathering should encompass some form of remuneration and

recognition, e.g. transport costs; honours; certificates, etc.

C. MULTIPLE STAKEHOLDER ANALYSIS

The steps and criteria for identifying multiple stakeholders are somewhat similar to
the selection of focus groups. However, stakeholders start at the top of the pyramid
at national/donor level and proceed down to the local/village levels as follows:

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 5

TABLE 1:

Category Insider Stakeholders (Control group) Outsiders

1. National/donor level – line ministry/donor – 1. Other donors/NGOs

admin. Authorities

2. Regional/local level 2. Staff from NGOs, private sector, other/similar

projects

3. Project management & front line staff – 3. Local front line staff – extension workers and
extension workers and local health/school local health workers, school teachers
workers

For each agro-ecological zone: From nearby and similar areas:

4. Community leaders & large landowners –

traditional, religious

5. Middle stratum -men - average households 3. Middle/lower stratum men

6. Lower stratum -men - poor/landless families 4. Middle/lower stratum women

7. Middle stratum -women - average households

8. Lower stratum women - poor/landless families

9. Youth (male and female) – from within the

project

10. Other minorities/ethnic classes (covered by 5. Other minorities/ethnic classes (covered by the
the project) – if not represented in the above project) – if not represented in the above
classifications classifications

N.B. This minimum of 15 categories of focus groups containing a wide range of

stakeholders needs to be covered. Additional stakeholders can be identified as the
fieldwork progresses, in consultation with management and project staff and/or the
various "outsider" categories.

It is important that the information given by project-related focus groups be cross-

checked through group meetings with a control group (or groups). Additionally, if
time and resources permit, attention should be paid to on-the-job training of a follow-
up M&E team (at least two staff members, a member of project management, and
selected stakeholders at village level). Finally, early in the project cycle, the mini-
survey questionnaire to be carried out if and when authorities at country level so
warrant it.

D. PROGRAMMING RAM INTO EVALUATION MISSIONS

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 6

Nearly all of the requirements for the use of RAM can be planned and written into
the TORs and mission field visits, in close consultation with project and national
authorities.

1. At national level, a large briefing meeting (to take place, if possible, on day 2) can
bring together all of the authorities and donor representatives engaged in project
activities and reduce the number of purely "formal" visits. However, it remains
necessary to complement this briefing meeting with a few key informant
interviews to further delineate the important issues.

2. From the above, a list of topics for further investigation can be drawn. This list,
together with the main findings and issues emanating from the literature review
(see section A.1) will form the basic list of questions to serve as a guideline for
subsequent key informant interviews and focus group meetings (see below).
These questions can be modified based on the results of early interviews.

3. After travelling to the field, a similar regional/local initial meeting can be arranged
to bring together a large number of stakeholders. It can include government and
NGO staff, private sector representatives and local authorities.

N.B. Discussions at these initial meetings focus on the objectives and scope of the
project and its relative comparative advantages and relevance to the
country/sector/region.

4. Once in the project's headquarters, an initial "brainstorming" session with all of

the project staff can be organized. Its purpose is to review together the
weaknesses and strengths/benefits and achievements of the project. Another
objective is to recruit people who will eventually become the stakeholders in the
evaluation process i.e. who will have a continuing commitment to an ongoing
M&E system.

5. At village level, selected focus group and key informant interviews need to be
planned beforehand to encompass the various categories and classes of
stakeholders. This is of course to be done on a case-by-case basis according to
the nature and geographical coverage of project interventions.

• However, in most cases at least the 10 main categories of "local stakeholders" (as
well as 5 control groups), per agro-ecological zone, should be programmed into the
field visits.

• Focus group interviews can be carried out by one or more of the mission
members in an iterative, interactive, question and answer mode with all of the focus
group members (see section A.4). Ideally, all focus groups in the first round of data
collection will also be used to familiarize the project staff who will manage the
ongoing M&E system, as well as the selected villagers, with the techniques involved
in running successful focus groups.

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 7

• Specifically, the mission should seek to provide on-the-job training to at least two
staff and two stakeholders at village level in order to leave behind a follow-up team
for continuous M&E.

Feed-back

6. Feedback throughout the various layers of the "pyramid" needs also to be

programmed through a series of group meetings to be used for restitution,
cross-checking, validation and eventually negotiation of the mission's main
findings.

7. At village level a town/local meeting can be programmed with members and

non-members of project activities. Discussions should lead to an understanding of
weaknesses and strengths present in the project, as well as to the planning of
consequent and agreed upon follow-up actions.

8. At national/regional level, an inter-institutional evaluation workshop can be

planned with representatives of insider and outsider organizations, as well as
project personnel ranging from field staff to top management.

• At both levels, discussions should lead to a better understanding of evaluation

requirements and a commitment to maintaining an M&E process as feedback
to both the community and the implementing institutions.

• In both cases, project results are discussed to identify:

a. The specific success criteria by which each of the stakeholder groups
judge the success of the project;
b. The use being made (or to be made) of project results i.e. potential effects;
and
c. The potential, opportunities and limitations to ensure continuity,
sustainability and/or institutionalization.

• In both cases, PRA visualisation techniques may be introduced, presented

and facilitated by the team members, in order not to taint the self-evaluation
process and initiate a process by which stakeholders have an interest in the
continuous review of project results.

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 8

Annex 1
Impact Evaluation PRA Sequence

Review of Secondary Data

• Desk review
• In-country sources and grey literature

Key Informant Interviews

• National level (insider/outsider)
• Regional (government, private sector, civil society)
• Front-line community workers (extension, etc.)
• Inter-institutional Seminar *

Zoning and Purposive Sampling of Communities

• Agro-ecological, ethnic, production systems

Key Informant Interviews at Community Level

• Leaders (traditional, modern)
• Upper stratum (rich households, leaders)
• Outsiders (schoolteacher, health post)

Focus Group Interviews with Project Beneficiaries

• Middle stratum (average households) Males
• Lower stratum (poor households) Males
• Middle stratum (average households) Females
• Lower stratum (poor households) Females
• Youth groups (Male and Female)
• Representatives of Other (ethnic minorities, etc.)

Focus Group Interviews with Control Group **

(non-participants in project; non-beneficiaries)
• Middle stratum (average households) Male
• Middle stratum (average households) Female
• Lower stratum (poor households) Males
• Lower stratum (poor households) Females
• Other (ethnic minorities, youth, etc.)

Supplementary Cross-checks
• Field Observation
• Content analysis of newspaper articles
• PRA exercises within a Community Setting

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 9

Household Interviews (few)

Feedback to Community at Village Meeting

• For Cross-checking and validation
• Restitution of Findings
• Negotiation of possible solutions

Feedback to Project Staff

• To Front-line workers (extensionists, etc.)
• Heads of line departments
• Project Management

Quantitative Survey ("last Step")

• Parametric data
• To cover unanswered questions or
• To fill gaps in data
• To Substantiate Controversial findings

Feedback to National Level

• National project team
• Minister
• Donors (FAOR, UNDP, etc.)
• Inter-institutional Seminar *

* PD - The Inter-institutional Seminar can be held ex-ante for impact evaluation to review:
• validity and pertinence of project goals and timing
• level of dissemination & institutionalisation of project results
• the probable effects of externalities and the nature of these on project results
• success criteria to be considered in addition to foreseen criteria.

The Inter-institutional Seminar can be held ex-post to the field visits to review:
• validity and pertinence of project goals to national policies and within the national context
• validity and pertinence of project activities to ongoing efforts in partner institutions
• level of dissemination & potential for institutionalisation of project results
• success criteria to be considered in addition to foreseen criteria

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 10

** Focus Group Modality = no more than 5 group members, through a semi-structured

interview with open and closed questions derived from the former 4 steps.

Seasonal Calendar
Facilitator /
Village Diagram/Transect
Interviewer
Value Matrix

Focus Group
male/female, adults/youth, -
participants/non-participants, etc.

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999
 11

Annex 2
How People May Participate
in Development Programmes and Projects2

1. PASSIVE PARTICIPATION
People participate by being told what is going to happen or has already happened. It is a
unilateral announcement by an administration or project management without any listening to
people’s responses. The information being shared belongs only to external professionals.
2. PARTICIPATION IN INFORMATION GIVING
People participate by answering questions posed by “extractive” researchers using
questionnaire surveys or similar approaches. People do not have the opportunity to influence
proceedings, as the findings are neither shared or checked for accuracy.
3. PARTICIPATION BY CONSULTATION
People participate by being consulted and external agents listen to their views. These external
agents define both problems and solutions, and may modify these in the light of people’s
responses. Such a consultative process does not concede any share in decision making and
professionals are under no obligation to take on board people’s views.
4. PARTICIPATION FOR MATERIAL INCENTIVES
People participate by providing resources, for example labour, in return for food, cash or other
material incentives. Much on-farm research falls in this category, as farmers provide the fields
but are not involved in experimentation or the process of learning. It is very common to see
this called participation, yet people have no stake in prolonging activities when the incentives
end.
5. FUNCTIONAL PARTICIPATION
People participate by forming groups to meet predetermined objectives related to the project,
which can involve the development or promotion of externally initiated social organization.
Such involvement does not tend to be at early stages of project cycles or planning but rather
after major decisions have been made. These institutions tend to be dependent on external
initiators and facilitators, but may become self-dependent.
6. INTERACTIVE PARTICIPATION
People participate in joint analysis, which leads to action plans and the formation of new local
institutions or the strengthening of existing ones. It tends to involve interdisciplinary
methodologies that seek multiple perspectives and make use of systematic and structured
learning processes. These groups take control over local decisions and so people have a stake
in maintaining structures or practices.
7. SELF-MOBILISATION
People participate by taking initiatives independent of external institutions to change systems.
They develop contacts with external institutions for resources and technical advice they need,
but retain control over how resources are used. Such self-initiated mobilisation and collective
action may or may not challenge existing inequitable distributions of wealth and power.

2
Excerpt from ”Regenerating Agriculture - Policies and Practices for Sustainability and Self-Reliance”, Jules N.
Pretty, Earthscan Publications Ldt., London, 1995.

Notes based on the Seminar given in FAO

th
By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999