M&E Workshop Handout Manual

CARE International
DM&E Workshop Series 1997
VOLUME 1:
HANDOUT MANUAL
By Richard Caldwell and Sofia Sprechmann,

With additions from Jim Rugh
TABLE OF CONTENTS
TABLE OF CONTENTS ........................................................................................................................... i
Handout 1.1: WORKSHOP INTRODUCTION ........................................................................................ 1
Handout 1.2: WORKSHOP OVERVIEW AND SCHEDULE ................................................................... 3
Handout 2.1: MONITORING ................................................................................................................... 6
Handout 2.2: EVALUATION.................................................................................................................... 7
Handout 2.3: INDICATORS 1 ................................................................................................................. 8
Handout 2.4: INDICATORS 2 ................................................................................................................. 9
Handout 2.5: PROJECT HIERARCHY ................................................................................................ 10
Handout 2.6: GLOSSARY.................................................................................................................... 16
Handout 2.7: WHO NEEDS INFORMATION ABOUT PROJECTS? ................................................... 21
Handout 2.8: EXERCISE ..................................................................................................................... 24
Handout 3.1: PROBLEM ANALYSIS HIERARCHY............................................................................. 25
Handout 3.2: CHILD MALNUTRITION PROBLEM ANALYSIS ............................................................ 29
Handout 3.3: USES FOR PROBLEM ANALYSIS................................................................................. 30
Handout 3.4: CAUSE AND EFFECT LOGIC IN A TYPICAL PROJECT ............................................. 31
Handout 3.5: LOGFRAME OF A TYPICAL CARE PROJECT............................................................. 32
Handout 3.6: WHAT ARE INDICATORS AND HOW ARE THEY USED ............................................ 36
Handout 3.7: INDICATORS ................................................................................................................. 39
Handout 3.8: SMALL GROUP EXERCISE .......................................................................................... 42
Handout 3.9: THREE CATEGORIES OF DATA .................................................................................. 44
Handout 3.10: HOW TO DETERMINE WHAT DATA TO COLLECT .................................................. 45
Handout 3.11: INDICATORS AND DATA WORKSHEET.................................................................... 46
Handout 3.12: DEVELOPING M&E MATRICES ................................................................................. 47
Handout 4.1: RESEARCH DESIGN FOR EVALUATION .................................................................... 49
Handout 4.2: EVALUATION RESEARCH DESIGNS .......................................................................... 50
Handout 4.3: EVALUATION RESEARCH DESIGN ISSUES .............................................................. 51
Handout 4.4: EXERCISE - EVALUATION DESIGN ............................................................................ 52
Handout 5.1: QUANTITATIVE DATA AND METHODS....................................................................... 53
Handout 5.2: QUALITATIVE AND QUANTITATIVE METHODS .......................................................... 54
Handout 5.3: CONDUCTING QUANTITATIVE BASELINE AND EVALUATION SURVEYS .............. 55
Handout 5.4: SURVEY PLANNING & DESIGN ................................................................................... 56
Handout 5.5: SAMPLING ..................................................................................................................... 58
Handout 5.6: TARGET POPULATION................................................................................................. 59
Handout 5.7: SAMPLING FRAME ....................................................................................................... 60
Handout 5.8: SAMPLING METHODS.................................................................................................. 62
Handout 5.9: SAMPLE SIZE ................................................................................................................ 68
Handout 5.10: EXERCISE - TARGET POPULATION AND SAMPLING FRAME ............................... 72
Hanout 5.11: QUESTIONNAIRE DESIGN........................................................................................... 73
Handout 5.12: TYPE OF QUESTIONS................................................................................................ 75
Handout 5.13: QUESTIONNAIRE LAYOUT, LENGTH AND CODING ............................................... 77
Handout 5.14: EXAMPLE - QUESTIONNAIRE WITH SEVERAL TYPES OF QUESTIONS .............. 79
Handout 5.15: TWELVE GUIDELINES FOR DEVELOPING............................................................... 81
Handout 5.16: DATA COLLECTION - TRAINING AND FIELDWORK ............................................... 82
Handout 5.17: QUANTITATIVE DATA ANALYSIS.............................................................................. 87
Handout 5.18: TOOLS FOR QUANTITATIVE DATA ANALYSIS ........................................................ 88
Handout 5.19: GRAPHING DATA........................................................................................................ 91
Handout 6.1: QUALITATIVE DATA ..................................................................................................... 93
Handout 6.2: WHEN TO USE QUALITATIVE METHODS .................................................................. 94
Handout 6.3: THE CASE STUDY ........................................................................................................ 96
Handout 6.4: QUALITATIVE EVALUATION CHECKLIST................................................................... 97
Handout 6.5: NATURE OF QUALITATIVE DATA ............................................................................... 98
Handout 6.6: QUALITATIVE DATA COLLECTION ............................................................................. 99
Handout 6.7: OBSERVATION ........................................................................................................... 105
Handout 6.8: CHOOSING A SAMPLE:.............................................................................................. 107
Handout 6.9: QUALITATIVE DATA ANALYSIS AND INTERPRETATION ....................................... 108
Handout 6.10: FIELD RESEARCH FOR QUALITATIVE STUDIES .................................................. 111
Handout 6.11: CODING AND ANALYSIS.......................................................................................... 112
Handout 6.12: QUALITATIVE DATA ANALYSIS EXERCISE ........................................................... 115
CARE 1997 M&E Workshop Series
Caldwell and Sprechmann
Handout 6.13: HEALTH PROJECT CASE STUDY ........................................................................... 118

Handout 6.14: DATA ANALYSIS ....................................................................................................... 121
Handout 7.1: PARTICIPATORY MONITORING AND EVALUATION ............................................... 122
Handout 7.2: PARTICIPATORY M&E 2.............................................................................................. 123
Handout 8.1: DISSEMINATION AND UTILIZATION ......................................................................... 124
Handout 8.2: POSSIBLE DISSEMINATION STRATEGIES .............................................................. 125
Supplemental Handout 1: PARTICIPATORY METHODS .................................................................. 126
Supplemental handout 2: RATE EXERCISE ...................................................................................... 129
Supplemental handout 3: DEVELOPING EFFECTIVE EVALUATION PLANS................................. 131
ii
Handout 1.1
Handout 1.1: WORKSHOP INTRODUCTION

INTRODUCTION
Welcome to the CARE Monitoring and Evaluation Workshop. This workshop has been
designed to enhance your knowledge and skills related to the monitoring and evaluation of
CAREs development assistance projects.
Every CARE office is challenged with assessing the extent to which it effectively implements
its projects. This includes day-to-day accountability of goods and services the project
proposes to deliver as well as overall accountability of achieving expressed intermediate and
final goals. Monitoring and evaluation are the two processes that we use to assess whether
or not intended goods and services as well as project benefits are being realized.
This workshop is supported by CARE USA and is intended to develop the capacity of
Country Office staff in monitoring and evaluation. The challenge, however, is in designing
and conducting a workshop that meets both the needs of individuals and the needs of
Country Offices. Since not all individuals and Country Offices come to this training with the
same knowledge and work experience in monitoring and evaluation, it is expected that there
will be differences in what each person receives from this training. We sincerely hope that
this 5-day training and the accompanying handouts increases your understanding of the
monitoring and evaluation process enables you to improve the M&E of CARE projects and
motivates you to want to learn more and more about monitoring and evaluation.
Richard Caldwell
Sofia Sprechmann
Handout 1.1
WORKSHOP OBJECTIVES
9
To gain a working knowledge of the fundamental concepts, definitions, and steps

involved in CARE monitoring and evaluation.
To acquire new skills in quantitative and qualitative methods related to monitoring

and evaluation.
To develop monitoring and evaluation frameworks based on project design logic.
To explore specific issues and discuss possible alternatives for a country office
strategy for improving monitoring and evaluation.
Handout 1.2
Handout 1.2: WORKSHOP OVERVIEW AND SCHEDULE

Goal: By the end of the workshop, participants will better understand the CARE monitoring and evaluation systems and have acquired new
basic skills for effective monitoring and evaluation.
Day
1
Sessions
1.
2.
Purpose
Workshop Objectives and

Schedule
To introduce participants to the workshop format and schedule and to
M&E Conceptual Framework
To discuss the purposes of and CARE's approach for monitoring and
Time
1 Hour
develop a set of group objectives.

2 Hours
evaluation.
To introduce key concepts and terms used in M&E.
3.
From Logframes to M&E

Systems
To explore and establish the linkages between project design and
4.
Research Design for Evaluation
5.
Quantitative Methods
5.
Quantitative Methods
monitoring and evaluation, paying particular attention to the outputs of

the design process that are incorporated into the M&E system .
To develop an M&E framework that can be used as a base for
incorporating M&E in project implementation.
To overview evaluation research designs and discuss advantages and
disadvantages of using quasi-experimental evaluation designs.
To explore the characteristics and use of quantitative data.
To review the steps involved in conducting baseline and evaluation
surveys, review sampling design, size and procedures, explore the

different types of questions used for structured/semi-structured
interviews, understand the issues involved in designing good
questionnaires, review the tasks involved in data collection and
understand the most commonly used techniques for quantitative data
analysis.
Continued
5 Hours **
(1 hour will be on
Tuesday morning)
2 Hours
4 Hours
7 Hours
Handout 1.2
Day
4
Sessions
6.
Purpose
Qualitative Methods
To explore the characteristics and use of qualitative data.
Time
6 Hours
To review the major methods in collecting qualitative data, understand
the issues involved in designing good discussion and interview guides,

review field tasks and understand the general approaches to
qualitative data analysis and some common features of analytic
methods.
7.
8.
9.
Participatory Techniques in
M&E
To review participatory techniques used for M&E and discuss the
Information Dissemination and
To explore ways of presenting information to decision-makers,
Country Office Strategies for

Monitoring and Evaluation
10. Workshop Evaluation and

Testing
1 Hour
benefits of involving stakeholders in the design of M&E frameworks,

data collection and analysis.
1 Hour
understand the importance of disseminating and using information,

and learn the different components of an action plan.
An open forum discussing current difficulties in M&E and exploring
strategies for improving M&E.
2 Hours
Participants will be provided an opportunity to express opinions about

the workshop and will take a short test to assess knowledge and skills
development.
1 Hour
Handout 1.2
M&E Workshop Handouts

1.
Workshop Objectives and Schedule
2.
M&E Conceptual Framework
3.
From Logframes to M&E Systems
4.
Research Design for Evaluation
5.
Quantitative Data and Methods
6.
Qualitative Data and Methods
7.
Participatory Techniques in M&E
8.
Developing Data Collection Instruments
9.
Field Tasks and Data Collection
10. Information, Dissemination and Utilization
Handout 2.1
Handout 2.1: MONITORING

What?
Monitoring is the process of routinely gathering information on all aspects of the project.
Monitoring provides managers with information needed to:
Analyze the current situation

Identify problems and find solutions
Discover trends and patterns
Keep project activities on schedule
Measure progress towards intermediate goals and formulate/revise actions necessary to
achieve these goals
Make decisions about human, financial and material resources
Monitoring is continuous. A monitoring system should be in place before project start-up.

Monitoring activities should be scheduled on the project's workplan.
Monitoring can be carried out through field visits, review of service delivery and commodities
records, and management information systems.
Monitoring reports should be timely, simple, concise and useful. They can be alerting using
yellow flags (an activity in danger of not being carried out as planned) or red flag (an action
that has not been completed as anticipated). They should include results, if achieved,
actions necessary to correct any deficiencies, and an activity schedule and reporting timely
for those actions necessary to correct an activity.
Who?
The first level of monitoring is done by project staff. Supervisors are responsible for
monitoring the staff and tasks under them, and the project manager is responsible for
monitoring all aspects of the project.
The second level of monitoring is done by donor(s). Through field visits and routine reports
from the project manager, the donor monitors progress and measures performance.
Why?
Monitoring provides managers with information needed to analyze the current project
situation, identify problems and find solutions, discover trends and patterns, keep projects on
schedule, and measure progress towards expected outcomes. It allows the project team to
formulate or revise future goals, make decisions regarding human, financial and material
resources, and minimize needless project costs.
How?
Monitoring takes the form of MIS, including quarterly and other reports. The use of
monitoring information includes the output of the MIS and the review of reports. Monitoring
is routinely carried out through field visits, review of service delivery and commodities
records, input from management information systems, and from review of project quarterly
reports and other reporting documents.
Handout 2.2
Handout 2.2: EVALUATION

What is evaluation?
Evaluation is the process of gathering and analyzing information to determine:
1. whether the project is generating its planned outcomes; and
2. the extent to which the project is achieving its stated goals through these activities.
How does evaluation differ from monitoring?
Timing
Focus
Level of detail
What is the purpose of evaluation?
To find out how effective the project is

To determine whether goals have been achieved
To learn how well things are being done
To learn from experience so future activities can be improved ("lessons learned")
When do we evaluate?
Periodically
Mid-term
At the end of the project (final evaluation)
After the end of a project (post-evaluation)
Who evaluates?
Internal evaluation can be carried out by the project manager and/or project staff.
External evaluations are carried out by donors or individual consultants.
What should we evaluate?
Progress in workplan
Establishment of systems
Implementation of planned activities
Achievement of goals
Effectiveness of project
Impact of project
Efficiency/cost-effectiveness of project
Handout 2.3
Handout 2.3: INDICATORS 1

What?
Indicators are measures used to ascertain or verify that a planned change has occurred.
DEFINING GOOD INDICATORS

Indicators have to be....
VALID
Actually measure the what they are intended to measure;
RELIABLE
Produce the same results when used more than once to

measure precisely the same phenomenon;
RELEVANT
Should apply to goal, intermediate goals, outputs;
SENSITIVE
Sensitive to the situation being observed, reflect changes of

the phenomenon under study;
SPECIFIC
Measure only what they are intended to measure;
OPERATIONAL
Be measurable or quantifiable with developed and tested

definitions and reference standards;
COST-EFFECTIVE
Results should be worth the time and money it costs to

apply/collect them;
TIMELY
It should be possible to collect data "reasonably quickly".
Handout 2.4
Handout 2.4: INDICATORS 2

CRITERIA FOR DECIDING WHAT TO MEASURE
Progress towards final goal, intermediate goals
Needed information - useful information
Data that has most potential to re-direct activities
Balance: Need to know ability to find out
CLASSIFYING INDICATORS IN TERMS OF IMPORTANCE AND EASE OF DATA

COLLECTION
Data collection feasibility
Importance of
indicator
EASY
FEASIBLE
DIFFICULT
HIGH
High priority
Worth collecting if
possible
Worth collecting if
possible
LOW
Worth collecting part

of instrument for
"important indicator"
Worth collecting part

of instrument for
"important indicator"
Low priority
Handout 2.5
Handout 2.5: PROJECT HIERARCHY

CAREs TERMINOLOGY FOR PROJECT HIERARCHY
Inputs
Resources needed by project (i.e., funds, staff, commodities, in-kind)
Process
Interventions or activities done by project utilising the input
Output
The direct result of process; products of project activities
Effect
Improvements in access to or quality of resources, and change in practices
Impact
Sustainable improvements in human conditions or well-being
10
Handout 2.5
TABLE I. HIERARCHY OF OBJECTIVES

CARE
terminology
Input
Previous
CARE
LogFrame
Input
Process
Activity
Output
Output
Effect
Intermediate
Goal
Impact
Final Goal
WHAT
resources
needed by
project
interventions
or activities
done by the
project
products
directly
produced by
the project
reactions and
actions of
target
populations as
a
consequence
of exposure to
project
interventions
sustainable
changes in
human
conditions or
well-being of
target
population at
household
level
CAUSED BY
WHOM
project staff
use them (and
are
accountable)
project staff do
it (and are
accountable)
CLAIMED BY
WHOM
100%
attributable to
the project
within the life

of project
(continuously)
100%
attributable to
the project
within the life

of project
(continuously)
project staff
produce it
(and are
accountable)
beneficiaries
do it, systems
reflect it
100%
attributable to
the project
within the life

of project
(when process
bears fruit)
within the life
of project (may
require special
study to
measure)
beneficiaries
experience it
attribution is
difficult, with
other
influences
substantial
and inevitable
should be
largely
attributable to
the project,
with other
influences
relatively
minor
TIME-FRAME
sometimes
measurable
within life of
project, but
more likely
requires postproject
evaluation
11
Handout 2.5
TABLE II. SECTORIAL EXAMPLES OF INDICATORS AT DIFFERENT LEVELS

Reproductive Health
example
INPUT
Resources used
PROCESS
Activities done utilizing
the input
OUTPUT
The direct result of the
process
EFFECT
Change in practices
IMPACT
Conditional changes at
household level
ANR example
SEAD example
Water example
grants, contracts, donations, commodities, supplies, other in-kind, staff hired and trained, vehicles
purchased, etc
# talks given
# home visits
# counselling sessions
# IEC materials
distributed
# communities needing
water identified
# organized to undertake
water system installation
# staff visits to farming

communities
# training sessions
organized
# staff visits to organize

communities
# village bank training
sessions held
# of methods distributed;
# of people trained
during sessions, visits
# groups of farmers
formed and trained
# clients receiving credit

# clients participating in
savings program
# of new or renovated
water systems installed
and functioning
# New FP acceptors;
# Couple Years of
Protection;
# families adopting biointensive crop

technology
y area covered with biointensive technology;
# household IGAs with

increased working
capital / improved
production technologies
# of target population
using sufficient and safe
water supply;
yincrease in per capita
consumption of water
yIncreased
Contraceptive
Prevelence Rate
% families who produce

enough food to cover
lean periods;
y decreased numberof
children who are
malnourished
yIncrease in net
household income
yPositive changes in HH
consumption patterns
yReduced morbidity and

mortality from water- and
excreta-related diseases
12
Handout 2.5
RELATIONSHIP BETWEEN CAREs PROGRAMS AND HLS FRAMEWORK
CARE
Goal
FUNCTION
Program
Goals
Sub
Sectors
Educational
Health Security
Clusters
Household Livelihood
Security
Respiratory Problems
Diarrhea
Measles
Birth Spacing
CYP
Health &
Population
Childrens
Health
Food Security
Literacy Rates by
Gender
Girls & Womens

Education
Adequate Access to
Food
Food Assisted
Projects
Reproductive
Health
Water, Sanitation &

Environmental
Health
Economic Security
Positive Income Flow
Income Cluster
(SEAD & ANR)
Financial &
Non-financial
Services
Formal & Informal

Education
Food for Work,

Cash For Work
& Safety Nets
Cropping &
Livestock
Systems,
Natural
Resources &
Biodiversity
STRUCTURE
Cross Cutting Themes:
Partnership
Community Participation
Sustainability Relief
Rehabilitation
Development
13
Handout 2.5
TABLE III. COMPARISONS OF DIFFERENT CARE REPORTING FORMATS

TOOL
Type of Indicator
(framework)
PIR
INPUTS
PROCESS
OUTPUTS
Main Purpose
Day to day
management at
the country level
Main Users
PM
CO
Other users
PAD Sector
Marketing (ER)
Donors
Periodicity
Quarterly
API
OUTPUTS
EFFECTS
1) Quantified
picture of what
CARE is doing
2) Report of key
indicators projects
and COs
achievements
3) Provide
aggregated data
of sectoral
portfolio
RMU, PAD Sector
Marketing (ER)
Program SVP
Donors
CO
PM
Annual
Optional
Format
Narrative,
Tables:
achievements vs.
targets
Questionnaire,
quantitative data
Data source
Field staff
Monitoring data
PIRs
Method of
collection
Continuous
Monitoring
Summary of
quarterly PIRs
Person
responsible for
Implementing
Project Manager /
Project MIS staff
Project Manager /
Project MIS staff
Person
responsible for
supervising
Country Director /
ACD
Country Director /
ACD
Project
evaluation
EFFECTS
potential IMPACT
Mid-term: assess
changes needed
in implementation
Final: Assess
systems and
behavioral
changes
accomplished
HLS Program
Evaluation
IMPACT
Assess
improvement in:
a) household
livelihood
security; b)
institutional
capacity
PM
CO
Donors
RMU
PAD Sector
CD
PAD
RM
HQ SMT
President
Board
Mid term
Final
Post-Project ( 3-5
yrs after the
project ends)
Narrative report
Research
Baseline
Every 3-5 years
Report
Project
Documents
External
assessment
May include
survey
Research / Survey
/ interviews
HLS Assessment
Post-Project
evaluations
Project Manager /
CO Sector
Coordinator / CO
M&E Coordinator
Country Director /
ACD
ACD / CO M&E
Coordinator +
PHLS
Research /
Survey
Country Director
Global: PHLS,
RMU + PAD
14
Handout 2.5
DATA FLOW
Processes: Project
manager
Activities aiming
for Purpose #1
1. W
2. X
3. Y
4. Z
Activities aiming
for Purpose #2
1. A
2. B
3. C
4. D
Activities aiming
for Purpose #3
1. M
2. N
3. O
4. P
Outputs: Project
manager, Country
director, TAG, RM
Effect: Project
manager, CD, TAG,
RM
Impact: CD, RM, TAG,

SMT, SVPs. President,
Board
Output #1
Effect # 1
Impact at the
Hopusehold
Level
Output #2
Effect # 2
Output #3
The effect can be

assesed by the
analysis of one or
several outputs.
15
Handout 2.6
Handout 2.6: GLOSSARY

Proposed Standard CARE Definitions of Key Terms
What follow are proposed definitions of some of the more common terms used with regard to
LogFrames and the API.
Impact: sustainable, significant improvements in human conditions or well-being, reflecting
the satisfaction of basic needs. This is the final goal level for CARE projects. Needs are
basic if they must be satisfied in order to secure the physical development of the individual
according to their genetic potential. Basic needs include food, health services, favorable
environmental conditions (potable water, shelter, sanitation), primary education, and
community participation. To obtain the essential resources necessary to meet basic needs,
households must have adequate access to finances, skills, time and social positions. These
conditional or well-being changes represent program level impacts. (It is recognized that
changes at the impact level are often influenced by other factors as well as those directly
addressed by a project).
Effect: changes in behaviors and practices that result from the use of training and services
provided by a project, or improvements in the access to or quality of resources. This is the
intermediate goal level for CARE projects. Effects can be seen in changed behavior (i.e.
when individuals put into practice what they have learned through a projects outputs).
There can also be effects seen through systemic changes (i.e. of institutions).
Note: Outcomes (or results) often refer to all that happens as a consequence of a projects
interventions. These are more specifically divided into effects and impacts.
Outputs: the direct results of project activities. Project outputs may refer to: 1) The results
of training, such as the number of women trained in improved nutrition, farmers in improved
agricultural techniques, etc. Note that this should include measures of changes in
knowledge and attitude. In other words, it is not sufficient to count how many people
attended a course, but some measure of how many actually learned what was being taught.
2) Capacity building, such as the number of extension staff trained, water systems built,
committees established, etc.; 3) Service outputs, such as an increase in the number of
program locations. 4) Service utilization, such as the number of people fed, or number or
patients treated. Outputs are the products the project produces. Indicators of outputs are
typically derived from the routine monitoring of project-based data. When compared to
resources used for inputs, a measure of outputs can be utilized to make at least a rough
assessment of the efficiency of project performance.
Processes: the intervention or set of activities through which project inputs are used to
obtain the expected results. This is the activity level for CARE. These activities include
management and supervision, counterpart training, service delivery, technical assistance
and information and evaluation systems.
Inputs: the set of resources that are needed by a project. These include the human and
financial resources, physical facilities, equipment, materials, logistics, in-kind contributions
and operational polices that enable program services to be delivered. The monitoring of
inputs are typical functions undertaken by most monitoring (or tracking) systems concerned
with basic management and accountability.
16
Handout 2.6
Indicators: primary or summary measures used to demonstrate change as a result of a

project intervention. Indicators may express quantitative elements (i.e. be written as
numbers -- which is useful for reports such as the API) or qualitative aspects (i.e. words -which may better describe the condition and quality involved). An indicator is a marker
which, when used over time, shows what progress has been made. It is a variable which
can be measured at various times, such as baseline, during project implementation, and final
evaluation.
Since this is such a key term, an extended description is warranted.
Note that there can be indicators for measuring progress at every level: input, process,
output, effect and impact. Note also that an indicator refers to what is being measured.
When a target for achievement by a certain time is attached to an indicator it becomes, by
definition, a goal or an objective.
The term indicator has been used by some to specifically refer to a sub-Intermediate Goal.
However, this alternative definition can cause confusion. Note that in the CARE Program
Measurement Framework and API descriptions there are frequent references to indicators
for all levels from input to impact. At any level an indicator can be turned into a goal or
objective by specifying a target and a date for achievement. The preferred term for a subIntermediate Goal would be a supporting objective.
At different levels indicators can be composites of lower-level indicators (or indexes or
variables). General indicators usually require more specific indicators or variables to clearly
define how they are to be measured. (State as measured by ... in Means of Verification
[MOV]).
Characteristics of good indicators include the following:
relevant (to the respective level of a logframe hierarchy)
measurable (assessable) (for the API indicators: need to be quantifiable, but
there can be qualitative indicators)
realistic (based on data which can be obtained or measured with reasonable
means)
objectively verifiable (someone else could agree with the same findings)
reliable (they consistently measure what theyre supposed to)
valid (they measure what they purport to measure)
meaningful (significant to what the project seeks to accomplish)
useable (by the managers and other people involved in a project)
comprehensible (simple to understand)
sensitive (capable of demonstrating changes in the situation being observed)
timely (it should be possible to collect and analyze the data reasonably quickly)
cost effective (the value of having the data is worth the time and effort required to
collect it)
Index: the product from a composite of variables or indicators that, through their
combination, or calculated by an industry-accepted formula, produce data that permits
comparisons between projects or programs.
Variable: a quantity of measurement which in any given situation may change by increase
or decrease. A variable can be thought of as a primary indicator; the thing one measures
directly. Usually obtained by direct observation, or through replies to a specific question by
persons being interviewed. Every variable represents a separate data point.
17
Handout 2.6
API: Annual Project Information (referring to the primary source of the data). (Also Annual
Portfolio Inventory, referring to its use in the aggregate form). A CARE reporting instrument
designed to measure and communicate project outputs/results. (It replaces the Basic Data
Questionnaire of previous years.) The API was initiated in response to needs by HQ, though
it is intended to be useful to others as well, including COs and PMs.
PIR: Project Implementation Report. A reporting instrument that updates progress in
relation to the projects annual implementation plan. It includes both a descriptive section
and a matrix in which the projects performance by indicators and intermediate goals is
quantified. Main users are those who need to do more detailed project analysis, which may
include Project Managers, CO Sectoral Coordinators, Country Offices, and (in come cases)
PAD Sectoral Coordinators. These reports are often useful for partners and donors. Ideally
data needed for the API are included within the PIR.
Direct Beneficiaries: Individuals who receive services or resources directly from CARE or
through a joint implementation partner. This category would typically include what projects
often refer to as participants, recipients, clients. (Note: It needs to be made clear which
beneficiaries are reached directly by CARE and which are reached directly by a partner
institution. This information will be recorded the partnership matrix on the API form.)
Net total Direct Beneficiaries: Net beneficiaries refers to those people who were reached
by one or more of the projects interventions. If, for example, in a multi-sectoral project a
woman gets prenatal care, a water system is installed near her home, she learns how to
grow vegetables, she takes literacy classes and she participates in a credit program, she is
only counted as one direct beneficiary, even though she benefited in much more than one
way. In other words, the net direct beneficiary figure is not a total of the beneficiaries
counted for each indicator. Each of those indicators has its purpose, but to add them all up
would be double counting.
This will need to be the PMs best estimate based on the records kept by the project.
If the projects information system keeps track of beneficiaries by name, it should be
possible to count the net total of persons reached, even though some of them were
reached in more than one way. Otherwise the project staff will need to make an
educated guess as to how many people benefited, discounting double counting
which would occur if simply adding up totals counted by indicators of different
interventions.
Note: The following will NOT be included on the API, due to the inconsistency with which
these numbers are generated:
Indirect Beneficiaries: Individuals who benefit indirectly from the project. This is
commonly determined by simply multiplying the number of direct beneficiaries by an
assumed average number of persons per family, or other factors. The ratios used by
individual projects have varied greatly. Though there may be legitimate and valid reasons
for these, where estimates of indirect beneficiaries are needed at the aggregate level (e.g.
by ER), past ratios of direct to indirect beneficiaries could be used by HQ.
Secondary Beneficiaries: Individuals who adopt an innovation which they learned about
from a direct beneficiary (second hand), such as neighboring farmers who learned about a
new technique from farmers who were trained by CARE. This is also known as the diffusion
effect or the multiplier effect, and is thus a desirable indicator of a projects wider
replication, effectiveness and sustainability.
It is important for CARE projects to account for the spread of a projects effect through
secondary beneficiaries, but it is not always feasible to obtain this information through
18
Handout 2.6
routine monitoring. This is why this data is not called for on the API. However, such
measures of the diffusion or multiplier effect should be included in periodic surveys such
as those conducted as a part of project evaluations.
(Another definition of secondary beneficiary is those persons reached directly by partner
institutions, even if not directly by CARE. As explained in the section on partnerships in the
API document, secondary beneficiaries defined in this way are to be included in the API
data. The fact that they were not reached directly by CARE will be noted on the Partnership
Matrix.)
Some additional definitions:
Disaster Response: A sum of decisions and actions taken during and after a disaster,
including immediate relief, rehabilitation and reconstruction. (Definition used by UNHCR)
Relief: As defined by the UNHCR, Relief is defined as assistance and/or intervention during
or after a disaster to meet the life preservation and basic subsistence needs. It can be of
emergency or protracted duration.
Relief has two aspects:
1) Emergency Response, quick fix activities. These are way stations, latrine trenches, high
protein biscuits, tents and other quick fix camp shelter stuff, emergency health care,
bladders for water supply, relief items, e.g. blankets, clothing, soap, etc..
2) Sustained Relief. These are longer term relief efforts, e.g. camps and camp activities for
IDPs or refugees. They can include health activities with displaced populations; free food
distributions (e.g. in Haiti or Ethiopia), where the population cannot sustain itself. There
might be other safety net activities, such as water or sanitation, e.g. providing water in
camps. There also could be some relief items, e.g. blankets, clothing, soap, CIKs like
soccer balls for camp populations.
Much of the above is a "we do for", but some is also done using development approaches to
working with people.
Rehabilitation The operations and decisions taken after a disaster with a view to restoring a
stricken community to its former living conditions, while encouraging and facilitating the
necessary adjustments to the changes caused by the disaster. (UNHCR)
Moving along the continuum from relief towards development. This includes getting people
home and responding to their critical needs. It may involve help with more permanent
shelters/schools, rehab or rebuilding community water systems, credit for building new
homes, provision of seeds and tools, de-mining. All of this should be done using
development methodologies, or at least keeping in mind the potential for longer term
sustainability.
Quality: Provision of superior services to all project beneficiaries/clients. Quality can be
ascertained by the range of choices that clients have, the completeness of the information
given to clients, technical competence of the provider, quality of interpersonal relations, and
appropriateness of services provided.
Quantitative data deals with numbers. It is easier to aggregate, do statistical analysis on
and display in tables than is qualitative data, but can be subject to misinterpretation (may
miss the point.) Quantitative methods commonly involve surveys and questionnaires which
ask for information which can be collected in numerical form.
19
Handout 2.6
Qualitative data is descriptive, complementing quantitative data with contextual information

to give a more holistic picture of a situation. Qualitative techniques commonly include indepth, open-ended interviews, direct observation, and written documents. (Even though
qualitative data is not as easy to analyze statistically, there are software programs which
enable rigorous analysis.)
(A combination of qualitative and quantitative information is needed for a fair evaluation.
One should aim for increased quality in the gathering and analysis of both quantitative and
qualitative data. Following the characteristics of good indicators listed above, and using
appropriate monitoring and evaluation methodologies helps assure such quality.)
Sustainability: 1) Potential for projects impact to continue after CAREs intervention
terminates. 2) Capacity of beneficiaries to be able to continue to practice an innovation or
technique without continued project intervention. 3) Capacity of local institutions to continue
project activities after the project ends. This may include self-financing the activities via
contributions of users of the goods and services provided, complementary funding from local
funding sources, and decreasing dependency on continued funding from external sources.
4) Sustainability also has an environmental protection aspect. In this case, sustainability
refers to the maintenance or enhancement of resource productivity on a long-term basis.
Sustainability, therefore, implies a respect and care for all forms of life, an improvement of
the quality of human life, a conservation of life-support systems and biodiversity, minimizing
the depletion of non-renewable natural resources, and enabling communities to care for their
own environment.
Scale: The extent to which the project serves large numbers of beneficiaries, or indirectly
has a positive effect on large numbers of people. Scale is essential to achieving both
sustainability of service delivery and significant impact among the target populations.
Equity: The extent to which the resources and opportunities generated by the project are
equally distributed within and among households. It pertains to the allocation of resources
according to gender, ethnic affiliation, social status and class.
Efficiency: The extent to which a project uses resources appropriately and completes
activities in a timely fashion.
Effectiveness: The extent to which a project makes desired changes or meets its
objectives through the delivery of services.
Cost-effectiveness: a measure of how much it costs to achieve results (usually in terms of
project outputs). Though in simple terms a ratio of benefits divided by costs, a benefit:cost
ratio is quite difficult to measure without making many assumptions, including attaching a
monetary value to benefits. Nevertheless, it is important to develop systems (including costaccounting and monitoring of benefits) and analytical frameworks which enable us to
determine relative cost-effectiveness of various interventions, even if doing so to any degree
of precision may be challenging.
20
Handout 2.6
Handout 2.7: WHO NEEDS INFORMATION ABOUT

PROJECTS?
Consider the range of participants and stakeholders who have needs for information about
CARE projects, and who are relied upon to provide and pass along portions of the data set.
Lets begin with the primary focus -- the individual household. Assuming that in that
households community there is a Community-Based Organization (CBO) involved, that CBO
asks the household to share certain information about what is happening in the lives of its
members -- basically indicators pertaining in some way to household livelihood security
(HLS). The CBO will need to have the kind of relationship with the household which
engenders a willingness to share this information. That includes making that household feel
a part in the ownership in what the CBO is doing, and providing it with appropriate feedback
on why the information being collected is needed and how it is being used. The CBO thus
collects information about all of the households in its community.
Let us further assume that there is a local Non-Governmental Organization (NGO) in the
area which relates to a number of CBOs. The NGO will ask the member CBOs to pass on
information about whats going on in the lives of their households. This, again, requires a
relationship in which the CBOs share a feeling of partnership with the NGO and understand
the mutual need for this information.
Further developing our hypothetical model, a CARE project in that part of the country has a
partnership relationship with one or more such NGOs. It will need to develop a relationship
with them which includes an appreciation for the need to gather and pass on certain kinds of
information, and shares in the analysis and use of the data generated. Appropriate forms of
feedback from the CARE project to the NGO can engender this relationship.
The Sector Coordinator in the CARE Country Office asks each of the projects in that sector
to pass on information about what is being accomplished by the project, which includes data
which originated from the household level. The Sector Coordinator performs a fairly detailed
level of analysis and provides feedback to the Project Manager.
Similarly, the Country Director (CD) and Assistant Country Director (ACD) for Program (and
the CO Senior Management Team) have needs for country-wide information to keep in touch
with how projects and sectors are performing. This information is needed for making
decisions at their level, including decisions about possible changes needed in the support of
present projects and plans for the design of future projects.
A number of stakeholders in the CARE Headquarters in Atlanta have needs for key
information about projects. These include the Regional Management Units (RMUs), the
Program Analysis and Development (PAD) Sector Coordinators, and those in the External
Relations (ER) Division who need to prepare fundraising proposals and reports to donors.
And the Senior Vice President (SVP) for Program needs information in a form useful for
reporting to the President and Board.
But the needs for information on CARE projects does not stop there. There are additional
audiences who are interested in various forms of reports. These include other CARE
International (CI) headquarters, other international NGOs and others who study and want to
learn lessons learned from CAREs relief, rehabilitation and development assistance
process. Of course the donors who support one or more projects are keenly interested in
information about achievement of results.
21
Handout 2.6
A depiction of these various levels of stakeholders and a simplified information-flow are

shown in the diagram on the next page. It needs to be remembered that similar
considerations of the quality of relationships between levels described at the HH-CBO-NGOProject levels pertains to other levels as well. I.e., each of us needs to have an
understanding of the need for data, how it is to be analyzed and used, and how that
information benefits our work. This includes appropriate forms of feedback to lower levels
to let them know how the information was used, which could include how they compare with
others in the bigger picture.
The purpose of our going through this listing of the various sources and users of CARE
project data to remind us of who else is involved. At each level there tends be an
assumption that data needs to be collected and aggregated for our own use. Thus we need
to recognize that the pyramid of information needs within CARE addressed by the API is
only a piece of a larger picture.
22
Handout 2.7
Other households . . .
Household
Other CBOs . . .
..
CBO
. Other CBOs
Other NGOs . . .
NGO
. . Other NGOs
Other projects . . .
Other COs
..
. . . Other households
CARE Project
. Other projects
CO: Sector Coordinator, ACD, CD
Other COs
Atlanta: RMU / PAD / SVP / ER / Board

Other lead countries . . .
. . . Other lead countries
CARE International
Bilateral Donors -- Individual Donors
KEY: Main flow

of information:
Other PVOs -- Students of Development

Feedback
From: Program Measurement Framework, CARE USA, January 1997
23
Handout 2.8
Handout 2.8: EXERCISE

Match the following terms with their definition
__ Impact
The process of gathering and analyzing information to

determine whether the project is generating its planned
activities and the extent to which the project is achieving its
stated goals through these activities.
Direct result of process, products of project activities.
__ Monitoring
Sustainable improvements in human conditions or well-being.
__ Indicators
__ Problem analysis
Describes a set of cause and effect relationships among

system variables
Resources needed by project.
__ Evaluation
__ Output
Improvements in access to or quality of resources, and

changes in practices
__ Effect indicators
The process of routinely gathering information on all aspects of

the project.
__ Process
Measures used to ascertain or verify that a planned change

has occurred.
__ Inputs
Measurements which describe the change in condition or

behavior as a result of achieving an intermediate goal.
__ Effect
Activities carried out by a project which convert inputs into

planned outputs.
24
Handout 3.1
Handout 3.1: PROBLEM ANALYSIS HIERARCHY

Problem Analysis describes a set of complex relationships among system variables
PROBLEM
on
Condition
Condition
Condition
Condition
SPECIFIC CONDITIONAL
CAUSES CONTRIBUTING TO THE PROBLEM
Multiple Levels
- Primary (Direct, Immediate)
- Secondary (Indirect)
- Tertiary
- etc
Usually identified
CARE projects usually not focused at this level
25
Handout 3.1
PROBLEM
Condition
Condition
Condition
Condition
Behavior
Behavior
SPECIFIC BEHAVIORAL CAUSES CONTRIBUTING TO CONDITIONS
Can also be multiple levels
Often requires Needs Assessment or other information to identify/verify
A common level for CARE projects to target
26
Handout 3.1
PROBLEM
Condition
Condition
Condition
Condition
Behavior
Behavior
Attitude
Knowledge
Belief
SPECIFIC ATTITUDES/KNOWLEDGE/BELIEFS CONTRIBUTING TO BEHAVIOR

Usually only one level identified
Often requires Needs Assessment to identify
CARE projects rarely target this level, but interventions can be designed around
attitudes, knowledge and beliefs
27
Handout 3.1
IDENTIFIED
PROBLEM
Condition
Condition
Condition
Condition
Behavior
Behavior
Knowledge
Attitude
Belief
General Social, Cultural and Political Factors
GENERAL FACTORS WHICH SHAPE THE CONTEXT

Usually beyond the scope of CARE projects to address
Project design, however, needs to take these factors into account
28
Handout 3.2
Handout 3.2: CHILD MALNUTRITION PROBLEM ANALYSIS

Farmers Unwilling
to Change Practices
Mothers Do Not
Perceive a Problem
in Malnutrition
Rural HHs are Very

Superstitious
A Low Value is
Afforded to
Education
Doctors Want a
High Income
People
Unaware of
Proper Sani-Practices
Mothers Do Not
Participate in
Child Feeding
Programs
Foods are Classified

on the Basis of
Folklore
Parents are
Uneducated
Most Doctors are

Unwilling to
Work in
Rural Areas
HHs Don't
Install Screens
or Plumbing in
Houses
High Soil , High Farm HHs Lack

Erosion, Inflation Income-Generating
Opportunities
Mothers Do
Not Have
Prenatal Care
Families Don't Attend

Nutrition Education
Programs
Low Agricultural
Production
Improper
Weaning
Lack of Nutrition
Knowledge
Farmers
Slash
& Burn
Men Control All

HH Income
Women Have No
Access to Credit/
Capital
Low Income
Inadequate Quantity
of Food Provided to
Children
High
Illiteracy
Rates
Few Gov't Clinics are Far

Health
From Villages
Care Programs
Inadequate
Health Care
Poor Nutritional Quality

of Food Prepared For and
Consumed By Children
Houses Open
To Flies &
Rodents
Unsanitary HH
Conditions
Predominance of
Infectious Diseases
HIGH MALNUTRITION RATES AMONG CHILDREN UNDER 5 IN MAWAWASI PROVINCE

Abnormally Low Physical
And Mental Child Growth
and Development
High Mortality in
Children Under 5
HH Incurs Higher
Health Care Costs
High Rate of Malnutrition

-Induced Diseases
Malnourished Children Do Not Become

Productive Members of HH
29
Handout 3.3
Handout 3.3: USES FOR PROBLEM ANALYSIS

USES FOR PROBLEM ANALYSIS IN THE DESIGN OF CARE PROJECTS
A hierarchical analysis of cause and effect
Development of guidelines for a needs assessment
Process for utilizing information obtained from needs assessment
Selection of causal level to address
Selection of appropriate output, effect and impact indicators
Exploration of sectoral interactions
Buy-in to project design for staff, partners, beneficiaries, donors, etc.
Assessment of relative contribution of various cause-effect streams to the problem
Assessment of impact potential
30
Handout 3.4
Handout 3.4: CAUSE AND EFFECT LOGIC IN A TYPICAL PROJECT

Levels of the
Problem Analysis
Output of the Problem

Analysis/Needs Assessment
Output of the Strategy

Selection
Hierarchy of Final
and Intermediate Goals
Cause and Effect

Logic
Broad Condition
(poverty)
Specific Condition
(low production)
Problem
(low income)
Direct (Immediate) Causes
(high soil erosion/low fertility)
Indirect (Secondary) Causes
(farmers plant steep slopes)
Problem
(low income)
Final Goal
(increase income)
Impact
(change in the condition)
Causes
(farmers plant steep slopes)
Intermediate
Goals
(farmers will adopt new
cropping practices)
Effects
(change in systems
or behavior)
Interventions
(extension education)
Outputs
(establish extension system)
Outputs
(goods and services
produced by project)
Behaviors
(poor planting practices)
Behavioral Causes
(farmers plant in rows; do not
practice contour planting)
Attitudes and Beliefs

(tradition must
be followed)
Social, Political, Economic,
Cultural, and Environmental
Factors
(no access to improved technologies)
Activities
Processes
(hire and recruit extension
(activities to turn
workers; organize farmer
inputs into outputs)
groups; conduct demonstration
tasks)
Resources (labor, cash,
M&E necessary to establish
extension education)
Inputs
(resources necessary to
achieve outputs)
31
Handout 3.5
Handout 3.5: LOGFRAME OF A TYPICAL CARE PROJECT

NARRATIVE SUMMARY
O.V. INDICATORS
MEANS OF
VERIFICATION
ASSUMPTIONS
Final Goal:
Improved reproductive
health of women and
men in Kabale, Kisoro
and Rukungiri Districts by
2001
* Statistically significant decrease in maternal mortality

and total fertility rates
* DHS 1995, 1999

* Surveillance reports
* National surveys
* Govt continues its commitment to a

national Reproductive Health Program
* Country continues to enjoy political and
civil stability
* Decentralization is working effectively
* Improving quality and access will
increase consumer use of Reproductive
Health Services
Behavioral Intermediate
Goals:
1. Increased use of
modern family
planning methods in
the project area
Increase at End of project in:

* New acceptors from 20,000 to 70,000
* Continuing users from 21,000 to 80,000
* CYP from 19,000 to 60,000
* CPR (modern methods) from 10.5% to 15%
* Service delivery
statistics from HMIS,
SDPs and CRHWs
reports
* survey
* Project mid-term
survey 1999
* Communities will support family

planning
* Difference between client visits and
new acceptors on HMIS of the MOH
form equals continuing users
* Clients go for services
2. Increased number of
women seeking
maternal health
services at a health
facility
Increase by End of project in:

* Women delivering at service delivery points from 24%
- 45% of all pregnant women in the project area
* Women attending anti natal services at least 2 time
per a given pregnancy to be 46,000
* Women attending post natal (PNC) services to be
50% of antenatal cases (ANC)
* Obstetric emergency presenting at a qualified obstetric
emergency unit is 40% of identified cases
* HMIS records
* CRHW records
* CREHP clinic and
CRHW data on ante
natal, post natal, and
EOC services
* Final evaluation
survey
* The new HMIS is operational in the 3

project districts
* Obstetric services are functioning and
acceptable and affordable
* ANC accessible, affordable and
efficient
* PNC incorporated into SDP services
* Improved quality of services and
knowledge will increase use
3. Increased demand,
prevention, diagnosis
and treatment services
for sexually transmitted
infections (excluding
HIV/AIDS)
50% increase by the End of Project in:

* Annual number of men, women, adolescents
presenting at SDPs for STIs diagnosis and treatment
due to increased awareness of the problem and
improved services
* Annual # of condoms distributed or sold to end users
* 30% reduction of women, men and adolescents
reporting risk behavior
* HMIS records
* Clinic treatment
records
* CREHP referral
forms
* CRHWs records
* World Bank STI program operates

effectively in all three districts
* Syndromic rather than laboratory
diagnosis is acceptable for clients
* Adequate STI drugs are at parish level
SDPs
* Men will use condoms to prevent
infection
32
Handout 3.5
4. Increased knowledge
in reproductive health
among women, men
and adolescents
* % of women and men who have knowledge of at least

2 benefits of family increased from 23% to 50%
* % of women &men who can mention 2 FP methods
increased from 35% - 60%
* % of women and men who know a source for
contraceptives increased from 50% to 80%
* 50% increase over baseline of women and men who
know the location of the closest EOC
can mention at least 2 benefits of ante natal care
can name 1 benefit of post natal services
* 50% increase over baseline of women and men and
adolescents who can name a source of STI treatment
* 1995 CREHP final

survey
* Operations research
on IEC
* Baseline/final survey
* There will be community interest and

participation
Systemic Intermediate
Goal:
Improved quality services
for family planning,
maternal health and
sexually transmitted
infections at service
delivery points and
community levels
By the end of the project:

* 50% of trained clinic health workers provide quality
selected Reproductive Health Services
* 60% of CRHWs are active (as defined by the project)
* 70% of selected service delivery points equipped with
reproductive health kits by the project have functioning
equipment in place
* SDPs and CRHWs have appropriate method mix
* 50% of clients express satisfaction with services
offered by service delivery points and CRHWs
*
*
*
*
*
Training reports
Supervision reports
QOC scores
CRHW records
Contraceptive
logistics records
* Client exit interviews
* Operations research
* SDP surveys
* DHT and communities support all

CREHP RH services offered
* DMOs will assume responsibility for
equipment maintenance
* GOU (MOH) will clear and exempt VAT
on imported CARE equipment
* MOH/DMO logistics system is in place
* District administration will provide funds
to transport STI drugs and
contraceptives
* GOU and district administration will
provide DMO health workers with a
living wage
* Better educated consumers will press
for improved quality services
* 25 DMO trainers trained in RH

* 375 health workers trained to provide integrated RH
care annually
* 34 supervisors trained to provide RH quality care
supervision
* 60% of trained health workers attain satisfactory QOC
scores at 6 months intervals post training
* Training reports
* CREHP and DMO
planning reports
* Supervision reports
* QOC forms
* DMOs will release trainers for training

workshops
* Health workers are motivated and
enabled to improve skills and quality of
care
Outputs
375 clinic-based health
workers trained to
provide high quality,
integrated services
33
Handout 3.5
420 CRHWs trained,

equipped and
encouraged to provide
quality basic RH
services
* 420 CRHW Trainers trained in basic RH, IEC, and

sales techniques
* 420 trained CRHWs regularly supplied with condoms
and OCs and equipped with standard project CRHW
kits
* 10 CRHW associations formed
* 245 community leaders trained in elementary RH care
and communications techniques
* 60% of CRHWs attain satisfactory QOC scores at 6
month intervals post training
*
*
*
*
Training reports
Field Officers reports
Supervisors reports
CREHP logistics
records
* CRHW records
* CBO surveys
* Communities will identify appropriate

CRHWs
* CRHW associations will actively
support members in their community
RH work
* Appropriate CBOs will be interested in
working with the project
Selected SDPs supplied

with RH kits
* 80 dispensaries, posts and private clinics equipped

with basic FP kits
* 25 Health Centers and maternity units equipped with
FP and basic obstetric kits
* 6 hospitals equipped with FP and obstetric emergency
kits
* 4 hospitals equipped with surgical contraception kits
* CARE records
* Funding approved
* VAT exemption by GOU on equipment
and supplies
50% of SDPs regularly

offer integrated basic
RH services at EOP
* 50% SDPs with CREHP trained staff and equipment

offer the three basic RH interventions
*
*
*
*
* DMOs and in-charges are committed to

integrated services
Supervision reports
SDP client records
Client exit interviews
Clinic surveys
34
Handout 3.5
How do monitoring and evaluation relate to the project logframe and the cause-effect
hierarchy? As with the Logframe, the structure of an M&E system is characterised by the
same hierarchy in exactly the same way. The following table shows how each level of the
cause-effect hierarchy links with specific monitoring and evaluation assessments.
RELATIONSHIP OF M&E TO PROJECT CAUSE-EFFECT LOGIC

Hierarchy of
Cause-Effect Logic
Types of
Information
Monitoring
Activities
Evaluation
Activities
Final Goal
Impacts
(fundamental
changes in the lives
of the target
population)
Few, if any
Baseline, then
summative or expost evaluation
Intermediate Goals
Effects (behavioral
and systemic
changes)
May use periodic

assessment to
measure change, but
more evaluative
Formative and
Summative
evaluation, annual
reviews
Outputs
Outputs (goods and

services produced
by the project)
Regular
measurement and
Reporting
Formative and
Summative
evaluation, annual
reviews
Processes
(Activities)
Activity targets
Regular
measurement and
Reporting
Usually assessed
thru use of key
questions; analysis
of monitoring data
Inputs
Planned Inputs
Financial Accounting
Analysis of financial
and other monitoring
data during
evaluation
35
Handout 3.6
Handout 3.6: WHAT ARE INDICATORS AND HOW

ARE THEY USED
Definitions of Indicators
To better understand what indicators are, it is worth reviewing some of the more common
definitions used for indicators. According to USAID:
An indicator is a unit of measurement which facilitates concise, comprehensive, and
balanced judgments about a situation. It is subject to the interpretation that if its level
changes in the "right" direction, things have gotten better and if the level changes in
the "wrong" direction, things are getting worse, or people are "worse off."
Indicators may be identical to the specific objective (direct), substitute for the
objective (indirect or proxy), or supplement the objective by describing certain
qualities.
Using a single indicator cannot give a comprehensive picture of change so multiple
indicators are often needed.
Practical Concepts Incorporated (PCI) in discussing "The Logical Framework" has this to say
about indicators;
Indicators are defined as those conditions that are so strictly associated with certain
other conditions (of the situation in question) that presence of or variation in the
former indicates the presence of or variation of the latter. Therefore, indicators are
plausible.
Indicators demonstrate results and are not conditions necessary to achieve those
results. They are independent. Indicators are not used to demonstrate achievement
by measuring the means to achieve the result.
In CARE, we will use the following definition:
A variable, measure or criterion used to assist in verifying whether a proposed event or
change has occurred.
In CARE project, we develop indicators for two reasons:
1. To measure attainment of inputs, processes, outputs, effects and impacts related to our
project design hierarchy.
2. To evaluate key questions in the evaluation of projects and programs.
Procedure for Determining Indicators

A convenient procedure for developing indicators involves the following steps:
1. Define the situation to be measured.
2. Determine the classification of analysis to be applied to investigate the situation. This is
done by converting the interrogative into the classification of analysis. Below are some
common interrogatives and their classification.
Interrogative
how much
how many
Classification of Analysis
amount, quantity, number
amount, quantity, number
(frequency)
(frequency)
36
Handout 3.6
how long
why
where
when
who
how
which
able to (can, could)
will
what
would, if
period of time
reasons
location
point or period in time
person, people
ways, methods, techniques
options
ability, potential, possibility
predictions
specifies, can mean which or why
possibilities
3. Develop the conditions that will be used to better understand the situation. Conditions
reflect the state of being of a particular being or thing (situation). They are the core of an
indicator and are used to give a concise, comprehensive, and balanced understanding of
the situation.
4. Determine the numerical form to be used to measure the conditions. Options for
numerical forms include frequency counts of the number of times events take place and
ratios, usually percentages and proportions.
5. Write the indicator:
INDICATOR = NUMERICAL FORM + CONDITION
In practice, the procedure might work the following way:
Question: How do farmers currently apply liquid pesticides?
1. Situation: After examining the question carefully, the situation in question could be
stated as application of pesticides.
2. Classification of Analysis: How implies ways, methods, or techniques.
3. Conditions: To get a concise, comprehensive, and balanced understanding of
methods farmers use to apply pesticide, the following conditions could be used:
a) use of hand pump sprayers.
b) use of the coke bottle method.
c) use other methods (specify methods).
4. Numerical Form: Percentage of farmers can be used to give a fairly clear picture
of how many.
5. Timeframe (optional): The question asks how farmers are currently applying
pesticides which implies at the present time. Current needs to be stated in a specific
timeframe which will dictate for when data must be collected. We might say during
the last 3 months or May, June, and July.
6. Indicators: So the indicators would be written in the following way:
a) % of farmers who properly used hand pump sprayers to apply pesticides.
b) % of farmers who properly used the coke bottle method to apply pesticides
two growing seasons after training.
c) % of farmers who used x,y,z (other methods) to apply pesticides at the end
of the project.
37
Handout 3.6
Questions, Situations, and Indicators

Below are some examples of questions, situations, and indicators. Examine the indicators
for each question and practice identifying the classification of analysis, conditions, and
timeframe (these steps have been omitted).
Key questions for analysis
1. How many women are
currently participating in the
small economic activities
program?
Situation
Women's participation in
the small economic
activities program.
Indicators
# of women who report they knew
about the program between June
and July, 1998.
# of active members in the program
between June and July, 1998.
2. How do farmers feel about

using fertilizer?
farmers' attitudes about

the use of fertilizer.
# of women who report during the

survey (August) their husbands
prefer they not participate.
% of farmers who believe during the
survey (August) fertilizer increases
production.
% of farmers who felt during the
survey (August) fertilizer is too
expensive.
3. How effective were the health

promoters in communities last
quarter?
Effectiveness of health
promoters in
communities.
% of farmers who report they used

fertilizer between June and July,
1998.
Average # days per month health
promoters spend in communities
between January and March, 1997.
Mothers' non-participation
in the last vaccination
campaign.
% of families who knew about the

vaccination program between June
and July, 1998.
% of mothers who can

demonstrate how to mix ORT
home solution during the survey
(April).
4. Why didn't mothers bring their
children to be vaccinated during
the last campaign?
% of mothers who had to travel more

than 2 kilometers to the vaccination
post between June and July, 1998.
% of mothers giving reasons x,y,z
(other reasons)
38
Handout 3.7
Handout 3.7: INDICATORS

A variable, measure or criterion used to assist in verifying:
whether a proposed change has occurred, and thus
whether a final or intermediate goal has been achieved.
Indicators are quantitative or qualitative criteria for success that enable one to measure or
assess the achievement of project goals. There are four general types of indicators:
input indicators - describe what goes into the project, such as the number of hours of
training, the amount of money spent, the number of contraceptives distributed;
output indicators - describe project activities such as the number of community workers
trained, the number of family planning acceptors, the number of women enrolled in
mothers' clubs;
effect indicators - describe the change in condition or behavior as a result of achieving

an intermediate goal
impact indicators - measure actual change in conditions of the basic problem identified,
including changes in livelihood status, health, wealth, etc.
Input and output indicators are easier to measure than effect and impact indicators, but they
provide only an indirect measure of the success of the project. They assume that the
achievement of certain activities will result in change, but they don't demonstrate it. They
also provide a standard against which to measure, or assess, or show, the progress of an
activity against stated targets or benchmarks.
Common indicators: input, output, process, effect and impact, leading, trailing, etc.
They can be direct or indirect (proxy). Indirect are often impact indicators and used when a
direct measure is not feasible or cost effective.
Criteria Checklist for Good Indicators:

Ideally, indicators should be:
Valid
they should actually measure what they are supposed to measure;
Reliable (i.e., verifiable or objective) - conclusions based on them should be the same if
measured by different people at different times and under different circumstances;
Relevant they should apply to final and intermediate goals;
Sensitive they should be sensitive to changes in the situation being observed;
Specific
they should be based on available data;
Cost Effective - the results should be worth the time and money it costs to apply them; and
Timely - it should be possible to collect the data reasonably quickly.
39
Handout 3.7
CHECKLIST FOR INDICATORS

_ Is a percentage or an absolute number more appropriate?
Use a percentage when you can accurately quantify the target group or your sample
size is adequate for inferring to the general population. Use absolute numbers if your
target group changes in size over the life of the project and when it is feasible to
sample.
_ Does the indicator reflect what is written in the goal statement? Does it really
measure what it is supposed to measure (i.e. - is it valid)?
For example, suppose your goal is to enable community health workers to properly
administer oral rehydration salts to children to combat dehydration. Now, if your
indicator was the # of children/month who suffered bouts of dehydration would this be
a valid indicator? No, although you are interested in seeing dehydration, you would
not be directly measuring the behavioral change. You would need another direct
indicator to measure the proper administration of salts.
_ Be sure the indicator is not in fact an activity or output.
One common error is to use the number of people trained as an indicator for
behavioral change. This, of course, is an output and not a measure of change.
_ Is it realistic?
For example, suppose the goal is to increase agricultural production 20% in the
project area and the indicator is going to be simply changes in agricultural
production, is this a realistic indicator given the large variation in yearly production
(especially in rainfed areas) and the difficulties in getting a large enough sample size.
_ Does the indicator contain terms that are unclear, or that can be interpreted in
different ways?
A CARE SEAD project focused on improving production activities for women by
providing them with skills training and credit. One indicator was % of women who are
able to improve the quality of their products. You can see that the term quality
could be interpreted in many different ways and would vary from product to product.
_ Can you realistically obtain reliable date regarding the indicator?
This is a very common problem with indicators and some examples include
measuring income, crop production averages, and child malnutrition rates. Each of
these is creates problems in getting reliable and cost effective data.
40
Handout 3.7
COMMON TYPES OF INDICATORS (modified from Feuerstein, 1986)

Type of
Indicator
Availability
Relevance
Accessibility
Utilization
Coverage
Quality
Effort
Efficiency
Effect
Impact
What it Shows
Example
These show whether something

exists and if it is available
How relevant or appropriate
something is
Whether what exists is actually
within reach of those who need it
Whether there is one trained local worker for

every ten households
Whether new stoves burn less fuel than old
stoves
A health post in one village may be out of
reach of other villages due to mountains,
seasonal flooding, inadequate transport, high
costs, etc.
How many non-literate villagers attend literacy
classes regularly; # of households using
newly constructed latrines
Of the number of people estimated to have
tuberculosis in a given area, what percentage
are actually receiving regular treatment
Whether water is free from harmful, diseaseinducing substances; whether water quality
meets national standards
The number of trained health workers it takes
to effectively vaccinate children in a given
geographical area in a given time
The number, frequency and quality of
supervisory visits after introducing bicycles to
replace heavy vehicles
The number of farmers adopting a particular
cropping practice
After a campaign against malaria, does the
incidence against malaria decline
To what extent something has

been made available is being used
for that purpose
Show the proportion of those who
are in need are actually receiving
services
Show the quality or standard of
something
How much and what is being
invested to achieve an output or
effect
Whether resources and activities
are being put to the best possible
use to achieve desired results
Whether behavioral or systemic
changes are taking place
Whether there is fundamental
change
41
Handout 3.8
Handout 3.8: SMALL GROUP EXERCISE

DEVELOPING INDICATORS
This exercise is meant to provide you with practice in developing indicators. On page 2, key
questions that are typical of those used for evaluation are listed in the left column. The task
is to develop 2 or 3 indicators for each question. Listed on page 3 are several intermediate
goals from actual CARE projects. For each intermediate goal develop at least two indicators
which could be used to measure effect changes. To do so, follow the procedure below for
developing indicators.
1. Examine the question carefully.
2. On a different piece of paper, state the situation in question and determine the
classification of analysis.
3. For each situation, ask what conditions would give a concise, comprehensive, and
balanced judgment about the situation to be able to answer the question.
4. List all possible conditions beside each situation.
5. Select the final combination of conditions which will give the most concise,
comprehensive, and balanced understanding of the situation.
6. Write the indicators next to the corresponding questions. Check the indicators to make
sure they satisfy the characteristics of indicators.
Key questions
1. Why don't men participate in community health education activities?
2. Did the income of the micro enterprise operators increase last quarter?
3. What was the agricultural production of farmers this quarter?
4. Has the nutrition status of children in the school nutritiion program improved?
Intermediate Goal #1:
80 farmers participating in the project organize and efficiently manage a communal seed
fund by the end of the first year of the project.
By September 1998, 700 women borrowers in enterprise management will be effectively
managing their small enterprises.
By the end of the project, 40% or more of village households will properly use improved
personal hygiene practices.
42
Handout 3.8
SMALL GROUP EXERCISE: M&E PLAN - STEP 1

DRY ZONE AGRICULTURAL DEVELOPMENT PROJECT
This exercise is meant to provide you with practice in developing the first part of an M&E
Plan based on the CARE/Sri Lanka Dry Zone Agricultural Development Project (DZADP)
proposal. You are required to read the information provided on the DZADP - background,
logical framework, and the monitoring and evaluation section. We will be developing an
M&E plan for this project following the M&E Matrix shown in Handout 3.12. We will base our
work on the existing Logical Framework and make modifications to this framework as the
week progresses. For this exercise, please complete the following.
Step 1. Your group will be assigned to work on the Wider Objective and one Immediate
Objective. Remember! In CARE the Wider Objective is also called the Final Goal and
Intermediate Objectives are also called Intermediate Goals. They are the same thing.
Carefully review the objectives your group has been assigned along with their indicators of
achievement (column 2) and how they will be measured (column 3).
Step 2. Examine each objective and determine if it is properly written according to CARE
standards. Is it clear in its meaning? Is it SMART? Your group should discuss how it wants
to write each objective statement and, once decided, print each on a flipchart.
Step 3. For each objective carefully examine the indicators. Determine first if these
indicators are sufficient for measuring the objectives. Follow guidelines on developing good
indicators.
HINT: It is often useful to define an overall indicator for measuring a goal, and next decide
what criteria (operational definitions) are included in the overall indicator. Based on the
criteria then develop sub-indicators that will actuall be measured.
Step 4. Decide which indicators are needed for each objective and list these on flipchart
paper in a column next to the objective. Be sure that each indicator is clearly written.
Pretend that you are going on vacation and before you leave you are giving this M&E plan to
your staff so that they can conduct a baseline survey. EACH INDICATOR MUST BE
WRITTEN SO THAT IT IS OBVIOUS WHAT IS BEING MEASURED.
Step 5. Be prepared to discuss and defend your work with the large group.
43
Handout 3.9
Handout 3.9: THREE CATEGORIES OF DATA

Data can be organized into three general categories; responsive, descriptive, and
documentary. The particular category determines the data source, collection process, and
their content and form. Below are brief descriptions of each category.
1. Responsive
Data which are characterized by responses by members of the target population. The
source for responsive data is people associated with the project. They are referred to as
respondents. The process for collecting these data is interrogative or questioning. Data
gathering tools most commonly used to collect responsive data are survey questionnaires,
group meetings or discussions, individual structured interviews, focus groups, conversational
interviews, ranking, rating, self-reporting instruments, and tests. The form of the data is
open, closed, or modified-closed. Open form refers to data which is in the respondents own
words (quotes) and is almost always qualitative. Closed form data are limited to a set
number of prearranged responses such as multiple choice questions, rating, or ranking.
Similar to closed form data are modified-closed form data. The major difference is that the
latter has the option for respondents to add to the pre-selected responses.
2. Descriptive (Observation)
Data which are characterized by observations. The source for descriptive data is people,
animals, or phenomena. Phenomena are occurrences or facts which can be detected or
observed. The primary process for collecting these data is observation. However, when a
physical phenomenon is the data source, such as water or soil, a physical sample must first
be collected, then observed. Data gathering methods frequently used to collect descriptive
data are diaries, observation and participant observation using schedules and guides,
narrative reports, and physical samples using microscopes or various physical/chemical
property analysis tests. Descriptive data forms are the same as those of responsive; open,
closed, and modified-closed. Open form data are descriptions by the observer without any
sort of pre-classification system. Examples of these data are feelings, interpretations,
hunches, and ideas reported in diaries or personal journals. Contrarily, closed data refer to
descriptions with a pre-classification system such as observation schedules. Modifiedclosed data adhere to a pre-classification system but with built-in flexibility for the observer to
make additions.
3. Documentary
Data which can be found in written form. The source is documents such as project
monitoring reports, evaluations, proposals, administrative records, and reference books.
The process for collecting documentary data is examination of the documents. Methods of
gathering these data include enumeration tallies and direct or narrative recording. Forms of
these data are the same as those for the responsive and descriptive categories. Open form
data are usually narrative recording without predetermined information needs while closed
form data were gathered with predetermined information needs such as enumeration tallies
or direct recording. Modified-closed data are those with information needs pre-established
but allows the examiner to make additions.
44
Handout 3.10
Handout 3.10: HOW TO DETERMINE WHAT DATA

TO COLLECT
1. Examine the indicator and identify its numerical form, condition, and timeframe.
Numerical form is a criterion that determines the quantity, capacity or dimension of the
condition. The condition is a particular state of being of a person or thing which is so closely
associated with the situation in question that its presence or any variation implies the
presence or variation in the situation. The timeframe refers to the period of time for which
the condition pertains. For the indicator, "percentage of households that stored drinking
water in a protected container between June 1 and July 30," the numerical form is
percentage (of households) while the condition is storage of drinking water in protected
containers. The timeframe is between June 1 and July 30.
2. Determine the data needed to construct the indicator. Most often these will be data
necessary to calculate the value of the numerical forms. For example, the indicator,
"percentage of households that stored drinking water in protected containers between June
1 and July 30" would require data to construct the percentage; number of households that
stored water in protected containers and the total number of households in the priority
population.
3. For data needed, determine the desired category and form. The category can be one or
a combination of the following; responses, descriptions, and documentation. Form can be
open, closed, or modified-closed (see Handout 3.9 for a more detailed discussion of data
categories and form). For the pieces of data from the water storage indicator, "number of
households that stored water..." and "number of households participating in the project," the
category for the first could be responses and documentation for the second. Responses
could come from a questionnaire survey while documentation could be gotten from
published census reports.
4. Specify the condition for the data. Conditions for data will either be identical or closely
related to those of the indicator. Following the water storage example, the data have 2
different but related conditions; "households that store drinking water in protected
containers" and "households participating in the project."
5. Express data in written form:
CATEGORY (FORM) + CONDITION
Data needed for the water storage indicator would be written like this: Responses (closed
form) of the number of households storing drinking water in protected containers AND
documentation (closed form) of the number of households participating in the project.
45
Handout 3.11
Handout 3.11: INDICATORS AND DATA

WORKSHEET
Listed in random order below are a question and its indicators and data. The task consists
of two parts. First, identify the question and write it under the question column, identify the
indicators write them beside the question under the indicator column; identify data for each
indicator and write them next to the indicator in the data column. You will notice the data are
incomplete. So for the second part of the task, determine and write the missing data. Use
Handout 3.10 as a guide.
_________________________________________________________________
Description (closed form) of number of health workers who take complete history of diarrhea.
Percentage of health workers who took complete history of diarrhea between April and June,
1999.
Responses (closed form) of number of health workers who counsel mothers.
How effective was the ORT health worker last quarter.
Proportion of health workers who managed diarrhea cases between April and June, 1999.
Description (closed form) of number of health workers who counsel mothers.
Documentation (closed form) of the number of health workers in the ORT program last
quarter.
Documentation (closed form) of the number of health workers who counseled mothers last
quarter.
QUESTION
INDICATORS
DATA NEEDED
46
Handout 3.12
Handout 3.12: DEVELOPING M&E MATRICES

Effective and efficient monitoring and evaluation requires careful planning. The optimal time
for developing an M&E plan is during the latter stages of project design. This plan can then
be easily modified at key points in implementation, such as after a baseline survey or during
preparation of specific evaluation plans.
All to often, however, projects are implemented without M&E plans. Usually when it is too
late, project managers suddenly realize that information is not being collected and used or
information that is being collected is not relevant to decision-making.
Logframes are useful tools to use as a starting point for developing M&E frameworks or
matrices. Logframes in and of themselves, however, do not contain all the required
information for a good M&E plan. Additional information needed includes:
How will the information needed be gathered? Who will collect it? When will it be
collected?
How will the gathered information be analyzed? Who will analyze it? When will the
analysis be completed?
Who will receive the results? In what format will they be distributed? What decisions
can be made?
Recognizing that the logframe does not contain all the useful information it is helpful (in fact,
essential) to develop a framework which includes all the information needed for planning
M&E. A useful way to do this is to develop a Monitoring and Evaluation Planning Matrix that
expands on the logframe and specifically addresses M&E needs.
47
Handout 3.12
BASIC FRAMEWORK FOR A MONITORING AND EVALUATION SYSTEM PLAN

Hierarchy
Indicators
Data Needed
Data Source
and Method
Frequency of
Collection
Person(s)
Responsible
Data Analysis
Dissemination
& Utilization
Final Goal
Intermediate
Goals
Outputs
Activities
Inputs
48
Handout 4.1
Handout 4.1: RESEARCH DESIGN FOR

EVALUATION
Choices of groups to be measured:
Experimental group only
Experimental group and a true (randomly assigned) control group
Experimental group and a non-equivalent (not randomly assigned) control group
Timing of measurements:
Pretest and posttest (baseline and final evaluation)
Posttest only (final evaluation only)
Time series (several measurements)
Vocabulary and symbols for research design

Observation =
Intervention or program being tested =
Alternative program (or no intervention) =
Experimental (project) group =
E>
Control (comparison) group =
C>
Randomization =
49
Handout 4.2
Handout 4.2: EVALUATION RESEARCH DESIGNS

1. No control group
No baseline
Baseline and final evaluation
2. With control group

No baseline
E>
C>
Baseline and final evaluation
E>
C>
O
O
3. Time series
Single group
OOO
Control group
OOO
OOO
OOO
50
Handout 4.3
Handout 4.3: EVALUATION RESEARCH DESIGN

ISSUES
PROBLEMS IN EVALUATION RESEARCH DESIGN WITH CONTROL GROUPS
External factors which change one group (E> or C>) but not the other, and which could
influence the outcomes.
Control group uses methods, materials, programs being tested on E-group.
People drop out of either E- or C- group for any reason.
Differences between E- and C- group in time spent on the program.
CRITERIA FOR SELECTING APPROPRIATE DESIGN FOR PROJECT

EVALUATION
Costs
Skills of staff to do surveys analyze results
Availability of staff
Project participants time and willingness to cooperate
Non- participants (control group's) time and willingness to cooperate
Ethics of using control groups
Accessibility to do a survey (security and other constraints)
Length of project
Need to prove attribution
51
Handout 4.4
Handout 4.4: EXERCISE - EVALUATION DESIGN

What evaluation designs were used to evaluate English programs in schools?
EVALUATION DESIGN
In School A teachers used visual aids for
English courses, in School B they did not.
The average score in the final test in School
A was 79 and in School B it was 64.
In School A the average score in the pre-test
was 63 and in the final English test it was 79.
In School A the average score in the final
English test was 79.
In School A teachers used visual aids for
English courses, in School B they did not.
The average score in the pre-test was 63 in
School A and 58 in School B. The average
final test scores was 79 in School A and 64
in School B.
In School A test were administered on a
regular basis (every week) in order to track
the effect of the introduction of a range of
teaching aids on students performance.
52
Handout 5.1
Handout 5.1: QUANTITATIVE DATA AND METHODS

What is quantitative data?
Quantitative data is data that can be analyzed using measures and techniques that can
summarize and describe information into usable numbers (percentages, ratios, rates, mean,
average, range). These summary and descriptive measures are also called statistics. The
aim of statistics is INSIGHT and not numbers. Statistics can condense attitudes, knowledge
and behavior of people in summary numbers that can be easily understood, remembered
and used as a basis for making decisions, setting baselines and evaluating projects.
While the aim of statistics is to help to make numbers more manageable, poor data or data
of low quality can not be saved using statistics.
"He uses statistics like a drunken man uses a lamp post, more
for support than for illumination."
Andrew Lang
Example of quantitative data sources
census
surveys
observation records
attendance numbers to health centers, schools, training, etc.
training pre- and posttests
What is a quantitative survey?

A survey is a method of collecting information directly from people about their feelings,
motivations, plans, beliefs, behaviors and their background. Surveys are usually conducted
using a questionnaire. Quantitative surveys use specially designed questionnaires for which
the range of answers is known in advance. Quantitative surveys can be used for BASELINE
and EVALUATION surveys which allow to quantify the IMPACT of a project.
A quantitative survey
provides scale and scope of behaviors, attitudes, knowledge studied;
obtains precise, statistical answers to defined questions;
obtains quantifiable information which can be extrapolated, generalized;
can collect information on a large population giving precise estimates.
53
Handout 5.2
Handout 5.2: QUALITATIVE AND QUANTITATIVE

METHODS
STRENGTHS AND WEAKNESSES OF QUALITATIVE AND QUANTITATIVE
METHODS
Time
Cost-effectiveness
Interview
participation
Flexibility of
protocol
Interviewer's skills
and experience
Statistical basis
Scope and scale
Type of information
QUANTITATIVE
Takes more time in data collection
Collection of data is more expensive,
higher yield in statistical data
Medium
QUALITATIVE
Takes less
Cost is higher
in analysis
High
Strict
High flexibility
Basic skills
needed
Valid and
statistically reliable
Generalizable
(i.e. 80% of young people ever heard
about HIV/AIDS)
Broader,
number based
Significant
experience
Credible
Inferential
(we can infer that ....)
Richer and
more in depth
CRITERIA FOR SELECTING QUANTITATIVE OR QUALITATIVE METHODS
Quantitative methods
Obtain precise, statistical answers to defined
questions
Collect information on a large population
giving precise estimates
Obtain quantifiable information which can be
extrapolated, generalized
Some information about the problem(s) and
issue(s) studied already available
Qualitative methods
Obtain rich information and understanding of
community life, peoples attitudes, opinions,
beliefs and behaviors
Explore attitudes
Research sensitive topics
Get "feel" for a problem
The best: a mixture of both quantitative and qualitative methods, as they provide different
perspectives, they have different advantages and allow cross-checking of information.
QUALITATIVE
If there is no qualitative
information available prior to a
major quantitative survey, it is
ESSENTIAL to conduct first
qualitative research, which will
allow to design a good survey
questionnaire.
QUANTITATIVE
Provides information on the
extent of the problems studied
and guides which research to
pursue.
QUALITATIVE
Allows further and more indepth insight and
understanding about
problems and issues found
in quantitative study.
54
Handout 5.3
Handout 5.3: CONDUCTING QUANTITATIVE

BASELINE AND EVALUATION SURVEYS
1. SURVEY PLANNING AND DESIGN
Prepare outline including survey objectives, research questions, target population,

coverage and reach of study, timeline and budget
Review existing information
Obtain permissions/buy-ins by key stakeholders and partners
Conduct preliminary qualitative assessment, obtain input from participants on issues,

questions to be asked, and categories of anticipated replies.
2. QUESTIONNAIRE DESIGN
Design draft questionnaire
Pre-test and review draft questionnaire
Finalize questionnaire
Draft tabulation plan
3. SAMPLING
Map target population and compile sampling frame
Design sample and decide on sample size
Select sample
4. TRAINING AND FIELDWORK

Select interviewers and supervisors
Train interviewers and supervisors
Complete fieldwork
Supervise fieldwork
Check and file questionnaires
5. DATA PROCESSING
Check forms
Code and edit questionnaires
Transfer data to computer
Clean and edit computerized data
6. DATA ANALYSIS AND REPORTING

Produce tables based on tabulation plan
Prepare charts and graphs
Study tables and draw conclusions from findings
Prepare draft and final reports
7. DISSEMINATION OF FINDINGS
Print and distribute appropriate reports for targeted audiences
Organize seminars, workshops, and discussions with project staff, beneficiaries,
partners, donors, and other stakeholders to communicate findings, get involvement in
determining follow-up action plans
Prepare action plan to carry out recommendations
55
Handout 5.4
Handout 5.4: SURVEY PLANNING & DESIGN

REVIEW EXISTING INFORMATION
Why?
It is of key importance to review existing information before conducting a survey in order to:
find out and learn what is already known about the issues and topics the survey aims to
address;
avoid duplication (don't waste time, effort and scarce resources!).
How?
Review all available and relevant information, including earlier studies and surveys,
papers, reports, policies and others
Discuss the research areas as widely as possible, both formally and informally. Contact
governmental and non-governmental organizations, community representatives,
community groups, key informants
SURVEY OUTLINE
Include in the survey outline:
Objectives and aims of the survey
Target population
Survey coverage
Methods (including sample design and size)
Data collection
Data processing and analysis
Budget
Timeline
Example: Survey timeline

Tasks
Planning and design
Review information
Prepare survey outline and budget
Obtain permissions
Weeks
4
5
Questionnaire design
Draft questionnaire
Pre-test draft questionnaire
Review and finalize questionnaire
Draft tabulation plan
etc.
56
Handout 5.4
Budget items to consider for a survey:
Personnel
Interviewers
Supervisors
Data entry clerks
Data analysis
Technical assistance (sampling, demography, etc.)
Time of permanent staff
Materials and supplies
Training of interviewers and supervisors
Travel (including lodging and per diem)
Transportation (including vehicle rental, fuel)
Computer
Translations
Communications
Printing of report
Public presentation, meetings, workshop
Contingencies
57
Handout 5.5
Handout 5.5: SAMPLING

What is a sample?
A sample is a subset of the population that is used to gain information about the entire
population. A good sample will represent the population well (REPRESENTATIVE
SAMPLE). How well a sample represents the target population depends on the sampling
frame, the sample size, and the sample design and selection procedures.
Why sample?
Sampling is efficient
Samples can be studied more quickly that target populations and are less expensive.
Sampling is precise
Sampling helps to focus the survey on precisely the characteristics of interest. For example,
if a study wants to compare rural communities and urban community, sampling strategies
are available (in this case stratified sampling) to obtain what is needed.
Checklist for obtaining a sample that

represents the target population
; Survey objectives are stated precisely
; Target population is clearly defined
; Rigorous sampling methods are chosen
58
Handout 5.6
Handout 5.6: TARGET POPULATION

Before choosing a sample, it has to be defined who is part of the target population for the
survey. This choice will depend on the purpose of the study. Inclusion and exclusion criteria
will have to be defined, that is, characteristics that include certain people and rule out certain
others.
Example: Target population and inclusion criteria

Research question:
What is the Contraceptive Prevalence Rate in

Province X?
Target population:
Women of reproductive age
Inclusion criteria:
Married
Between ages 15 and 49
Example: Target population and inclusion criteria

Research question:
Are antenatal care clients satisfied with the care

received in health centers?
Target population:
Pregnant women
Inclusion criteria:
Pregnant in the 6 month period prior to the

survey
Visited health center at least once for antenatal
care in the 6 month period prior to the survey.
59
Handout 5.7
Handout 5.7: SAMPLING FRAME

Once the target population is defined, the next question is whether or not there are complete
lists or maps of the target population from which a sample can be selected. Such list or map
is called the SAMPLING FRAME. The sampling frame is the set of people that has a chance
to be selected. A sample can only be representative of the population included in the
sampling frame. One design issue is how well the sampling frame corresponds to the
population the researcher wants to describe.
Most sampling frames fall into 3 general classes:
Sampling is done from a more or less complete list of individuals in the population to be
studied.
Sampling is done from a set of people who go somewhere or do something that enables
them to be sampled (example: patients that received medical care in a health center, or
people who attended a meeting). In this cases there are no advance lists from which
sampling occurs. The creation of the list and the process of sampling occur
simultaneously.
Sampling is done in two or more stages, with the first stage involving selecting
something other than the individuals finally to be selected (example: village selected first,
than individuals in selected villages).
Examples of sampling frames
a list of villages
a list of antenatal care clients
a list of government health staff trained in HIV/AIDS prevention
a village map showing individual dwellings
a list of household heads
There are three characteristics of a sampling frame that a researcher should evaluate:
Comprehensiveness
A sample can only be representative of the sampling frame, that is, the population that
actually had a chance to be selected. A key part of evaluating any sampling scheme is
determining the percentage of the study population that has a chance of being selected and
the extent to which those excluded are distinctive.
Examples of incomplete sampling frames
recent migrants not listed (out-of-date)
compiled in 1965 (out-of-date)
ethnic minorities not included (biased)
squatter population not included (biased)
60
Handout 5.7
Probability of selection
Is it possible to calculate the probability of selection of each person sampled? For example,
if a sample is drawn from clinic visit records over a 6 month period, it will give individuals
who visited the doctor numerous times a higher chance of selection than those who saw the
doctor only once.
Efficiency
In some cases, sampling frames include units that are not among those that the researcher
wants to sample. However this can be solved. For example, if a survey wants to interview
women of reproductive age, a household sample can be selected and within the household
all women of reproductive age can then be interviewed.
61
Handout 5.8
Handout 5.8: SAMPLING METHODS

Probability sampling
What is probability sampling?

Probability samples have the characteristic that each unit in the target population has a
known, non-zero probability of being included in the sample. Probability samples use
RANDOM selection mechanisms. The following are the most commonly used random
selection mechanisms:
a. Simple random sampling
Obtain a list of individuals from which to select a sample. If the sample is to be

representative of the population, the sampling frame must include all or nearly all
members of the population.
Use a random number table and select individuals or "sampling units". Each individual
has the same chance of being selected from the list. Members of the target population
are selected one at the time.
Once members have been selected, they are not eligible for a second chance and they
are taken out of the sampling frame for the selection of subsequent members to include
in the sample for this exercise.
b. Systematic sampling
Obtain a list of individuals or a map from which to select a sample.
Choose a random start, and then select every nth unit (i.e. every 8th or every 125th).
The random start is an essential component of the process. Without using a random
start some members have zero probability of selection and it can not be considered a
probability sample.
If you have a population of 10,000 and you want to select a sample of 250 individuals,
the sample interval n would be 40 (10,000/250). After choosing a random start every
40th unit will be selected.
c. Stratified sampling
A stratified random sample is one in which the population is divided into subgroups or
"strata", and a random sample is then selected from each subgroup.
Divide the members of your target population into groups which are different in ways
which are significant to the issue being studied. Each member is assigned to one and
only one group.
Select independent random samples for each of the groups, using simple random or
systematic sampling.
62
Handout 5.8
Examples of criteria for dividing the population into subgroups
groups by residence:
1. urban
2. semi-urban
3. rural
groups by marital status:
1. single
2. married
3. divorced or widowed
d. Cluster/multistage sampling
What is a cluster? A cluster is a naturally occurring unit such as a school, a village, a
hospital. Cluster sampling is usually used for large surveys.
For selecting a cluster sample:
Obtain or compile a list of clusters (i.e. list of schools).
Select randomly a certain number of clusters (i.e. 10 out of 100 schools).
Include all members of the selected cluster in the sample (i.e. all school teachers in the
10 schools).
Multistage sampling is an extension of cluster sampling. After selecting randomly clusters

the next step is the following:
Select a sample from the cluster using simple random or systematic sampling.
PROBABILITY SAMPLING METHODS

Sampling
Method
Simple
random
Description
Advantages
Each member of the study

population has an equal probability
of being selected.
Simple, self-
Disadvantages
Lists (sampling frames)
weighting.
Systematic

population is listed/mapped, a
random start is designated, the
members of the population are
selected at equal intervals
(sampling interval).
Simpler and faster.
may not be available or

incomplete.
Members of a subgroup of interest may
not be included in
appropriate proportions.
Samples may be very
dispersed.
Lists (sampling frames)
may not be available or
incomplete.
Must watch for recurring
patterns within the
sampling frame
(example: lists arranged
by age, sex).
Samples may be very
dispersed.
63
Handout 5.8
Stratified
Cluster
Multistage

population is assigned to a group
or stratum according to
meaningful characteristics, then a
simple random or syste-matic
sample is selected in each group.
Allows sub-
Must calculate sample
population analysis
Sample is more
likely to reflect the
population,
improves
efficiency.
size for each subgroup,

increasing sample size,
cost and time.
Weighting needed.

population is assigned to a cluster,
then clusters are selected
randomly and all members of the
selected cluster are included in the
sample.
Does not require
Increases sampling
Clusters are selected as in cluster

sampling, then sample members
are selected within each cluster by
simple random sampling.
Clustering may be done at more
than one stage.
Does not require
listing of full
population.
Less geographical
spread of sampling
units and therefore
saves time and
money.
error.
Clusters may not be
representative
Increases sampling
listing.
Less geographical
spread of sampling
units and therefore
saves time and
money.
error.
Example of Simple Random Sampling

1
11
21
31
41
51
61
71
81
91
12
22
32
42
52
62
72
82
92
13
23
33
43
53
63
73
83
93
14
24
34
44
54
64
74
84
94
15
25
35
45
55
65
75
85
95
16
26
36
46
56
66
76
86
96
17
27
37
47
57
67
77
87
97
18
28
38
48
58
68
78
88
98
19
29
39
49
59
69
79
89
99
10
20
30
40
50
60
70
80
90
100
Sample Frame: 100 units

Sample Size: 9 units (9%)
Each sample randomly selected
In this example units selected were: 15, 51, 95, 2, 55, 24, 25, 4, 92
64
Handout 5.8
Example of Systematic Sampling

1
11
21
31
41
51
61
71
81
91
12
22
32
42
52
62
72
82
92
13
23
33
43
53
63
73
83
93
14
24
34
44
54
64
74
84
94
15
25
35
45
55
65
75
85
95
16
26
36
46
56
66
76
86
96
17
27
37
47
57
67
77
87
97
18
28
38
48
58
68
78
88
98
19
29
39
49
59
69
79
89
99
10
20
30
40
50
60
70
80
90
100
Sample Frame: 100 units

Sample Size: 9 units (9%)
Formula:100/(9-1) = 12 (appx) Select every 12th unit, beginning with randomly selected
number between 1 and 12. In this example start number randomly selected was 4. Thus
units selected were 4, 16, 28, 40, 52, 64, 76, 88, 100
Example of Stratified Random Sampling

A1
A6
A11
A16
A21
B1
B6
B11
B16
B21
A2
A7
A12
A17
A22
B2
B7
B12
B17
B22
A3
A8
A13
A18
A23
B3
B8
B13
B18
B23
A4
A9
A14
A19
A24
B4
B9
B14
B19
B24
A5
A10
A15
A20
A25
B5
B10
B15
B20
B25
C1
C6
C11
C16
C21
D1
D6
D11
D16
D21
C2
C7
C12
C17
C22
D2
D7
D12
D17
D22
C3
C8
C13
C18
C23
D3
D8
D13
D18
D23
C4
C9
C14
C19
C24
D4
D9
D14
D19
D24
C5
C10
C15
C20
C25
D5
D10
D15
D20
D25
65
Handout 5.8
Sample Frame: 4 Strata with 25 units in each

Sample Size: 3 units per strata (12%)
Three units each randomly selected within each strata. In this example units selected were:
A23, A2, A25; B12, B20, B1; C13, C8, C18; D5, D25, D21
Example of Cluster Sampling

Cluster W
1
11
Cluster Y
21
31
41
51
61
71
81
91
12
22
32
42
52
62
72
82
92
13
23
33
43
53
63
73
83
93
14
24
34
44
54
64
74
84
94
15
25
35
45
55
65
75
85
95
16
26
36
46
56
66
76
86
96
17
27
37
47
57
67
77
87
97
18
28
38
48
58
68
78
88
98
19
29
39
49
59
69
79
89
99
10
20
30
40
50
60
70
80
90
##
Cluster X
Cluster Z
Sample Frame of clusters: list of 4

Sample Size: 9 units (9% of that cluster)
Example of Multistage Sampling

Cluster W
C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
Cluster Y
C11
C12
C13
C14
C15
C16
C17
C18
C19
C20
C21
C22
C23
C24
C25
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
D11
D12
D13
D14
D15
D16
D17
D18
D19
D20
Cluster X
D21
D22
D23
D24
D25
Cluster Z
One cluster randomly selected out of 4 clusters

Within Cluster "W" 2 sub-clusters randomly selected
Within these sub-clusters: Sample frame 50 (25 in each)
(Note: lists of units needed only for these two clusters)
Sample size: 6 each; total 12 units sampled (24% of units within these two sub-clusters)
66
Handout 5.8
Non-probability sampling
What is non-probability sampling?

Non-probability sampling includes several sampling approaches where subjective judgments
play a role in the selection of the sample. No randomized method is used. Non-probability
samples are not representative of the target population.
When are non-probability samples useful?
Exploratory research or pilot surveys

Surveys of special/specific populations (i.e. traditional medicine users)
Surveys of hard-to-identify groups (i.e. drug users)
NON-PROBABILITY SAMPLING METHODS

Method
Convenience
sampling
Most similar/ dissimilar
cases
Typical cases
Critical cases
Snowball sampling
Quota sampling
Description
Select cases based on their availability for the study.
Select cases that are judged to represent similar conditions or, alternatively,
very different conditions.
Select cases that are known beforehand to be useful and not to be extreme.
Select cases that are key or essential for overall acceptance or assessment.
Respondents identify additional members to be included in the sample.
Interviewers select sample that yields the same proportions as the
population proportions on easily identified variables.
67
Handout 5.9
Handout 5.9: SAMPLE SIZE

Factors to consider in the choice of a sample size:
precision or degree of accuracy (tolerable error),
confidence level (usually 95%, meaning that we can be 95% confident that the estimate
will fluctuate as much as tolerable error, that is, that 95 out of 100 samples will be
representative of the population);
variance or standard deviation of main variable(s) studied (if not available: use "worst
case scenario" p=0.5);
resources and time available;
skills of staff.
The size of the population from which a sample is selected

has virtually no impact on how well that sample is likely to
describe the population. For example, a sample of 150
people will describe a population of 15,000 or 15 million with
virtually the same degree of accuracy, assuming that all other
aspects of the sample design and sampling procedures were the
same.
Sample size for simple random sampling for 95% confidence level, (binominal
distributions)
n = (z/standard error) (p) (1-p)

n
= sample size
z
= standard score corresponding to a given confidence level
(z = 1.96 for the 95% confidence level)
p
= expected proportion with the characteristic
(1-p)
= expected proportion without the characteristic
Example: Simple random sampling size calculation for 95% confidence level
The CARE Cambodia team planning a health survey in Pursat does not have any
estimate on the contraceptive use in Pursat, which they want to study. They therefore
assume a 50% - 50% distribution or the "worst case scenario" (p=0.5) They want the
results to have a maximum of 7 standard error in the 95% confidence level.
n = (1.96/0.07) (0.5) (1-0.5)
n = 196
The sample size required is of 196 women of reproductive age.
TIP: If population is very small (i.e. 30 individuals), do not use a sample.
68
Handout 5.9
Sample size and standard errors for simple random sampling for the 95%
confidence level.
Sample
size
50/50
100
200
300
400
500
600
700
800
900
1,000
1,100
1,200
1,300
1,400
1,500
1,600
1,700
1,800
1,900
2,000
10.0
7.1
5.8
5.0
4.5
4.1
3.8
3.5
3.3
3.2
3.0
2.9
2.8
2.7
2.6
2.5
2.4
2.4
2.3
2.2
Binominal percentage distribution

60/40
70/30
80/20
9.8
6.9
5.7
4.9
4.4
4.0
3.7
3.5
3.3
3.1
3.0
2.8
2.7
2.6
2.5
2.4
2.4
2.3
2.2
2.2
9.2
6.5
5.3
4.6
4.1
3.7
3.5
3.3
3.1
3.0
2.8
2.6
2.5
2.4
2.4
2.3
2.2
2.2
2.1
2.0
8.0
5.7
4.6
4.0
3.6
3.3
3.0
2.8
2.7
2.5
2.4
2.3
2.2
2.1
2.1
2.0
1.9
1.9
1.8
1.8
90/10
6.0
4.2
3.5
3.0
2.7
2.4
2.3
2.1
2.0
1.9
1.8
1.7
1.7
1.6
1.5
1.5
1.4
1.4
1.3
1.3
Adjustments for design effects*

Sampling method
Adjustment range
Stratified sampling
Cluster sampling
Multistage sampling
* Based on world-wide survey experience
0.50 to 0.95
1.50 to 3.00
1.25 to 1.50
As can be observed in the above table, the design effects for cluster sampling vary from 1.5
to 3, which of course has an important implication for sample size, cost and time needed to
complete the fieldwork. For cluster sampling the exact design effect depends on the number,
size and homogeneity of clusters. As a general rule, it is better to have a larger number of
small sized clusters than a small number of large sized clusters.
Example: Adjustment for design effect
The CARE Cambodia team decided to use a stratified cluster sampling design and the
sample size of 196 has to be adjusted, following the advice of an expert, by 1.3.
n = 196 x 1.3
n = 255
69
Handout 5.9
Adjustments for non-response

Most common reasons for non-response:
inability to contact the respondent (i.e. respondent not at home at the time of the survey)
inability of respondent to complete the interview (i.e. respondent is ill, interviewer does
not speak respondent's language)
refusal of respondents to answer to answer to the survey questionnaire

There are certain techniques to minimize non-response, which will be discussed in other
sections. Non-response has implications for the sample size calculation. If, for example, a
response rate of 90% is expected, than the sample size will have to be adjusted by 1.10.
Example: Adjustment for non-response
The CARE Cambodia team adjusted the sample for non-response, expecting that
about 10% of the selected sample of the target population would not be found at the
time of the survey.
n = 255 x 1.10
n = 281
Sampling error
Standard error for 95% confidence level.
_________
SE = 2 x (p x (1-p) /n
SE
standard error of the mean
proportion with the characteristic
(1-p)
proportion without the characteristic
sample size
Example: Standard error calculation

The CARE Cambodia team found in their survey that the contraceptive prevalence
rate in Pursat was of 17%. The actual number of respondents to the survey
questionnaire was of 248.
______________
SE = 2 x 0.17 (1-0.17) /248
_________
SE = 2 x 0.1411/248
_________
SE = 2 x 0.0005689
SE = 4.7
The survey had initially calculated a sample size and error based on the "worst case
scenario". Now that the results are known, the error estimation for current
contraceptive use is of 4.7. This means that the CARE Cambodia team can be 95%
sure that the current contraceptive use in Pursat province lies between 12.3 and 21.7.
70
Handout 5.9
Non-sampling errors
Imprecision in the definition of the target and study population

Errors in survey design
Non-response
Measurement errors (i.e. poorly worded questions and response choices, inadequately
trained interviewers)
Errors in data processing
71
Handout 5.10
Handout 5.10: EXERCISE - TARGET POPULATION

AND SAMPLING FRAME
TARGET POPULATION
Define who is part of the target population for the survey
Research question:
Target population:
Inclusion criteria:
From where could you select a sample of the target population? Discuss which sampling
frames could be used for the your survey.
Exercise: Sample size and sampling error

Simple random sampling size calculation for 95% confidence level
The CARE Cambodia team planning a health survey in Pursat province does not have
any estimate on the contraceptive use in Pursat, which they want to study. They
therefore assume a 50% - 50% distribution or the "worst case scenario" (p=0.5) They
want the results to have a maximum of 6 standard error in the 95% confidence level.
Sample size:
n =
Adjustment for design effect

The CARE Cambodia team decided to use a stratified cluster sampling design. A
sampling expert advised to increase the simple random sampling sample size by 1.3.
Sample size adjusted for design effect:
n=
Adjustment for non-response

The CARE Cambodia team adjusted the sample for non-response, expecting that
about 10% of the selected sample of the target population would not be found at the
time of the survey.
Sample size adjusted for non-response:
n=
Standard error calculation

The CARE Cambodia team found in their survey that the contraceptive prevalence
rate in Pursat was of 17%. The actual number of respondents to the survey
questionnaire was of 362.
Standard error =
72
Handout 5.11
Hanout 5.11: QUESTIONNAIRE DESIGN

DESIGN OF QUESTIONNAIRES FOR STRUCTURED INTERVIEWS
Data collection instruments are the "thermometers" used to assess facts, knowledge,
attitudes, behaviors, perceptions, opinions.
It is advised to follow the these steps for designing instruments:
1. Review previous instruments used for similar studies, conduct focus group and/or key
informant interviews with representatives of the target population to get agreement on
purpose for and basic design of survey.
2. Brainstorm a first draft instrument
3. Get comments from others and reduce the number of questions to a minimum set
needed to measure indicators
4. Review the first draft and prepare second draft
5. Pre-test the instrument (if applicable) for :
sequence of questions/flow
comprehension of questions
appropriateness of questions
coding to fit responses
timing
skip-pattern
6. Review second draft based on the pre-test findings an prepare third draft
7. (If applicable) translate the instrument from the language it was designed to the
language for interviewing and back by two independent translators.
Designing questions to be good measures

Good questions are:
reliable (providing consistent measures in comparable situations);
valid
(answers correspond to what they are intended to measure).
73
Handout 5.11
TIPS FOR DESIGNING A QUESTIONNAIRE
Use simple language.
Avoid asking for more than one piece of information in a single question. (Bad question:
"Were you pleased with the cleanliness and the hours of service of the clinic?")
Avoid emotionally charged questions. (Bad question: "Do you think that condoms are
only for husbands who cheat their wives?")
Avoid superficial questions that provoke near-unanimous agreement. (Bad question: "Do
you think that the interest rate charged by the credit program should be lower?").
Do not assume knowledge or agreement. (Before asking "How many times did you go to
the health center?" ask "Did you ever go to the health center?).
Limit questions to respondents' own knowledge, attitudes and practice.
Pre-code responses.
Pre-test on a population similar to the target population!
74
Handout 5.12
Handout 5.12: TYPE OF QUESTIONS

1. Open and closed-ended questions
Closed-ended questions have set responses, which allow easier data processing. Openended questions have space to record exact quotes or paraphrase the respondent. They
give respondents the opportunity to state a position in their own words.
While structured interviews should used closed questions as much as possible, open
questions are very useful for exploratory or pre-testing work, when the full range of possible
answers is not known. Structured interviews can contain some open questions, however
this means that responses will have to be coded prior to data entry. It is important to then
consider the time that will be needed for analysis.
Example: Open-ended questions
Question
What did you like most about your visit to the health
center?
Codes
________________________________
________________________________
And what did you like least?
________________________________
________________________________
Example: Closed-ended question

Question
Did your child have diarrhea in the last two weeks?
(IF YES:)
How many times?
Codes
No diarrhea episode in past 2 weeks...........1
1 time.........................................................2
2 times.......................................................3
More than 2 times.......................................4
2. Answer lists
The answer list is a way of measuring strength of feeling among a standard range of
possible answers. Respondents choose which answer(s) they agree with.
Example: Answer list
No
1.
Question
Which of the following sicknesses do you consider the most
important problem for your children? (READ OUT)
Meningitis?
Malaria?
Diarrhea?
Respiratory infections?
Skin infections?
Measles?
Other? (specify: _______)
Codes
YES......................................NO
1...............................................2
1...............................................2
1...............................................2
1...............................................2
1...............................................2
1...............................................2
1...............................................2
75
Handout 5.12
Example: No answer list (spontaneous response)

No
1.
Question
Which sicknesses do you consider the most important
problem for your children?
Codes
Meningitis.................................1
Malaria.....................................2
Diarrhea...................................3
Respiratory infections...............4
Skin infections..........................5
Measles.................................6
Other (specify: _______)........7
The advantage of answer list is, that respondents often forget to mention in the short time
they have available in an interview to mention all important answers.
Answer lists have disadvantages, it can happen that respondents that do not understand
them answer at random and may not give their true answer if it is not listed. Even with an
"other" category, items not on the list are less likely to be mentioned. Lists are most
effective when respondents can read and look back over the items. When a respondent can
not read, so that the list has to be read aloud, they may not consider all the alternatives.
3. Measuring attitudes
It is harder to collect reliable data on attitudes than on more factual matters. It is very
important to treat such topics with caution. It is advisable to explore attitudes through indepth case studies, interviews and focus groups. Even well designed questionnaires can
only scratch the surface of what people really think, perceive, feel.
One option for collecting data on attitudes is the use of scales. Scaling means constructing
an ordered list of opinions or attitudes. Respondents select the statements they agree with:
Example: Measuring attitudes
No
1.
Question
In general, do you approve or disapprove of couples using a
method to avoid getting pregnant?
Codes
APPROVE..............................1
DISAPPROVE.........................2
DK..........................................8
It is important that interviewers always ask the questions in the same way, keeping the exact
wording. Changes could make the questions useless.
4. Measuring intentions
To find out about intentions rather than opinions, a time scale can be used.
Example: Measuring intentions
Question
Do you want to use a contraceptive method in the future? (IF YES:)
When do you plan to start using a contraceptive method?
Codes
Within one month.....................1
1 month to 5 months.................2
6 months to 1 year....................3
More than one year...................4
Undecided................................5
Does not plan to use.................6
Other: _______________......7
76
Handout 5.13
Handout 5.13: QUESTIONNAIRE LAYOUT, LENGTH

AND CODING
Introducing the survey
Before asking the first question, interviewers should give respondents some information.
Include in the introduction the name of the organization conducting the survey, the name of
the interviewer, the purpose of the survey and how results will be used. Avoid making false
promises or comments that will raise expectations.
Order of questions
Start with questions that are easy to answer.
This will help to establish trust and reassure the respondent. It is better to leave questions
about attitudes, beliefs and intentions to the later stages of the interview.
Work from the particular to the general.
For example, in a question about water use questions about the source of water supply
would come first, and questions like "Overall, are you satisfied with the place you collect
your water?" would come towards the end of the interview.
Questionnaire length
Even when the research topic demands a long interview, do not expect to give people more
that 30 to 45 minutes of their time. Long interviews increase the risk of error.
Advantages of short interviews
Less mistakes
Respondents give usually more meaningful and "true" answers and are more willing to
collaborate with the survey
The sample size can be increased
Analysis is simpler
Codes
Coding means to give a number (or other symbol) to each possible answer. Codes are
useful in order to summarize a large amount of information. They are essential, even for
small samples, when data is to be analyzed by computers.
Closed-ended questions are pre-coded, open-ended questions have to be coded once forms
are returned to the office. For open-ended questions clear coding instructions are needed.
Always include an "other" and "no response" and "don't know" (if it applies) category in order
to classify all possible responses
Skips and filters

Think about the following question. What is wrong with it?
Have you ever borrowed any cash? If yes, from whom did you borrow
money and how much the last time you did so?
This question aims at finding out too many things at the same time, it can be divided up into
3 clear and precise questions. Also, those who never borrowed money do not need to be
asked how much they borrowed, etc. This can be solved using a SKIP.
77
Handout 5.13
Example: Skips
No.
1.
Question
Have you ever borrowed cash?
2.
Lets talk about the last time you borrowed cash.

From whom did you borrow it?
3.
What amount did you borrow at that time?

(Record in local currency or in US$)
Coding
Yes.................................1
No...................................2
Bank................................1
Money lender...................2
Relative or friend.............3
Other: ____________.....4
Skip
4
Local currency __ __ __ __
US$ __ __ __ __
4.
5.
Have you ever borrowed anything else besides

cash?
What did you borrow?
(Check all that apply)
Yes..................................1
No...................................2
Food................................1
Farm inputs.....................2
Farm tools.......................3
Other:___________.........4
10
78
Handout 5.14
Handout 5.14: EXAMPLE - QUESTIONNAIRE WITH

SEVERAL TYPES OF QUESTIONS
Birth Spacing Campaign Questionnaire
(Introduction: Explain that you work for CARE and you are conducting a survey about health
in several provinces. Ask to speak to a married man or women, aged 15 to 44. If none in the
household or not available DO NOT CONDUCT THE INTERVIEW).
A. Identification of questionnaire
No.
1.
Question
Questionnaire number
3.
Rural or urban
4.
Record gender
Coding
__ __ __
Urban..............................1
Rural...............................2
Female............................1
Male................................2
B. Interview
No.
5.
How old are you?
Question
Coding
__ __
6.
Are you married?
7.
Have you ever attended school? (If yes:) What is

the highest level of school you attended?
8.
Does your household have.............
Yes.................................1
No..................................2
Never attended................1
Primary incomplete..........2
Primary complete............3
Secondary incomplete.....4
Secondary complete+......5
Yes...............................No
1....................................2
1....................................2
Daily................................1
Weekly............................2
Less than weekly.............3
Never..............................4
Daily................................1
Weekly............................2
Less than weekly.............3
Never..............................4
__ __
.....a radio?
........a TV?
9.
How often do you listen to the radio?
10.
How often do you watch TV?
11.
Do you have children? (If yes:) How many

children do you have?
How old is your youngest child?
12.
Skip
END IF
<15 or
>44
END
__ __
79
Handout 5.14
No.
14.
15.
16.
17.
18.
19.
Question
Have you ever heard about any method to delay
or avoid a pregnancy? (If yes:) Have you heard
about the........
...........daily pill?
......monthly pill?
..........injection?
.................IUD?
..........condom?
............female sterilization?
..............male sterilization?
....periodic abstinence/rhythm?
..................withdrawal?
......other: __________?
Do you think that couples who want no more
children or who want to wait longer time until
having a baby should use a birth spacing
method or not?
Have you or your wife/husband ever used a birth
spacing method? Which one(s)?
Are you or your wife/husband currently using any

birth spacing method? Which one?
When did you start the use of the method you

are using now?
From where/whom did you first hear about birth
spacing?
Coding
Yes...............................No
1....................................2
1....................................2
1....................................2
1....................................2
1....................................2
1....................................2
1....................................2
1....................................2
1....................................2
1....................................2
Skip
If
respondent
does not
know
any
method:
END
Yes..................................1
No...................................2
Not sure...........................3
Daily pill...........................1
Monthly pill......................2
Injection...........................3
IUD..................................4
Condom...........................5
Female sterilization.........6
Male sterilization..............7
Periodic abst./rhythm.......8
Withdrawal......................9
Other: __________........10
Never used....................11
Daily pill..........................1
Monthly pill......................2
Injection...........................3
IUD..................................4
Condom...........................5
Female sterilization.........6
Male sterilization..............7
Periodic abst./rhythm.......8
Withdrawal......................9
Other: __________........10
Not using.......................11
___ /19__ __
19
19
Radio or TV.....................1
Friends/relatives..............2
Health staff......................3
Other: ____________.....4
Thank you very much for your time and cooperation!
80
Handout 5.15
Handout 5.15: TWELVE GUIDELINES FOR

DEVELOPING
GOOD QUESTIONS
1.
AVOID LOADED OR EMOTIONALLY CHARGED WORDS. (e.g. Do you agree with

the fanatical opinion of government regarding the marketing of agricultural surplus?)
2.
AVOID SUPERFICIAL, PAT QUESTIONS THAT ENCOURAGE STEREOTYPICAL,

UNIFORM RESPONSES. (e.g. Do you like the extension services?)
3.
AVOID DOUBLE-BARRELED QUESTIONS. (e.g. Do you attend meetings of the

cooperative and take loans from it?)
4.
AVOID QUESTIONS THAT PRESUME KNOWLEDGE, EXPERIENCES OR PAST

PRACTICES. (e.g. Are you continuing to use pesticides?)
5.
AVOID ESOTERIC OR TECHNICAL WORDS. (e.g. What should you do when your
child has an acute upper respiratory infection?)
6.
AVOID NEGATIVELY WORDED QUESTIONS. (e.g. Are you not now leaving your
water storage containers uncovered?)
7.
AVOID CHARACTERIZING THE ISSUE IN A WAY AS TO PREJUDICE THE

RESPONSE. (e.g. Which do you prefer, the improved extension service or the old
extension service?)
8.
AVOID SWEEPING QUESTIONS. (e.g. Are you in favor of modern farming

practices?)
9.
AVOID QUESTIONS THAT ARE AMBIGUOUS. (e.g. When did you first become
interested in planning your family?)
10.
AVOID UNNECESSARILY COMPLEX QUESTIONS. (e.g. Would you or another

member of your family be interested in attending a meeting of potential entrepreneurs
to learn about bookkeeping methods that can be used for small businesses and how
to gain access to credit through cooperative methods?)
11.
AVOID QUESTIONS THAT MAY REQUIRE GUESSING. (e.g. How much money did
you spend on contraceptives last year?)
12.
AVOID QUESTIONS THAT DO NOT ADEQUATELY DEFINE THE EXTENT OF

DETAIL OR THE DEGREE OF THOROUGHNESS OF THE DESIRED ANSWER.
(e.g. What are some of the things about the project that you like?)
81
Handout 5.16
Handout 5.16: DATA COLLECTION - TRAINING

AND FIELDWORK
Interviewers and supervisors play a major role in surveys. Their skills are essential for
obtaining GOOD QUALITY DATA and therefore it is of key importance to select and train
interviewers and supervisors well. Different interviewer skills and experience are needed for
quantitative surveys than for in-depth interviews, focus groups or other qualitative methods.
FIELD RESEARCH FOR QUANTITATIVE STUDIES

THE INTERVIEWER
1. Role of the interviewer
The interviewer occupies the central position in surveys, as she/he is the one who collects
information from the respondents. Therefore, the success of the survey depends on the
quality of each interviewer's work. The responsibilities of an interviewer include:
to locate and enlist cooperation of selected respondents;
to ask questions, record answers, and probe incomplete answers to ensure that
responses meet the question objectives;
to check completed interviews to be sure that all questions were asked and the
responses legibly recorded.
2. Training and supervision of interviewers

Training of interviewers generally consists of a combination of "classroom" training and
practical experience. Training is a continuous process. Observation and supervision
throughout the fieldwork are part of the training and data collection process. Supervisors
play a very important role in continuing interviewer's training and ensuring the quality of the
survey.
3. Conducting an interview
Successful interviewing is an art and should not be treated as a mechanical process. Each
interview is a new source of information. It is very important to make the interview interesting
and pleasant. The art of interviewing develops with practice, but there are certain basic
principles which are followed by every interviewer in order to be successful:
Building rapport with the respondent
Make a good first impression
Always have a positive approach

Do not use words such as "Are you too busy?", "Do you have a few minutes?" or "Would
you mind answering some questions?". Such questions invite refusal before you start.
Rather, tell the respondent: "I would like to ask you a few questions" or "I would like to
talk to you for a few moments".
Stress confidentiality of responses when necessary

If the respondent is hesitant about responding to the interview or asks what the
information will be used for, explain that the information you collect will remain
confidential, no individual names will be used. Also, you should NEVER mention other
82
Handout 5.16
interviews or show completed questionnaires to other interviewers or supervisors in

front of a respondent or any other person.
Answer any question from the respondent frankly

Before agreeing to be interviewed, the respondent may ask you some questions about
the survey or how she/he was selected to be interviewed. Be direct and pleasant with
your answer.
Conduct interviews privately

The presence of a third person during the interview can keep the respondent from giving
frank, honest answers. It is, therefore, very important that individual interviews be
conducted privately and that all questions are answered by the respondent her/himself.
Tips in conducting an interview
Be neutral throughout the interview

Most people are polite and will tend to give answers that they think you want to hear. It is
therefore very important that you remain absolutely neutral as you ask the questions.
Never, either by expression on your face or by the tone of your voice, allow the
respondent to think that she has given the "right" or "wrong" answer to a question. Never
appear to approve or disapprove of any of the respondent s replies.
Never suggest answers to the respondent

If a respondent's answer is not relevant to a question, do not suggest her an answer by
saying something like : "I suppose you mean that .... is that right?" Rather, you should
probe in such way that the respondent her/himself comes up with the relevant answer.
Do not change the wording or sequence of questions

The wording of the questions and their sequence in the questionnaire must be
maintained. If the respondent has misunderstood the question, you should repeat the
question slowly and clearly. Provide only the minimum information required to get an
appropriate response.
Be patient with hesitant respondents

There will be situations where the respondent simply says "I don't know", gives an
irrelevant answer, acts very bored or detached, contradicts something she/he has
already said, or refuses to answer the question. In this cases you must try to re-interest
her/him in the conversation. Spend a few moments talking about things unrelated to the
interview (for example, about the town or village, the weather, etc.).
Do not form expectations

You must not form expectations as to the ability and knowledge of the respondent. Do
not assume, for example, that women from rural areas or those who are less educated or
illiterate do not know about family planning.
Do not hurry the interview

Ask the questions slowly to ensure the respondents understands what they are being
asked and give them time to answer questions.
Do not make any promises

Make no promises like "CARE is planning to construct wells in your village" or similar.
83
Handout 5.16
4. Fieldwork
Field procedures and problems
Field work should proceed according to a time schedule, and it is important that each
interviewer understands and follows field procedures and how to handle various problems
that may be experienced during the field work.
In most quantitative surveys, clear instructions have to be made for the following procedures
and problems:
a. Locating sample households
b. Problems in locating a household
No one at home at time of the call
Assigned household inaccessible
The house is all closed up and neighbors say that no one lives there
The dwelling is non-residential
The dwelling is a building with several apartments

c. Identifying and interviewing eligible respondents
No eligible respondents
Eligible respondent not available
Respondent refuses to be interviewed
Interview not completed
Checking completed questionnaires

It is responsibility of the interviewer to review each questionnaire when the interview is
finished. Interviewers have to make sure that every appropriate question was asked, that all
answers are clear and reasonable, and that the skip instructions were followed correctly.
NEVER RECOPY QUESTIONNAIRES, this increases the chance of mistakes.
THE SUPERVISOR
1. The role of the supervisor
The main responsibility of the field supervisor is TO REDUCE NON-RESPONSE. Some
other important responsibilities are (not all apply to all surveys):
to direct and supervise the work of team members;

to check if the selection of households has been done according to the method of
selection developed for the survey;
to observe several interviews and discuss with the interviewer's about their work after
each survey day;
to advise interviewers if problems arise;
to contact local authorities just before the survey takes place in the areas and to express
thankfulness before leaving;
to do a simple mapping of the villages;
to make well a publicity campaign, explain about purposes and meanings of the survey
to the people
to keep well all the questionnaires, properties and supplies provided
2. Reducing non-response
One of the most important duties of the supervisor is to try to minimize non-response. There
are mainly three types of non-responses:
84
Handout 5.16
Type 1: The interviewer is unable to locate the selected household.

Type 2: The interviewer is unable to locate the eligible respondents.
Type 3: The respondent refuses to be interviewed.
3. Monitoring interviewer performance
Systematic spot-checking of household selection and/or composition
Observing interviewers
Evaluating interviewer performance. This may include returning to a small sample of

households and asking a few key questions to verify that the questionnaire form was
correctly recorded.
The Art of Questioning: An Exercise to Increase

Awareness of How Questions Are Asked
A CARE employee has been requested to travel to a village and interview a group of farmers
to learn about their rice production practices. Specifically, CARE wants to know more about
the use of inputs (in particular improved seed and fertilizer) and the amount and use of
marketable surplus which leads to increased household income. Listen to the questions the
interviewer asks the farmers. After each question the group will discuss the characteristics
of the question and note if there are faults with the way the question was formulated. If there
are faults then corrections will be made by the group.
Hello, Im from CARE and I would like to ask you some questions about your rice
production...
Fault:
Corrections:
Im interested in details of your rice production activities. First, do you use inputs?
Fault:
Corrections:
Okay, could you describe in general terms the cycle of rice production in this area?
Fault:
Correction:
Are you continuing to use pesticides?
Fault:
Correction:
85
Handout 5.16
I dont think use of pesticides is environmentally responsible, would you agree?

Fault:
Corrections:
Do you attend meetings of the farmers association and take loans from it to purchase
inputs?
Fault:
Correction:
What has been the extent of your use of di-dedra tetracycline (DDT) as a pesticide for corn
stem borer?
Fault:
Corrections:
Are you not now using improved rice seed?
Fault:
Correction:
Do you think that if you use more pesticides you production of rice will increase?
Fault:
Correction:
What should be the duties of an agricultural extension worker?
Fault:
Has the CARE project been successful?
Fault:
Correction:
What would be the effect on you if the cost of pesticides were to rise?
Fault:
86
Handout 5.17
Handout 5.17: QUANTITATIVE DATA ANALYSIS

1. Ways of looking at data
Time series record developments over time, for example, monthly attendance to a
health center.
Cross-sectional data are snapshots which capture a situation at a moment in time,

such as the percentage of farmers that use IPM (integrated pest management at a given
moment of time.
2. Nominal, ordinal and numerical scales and data

Nominal or categorical scales and data
Nominal or categorical data identify classifications. The answer is the "name " of the
category into which the data fit. The numbers are arbitrary and have no inherent value.
Example: Nominal scales that produce nominal data
1. Circle respondents gender
2. Circle respondent's water source
Female..................................1
Male......................................2
Well......................................1
Piped water...........................2
River or pond........................3
Other:______________.........4
Ordinal scales and data

Categories can be sorted into a meaningful order, but differences between ranks are not
necessarily equal.
Example of ordinal scale that produces ordinal data
Have you ever attended school? IF YES:
What is the highest level of school you
attended?
NEVER ATTENDED...................1
PRIMARY INCOMPLETE...........2
PRIMARY COMPLETE..............3
SECONDARY...........................4
UNIVERSITY.............................5
Numerical scales and data

When differences between numbers have a meaning on a numerical scale, they are called
numerical. Age, for example, is a numerical variable. Means and other statistical measures
can be used to analyze and summarize this kind of data.
Example of numerical scales that produce numerical data
How many cows do you have?
__ __
And how many pigs?
__ __
87
Handout 5.18
Handout 5.18: TOOLS FOR QUANTITATIVE DATA

ANALYSIS
a. Percentages and proportions
Percentages and proportions are widely used and known, they are one of the most important
tools for quantitative data analysis. Proportions are expressed relative to 1, percentages in
relation to 100. Put in another way, a percentage is a proportion multiplied by 100.
TIPS: Include the number that you are basing the percentage on.
Don't calculate percentages for less than 30 cases.
b. Tables
One way tables or frequency distribution table (using one variable)
Example: Percent distribution of women who do not want to use a contraceptive method in
the future by reason.
Reason
Wants children
Lack of knowledge
Fear side effects
Leave it to nature
Husband opposed
Too expensive
Percent
48.4
16.8
14.9
12.7
5.7
1.5
Total percent
100.0
Two-way table or cross-tabulation (using two variables)

Two-way tables or cross-tabulations are the basic tool to show relationship between two
variables.
Example Percent distribution of currently married women who do not want to use a birth
spacing method in the future by reason, according to age.
Age
Reason
Wants children
Lack of knowledge
Husband opposed
Too expensive
Fear side effects
Leave it to nature
Husband absent
Difficult get pregnant
Other
Total percent
<30
66.1
4.4
4.3
0.8
9.2
9.5
0.0
0.8
4.8
30+
37.7
6.9
5.4
1.4
18.4
14.5
1.2
0.0
14.4
Total
48.4
6.0
5.0
1.2
14.9
12.7
0.7
0.3
10.8
100.0
100.0
100.0
88
Handout 5.18
Independent and dependent variables

A variable is a characteristic that is measurable, it can be numerical or non-numerical.
When data form surveys, attendance records or other is used to analyze behaviors,
attitudes, opinions, perceptions and beliefs, some variables will be used as "explanatory" or
independent variables, which will help to explain the result of a dependent variable. In
the example below, the perception of change in the economic situation in the household is
the dependent variable, which is analyzed by 4 independent or explanatory variables: age,
gender, education and economic status. Note that the percentages have to be created in
the direction of the independent variable. This is the most basic tool to compare and explain
differences between subgroups of the target population.
Example Percent distribution of respondents by perception of change of the economic
situation of the household in the year preceding the survey by background characteristics.
Background
Improved
No change
Worsen
Total
characteristics
percent
Age
15 to 34
15.6
50.9
33.4
100.0
35 to 49
14.3
40.0
45.6
100.0
49+
10.2
44.4
45.4
100.0
Gender
Female
Male
9.7
18.1
39.8
51.0
50.4
30.8
100.0
100.0
Education
No schooling
Primary incomplete
Primary complete+
3.7
15.9
20.3
42.6
47.5
42.9
53.7
36.7
36.8
100.0
100.0
100.0
Economic status
Rich or well off
Poor or very poor
30.2
6.9
58.8
39.8
11.0
53.3
100.0
100.0
Total
13.5
44.8
41.7
100.0
c. Descriptive measures
Average or mean
The average or mean is used for numerical variables. It is obtained by adding all scores or
responses together and dividing by the number of observations.
_
Mean ( X ) = X/n
= Greek letter sigma, meaning add or sum
X = each individual observation
n = total number of observations
Example: Calculating the mean
Name
Age
45
Ahmed
23
Peter
34
Ann
42
Maria
38
Alex
Mean age = (45+23+34+42+38)/5 = 36.4
89
Handout 5.18
Disadvantage of the mean: it is easily influenced by extreme values.

Example: Mean influenced by extreme values
Name
Income per month
75
Ahmed
40
Peter
85
Ann
60
Maria
780
Alex
Mean income = (75+40+85+60+780)/5 = 208
Median
The median is the middle observation, it says that half of the observations are smaller and
half are larger than the median. It is not influenced by extreme values. For example, the
median of the following numbers is 7, because half of the scores are below and half are
above.
3,6,6,7,9,13,17
Example: Finding the median
Name
Income per month
75
Ahmed
40
Peter
85
Ann
60
Maria
780
Alex
Income arranged in order: 40, 60, 75, 85, 780
Median = 75
Mean = 208
Example Mean and median amount (in US$) borrowed among respondents who borrowed
in the year preceding the survey by gender .
Gender
Mean US$
Median US$
Female
Male
108.6
186.1
40
60
Total
144.0
45
Measures of spread
Mean and median give idea of center, but no idea of how dispersed or compact the
distribution is. For example, the following groups of data have the same mean and median,
however different range and spread.
8, 9, 10, 11, 12 (Range= 8 to 12)
0, 5, 10, 15, 20 (Range= 0 to 20)
Measure of spread commonly used: standard deviation
90
Handout 5.19
Handout 5.19: GRAPHING DATA

Bar, pie and scatter graphs are the most commonly used graphs for presenting quantitative
date. For two numerical variables, it is very useful to use scatter graphs in order to visualize
their relationship.
Example: Positive relationship
Average monthy income in US$ by
years of schooling in Region X
Average income
250
200
150
100
50
0
0
10
12
14
16
Years of schooling
Example: Negative relationship

Mean number of children ever born among women
aged 40 to 49 by number of years of schooling in
Region X
Mean number of children
8
7
6
5
4
3
2
1
0
0
10
Years of schooling
91
Handout 5.19
Example: No relationship
Monthly interest rate paid
Monthly interest rate paid on last loan

by years of schooling
5
4
3
2
1
0
0
10
Years of schooling
92
Handout 6.1
Handout 6.1: QUALITATIVE DATA

1. Characteristics and Use
Quantitative methods produce numerical data and qualitative methods result in information
which can best be described in words. Examples of qualitative data include descriptions of
situations, events, people, interactions and observed behaviors; direct quotations from
people, and excerpts or entire passages from documents, correspondence, records and
case studies. Qualitative methods focus on the signs and symbols that decode the reality
seen by the target population. Qualitative methods are iterative, there is an ongoing
opportunity to revise interview protocols, guides, and observation record forms as a study
progresses and new facts are brought to life.
One of the most important sources of information for M&E of CARE projects is qualitative
interviews. Projects are conducted in complex sociological, ecological, cultural and political
settings. Systems are noisy and not as amenable to quantitative procedures.
Comparison of Qualitative and Quantitative Data

QUALITATIVE
Answers the question Why?
A process of discovery
Helps to understand what is going on
Interpretive
QUANTITATIVE
Answers the question How many?
Looks for evidence
Measures
Descriptive
Themes in Qualitative Methods

Naturalistic Inquiry
- evaluator does not attempt to manipulate the program or its participants for
purposes of the evaluation; want things to take their natural course as
opposed to experimentaton
- particularly useful for studying variations in program implementation (i.e. what happens in a program often varies over time as participants and
conditions change)
Inductive Analysis
-Qualitative methods are particularly oriented toward exploration, discovery,
and inductive logic
- begin with specific observations and build toward general patterns
- qualitative analysis is guided not so much by hypothesis but by questions,
issues and a search for patterns
Direct Contact With the Project (Going Into the Field)
- having direct and personal contact with people in the project in their own
environments
Use of Case Studies
- depth and detail in qualitative methods typically derive from a small number
of case studies
- case studies are useful when the evaluation aims to capture individual
differences or unique variations from one program to another.
- The more a project aims at individualized outcomes, the greater the
appropriateness of qualitative case methods
93
Handout 6.2
Handout 6.2: WHEN TO USE QUALITATIVE

METHODS
Process Evaluations
aimed at understanding the internal dynamics of projects and their activities

focus on the following kinds of questions
- what are the factors that come together to make this project what it is?
- what are the strengths and weaknesses of the project?
- how are beneficiaries brought into the project and how do they
participate?
- what is the nature of CARE-Partner-Beneficiary interaction?
the need to generate an accurate and detailed description of program
operations particulrly lends itself to the use of qualitative methods
Evaluating Individualized (Target Group) Outcomes
matching project goods and services to the needs of individuals or target

groups
(e.g. - many educational projects place emphasis on the unique and individual
needs of a child; often income differences make target groups different and
thus require different needs)
reproductive behavior projects are a good example of the type of programs
CARE implements where one may want to evaluate outcomes based on
individual or group response
whenever one seeks behavioral change there will be differences among
individuals and among various target groups
Case Studies
evaluating individualized outcomes (described above) is one area where case

studies are used, but there are others
we use case studies to evaluate peculiar occurences - unusual successes or
failures, dropouts to projects, etc.
documenting pilot projects, especially the testing of interventions
detailed sampling schemes exist for selecting critical and relevant cases for
study
Implementation (Formative) Evaluation

tests whether a project is progressing according to design
conducted for the purpose of improving programs

when outcomes (goals) are evaluated without knowledge of implementation,
the results do not provide decision makers with information about what
actually produced the outcomes, a black box approach to evaluation
an important way of evaluating implementation is to gather detailed,
descriptive information about the project in order to answer questions such
as:
- whats working and whats not
- what do beneficiaries in the project experience and what are their
perceptions
- what services are being provided to target groups
- what do staff do to produce goods and services
- what is it like for beneficiaries to be part of the project
aimed at improving the quality of project activities and outcomes, not just
levels of attainment, and judgements about quality often require qualitative
data
94
Handout 6.2
Adding Depth, Detail and Meaning to Quantitative Analyses
produces new insights or at least forces people to think about old insights in a
new way
adds description to why things are the way they are
Summative Evaluation
key questions which answer specific implementation questions
aimed at producing lessons learned
95
Handout 6.3
Handout 6.3: THE CASE STUDY

Case studies are detailed examinations of a relatively few persons or items. In most case
studies, subjects are not chosen by a formal sampling process. Often subjects are selfselected or selected on the basis of relevant features.
In the type of work CARE does, case studies typically focus on an organization, community
or target group.
Some common types of case studies are:
1.
community studies: the unit of enumeration is the community and the number
of units studied may be one or more;
2.
trace studies: the study is based on recorded cases obtained from clinics,
social workers; health workers, village monitors, etc. The study involves
tracing the individuals concerned and relating their history and background to
the phenomenon that led to the making of the record;
3.
pilot surveys: the testing of the methodology, questionnaire, and interview

techniques to be used in a larger survey on a few respondents chosen to
represent a wide range of types;
4.
detailed activity studies: focus on a particular behavior or activity of groups or

organizations, for example the study of labor inputs into cropping cycles or
the study of personal hygiene habits in households;
5.
supplementary surveys: for example, a village-level survey of nutritional levels

may be supplemented by detailed observation and questionning of a small
sub-sample.
Strengths of Using Case Studies

provides in-depth, detailed analysis
Guidelines:
Interviews must be conducted by those skilled both in the subject matter and in the art of indepth interviewing.
96
Handout 6.4
Handout 6.4: QUALITATIVE EVALUATION

CHECKLIST
Qualitative methods are not valid for every evaluation situation or for the study of all
evaluation questions. Many of CAREs indicators are purely quantitative in nature, for
example. The following is a checklist of questions which can be used to help decide if
qualitative methods are an appropriate evaluation strategy. If the answer to any of the
questions is yes, then the collection of some qualitative data is likely to be appropriate.
1.
Does the project emphasize individualized outcomes, - that is, are different
participants (or target groups) expected to be affected in qualitatively different ways?
And is there a need or desire to describe and evaluate these individualized
outcomes?
2.
Are decisions makers interested in finding and understanding the internal dynamics
of the project - project strengths, weaknesses, and the overall process that is (was)
followed? This gets to what we call lessons learned.
3.
Is detailed, in-depth information needed certain target group cases or project sites,
for example, particularly successful cases, unusual failures, or critically important
cases for programmatic, political, financial or other reasons?
4.
Is there interest in focusing on the diversity among individual beneficiaries or target

groups?
5.
Is information needed about the details of project implementation, including items

such as services, organization, management, partnering, staff activities, etc.
6.
Is there interest in formative evaluation (i.e. - finding out how to improve the project)?
7.
Is there a need for information about program quality - descriptive information about
the quality of the project activities and outputs, not just levels, amounts, or quantities
of program activity and outcomes?
8.
Are decision makers or donors interested in having evaluators conduct project site
visits so that evaluators can be the surrogate eyes and ears for decision makers and
donors who may be too busy to make such visits themselves?
9.
Is the obtrusiveness of evaluation a concern? Will the collection of qualitative data

generate less negative reaction among project participants? This applies often, for
example, for evaluating changes in income.
10.
Are the goals of the project vague, general, and nonspecific, indicating the possible
advantage of using certain goal-free approaches?
11.
Is there a possibility that the project may be effecting beneficiaries in unanticipated

ways (both positive and negative) so that a discovery method is needed?
12.
Is the evaluation exploratory? Is the project at a pre-evaluation phase, where goals

and project content are not fully realized?
13.
Is there a need to add depth, detail, and meaning to statistical findings or survey
generalizations?
14.
Has the collection of quantitative evaluation data become so routine that no one pays
much attention to the results anymore, suggesting a possible need to break the old
routine and introduce something new?
97
Handout 6.5
Handout 6.5: NATURE OF QUALITATIVE DATA

General Nature
In some senses, all data are qualitative; they refer to things about people, objects, and
situations. We have a raw experience which is converted into words (His face is
flushed....He is angry.) or into numbers (Six voted yes, for voted no....the thermometer
reads 74 degrees.).
But qualitative data is in the form of words - that is, language in the form of extended text.
(Qualitative data also can appear as still or moving images). The words are based on
observation, interviews, or documents (or as others have put it, watching, asking, or
examining). These data collection actvities typically are carried out in close proximity to a
local setting for a sustained period of time.
This results in three basic kinds of data collection: (1) direct observation; (2) in-depth, openended interviews; and (3) summarizing or extracting from written documents, including such
sources as open-ended written items of questionnaires, personal diaries, project evaluations,
and program records, to name only a few. The data from open-ended interviews consist of
direct quotations from people about their experiences, opinions, feelings, and knowledge.
The data from abservations consist of detailed descriptions of program activities,
participants behaviors, staff actions, and the full range of human intercations that can be
part of program experiences. Document analysis yields excerpts, quotations, or entire
passages from records, correspondence, official reports, and open-ended surveys.
98
Handout 6.6
Handout 6.6: QUALITATIVE DATA COLLECTION

INTERVIEWS
I. QUALITATIVE INTERVIEWS
One of the most important sources of information for needs assessment is qualitative
interviews. They are the most useful tool in understanding the felt needs, perspectives,
attitudes and beliefs of the target population. Qualitative interviews can also be used to
generate ideas and themes which can then be tested on a wider population using a
structured questionnaire.
IA. Types of Qualitative Interviews

1. Informal, Conversational Interviews
The interviewer has complete freedom to explore a broad range of subjects with the
respondent. Issues as they emerge can be further explored. The interviewer generally takes
very few notes.
More than just a casual conversation. The interviewer has a purpose in mind and must
control the conversation to serve that purpose.
Limitations:
1. time consuming
2. the conversation can become unfocused and wander in circles
3. information gathered from one respondent may not be comparable to that
from another
4. highly susceptible to "interviewer effect"
Strengths:
1. allows a wide range of issues to emerge

2. more may be revealed than in a more formal setting
3. especially useful in diagnostic surveys
2. Topic-focused Interviews (Topical Interviews)

Topic-focused interviews are conducted using an interview guide which is also called a
topical outline. It lists the main topics and sub-topics to be explored. The interviewer,
however, uses her/his judgment on how to use the guide in a way to permit smooth flow of
discussion. Although the guide is used, it does not prohibit exploration of other topics.
Advantages: 1. Since interviewers cover the same topics the information is more
comparable.
2. Subjects stay within the context of interest and saves time.
Example of an Outline for a Topic-focused Interview
A project is promoting use of improved millet that requires the application of a new fertilizer.
Monitoring data from the project suggests that use of the new seed and fertilizer is not
increasing as rapidly as expected. The project staff decides to interview farmers and ask
them why they are or are not using the new technology. The interview guide would list topics
and sub-topics similar to the following:
1.
2.
The farmer's understanding of the composition of the technical package.

(What is it all about? What does it involve? What does one have to do in
order to adopt it?)
The farmers perceptions regarding the advantages and costs of the technical
package. (What may be gained by using the new millet variety? Is it
economical (profitable)? What are the risks? How much investment does it
99
Handout 6.6
3.
4.
5.
6.
require? How easy or difficult is it to obtain the new millet seed and
fertilizer?)
The farmer's opinion about the relevancy of the new technology. (Are there
constraints on land, credit, or labor that prevent adoption?)
The farmer's views regarding the availability of services. (Is there somebody
to show them how to use the new technology? Do they have confidence in
these services?)
The farmer's assessment of the risks involved. (Is the new variety of millet
seen as less reliable than the traditional variety? Are there consumption,
preparation, or taste problems?)
The farmer's assessment of the potential rewards. (Is he interested in
increasing production? What would he do with any surplus production?)
Background information. (Availability of labor in the HH, cropping patterns,
etc.)
3. Semi-structured, Open-ended Interviews

Semi-structures, open-ended interviews are the most structured form of qualitative
interviews. They use an open-ended questionnaire which lists the specific questions to be
asked. They are similar to interviews conducted for structured surveys but differ from them
in three ways.
1.
Semi-structured interviews use open-ended questions, respondents
encouraged to express themselves fully rather than respond to predetermined
options.
2.
The sequence (order) of the questions is not predetermined and the
interviewer has control over which questions are asked and in what order.
3.
The interviewer can ask additional questions to explore topics further.
Strengths:
1.
2.
3.
4.
The information obtained specifically answers certain questions that project

managers wish to address.
The information from different interviewers is comparable enough to generate
simple frequencies, although the main emphasis will still be placed on an
in-depth understanding of the respondents.
Compared to other qualitative interviews, success is less dependent on the
interviewer's communication skills and experience.
Can be conducted faster than the other types of qualitative interviews.
5. General Limitations of Qualitative Interviews

1. They do not generate quantitative data that can be summarized to make general
statements about the population. For example, it is difficult to say that 60% of the farmers
are satisfied with the existing extension services.
2.
It is difficult to use qualitative interviews based on probability samples. This
means that the selection of respondents is often biased. One common
mistake is that interviewers use respondents of higher social or economic
status.
3.
Findings can be easily based on biases of the interviewers. Common for
interviewers to hear more information when it conforms to their own thinking
or opinions.
6. Guidelines for Qualitative Interviews

Initial Contact:
very important; establish the basis for a comfortable

interview; appearance, style, manner of introduction,
clothing, gender can all make a difference
100
Handout 6.6
Sequencing of Questions: polite to begin with general conversation; can volunteer

information about yourself, the project; begin with
simple questions
Wording of Questions:
questions must be posed in an understandable way;

language and expressions obviously important; phrase
questions in a way to encourage detailed responses
(generally avoid yes or no questions); do not pose two
or more questions simultaneously
Example: Form of Questions Leading to Varying Responses

Leading to
Leading to
yes or no response
detailed response
Have you heard of the extension services
What do you know about the extension
operating here?
services available in this area?
Do you think that if you use _______
your production of millet will increase?
What is your opinion about the risks and

benefits of using _____ in producing millet?
It is sometimes said that those marketing

seeds only deal with the large farmers?
Is this true?
What is your view regarding how seeds

Are marketed in this area?
Has the CARE project been successful?
What effects has the CARE project had on

you and your family? On your neighbors?
Do you have difficulty in obtaining improved

millet seed?
Please describe how you go about obtaining

improved millet seed?
Would you grow more millet if the government What would be the effect on you if the
raised the price of millet?
government raised the price of millet?
Role Playing:
role playing can be very effective; lets respondent assume the

role of somebody else (e.g. - asking a farmer What should be
the duties of an agricultural extension worker?, or Suppose
somebody wanted access to family planning services, where
would they go?); interviewer can also assume a role (What
advise would you give me to improve the marketing of millet
seed?); dont ask people to assume roles for which they cant
readily identify
Vignettes:
respondents are given imaginary people in imaginary

situations and asked to decide what those people should do.
Example: a) Suppose a poor community needed to obtain a teacher for its school.
Do you think that members of the community should have to collectively pay to hire
the teacher? b) Suppose one or more households could not contribute. Do you think
their children should be allowed to attend school? c) Do you think everyone in the
community should contribute the same amount, irrespective of their income?
Irrespective of whether or not they have children who would be attending school?
Probing:
essential skill for effective interviewing; often necessary to

encourage respondent to be more specific; use encouraging
101
Handout 6.6
words or nods of the head; think of follow-on questions that

begin with How, Why, Where, etc.
Controlling
Conversations:
some respondents get off the point and start giving you
opinions on everything; try non-verbal communication (e.g. quit nodding your head) or interrupt politely
Neutral Attitude:
avoid giving the impression of having strong views on the

subject; sometimes you can state both sides of an issue to
demonstrate your neutrality (e.g. - loan pay backs)
Recording the
Interview:
use of a tape recorder is optimal for recording but may inhibit

some respondents; have second person take good notes; ask
before recording and reassure that information is confidential;
write up interview as soon as possible
II. GROUP INTERVIEWS

There are two general types of group interviews: community interviews and focus-group
interviews. Community interviews generally use an interview guide and have more than 15
participants. Usually only a small number of questions are asked and each participant is not
expected to answer all the questions individually. An interdisciplinary team is often used to
ask the questions. These types of interviews are often exploratory. A focus-group interview
is conducted with a small group, usually 6-10 participants. One of the real features is that
respondents discuss ideas, issues, insights and experiences among themselves. If properly
run, a focus-group interview can provide very rich detail on a subject. Participants are
selected based on established criteria. There are three main reasons why group interviews
may be preferable to individual interviews:
1.
2.
3.
Group interviews allow you to gather information in a fast and cheap (economical)
manner. One can interview 8-10 people in one or two hours, whereas to interview
8-10 people individually may take two days.
Group participation often reduces individual inhibitions and may reveal information
not otherwise revealed. Of course, the opposite may also occur depending on
subject matter.
Information from a group interview can represent a consensus (or average) and be
more representative of the general population, especially if structured formats are
used and the group has to agree on a common answer. In group interviews,
respondents are reluctant to give wild or inaccurate answers.
Example: Group Interview for Generating Quantitative Data

In a research project in Costa Rica, a structured questionnaire was used in combination with
a topical outline for interviews with 860 communities for generating community-level
statistics. Below are some examples of the questions used.
7. What is the daily wage of an agricultural worker in this area?_____
7.1 For how many hours?_____
7.2
Does this include:
Yes
No
Food:_______ _____
Housing:____
_____
Land:_______ _____
9. What are the tree main crops grown: __________ __________ __________
9.1 Which is the most important? __________
The second most important? __________ The third most important? __________
9.2 How much is sold commercially?
Almost
More than
Less than
Little or
102
Handout 6.6
Crop
all
half
half
none
1
2
3
14. Where do people generally go to buy the things they cannot buy here locally?
Community Market __________
District Market __________
Provincial Market __________
Other
__________
14.1 How do they go there? __________
14.2 How long does it take? __________
The qualitative data generated through community interviews can be aggregated in two
ways. First, individuals can be treated as cases. For example, suppose ten group meetings
are attended by a total of 200 farmers. If 80 farmers say that they grew improved millet
seed, it can be reported that 40% of farmers interviewed grew improved millet seed.
Second, each group can be treated as a case. For example, in the above example you may
be able to report that in four out of ten villages the majority of farmers are using improved
millet seed.
Be careful, however, how you evaluate aggregated findings generated by community
interviews. They have some validity only if certain conditions are met, for example:.
1.
Participants must be representative of the target population.
2.
Group processes must not inhibit free expression of feelings or preferences.
3.
The questions must not be politically or culturally sensitive.
III. FOCUS-GROUP INTERVIEWS

A focus-group interview is a type of group interview, but there are some important
differences. Focus group interviews are difficult to conduct and require a skilled moderator.
They are much more than simply a conversational interview with a group. Other differences
include: optimal group size; selection of respondents; their expected contribution; and the
nature of spontaneous reaction.
A focus group is commonly used when you want to explore ideas, reactions, or
recommendations from a collection of respondents whom you know have information and
opinions about the focus group topic. You use purposeful sampling to select members of the
focus group based on a set of criteria that makes the group similar or homogeneous(based
on, for example, age, sex, interests, income, employment). The criteria depend on the
discussion topic.
You need a skilled moderator to guide a discussion group to ensure that:
the discussion does not stray from important topics
issues are explored in detail
some members dont talk too much and others too little
points are made clear
You also need a recorder to take notes about what is being said and to observe respondents
behaviors during the session. Tape recorders are recommended for recording information
that can later be transcribed.
Focus groups provide qualitative data about how the group members feel about a given
topic. They are not evaluative, therefore results cannot be generalized to the general
population and are not statistically valid. It is important to transcribe and code your data
as soon as possible after the interview.
103
Handout 6.6
Strengths:
useful for: identifying prolems and their causes

developing messages for communication or social marketing strategies
understanding felt needs
determining areas to focus further data collection activities on
information can be gathered efficiently since you know group members will
have knowledge and opinions of topic
exploring behaviors, beliefs, and attitudes
Weaknesses:
cannot generalize findings because members are not selected based on

mathematical probability. Nor should you try to quantify responses or
generalize them to the population. For example, it is inaccurate to say that
because 25% of the focus group reported using a family planning method,
you can conclude that 25% of the population uses a family planning method.
takes a skilled moderator
Guidelines:
Interview Guide:
only type recommended is a short checklist of topics and is less

structured than other guides; one of primary objectives is to explore
each topic in greater depth than other interviews; the topics that
generate the real interest and excitement may not have been
anticipated.
In a focus group on use of a specific family planning method, one of
the participants may make a casual remark that more people would
use a method if the information were made more available. This may
lead to a discussion of social marketing for family planning or other
styles of communication and information dissemination.
Size & Composition optimal generally ranges from 6-10; in smaller groups people often
of the Group:
feel more pressure; larger groups leave too little time for individual
expression; range of socio-economic backgrounds and social status
should be considered; better if the members do not know each other
well
Selection of
Group:
consult key informants who are knowledgeable about local conditions;

select subset from a long list; try to include diverse participants
Seating, etc.:
Use round table or circle when convenient; avoid large crowds by not
holding in open; should not exceed 2-3 hours
Controlling Flow:
dont let a few dominate discussions; minimize group pressure

(important to let those with minority viewpoint speak); set ground rules
early; encourage group to consider alternatives if consensus is too
quick
104
Handout 6.7
Handout 6.7: OBSERVATION

Observation is the systematic method of watching or observing peoples behavior or other
phenomena and recording the results. It not only includes sight, but also smell, feelings,
touch, and other sensory perceptions. The data collector is the observer who carefully
watches or senses what others are doing. Some examples include:
mothers feeding their children

farmers applying fertilizer
consumers making purchases at the market
grain storage facilities
women interacting while drawing water from a standpipe
You can collect both qualitative and quantitative data using observation. When you use a
structured observation instrument, the observer records events as being present or absent,
having occurred or not occurred, or how many times they occurred. These result in
quantitative data.
In using an unstructured instrument, the observer takes extensive notes about what he or
she sees or senses. Later these notes are organized and the event is described. This
usually results in qualitative data. There are two general types of observation: participant
and non-participant.
Participant Observation is relatively rare in development assistance projects. The observer
lives with and takes part in the daily activities of the subjects. The participant observer may
help take care of the children, cook, work in the fields, or do some other activity with the
people being observed. Ethnographic studies often employ this technique.
Those being observed generally do not know the extent to which the observer is studying
their behavior. That is important because you want them to go about things as they would if
the observer were not there. Also, the observer needs to record the observation out of the
sight of those being observed. When people know they are being watched, they tend to act
differently. This can give you unreliable data.
What steps do you take and what do you say to get the subjects to let the observer live with
them? You can explain that you want to learn more about the community.
Non-participant Observation does not require the observer to live with the subjects. The
observer often uses some sort of observation guide to make and record observations.
The major advantage of observation is that the data are generally very reliable, because
observation is unobtrusive. It gives a clearer picture of the real situation. It is harder to get
a real picture with interviewing techniques because people may not answer questions
truthfully.
There are, however, a couple of disadvantages you should consider before deciding to use
observation: cost, and the objectivity of the observer.
Some observation methods, especially participant observation, require that observers spend
long time periods collecting information. This can cost a lot of money in terms of time and
effort. Calculate and compare the cost of conducting observation to the value of the
information you expect to get. Does the information obtained justify the cost?
105
Handout 6.7
Another drawback is that the reliability of the data is dependent on the objectivity of the
observer.
Field Visits. Project managers often use an informal type of observation during field visits.
They keep a mental check list of events they wish to observe. After the visit they recall the
observations and record them or discuss them with staff.
Another type of observation technique is photography. You can take before and after photos
of some phenomenon and examine them to determine whether or not change took place.
OBSERVATION SCHEDULE FOR PARTICIPANT OBSERVATION

(USE ONLY ONE FORM PER DAY)
Name of Observer________________________
Date_________________
Name of head of household_________________________________________

Number of people living in household_________________________________
1. How many people did you observe today eating breakfast?_______
2. How many people did you observe today washing their hands with soap, citron, or ash
before they ate breakfast?_______
3. How many people did you observe today eating dinner?_______
4. How many people did you observe today washing their hands with soap, citron or ash
before they ate dinner?_______
5. How many people did you observe today using their latrine?_______
6. How many people did you observe today washing their hands with soap, citron or ash
after using the latrine?_______
7. How many trips to get water did you observe today?_______
8. In how many of these trips did you observe someone in your household using a clean
water vessel?_______
106
Handout 6.8
Handout 6.8: CHOOSING A SAMPLE:

THE LOGIC OF PURPOSEFUL SAMPLING
Purposeful sampling for qualitative data is different than probableistic sampling common for
quantitative data which relies heavily on statistics. The power in quantitatve data depends
on selecting a truly random and representative sample which permits confident
generalization from the sample to the larger population (usually the target group, a village, or
a class of people such as wealthy or poor, farmers or non-farmers, etc.). The power in
purposeful sampling for qualitative data is in selecting information-rich cases for in-depth
study. Information-rich cases are those from whom one can learn a great deal about issues
of central importance to the evaluation.
For example, if the purpose of an evaluation is to discover how a project is effecting groups
of lower socio-economic status, one can learn more by focusing on understanding the
needs, interests and incentives of a small number of carefully selected poor families than
gathering information from a large, random and statistically significant population.
Strategies for Selecting Purposeful Samples
1. Extreme or Deviant Case Sampling
2. Maximum Variation Sampling
3. Homogeneous Samples
4. Typical Case Sampling
5. Critical Case Sampling
6. Snowball or Chain Sampling
7. Criterion Sampling
8. Confirmatory or Disconfirming Cases
9. Convenience Sampling
10. Random Purposeful and Stratified Purposeful
107
Handout 6.9
Handout 6.9: QUALITATIVE DATA ANALYSIS AND

INTERPRETATION
Analysis is the process of bringing order to the data, organizing what is there into patterns,
categories, and basic descriptive units. Analysis begins even while data is being collected.
Evaluator should become particularly sensitive to looking for alternative viewpoints and
contrary patterns.
Interpretation involved attaching meaning and significance to the analysis, explaining
descriptive patterns, and looking for relationships and linkages among descriptive units.
The most serious and central difficulty in the use of qualitative data is that methods of
analysis are not well formulated. The analyst faced with a stack of qualitative data has few
guidelines for protection against self-delusion, let alone the presentation of unreliable or
invalid conclusions to scientific or policy-making audiences.
The evaluator has two primary sources to draw from in organizing the analysis:
1.
the evaluation framework (questions) that were developed during project
design (this could include indicators and key questions) and
2.
analytic insights and interpretations that emerge during data collection.
Steps in qualitative data analysis
1. Focus the Analysis
- focus comes from the project design: indicators and key questions
- negotiate early around the purpose of an evaluation
2.
Organizing Qualitative Data for Analysis

- data are usually voluminous, and sitting down to make sense out of pages of
data can be overwhelming
- first thing to do is make sure its all there!
- transcribe the interviews
- make a copy of the data (will need to cut and paste later)
a. Qualitative Description
- pure description of the project and experiences of people in the project, often
written in narrative form
- purpose of the description is to let the reader know what happened in the
project and what participants experienced
- evaluator tries to look for the typical project experience based on his/her
simple interpretation
b. Case Analysis
- cases can be people, target groups, critical events, communities, project
sites, etc., but the case is the basic unit of analysis
- first step is to pull together the data relevant for each case and to write a
descriptive, holistic case study, sometimes the entire analysis is only one
case study
- at the individual level, case data can include clinical records, background
information, interviews, observations
- at the project level, case data can include project documents, project reports,
interviews with beneficiaries and staff, observations of the project, and project
histories.
108
Handout 6.9
analysis starts with writing a case record which pulls together and organizes
the data; information is edited, redundancies are pulled out, and parts are
fitted together
c. Content Analysis
- content analysis involves identifying coherent and important examples,
themes and patterns in the data
- look for qoutations or descriptions that closely match and that represent a
theme, issue, problem or concept
- begin by reading through field notes, records , interviews, case studies, etc.
while writing comments in the margins indicating what can be done with the
different parts of the data; this is the beginning of organizing data into topics
(like constructing the index for a book or labels for a file system
- where more than one person is working on the data it is important to have
each do their own content analysis and then compare
d. Inductive Analysis
- inductive analysis means that themes, patterns or categories emerge from the
data rather than being decided prior to data collection and analysis
two kinds of patterns can emerge from data:
- analyst can use the categories developed by people in the project (indigenous
typologies)
Example: hamburgers - hamburgers can vary a great deal; there are many ways
to prepare them or to add to them, and yet they are still called hamburgers.
However, when a piece of cheese is added to the meat, we call it a
cheeseburger. The task for the evaluator is to find out what separates
hamburger from cheeseburger. How people construe their world from the way
they talk about it.
Example: evaluting a project which aims at keeping girls in school longer. In
observations and interviews it became important to understand the way teachers
categorized students. With regard to problems of dropouts, teachers labeled
young girls as self-motivated or victims. The low-motivated students were ones
who would not have continued school regardless of HH circumstances because
they werent motivated and did not value education. The victims were students
who sincerely wanted to learn but whose parents could not afford to keep them in
school or didnt value education. Important in the project to understand
differences between the two groups.
-
analyst can develop own terms based on his/her interpretation of the data
(analyst-constructed typologies)
The primary purpose of typologies is to describe and classify.

(See page 150, How to Use Qualitative Methods in Evaluation)
e. Logical Analysis
One of the most important sources of information for M&E of CARE projects is qualitative
interviews. Projects are conducted in complex sociological, ecological, cultural and political
settings. Systems are noisy and not as amenable to quantitative procedures.
109
Handout 6.9
Analysis - challenge is to separate description and explanation. Coding: Lofland provided a

classification of social phenomena which can usefully be employed as the basis for a
coding scheme:
1.
Acts. Action in a situation that is temprally brief, consuming only a few
seconds, minutes or hours.
2.
Activities. Action in a setting of more major duration - days, weeks,
months - consuming significant elements of persons involvement.
3.
Meanings. The verbal production of participants that define and direct
action.
4.
Participation. Persons holistic involvement in, or adaption to, a
situation or setting under study.
5.
Relationships. Interrelationships among several persons considered
simultaneously.
6.
Settings. The entire setting under study conceived as the unit of
analysis.
110
Handout 6.10
Handout 6.10: FIELD RESEARCH FOR

QUALITATIVE STUDIES
Significant skills and experience required in:
Listening techniques
Interviewing
Ability to handle group dynamics
Ability to reflect and summarize
Open attitude
Excellent interpersonal skills
Ability to adapt on the spot
Analysis ability
111
Handout 6.11
Handout 6.11: CODING AND ANALYSIS

Coding is often a difficult step in the analysis of qualitative data and is time consuming. A
common way to code qualitative data is to let the indicators determine categories, then
assign codes to strings of quotes or descriptions that belong to a category. You will want to
code close-ended or structured responses when the instrument is made. Code open-ended
responses, especially when there is a lot of qualitative data, after data reduction.
Transcribe qualitative data from field notes or cassettes to summary tables or computer,
then code and categorize data. Matrices are popular summary tables for qualitative data. In
the matrix below, quotes would be organized by who said it (project manager, farmers, and
field staff) and what they said about pesticides, training and technical assistance.
SAMPLE SUMMARY MATRIX

Project Manager
Farmers
Field Staff
Pesticides
Pesticides from
local markets are
not good choices
Made me sick
I got a rash
Farmers need
more training in
safety measures
Training
Trainers too
directive
Too short
Too much written

material
Technical
Assistance
Field staff need

more training
Very helpful
Increased yield
Need motorcycles
to go to the field
To analyze qualitative data, you want to get percentages, rates, or frequency counts and still
not lose the quotes or descriptions. Here is one way to do it:
Develop categories based on indicators or key question.
Assign qualitative data such as quotes, descriptions or summaries to the

appropriate category.
Calculate values by counting, for example, how many people responded a

certain way or behaved a certain way.
Use actual quotes or descriptions to support the values.

Example:
Key Question: Why dont some families of the project grow trees on their land?
Indicators:
Percentage of farm families who say they dont have enough money to
purchase cuttings.
Percentage of farm families who say they dont want to take the risk of
growing trees.
Percentage of farm families who say they are afraid the government
will take their trees once they are productive.
Percentage of farm families who give another reason.
Then organize these categories into a matrix and list select quotes under each category.
112
Handout 6.11
ANALYZING QUALITATIVE DATA

Investment
Risk
Government
Other
I have to pay
school fees
What if the
trees die?
I dont trust the

government
Too busy
My daughters
getting married
next year
Trees die if they

dont have water
My brothers trees
were cut by soldiers
Nobody to do
the work
Count the number of respondents with quotes in each of the categories. Lets say we
counted quotes and came up with the following numbers.
Investment
Risk
Government
Other
=
=
=
=
55 respondents
22 respondents
35 respondents
9 respondents
TOTAL =
121 respondents
To calculate indicator values, figure the percentages: divide number of respondents in each
category by the total respondents (121) and multiply by 100:
45.5% of respondents say they dont have enough money to invest in trees.
18.2% of respondents say they dont want to take the risk of growing trees.
28.9% of respondents say they dont trust the government enough.
7.4% of respondents give other reasons.
To answer the key question, we can say that our information suggests that the primary
reason that families dont grow trees is that they dont have enough money to invest. This
information is important and useful. However, the quotes from families add a whole other
dimension of information that can help us decide what to do about promoting tree production
on farmers fields.
PRESENTATION OF QUALITATIVE DATA

One of the simplest ways of presenting qualitative data is using a matrix similar to many
tables used to present quantitative data. There are two basic types: simple and compound.
SIMPLE MATRIX
Key Question: Why dont mothers bring their children to be immunized?
Health center
is too far
(30%)
Nurses treat
mothers badly
(55%)
Mothers did
not know about
services (5%)
Other
(10%)
Its a three
hour walk
She acted like I

was stupid
I have never
heard of this
thing
Clinic is
dirty
I cant afford
They made me
Needles
113
Handout 6.11
transport
wait six hours
cause AIDS
I cant take a
full day
She just grabbed

the baby
The children
have to work
A compound matrix has two sets of categories. One is listed horizontally and the other is
listed vertically. Place data in the cells that go with each set of categories.
COMPOUND MATRIX
Key Question: Why dont mothers bring their children to be immunized?
Mothers
Province
Reasons
Health center
too far away
(35%)
Nurses treat
mothers badly
(30%)
Mothers did
not know
(15%)
Other
(20%)
North (22%)
South (18%)
(Quotes go here)
East (27%)
West (33%)
Interpretation is an excellent opportunity to bring together key stakeholders to examine data

and make judgements about goals, outputs, and activities or key questions.
Qualitative data are often richer and more helpful than quantitative data. For example,
specific quotes from respondents about why they dont use credit is more meaningful than
the number of respondents who say they dont use credit because it is too expensive.
The advantage of qualitative data becomes a disadvantage during data analysis. Quotes
from 100 different people about why they dont use credit are much more difficult to make
sense of then 100 responses to a set of closed questions about using credit. Analysis of
qualitative data is a lot easier if you focus data and determine how to organize data ahead of
time. For example, if you want to know why some families dont want to grow trees, you
develop several indicators for this question. Indicators reflect your guesses about why
families dont grow trees or they come from a first-order analysis of your data.
114
Handout 6.12
Handout 6.12: QUALITATIVE DATA ANALYSIS

EXERCISE
Below are quotes from field studies which were conducted during an evaluation to answer a
key question about community problems. Code the data according to themes or categories
that you find. Next, construct a compound matrix to show results, including percentages for
each category you developed.
Hint: First code the data in the margins according to major themes that you discover. I
something is only mentioned once it probably does not represent a theme and you may want
to put it into a category called Other.
Key Question: What do members of the community feel is their key problem?
VILLAGE A
We need education for all of our children. How are they supposed to get jobs if they have
no education?
Our farm land is getting worse. Twenty years ago we had twice the rice production we have
today
My two daughters want to go to school but the nearest school is over twenty kilometers, and
besides, I need them to help me here in the home
When people get sick they just die. This has got to stop.
If I only had some savings I could buy more land and other things for my family
People round here dont have enough money to buy food and things
We need building materials to strengthen our houses
The health clinic is too far away.
Many children are sick since the flooding last year. We need our children to be healthy.
I sure wish the extension services would show us how to grow more and better food.
I want to send all of my children to school so they can go to Phnom Penh and get good
jobs.
If I only had some savings I could buy more land and other things for my family
Prices are going up all the time but our income stays the same. We have no money to
purchase oil and other things for cooking.
My rice crop has failed two years in a row. I cant understand why.
We barely grow enough to feed our family, much less have surplus to sell for cash.
You see that people are sick and our children are skinny. What can we do about this?
The nurses at the clinic do not know anything about diseases.
115
Handout 6.12
Our only teacher left last year to get a better job in the city and we need a new teacher.
There is so much fighting around here nobody feels safe.
We need money for food and better shelter.
Many women die while giving birth and I think this is a serious problem.
We need higher incomes so that we can provide for our families better.
People here need to find employment in the dry season to earn money.
We need a new clinic with free medicines.
My children have diarrhea almost every week.
I want to have my leg fixed. It was broken two years ago and has hurt since.
The land is not as fertile as it once was and we need to use more fertilizer.
Our biggest problem is lack of good health services.
My son graduated last year from school but now wants to go to university. I dont have
enough money to send him to university in Phnom Penh so I dont think he will go there.
Maybe its better if he stays here and helps with the farm.
People die because they cant get to a doctor. Its a real shame.
Nobodys health here is very good.
We need more money to purchase food and stay in good health.
You see here in my house that I have nothing. I have no money to buy my kids shoes.
What can I do?
VILLAGE B
If I only had some money I could buy more land and other things for my family
We need water for our crops. Last year the irrigation canal was almost dry and many
farmers could not grow good crops.
We could use a new school. Our school is very crowded and children have to sit on the
floor where its dirty.
If I only had some savings I could buy more land and send my children to college.
You see that people are sick and our children are not healthy. We need better village
health care workers.
The nurses at the clinic do not know anything about diseases and they treat us badly.
We need loans to start up new business. Many people have no or very little income.
Money. Money. Money. The world runs on money you know!.
116
Handout 6.12
Better health facilities. No question about it.

Many people will tell you the problem is health, but I say its lack of money. If you have
money you can use good doctors.
We need to be able to grow more food. This requires fertilizer, good seed, and plenty of
water for our crops.
I want a job for each of my kids. They dont want to farm and if they cant make money here
I fear they will move to the capital.
We all are very poor in this community. Happy, but poor.
I had 100 kg of rice to sell last week but received a very low price. How do I survive on so
little money? Everything now is very expensive.
I think our biggest problem is that everyone in this community is too lazy.
We need more and better doctors.
Most people around here would probably tell you that health is a major problem, but I think
its income. Most people are very poor.
We need affordable medicine. People used to know local medicines from plants, but now
nobody remembers how to cure themselves and must go to the clinic.
We need to grow more rice so everybody has a full belly and we can sell the rest in the
markets at a good price.
My only son died least month because we couldnt find a doctor. So you tell me what the
problem is and what you can do to help us.
I heard an American once say Money cannot buy happiness. I would like to find out for
myself if that is true because I have no money.
117
Handout 6.13
Handout 6.13: HEALTH PROJECT CASE STUDY

by Dan O'Brien
The following is a case study from an actual child survival project managed by CAREBolivia. The study begins with an overview of the project which includes project areas,
priority population, major activities, and staffing. The second part of the study contains a list
of 8 major observations made by an outside consultant during a visit in the first year of the
project.
Pretend your group is managing this project and just received the list of observations from
the consultant. Make a list of 10 questions for which your team would like to have answers
to address the observations with the intention of improving the project.
BACKGROUND OF THE PROJECT

The Rural Bolivian Health Education Project (RUBHEP) is a three year $822,000 project
partially funded by the Agency for International Development (AID) Child Survival Grants
Program and managed by CARE-Bolivia. The aim of RUBHEP is to lower childhood
mortality rates in rural Bolivia, the highest in the western hemisphere, by focusing on
immuno-preventable disease, diarrhea and resulting dehydration, and malnutrition. Project
activities are underway in 123 communities located in the departments of Chuquisaca,
Potosi, and Tarija. The total estimated size of the priority population is 10,230. The principal
activities include immunizations, oral rehydration therapy (ORT), growth monitoring and
nutrition education, and hygiene education. Immunizations follow the Bolivian Ministry of
Health (MOH) recommended schedule of BCG, DPT, Polio, and Measles. ORT includes
both packets donated by UNICEF and home mix. In addition, project staff work with
communities to provide iodized salt in an effort to prevent iodine deficiency diseases (IDD)
prevalent in the highlands.
The activities depend primarily on project staff consisting of 18 health educators, three
supervisors, and the project manager. The health educators are mostly nurses and health
technicians while the supervisors are physicians. The role of the health educators is twofold.
First, to provide immunization and growth monitoring services and health education to
communities. And second, to work with communities to identify, select, and train community
volunteers to assume responsibility for growth monitoring and health education activities.
The supervisors provide technical and administrative support to the educators and
volunteers.
The causes of childhood mortality are many and their relationships complex. To maximize
the impact the health projects have on lowering childhood mortality, the important causes
must be addressed. For this reason, RUBHEP is attempting to combine the typical child
survival interventions with those related to safe water and sanitation. An important criterion
for selecting communities to participate in the project is that they have a drinking water
system which in many instances was built with assistance from CARE-Bolivia. By doing so,
project staff anticipate RUBHEP will be more effective in the long run.
CONSULTANT'S OBSERVATIONS
1. I discovered while working with similar vaccination campaigns that merely boiling needles
and syringes is often not sufficient to kill bacteria. Sterilizing syringes by the boiling method
resulted in abscesses which damaged the credibility of the program since mothers were
afraid to bring their children back for subsequent immunizations. Once steam sterilization
was initiated with the use of pressure cookers, there were no more reports of abscesses.
118
Handout 6.13
During my field visit, I observed that the health educators were using the boiling method to
sterilize syringes and needles.
2. The health educators are spending 8 days every 3 months immunizing children under 4
years. This is approximately 15% of their time in the field. Their salary is about $250/month.
On the other hand, MOH personnel receive $20/month and do the same work during the
national vaccination campaigns. CARE/Bolivia could easily increase the cost-benefit ratio of
cost to each child immunized by having each health educator train and supervise 5 - 10
community volunteers to help in the national vaccination campaigns. Futhermore, the
Bolivian MOH has recently been trying to recruit and train volunteers for the vaccination
campaigns. CARE/Bolivia, by training volunteers to immunize, would not only improve its
cost-benefit ratio, but would be cooperating with the Bolivian MOH's new policy.
3. I observed health educators and spoke with several more who have been weighing
children and having the mothers plot the weights on the growth curve without any sort of
explanation as to why the children were or were not growing well. I believe there exists
enormous educational potential in the growth monitoring component.
4. In the majority of the communities that I visited, no latrines existed. Therefore, the health
educators defecate in the fields as do the rest of the people in the community. Nevertheless,
the health educators are involved in teaching basic sanitation education. I believe that until
materials are available for the construction of latrines in the community, the health educators
could begin setting an example of good sanitation hygiene practices by building and using a
latrine in the community where they live.
5. Two of the regional water and sanitation engineers told me that the criteria a community
must meet in order to be considered for a water system are: a) a sufficient supply of water;
b) acceptable water quality; c) topographical conditions conducive to building the water
system and; d) a community water committee. The engineers went on to tell us that at times
they help organize the village water committee if the other criteria are met. I believe this is
precisely the problem. Maybe it is too tempting for water and sanitation engineers to quickly
organize a village water committee in order to take advantage of an ample supply of good
water and ideal topographical conditions so that a water system can be built. Once the
water system has been built, the hastily organized village water committee is unable to
maintain the system (see the WASH evaluation for CARE/Bolivia water systems). A
community that is able to raise enough money or in-kind payment (fee for service as an
indicator) would demonstrate a certain level of commitment and organization that might be
better suited to maintain the water system once the CARE water and sanitation engineers
have pulled out of the community.
6. I was able to sit in on 3 health education classes where health educators taught women
in mothers' clubs. In each case, the health educator gave a 15 - 20 minute presentation on
a child survival subject, asked if anyone had questions, then ended the session. There was
no participation from the group. Based on research and my own experiences, I believe that
adults learn best when they are actively involved in the learning process. I feel that a
serious problem with the health education component of RUBHEP is the lack of skills the
health educators have in actively involving the group in the learning process.
7. Studies have demonstrated that the participation of community volunteers in projects
decreases proportionately to the amount of time the volunteers participate without
compensation for their services. I have witnessed this phenomenon in the Dominican
Republic, Nepal, and Guatemala. During my field visit I spoke with 6 health educators who
have already lost 50% of their original community volunteers. The most common reason
given by the community volunteers for leaving the project was no compensation for their
work. The 50% attrition has occurred despite the supervision and support of the health
119
Handout 6.13
educators. What will happen in 4 years when the project funds can no longer be paid to
supervise and support the community volunteers?
8. CARE/Bolivia received 822,000 dollars for RUBHEP of which approximately 300,000
dollars is budgeted for salaries and personnel support. The rationale for the huge
investment in personnel is that in 3 years the health educators will have been able to change
enough of the target population's health related behaviors to make a substantial difference in
childhood mortality and that there will be trained community health volunteers to carry on the
project. Similar projects have shown that the time it takes to sustain a change in health
related behavior often takes longer than 3 years. Then the change in behavior needs to be
continuously reinforced. It appears that the projects will rely on the uncompensated
community volunteers to sustain any change in behavior. For the reasons mentioned
previously, I would not count so heavily on the community volunteers as long as they are not
being compensated for their work.
120
Handout 6.14
Handout 6.14: DATA ANALYSIS

Source:
Casley, Dennis J. and Krishna Kumar. 1988. The Collection, Analysis, and
Use of Monitoring and Evaluation Data. Johns Hopkins University Press.
1. Exploratory Data Analysis

Exploratory analysis is useful in and of itself in that it gives those with little or no formal
training in statistics an opportunity to understand the data, which in turn improves their ability
to make decisions based on the M&E system. For monitoring purposes, simple exploratory
data analysis is usually all that is needed. Evaluations often require more sophisticated
techniques, but even then exploratory analysis is a first step. Exploratory data analysis
looks for simple structures and patterns in the data, and helps to determine if the data has
errors.
1.1 Graphing
Graphs are used to detect possible patterns, not to present the results more
simply.
Usually plotting the main variables against time or against each other will
provide useful information.
Can assess noise in the data.
Useful for detecting outliers.
1.2 Ordering of Data
Usually the ordering of cases is more or less random before data processing.
If data is computerized, it can be easily ordered and measures of central
tendency can be easily calculated.
Can take the form of a grouped or non-grouped frequency distribution.
Look for whether the data is unimodal or bimodal, and whether it has long
tails.
Can facilitate assessing central tendencies (mean, median, mode).
1.3 Dispersion
Simplest measurement is the range, but usually of limited use.

Most common is the standard deviation of the distribution. This is the square
root of the variance.
The variance is useful in informing one of the noise in the system being
studied.
1.4 Linear Relations
Useful to explore the linear relationshiop between two variables, in the form of
y = a + bx. This is one of the basic statistical calculations known as leastsquares regression.
You can use the quartile method to easily do this by hand. Order the data
according to ascending values of x, calculate the means of the variables for
the lower quartile and upper quartile, respectively. Plot these points, and
draw a regression line through them. This is very quick to do and gives a
reliable estimate of the true linear relationship.
121
Handout 7.1
Handout 7.1: PARTICIPATORY MONITORING AND

EVALUATION
What?
"Traditional" monitoring and evaluation is initiated from the top and carried out for the
people. Participatory (PME) is carried out and belongs to the project or program participants
(providers, partners, beneficiaries, and other interested parties). PME requires the
involvement of project participants in:
deciding what to monitor and evaluate
selecting indicators for M&E
selecting data collection methods
processing data
analyzing data
using PME information for their own purpose.
Advantages and disadvantages of PME

A participatory approach to M&E empowers people to manage their resources, therefore
increasing control over their lives as well. PME helps achieve results while increasing
people's understanding and ability to solve their own problems.
Advantages of PME:
examines relevant issues by involving key players in the evaluation design
promotes participants' learning about the program and its performance
Improves participants M&E skills
Enhances teamwork and builds a shared commitment to act on evaluation

recommendations.
Increases likelihood that M&E information will be used to improve performance.
Disadvantages of participatory evaluations
Be viewed as less objective because staff and beneficiaries that might have their own
interests participate in evaluations
Requires considerable time and resources to identify and involve a wide range of
stakeholders
Takes participating staff away from ongoing activities
122
Handout 7.2
Handout 7.2: PARTICIPATORY M&E 2

How to encourage participation in M&E
use active rather then passive methods
start with monitoring and evaluation activities that are of interest to all
use small groups
use simple graphics and tables
facilitate access to more information
exchange ideas and information with group members about:
the elements of the M&E system

who should be responsible for each task
how the data will be used
where the system will be physically based
when to begin and to end the process
Examples of data collection activities that encourage participation
creating a village or community map
educational games and role plays in groups
participatory monitoring wall charts
group field visits and study tours
use of case studies
group presentation of important findings
use of stories, drama to present findings
group analysis of research reports
123
Handout 8.1
Handout 8.1: DISSEMINATION AND UTILIZATION

1. Define the audience the evaluation is intended for (users).
2. Clearly describe the purpose of the evaluation and what approach, model, or framework
was used to provide direction?
3. Make use of tables and matrices to summarize qualitative and qualitative data analysis
4. Avoid unsupported statements and recommendations
5. Provide practical and constructive comments
6. Include a separate section on lessons learned.
124
Handout 8.2
Handout 8.2: POSSIBLE DISSEMINATION

STRATEGIES
Formal written evaluation reports (various lengths and complexities).
Workshops with counterparts and/or staff (action plan as the output).
Information pamphlet with evaluation highlights.
Case studies.
Specific reports on Lessons Learned.
Video or slide presentations.
Meetings with donor/counterparts/staff.
Community meetings.
Formal presentations of information in graphs and charts.
One on one discussions with stakeholders and what to do next.
125
Supplemental Handout 1
Supplemental Handout 1: PARTICIPATORY

METHODS
By Jim Rugh
NAMES FOR PARTICIPATORY / RAPID APPRAISAL APPROACHES
PME = Participatory Monitoring & Evaluation
PRA = Participatory Rural Appraisal
PLA = Participatory Learning & Action
FPR = Farmer Participatory Research
PALM = Participatory Analysis and Learning Methods
PAR = Participatory Action Research
PRM = Participatory Research Methodology
PRAP = Participatory Rural Appraisal & Planning
PUA = Participatory Urban Appraisal
RAP = Rapid Assessment Procedures
RFSA = Rapid Food Security Assessment
RRA = Rapid Rural Appraisal
Source: Jules Pretty, Irene Guijt, John Thompson & Ian Scoones A Trainers Guide for
Participatory Learning & Action, IIED (1995)
Brief history of participatory approaches

Activist participatory research: inspired by Paulo Freire (1968); uses dialogue and joint
research to enhance peoples awareness and confidence and to empower them to action.
Rapid rural appraisal: written about by Robert Chambers (late 1970s); reaction to biases
inherent in the rural development tourist approach, which tended to hide the worst poverty
and deprivation. Sought to enable outsiders to gain insight and information from rural people
about rural conditions in a cost-effective and timely manner.
Applied anthropology: (1980s) useful in helping development professionals to appreciate
better the richness and validity of rural peoples knowledge; benefits of unhurried participant
observation and conversations and the importance of attitudes, behavior and rapport.
Field research on farming systems: (1980s) recognized the rationality of small and poor
farmers and their activities as experimenters.
Agroecosystem analysis: developed by Gordon Conway (1987); draws on systems and
ecological thinking; uses transects, informal mapping and diagramming, and the use of
scoring and ranking to assess innovations.
Sources: Andrea Cornwall, Irene Guijt and Alice Welbourn (1993); Robert Chambers (1992),
cited in A Trainers Guide to Participatory Learning & Action, IIED (1995)
Some examples of PRA tools

Semi-Structured Interviews
Mapping
Transect walks
Seasonal calendars
Diagramming & Visualizations
Venn organizational diagram
Flow diagrams for systems
126
Ranking & Scoring

Preference ranking
Wealth ranking
Matrix of Criteria
Criteria for trustworthiness
Do participatory approaches meet the four criteria used by conventional researchers?
How can we be confident about the truth of the findings (internal validity)?
Can we apply these findings to other contexts or with other groups of people
(external validity)?
Would the findings be repeated if the inquiry were replicated with the same (or
similar) subjects in the same or similar context (reliability)?
How can we be certain that the findings have been determined by the subjects and
context of the inquiry, rather than the biases, motivations and perspectives of the
investigators (objectivity)?
Source: Y.S. Lincoln and E.G. Guba, Naturalistic Inquiry, Sage (1985)
A typology of participation
1. Passive participation
2. Participation in information giving
3. Participation by consultation
4. Participation for material incentives
5. Functional participation
6. Interactive participation
7. Self-mobilization
Source: J.N. Pretty, Regenerating Agriculture: Policies and practice for Sustainability and
self-reliance (1994), adapted from Adnan et al (1992)
Applying participatory techniques to M&E processes

Review some of the techniques weve learned this week. Which of them lend themselves to
participation by beneficiaries?
Needs assessment
Project design
Planning M&E system
Baseline study
Monitoring system
Conducting evaluations
Data analysis
Quantitative methods
Qualitative methods
Communicating findings
Making decisions, action plans
127
Thinking participatively
1. How can we recognize good practice in participatory development?
2. What external conditions make it difficult to adopt participatory methodologies?
3. Under what conditions and in what situations are participatory approaches appropriate for
CARE projects?
4. What internal conditions in CARE encourage or discourage innovation and adaptation of
participatory approaches?
128
Supplemental handout 2: RATE EXERCISE

(Health/population examples)
DEMOGRAPHICS
Population: 8,330
Families: 990
Total females: 4,000
Females between 15-49: 2100
Males between 15-49: 2000
Males above 15: 2404
Live births last year: 375
Children under 5: 1,660
Children 12-23 months old: 323
Pregnant women: 134
1. Last year there were 47 deaths of children under one year of age. What was the Infant
Mortality Rate?
2. Last year there were 60 deaths of under-five children. What was the Under-5 Mortality
Rate?
3. There are 1,320 women of child-bearing age who are using contraceptives (modern and
traditional). What is the Contraceptive Prevalence Rate?
4. Of 12-23 month olds, 255 were immunized. What is the under-one year immunization
coverage rate?
5. Of 300 mothers with children under 2 who reported that their child had diarrhea in the
past 2 weeks, 200 said that they have used ORT (oral rehydration therapy) at some time.
100 said they used ORT during the past 2 weeks. What is the ORT usage rate?
6. Among children 12-23 months of age, 34 were treated with chloroquine for malaria.
What is the percentage that were treated correctly?
7. Of the 1,660 under-five children 250 were weighed during the past 3 months. 200 gained
weight. What percentage of children gained weight?
8. There are 200 men who have had vasectomies or who report using condoms regularly
and who do not want another child in the next year. What is the contraceptive
prevalence rate?
9. 500 adults were referred to clinics for STD (sexually transmittable diseases). What is the
percentage of those with STD reporting for treatment?
10. Rate Exercise (DZADP examples)
NUMBERS GIVEN IN PROPOSAL
Districts
Divisions - Agrarian Services Centres
(ASC) (at least 1 per Division)
Farmer Organizations (FO) per ASC
Average Farmers per FO
Estimated Total number of farmers
Total farmers cultivating under irrigation in
the DZADP Districts
Other numbers given:
Total
6
65
Phase I target
6
8
Phase II target
6
16
30
42
5
42
10,080
10
42
40,320
78,000
129
1. Output for Immediate Objective (IO) #2: Farmer Field Schools established: 24,000
farmers trained in the use of improved agricultural practices, post harvest methodologies,
livestock and agroforestry. What % of targeted farmers to receive training at Farmer
Field Schools?
2. IO #2: Improved farming systems and post harvest methodologies adopted by 80% of
farm households. How many farm households will need to adopt these methodologies
to fulfill this objective?
3. Assuming a 20% drop-out rate (farmers not applying what they were taught), how many
farmers will need to be trained in Farmer Field Schools in order to meet IO #2?
4. IO #1.3: 40% of farm households have access to improved irrigation infrastructure.
What % of farmers in the target area already have access to irrigation infrastructure?
130
Supplemental handout 3: DEVELOPING EFFECTIVE

EVALUATION PLANS
By Jim Rugh
Introduction
Objective 1: Participants will be given a very brief overview with basic experimental designs
Objective 2: Participants will develop criteria for determining appropriate research/
evaluation designs for CARE projects
Questions to keep in mind:
What is the role of research in the design of a CARE project?

Under what conditions are more sophisticated research designs warranted?
How do these considerations influence the M&E plans for a project?
Agenda
Step 1: Review different experimental/research designs
Step 2: Develop criteria for determining appropriate design for a projects evaluation plan.
1. Overview of research designs
Purposes of evaluation/research design:

To prove impact attributable to a particular intervention
To test hypothesis that interventions outputs lead to impact
What are the elements of research design?
Vocabulary:
Observation = O
Intervention being tested = X
Alternative program (or no intervention) = C
Pretest ... Posttest (baseline ... final evaluation)
Experimental (project) group = E>
Control (comparison) group = C>
Randomization = R
Evaluation Research Designs

NO CONTROL GROUP
No baseline
X O
Before + After
O X O
WITH CONTROL GROUP
Posttest Only E>
X O
C>
O
Pretest +
E> O X O
Posttest
C> O
O
TIME SERIES
Single Group
Control Group
OOOX OOO
OOO OOO
131
Classical Experimental Designs

True Control Group Pretest-Posttest
Time
1 (pre)
2 (post)
Experimental Group E> O
X O
Population >
R
Control Group
C> O
C O
Time Series with Non-Equivalent Control (Comparison) Group
Time
1
2
3
4
Experimental
Group E>
O X O X O X O
Control
Group C>
O C O C O C O
Major Threats to the Implementation of Experimental Designs
Confounds: extraneous effects which happen to one group (either E- or C-group) but
not the other, and which could influence the outcome measures.
Large sample size provides some protection
Account for other influencing factors
Contamination: supposed control group uses methods or materials being tested on
E-group.
Difficult to isolate control group
Attrition: people drop out of either program or control group for any reason.
Make sample size large enough to compensate
Differences between E- and C-groups in time spent on the program (intervention).
Develop criteria for determining appropriate design for a projects evaluation

plan
What are some of the considerations?
For what reasons should experimental or quasi-experimental research designs be used?
When might they not be necessary?
Suggested criteria for determining appropriate design for a projects

evaluation plan
What are some of the considerations?

Costs (funds, staff time or outside expertise required)
Skills available (among staff or outsiders) to do surveys, analyze results
Availability of staff (or outsiders)
Participants time and willingness to cooperate
Non-participants (control groups) time and willingness to cooperate
How to keep control group pure?
Ethics of using control group
132
Accessibility to area to do survey (security considerations)

Intervention time-line (i.e. length of project)
For what reasons should experimental or quasi-experimental research designs be
used?
When it is necessary to prove impact and attribution
Pilot project to serve as model for wide multiplication
If new, untested technology/intervention (test alternatives)
To test hypothesis (i.e... correlation between interventions outputs and impact)
If training and building capacity of partners in evaluation research is a goal of project
When might they not be necessary?
Proven intervention, impact previously ascertained
Reliable, valid and relevant secondary data available
Only need to verify that implementation and outputs complies with standards
Short-term (i.e. emergency)
Questionable security situation
When sophisticated research would be nice but not necessary
POSSIBLE EVALUATION DESIGNS WHEN A CONTROL IS NOT APPROPRIATE

POSSIBLE EVALUATION DESIGNS WHEN THERE IS NO BASELINE
133

M&amp;E Workshop Handout Manual

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

M&amp;E Workshop Handout Manual

Uploaded by

Copyright:

Available Formats

CARE International

DM&E Workshop Series 1997

By Richard Caldwell and Sofia Sprechmann,

CARE 1997 M&E Workshop Series

Caldwell and Sprechmann

Handout 6.13: HEALTH PROJECT CASE STUDY ........................................................................... 118

CARE 1997 M&E Workshop Series

Caldwell and Sprechmann

Handout 1.1: WORKSHOP INTRODUCTION

CARE 1997 M&E Workshop Series

Caldwell and Sprechmann

To gain a working knowledge of the fundamental concepts, definitions, and steps

To acquire new skills in quantitative and qualitative methods related to monitoring

To develop monitoring and evaluation frameworks based on project design logic.

CARE 1997 M&E Workshop Series

Caldwell and Sprechmann

Handout 1.2: WORKSHOP OVERVIEW AND SCHEDULE

Workshop Objectives and

To introduce participants to the workshop format and schedule and to

M&E Conceptual Framework

To discuss the purposes of and CARE's approach for monitoring and

develop a set of group objectives.

From Logframes to M&E

To explore and establish the linkages between project design and

Research Design for Evaluation

CARE 1997 M&E Workshop Series

monitoring and evaluation, paying particular attention to the outputs of

surveys, review sampling design, size and procedures, explore the

Caldwell and Sprechmann

To explore the characteristics and use of qualitative data.

To review the major methods in collecting qualitative data, understand

the issues involved in designing good discussion and interview guides,

To review participatory techniques used for M&E and discuss the

Information Dissemination and

To explore ways of presenting information to decision-makers,

Country Office Strategies for

10. Workshop Evaluation and

CARE 1997 M&E Workshop Series

benefits of involving stakeholders in the design of M&E frameworks,

understand the importance of disseminating and using information,

Participants will be provided an opportunity to express opinions about

Caldwell and Sprechmann

M&E Workshop Handouts

Workshop Objectives and Schedule

M&E Conceptual Framework

From Logframes to M&E Systems

Research Design for Evaluation

Quantitative Data and Methods

Qualitative Data and Methods

Participatory Techniques in M&E

Developing Data Collection Instruments

Field Tasks and Data Collection

10. Information, Dissemination and Utilization

CARE 1997 M&E Workshop Series

Caldwell and Sprechmann

Handout 2.1: MONITORING

Analyze the current situation

Monitoring is continuous. A monitoring system should be in place before project start-up.

CARE 1997 M&E Workshop Series

Caldwell and Sprechmann

Handout 2.2: EVALUATION

How does evaluation differ from monitoring?

What is the purpose of evaluation?

M&E Workshop Handout Manual

M&E Workshop Handout Manual