Data Sources for M&E:

Activities and Outputs

Saturday June 14th

11:30am
Learning Objectives

By the end of this session you will be able to:

 Determine data sources for monitoring your project’s
activities
 Determine potential data sources for measuring whether
your project has achieved intended outputs
 Explain the pros and cons of a number of different non-
routine data sources
 Understand the link between your M&E framework
(Logframe) and data sources…

2
 Session Overview

 Activities monitoring
• Tools
 Measuring progress towards intended outputs
• Data sources / means of verification
 Surveys and sampling

3
 Activities Monitoring
Objective: Access to quality services increased
Indicators & Metrics Means of Verification
Output A: Information and
services provided
Activity A.1: Develop and # of IEC programs on
run IEC programs on radio radio in past year
Activity A.2, A.3, A.4…. …
Output B: Clinical training
carried out
Activity B.1: Develop Curriculum developed
curriculum for training
Activity B.2, B.3, B.4… …

4
Tools for Activities Monitoring

 Sign-in / registration logs

 Program activity forms
 Logs and tally sheets
 Patient charts
 Direct observation checklist

5
 MOV for Activities: Was the soup stirred?
Objective: Access to quality services increased
Indicators & Metrics Means of Verification
Output A: Information and
services provided
Activity A.1: Develop and # of IEC programs on ???
run IEC programs on radio radio in past year
Activity A.2, A.3, A.4…. …
Output B: Clinical training
carried out
Activity B.1: Develop Curriculum ???
curriculum for training developed

Activity B.2, B.3, B.4… …

6
 MOV for Outputs: Was the soup made?
Indicators & Metrics Means of Verification
Objective: Access to quality services increased
Output A: Information and 1. % of facilities ???
services provided providing all
targeted services
2. % of facilities
providing IEC
material
Activity A.1, A.2, A.3, A.4…. …
Output B: Clinical training # of providers who ???
carried out have completed
clinical training

Activity B.1, B.2, B.3, B.4… …

7
 Step 1: Understanding your indicators

Remember metrics?
What do these indicators mean?
What data do we need?

# of IEC programs on radio in past year

% of facilities providing all targeted services
# of providers who have completed clinical training

Spend 5-10 minutes discussing both of these in your groups and

be prepared for group discussion

8
Step 2: Determine data sources

 How can we get this data?

• Routine sources (continuous data collection)
• Non-routine sources

9
 Non-routine Methods

 Quantitative
• Counting
• Surveys: bio, behavioral, GIS
• Costing
 Qualitative
• Interviews: in-depth, semi-structured
• Focus groups
• Observation
• Documentary & Policy Analysis

10
Levels of non-routine data collection

 Facility / service
 Provider / staff
 Client
 Population

11
Data Sources: Facility & Provider level

 Facility audits, inventories

 Facility surveys
 Provider interviews
 Provider-client observation
 Provider training records
 Situation analysis
 Others?

12
Data Sources: Client level

To measure “program exposure”: utilization, service experience,

quality of care/service delivery, disease surveillance

 Client (exit) interviews

 Client focus groups
 Case surveillance
• e.g. Child Status Index (MEASURE)
 Provider-client observation
 Patient records and registers
• e.g. Basic HIV care / ART patient monitoring tools (MEASURE)
 Community-based monitoring, e.g. CLPIR (MEASURE)

13
Data Sources: Population level

To measure program exposure, behavior, and other

characteristics of the general or target population

 Census
 Vital registration
 Household surveys, e.g. DHS
 Targeted population surveys
 e.g. (Bio-)BSS, Framework for monitoring MARPs
(MEASURE)

14
Choosing MOV: Issues to consider

 Different data collection methods deliver

different products
 Methods present different degrees of cost &
complexity
 Indicators dictate methods
 Other factors, e.g. resources, capacity, time
etc, also influence the choice of methods

15
 Data Sources: Trade-offs

Specific sample
Cost

Indicator selection needs to surveys

consider the resources
Focus group
available for data collection
discussions

Key informant
Adapted interviews
routine
Existing records Routine statistics The effort should match
(e.g. household statistics the improvement in
decision-making
lists)

Complexity
16
Triangulation: Multiple MOV

Triangulation = the process of collecting, examining,

and interpreting data from multiple sources to
improve our understanding of a public health
problem and to guide decision-making

Sometimes we need to use multiple MOV to measure

progress against one output

Can you think of why / when?

17
Rule of Thumb

 Whenever possible, use routine data

 Assess gaps in routine data available against your
M&E objectives
 Determine priorities
 Develop Data Collection Plan for non-routine
sources of data

18
 Group work
Indicators & Metrics Means of Verification
Objective: Access to quality services increased
Output A: Pop’n 50% increase in the
awareness of services number of people
increased reporting awareness of
4 out of 5 key services
offered
Activity A.1, A.2, A.3, …
A.4….
Output B: Practitioners’ All practitioners trained
skills / knowledge demonstrate increased
increased skills and knowledge 6
months after training
Activity B.1, B.2, B.3… …

19
 Means of Verification for Outputs
Indicators & Metrics Means of Verification
Objective: Access to quality services increased
Output A: Pop’n 50% increase in the Client exit survey
awareness of services number of people reporting
increased awareness of 4 out of 5
key services offered
Activity A.1, A.2, A.3… … …
Output B: Practitioners’ All practitioners trained Provider observation
skills / knowledge demonstrate increased
increased skills and knowledge 6 Provider pre-post test
months after training with 6 mos. follow-up
Activity B.1, B.2, B.3… … …

20
 What is a Data Collection Plan?
A narrative that outlines:
 All data sources, linked to all indicators
 Who will be responsible for data collection and its supervision
 Who will be responsible for ensuring data quality
 How will data be collected, compiled & analyzed?
• What tools/forms will be used, if any?
• Who will do this?
 How often will the data be collected, compiled & and analyzed
 What resources (staff, office supplies, computers,
transportation) are needed?

Summarize with a Data Collection Matrix

21
 Data Collection Matrix

Timeframe Person Responsible

Data source Q1 Q2 Q2 Q4 Data Analysis

Collection

Client exit survey

Provider observation

Provider pre-test /
post-test & 6 month
follow-up
…

22
Surveys & Sampling

23
Surveys: When are they appropriate?

 Surveys are especially useful

• If other data are not available or inadequate
• If tailored to fit specific measurement objectives
 Yield cost-efficient data on population and services
 Good sampling techniques produce
representative results
 Expensive, but versatile & widely used

24
Facility Surveys

Provide information on:

 Facility profile
• Number, types, services, staffing, location, fees
• Service readiness, quality of care
 Whether access and quality of care affect:
• Client choice of health care providers
• Health outcomes
 Cost-effectiveness
• Barriers to efficiency

25
Facility Surveys as a Source of M&E Data

 Strengths
• Timing can coincide with program implementation
• Can combine with population survey for outcome
evaluation

 Limitations
• Design and analysis may be complex
• Expensive, time-consuming
• Information rapidly outdated, unless repeated

26
Population surveys

 Indirect data source for client behavior

 Can ask about:
• Past, current, intended use of services
• Preferences

 Strengths: Can compare client and non-client

preferences/choices & past use
 Limitations: recall bias, very expensive, very
complex

27
 Sampling

 Most research methods, and all surveys, require a

sampling strategy
 Various sampling strategies
 Convenience
 Snowball
 Respondent-Driven Sampling (RDS)
 Random (probability) – quantitative only

28
Random Sampling

Why?

= Random samples allow us to

generalize our results

How?

29
 Step 1: Define sampling frame

 A sampling frame is a complete list of every

“element” in your population

 An example research questions

“What percent of adults in Abuja drive SUVs?”

What terms need defining in the question?

30
Sampling Frame

 Define your Sampling Frame: Inclusion &

Exclusion Criteria
• All adults in Abuja?
• All licensed drivers in Abuja?
• All licensed drivers who permanently reside in Abuja?
 If there are 100,000 licensed drivers in Abuja, then
your sampling frame should consist of exactly
100,000 unique names of these people*
 Can’t draw a random sample without a sampling
frame

31
 SAMPLING FRAME
All Adults in Abuja
N=??

Every population
member has an equal
chance of being
selected into your
sample

SAMPLE
Random selection
N=100

32
 The Parameter is the
actual, true value that
you are trying to find:
Actual % of Abuja
drivers who drive SUVs
Sample
What is the statistics
function of a are
statistic? estimates
of the
parameter

Sample Statistic:
20% drive SUVs
20% is your estimate
of the “true” value
33
 Probability Theory: How Closely are the Sample Statistics
Clustered Around the True Mean (Parameter)?

100
Number of
random
samples
drawn
50

0% 10% 20% 30% 40% 50%

% of Abuja drivers reporting: “I drive an SUV.”

34
 How To Draw A Random Sample

1. Name 16. Name 31. Name 46. Name 61. Name

2. Name 17. Name 32. Name 47. Name 62. Name
3. Name 18. Name 33. Name 48. Name 63. Name
4. Name 19. Name 34. Name 49. Name 64. Name
5. Name 20. Name 35. Name 50. Name 65. Name
6. Name 21. Name 36. Name 51. Name 66. Name
7. Name 22. Name 37. Name 52. Name 67. Name
8. Name 23. Name 38. Name 53. Name 68. Name
9. Name 24. Name 39. Name 54. Name 69. Name
10. Name 25. Name 40. Name 55. Name 70. Name
11. Name 26. Name 41. Name 56. Name 71. Name
12. Name 27. Name 42. Name 57. Name 72. Name
13. Name 28. Name 43. Name 58. Name 73. Name
14. Name 29. Name 44. Name 59. Name 74. Name
15. Name 30. Name 45. Name 60. Name 75. Name

35
 Simple Random Sample
Generate 10 random digits between 1 and 75.
1. Name 16. Name 31. Name 46. Name 61.name
2. Name 17. Name 32. Name 47. Name 62. Name
3. Name 18. Name 33. Name 48. Name 63. Name
4. Name 19. Name 34. Name 49. Name 64. Name
5. Name 20. Name 35. Name 50. Name 65. Name
6. Name 21. Name 36. Name 51. Name 66. Name
7. Name 22. Name 37. Name 52. Name 67. Name
8. Name 23. Name 38. Name 53. Name 68. Name
9. Name 24. Name 39. Name 54. Name 69. Name
10. Name 25. Name 40. Name 55. Name 70. Name
11. Name 26. Name 41. Name 56. Name 71. Name
12. Name 27. Name 42. Name 57. Name 72. Name
13. Name 28. Name 43. Name 58. Name 73. Name
14. Name 29. Name 44. Name 59. Name 74. Name
15. Name 30. Name 45. Name 60. Name 75. Name

36
 Systematic Random Sample
Move through the list based on the sampling interval k

1. Name 16. Name 31. Name 46. Name 61.name

2. Name 17. Name 32. Name 47. Name 62. Name
3. Name 18. Name 33. Name 48. Name 63. Name
4. Name 19. Name 34. Name 49. Name 64. Name
5. Name 20. Name 35. Name 50. Name 65. Name
6. Name 21. Name 36. Name 51. Name 66. Name
7. Name 22. Name 37. Name 52. Name 67. Name
8. Name 23. Name 38. Name 53. Name 68. Name
9. Name 24. Name 39. Name 54. Name 69. Name
10. Name 25. Name 40. Name 55. Name 70. Name
11. Name 26. Name 41. Name 56. Name 71. Name
12. Name 27. Name 42. Name 57. Name 72. Name
13. Name 28. Name 43. Name 58. Name 73. Name
14. Name 29. Name 44. Name 59. Name 74. Name
15. Name 30. Name 45. Name 60. Name 75. Name

37
 Stratified Random Sample
Group 1 Group 2 Group 3 Group 4 Group 5
1. Name 16. Name 31. Name 46. Name 61.name
2. Name 17. Name 32. Name 47. Name 62. Name
3. Name 18. Name 33. Name 48. Name 63. Name
4. Name 19. Name 34. Name 49. Name 64. Name
5. Name 20. Name 35. Name 50. Name 65. Name
6. Name 21. Name 36. Name 51. Name 66. Name
7. Name 22. Name 37. Name 52. Name 67. Name
8. Name 23. Name 38. Name 53. Name 68. Name
9. Name 24. Name 39. Name 54. Name 69. Name
10. Name 25. Name 40. Name 55. Name 70. Name
11. Name 26. Name 41. Name 56. Name 71. Name
12. Name 27. Name 42. Name 57. Name 72. Name
13. Name 28. Name 43. Name 58. Name 73. Name
14. Name 29. Name 44. Name 59. Name 74. Name
15. Name 30. Name 45. Name 60. Name 75. Name

38
What is cluster sampling (CS)?

 Multi-stage sampling process used when a

complete list of members of a population of
interest is not available, but when members can
be associated with geographic locations,
institutions, or other “clusters” and a list of such
sites is available or can be feasibly constructed

39
What is cluster sampling? (cont’d)

 Strategy – select a sample of such “clusters” at the first

stage of sample selection, make a list of sub-population
members only for sample clusters, and choose a random
sample at the second stage of sample selection
 Note: some variants of cluster sampling do not require
making a list of sub-population members
 Sampling can consist of 2, 3, 4 or more stages (multi-
stage cluster sampling)

40
What is cluster sampling? (cont’d)

 First stage sampling units are referred to as

Primary Sampling Units, or PSUs
 Sampling units at subsequent stages are referred
to as secondary, tertiary, etc. sampling units
(SSU’s, TSU’s, etc.)
 Most common cluster sample design is the two-
stage design

41
Two-stage cluster sample

Stage I:Selection of Clusters/PSUs

Stage II:Selection of Respondents

42
Primary Sampling Units (Clusters)

PSU/Cluster - any identifiable site or location where

population sub-population members congregate

Group Possible PSU

General pop Census EA’s, villages/blocks, HH
Youth Same as above, plus schools
FSW (BB) Brothels, massage parlours
FSW (NBB) Streets, bars, hotels
MSM Cruising sites, hotspots
IDU Shooting galleries, injecting sites
Truckers Loading/unloading/halting points

43
PSU’s/Clusters (cont’d)

 However, for many of these groups cluster

sampling is less than optimal because:
 Some/many members may not appear at such clusters
 Some/many members may be associated with multiple
clusters
 Some/many members may appear at clusters only
irregularly, and finding them “off-site” is difficult

44
Sampling frame development

 List of PSUs must be relatively complete

(incomplete frame = potential bias)
 This will require preliminary fieldwork to identify
relevant sites/locations
 The results of program planning and pre-
surveillance assessment activities may be useful
in constructing a sampling frame, but more
focused efforts to identify specific locations may
be necessary

45
Strengths of Cluster Sampling

 Provides a feasible means of sampling some

sub-populations for which no sampling frame
exists
 Fixed sample selection procedures minimize
sampling bias
 State of the art in general population and
institutional surveys

46
Limitations of Cluster Sampling

 Only reaches the sub-populations that congregate and,

within these sub-populations the most “visible” subset of
the groups (i.e. those who come into contact with the
locations where sampling is done)
 Difficulties of accessing sub-population members
associated with sample sites who may not be present at
time of data collection further limits its utility for population
surveys

47
Limitations of Cluster Sampling
(cont’d)
 Difficulties in obtaining accurate measures of size
minimizes feasibility of PPS selection

48
Why did we spend time discussing CS
if it is not useful?
 The preferred choice for some populations
 Provides “gold standard” of sorts for comparison
for the other sampling methods discussed

49
Learning Summary

 Data sources and data collection becomes more

complex as we move from activities to outputs…to
outcomes & impact
 Indicators need to be operationalized
 First look for routine data sources to obtain data
 Consider pros and cons of non-routine data sources
 Develop a data collection matrix
 There are many sampling methods
• Gold standard=random sampling

50
MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) through
Cooperative Agreement GPO-A-00-03-00003-00 and is
implemented by the Carolina Population Center at the
University of North Carolina in partnership with Futures
Group, John Snow, Inc., ORC Macro International, and
Tulane University.

Visit us online at http://www.cpc.unc.edu/measure.