Ethical Challenges

American Journal of Evaluation

31(4) 549-556
ª The Author(s) 2010
The Ethical Implications of Reprints and permission:
sagepub.com/journalsPermissions.nav
Underfunding Development DOI: 10.1177/1098214010373376
http://aje.sagepub.com
Evaluations

Michael Hendricks1 and Michael Bamberger2

Each year a great many evaluations are conducted of international development efforts around the
world. These development evaluations study projects, programs, country-wide portfolios, policy
reform efforts, and other topics of interest to funders, governments, program managers, and other
involved stakeholders. Although some of these evaluations are well funded, and while a rare evaluation
might possibly even be overfunded, the majority of individual development evaluations are probably
underfunded. That has certainly been our own personal experience working on evaluations for a wide
range of development agencies—experience that has been confirmed by the majority of development
practitioners who attend the evaluation workshops we organize and with whom we consult.
This underfunding can produce several implications for a development evaluation: logistical
implications such as the number of evaluators that can be involved and the amount of travel possible,
methodological implications such as the types of study designs possible and the types and amount of
data that can be gathered, and even political implications such as who can be involved in the
evaluation, at what stages, and in what ways.
Many of these implications are well known, especially among experienced development evalua-
tors, but we wonder if there might also be one additional set of implications of this underfunding—
implications which we believe have not been sufficiently discussed to date. Specifically, we wonder
if there might also be ethical implications of underfunding development evaluations.
Please note that this article does not address the macrolevel ethical implications of underfunding
an entire evaluation function, either within or across organizations. This is certainly an important
issue, but one which Vinod Thomas addresses in the accompanying article. Instead, this article
focuses attention on what we are calling the microlevel (project and program level) ethical implica-
tions of the underfunding of individual development evaluations.

What Makes an Implication ‘‘Ethical?’’

As we begin, it is incumbent on us to define what we mean by an ‘‘ethical implication.’’ This is not
simple, and the chief difficulty is how to determine when an implication crosses the line from

1
Independent Evaluation Consultant
2
Independent Consultant

Corresponding Author:
Michael Hendricks, PO Box 920, Hood River, OR 97031, USA
Email: mikehendri@aol.com

549
550 American Journal of Evaluation 31(4)

logistics, methodology, or politics and enters the realm of ethics. For example, not being able to fly
first class to the evaluation site certainly has logistical implications for the evaluation team, but few
people (and certainly not we) would claim that it has ethical implications. And while not being able
to videotape focus groups might have methodological implications for the evaluation’s analysis, we
would see few ethical implications there, either.
What, then, does distinguish an ethical implication from other types of implications? One diction-
ary defines ethics as ‘‘the discipline dealing with what is good and bad and with moral duty and obli-
gation’’ (Merriam-Webster, 2010). Applied to development evaluation, this definition might suggest
that underfunding has ethical implications if it prevents a development evaluation from operating
according to what is good and bad and within its moral duty and obligation. However, we recognize
that this definition itself is open to considerable interpretation. What is good versus bad? What is an
evaluator’s or a program’s moral duty or obligation?
We also recognize that there are few, if any, moral absolutes and that evaluators are often asked to
play different roles at different times (cf. Bamberger, Rugh, & Mabry, 2006, Chapter 6). Are the
ethical issues the same for an evaluator conducting a multiyear randomized experiment as for an
evaluator conducting a brief needs assessment? If so, would all evaluators agree on the common
ethical issues?
Another set of questions concern the extent to which the evaluator must assess a program strictly
in terms of its stated objectives versus the circumstances under which the evaluator, or the funding
agency, can introduce ethical criteria that they consider important. For example, many managers
complain that it is not fair to assess their program in terms of gender equality if this was not one
of the stated objectives when the program was designed. Similarly, is it appropriate to criticize
an economic development program for not ensuring an adequate participation of the poorest farmers
or households if the goal of the program was to promote economic growth but with no reference to
equity objectives?
We suspect that there is no universal agreement on these questions, and so we believe that individual
evaluators and their funders will need to decide on a case-by-case basis. However, to begin the con-
versation on this issue, and after consulting with a leading expert on evaluation ethics (M. Morris, per-
sonal communication, March 16 and 27, 2010), we will propose four dimensions of ethical
implications. We should add that although there may not be universal agreement on these criteria, they
are all drawn from a review of evaluation standards and guidelines and from the stated policies of many
development agencies. Specifically, we suggest that an implication has an ethical dimension if it inter-
feres with an evaluation’s obligation to accomplish any of the following four tasks:

Evaluations Should Provide Valid and Useful Information About the Program and its Effects
In our opinion, this is the essence of the evaluator’s role, and it is no coincidence that the first of the
five American Evaluation Association (AEA) Guiding Principles for Evaluators (AEA, 2004) is that
‘‘Evaluators conduct systematic, data-based inquiries.’’ If the reality of a typical real-world situation
is not obvious but is hidden behind subtle complexities, and if the evaluation is not sufficiently
funded to uncover those complexities, the program will neither learn about nor be able to act upon
those important findings.
For example, a country program evaluation of the UN’s International Labor Office (ILO) in
Indonesia interviewed over 100 different persons, each with a unique perspective on the situation.
Since the evaluation funded only a few days for analyzing all data gathered during field work and
preparing a draft report, it was suggested that these 100 interviews be grouped into four large cate-
gories for analysis.
However, the evaluators felt that using only four groupings might mask important differences of
perspective, so instead they created nine separate respondent categories and analyzed the responses

550
Hendricks and Bamberger 551

at that more-detailed level. By doing so, they were able to show differences of opinion, including
within the ILO office, on such important issues as the proper role of the ILO in Indonesia, the current
level of collaboration with local constituencies, the maximum effective workload possible, and the
importance of results-based management (Hendricks, 2009). Each of these findings led directly to an
evaluation recommendation and hopefully to program improvements. Yet if the analysis had been
conducted at only the funded level of detail, these findings would never have surfaced, with the
obvious ethical implications.

Evaluations Should Identify Harm that May be Resulting

If persons (or their environments, cultural heritages, etc.) are potentially being harmed by a program,
and if the evaluation is not sufficiently funded to detect that harm, the program can do nothing to
ameliorate that harm or prevent it in the future.
For example, evaluators studied the impacts of a food-for-work program on women’s economic
and social empowerment in Central America, and they met with groups of women participating in
the program and with many of their husbands. It was apparent that the program had produced sig-
nificant increases in the income of the women, that their husbands were very supportive of their
wives’ economic activities, and that there was convincing evidence of the women’s economic
empowerment and increase in their self-confidence and social empowerment. In most communities
the evaluation ended at this point, and a very positive report was produced.
However, in some communities the evaluators were able to obtain additional information about
the experiences of other women who had not been available for interviews during the main evalua-
tion. By taking this extra step, the evaluators learned that quite a few women who had attended the
program’s initial meetings had later been beaten by their husbands for attending without permission.
In these cases, the husbands were unemployed and apparently felt humiliated that they were not able
to fulfill their traditional role of providing economically for their family.
In other cases women were identified who had not even been permitted by their husbands to
attend the orientation meeting. Learning about these excluded women was not funded in the
overall evaluation, and the program came very close to not learning about actual physical harm
it was causing for some of its targeted beneficiaries. We believe this represents a clear
ethical implication of not interviewing a more representative sample of the target population
(Bamberger, 2009).

Evaluations Should Promote Equity

We realize that some readers may agree with our first two dimensions but may be less convinced of
this one. ‘‘Why do evaluations have an obligation to promote equity?’’ In our opinion, this is simply
an extension of identifying harm that may be resulting, because inequity is a form of harm. If some
beneficiaries are benefitting more from a program than other beneficiaries, and if the evaluation is
not sufficiently funded to detect that inequity, the program can do nothing to distribute the benefits
more equitably in the future.
In Bolivia, for example, evaluators studying a microcredit program promoting the manufacture
and marketing of traditional carpets interviewed a sample of carpet makers who had received loans.
It was found that, on average, their income from the sale of carpets had increased significantly. It
was concluded that microcredit was an effective instrument for increasing the income of traditional
artisans and reducing poverty.
However, when carpet manufacturers who had not received loans were interviewed it was found
that on average their income had declined. One of the reasons was that loan recipients were able to
rent or purchase a vehicle to get their carpets to market more quickly and cheaply, giving them a
competitive advantage. On the basis of this additional information, it was estimated that the total

551
552 American Journal of Evaluation 31(4)

sales of carpets had probably not increased very much, but that a larger share of the market was now
controlled by loan recipients. If the evaluators had not been sufficiently funded to be able to locate
and interview carpet manufacturers outside the direct program participants, the program would not
have realized that it was inequitably shifting benefits from one subgroup of the target group to
another. Frequently the excluded or disadvantaged groups are the poorest and most vulnerable
(Bamberger, 2009).

Evaluations Should Give Voice to All Affected Persons

This may be the least agreed upon of our four dimensions, but after considerable thought, we believe
it does represent a moral imperative for an evaluation. Two other of the AEA Guiding Principles for
Evaluators are that ‘‘Evaluators respect the security, dignity and self-worth of respondents, program
participants, clients, and other evaluation stakeholders’’ and that ‘‘Evaluators articulate and take into
account the diversity of general and public interests and values that may be related to the evalua-
tion.’’ In our opinion, these principles cannot be upheld if an evaluation allows certain voices, typi-
cally the more-powerful elites, to dominate the evaluation to the detriment of voices from those who
are less powerful.
For example, an underfunded evaluation rarely has the opportunity to visit beneficiaries living far
from the capital city or from the center of the village, or even to randomly sample those beneficiaries
living nearby. Instead, it is common practice for the interviews with beneficiaries and other stake-
holders to be arranged by program officials or staff. Perhaps not surprisingly, these interviewees
often have positive views of the program and its effects, yet one can wonder if the views of persons
not selected by the program would have been equally positive. To us, there are ethical implications
of not providing a voice to a fully representative cross-section of beneficiaries and other
stakeholders.
In summary, then, we propose that the ethical implications of underfunding development eva-
luations can reasonably be viewed through the four dimensions of providing valid and useful infor-
mation, identifying harm that may be resulting, promoting equity, and giving voice to all affected
persons.

Our Approach to Identifying the Ethical Implications of Underfunding

Development Evaluations
Armed with these four dimensions of an ethical implication, we begin our analysis well aware that
addressing a question of this sort could easily devolve into personal opinion, philosophical justifi-
cation, political ideology, or simple speculation. To avoid those missteps and to ground our analysis
to the extent possible in objective considerations, we developed a two-step framework for uncover-
ing possible ethical implications:

1. As Table 1 shows, we first document (in column one) possible effects on the conduct of an eva-
luation when the necessary time and activities are underfunded. (We use the word ‘‘possible’’ to
reflect honestly that these effects are not always inevitable.) Specifically, we consider the pos-
sible effects of underfunding at five different stages of an evaluation: initiation/commissioning,
particularly as specified in the funder’s initial request for proposal (RFP) and/or terms of refer-
ence (TOR); design; data collection; analysis; and reporting/use.

In our opinion, only by considering these specific effects on the evaluation can we consider with
any degree of objectivity whether or not ethical implications exist. For example, a common

552
Hendricks and Bamberger 553

Table 1. A Framework for Identifying Ethical Implications and How Three Groups Might Respond

Some effects of underfunding Some ethical implications of Some possible actions to

on the conduct of the under-funding for the minimize these ethical
evaluation program implications

Initiation/Commissioning
* The evaluation cannot study
all relevant and important
issues
* The evaluation team cannot
include all required skills
Design
* Not possible to review in
advance all relevant
documents or data
* For impact evaluations, not
possible to develop a credible
counterfactual
Data Collection
* Cannot interview a
representative sample of all
target and comparison
beneficiaries
* Cannot interview the full
spectrum of non-beneficiary
stakeholders
Analysis
* Superficial analysis of rich
data sets from field work
* Not possible to study key
elements of the project’s larger
context
Reporting/Use
* Not all stakeholders can
receive the report or attend
briefings
* Cannot translate report into
local languages
Not receive valid and useful
information about the program and
its effects
* The program may not learn about
important aspects of its operations
and effects
* The program may receive an
inaccurate estimate of its effects
* The program may not learn its
place in the larger, surrounding
context
Not understand harm that may be
resulting
* The program may not learn about
actual harm it has produced
(human, cultural, environmental)
Not learn about inequity
* The program may not learn about
the inequitable distribution of its
benefits
*The program may not learn about
vulnerable groups who cannot
access program benefits
Not hear voices of all affected
populations
* The program may learn the
perspectives of only powerful, elite
stakeholders
* The program may not hear from
those most affected by its
operations
Internal or external evaluators might…
* Review the RFP and TOR with an eye to ethical
implications, and discuss these with the funder
along with viable alternatives
* Remain aware of the possible ethical
implications throughout the evaluation, and
make creative efforts (mixed methods, real-
world evaluation techniques, alternatives to the
conventional counterfactual) to avoid or
ameliorate each possibility
* Describe in the final report the limitations faced
while conducting the evaluation, the ethical
implications these limitations raise, and any
needed cautions against misinterpretation.
Funding agencies might…
* Revise RFPs and TORs to encourage that all
sectors of the target population are identified
and interviewed
* Increase use of local consultants (cheaper and
more knowledgeable on many topics) to ensure
a deeper understanding of the program and its
context
* Require evaluators to include in every
evaluation report a discussion of possible ethical
implications
* Ensure the report is accessible to all
stakeholders and that feedback is obtained
from all groups
The global evaluation community might…
* Raise the visibility of the ethical implications of
under-funding development evaluations
* Gather case studies of under-funded
development evaluations and the resulting
consequences
* Make clear declarations that it is unacceptable
to conduct evaluations with such ethical
implications
* Propose practical guidelines for how the most
serious of these ethical implications could be
avoided or minimized

effect of underfunding is that the evaluation contacts only the more elite, more advantaged sta-
keholders who are logistically the easiest to reach.

2. Next we consider (in column two) the ethical implications for the program that flow from each
of these limiting effects on the evaluation. For example, because the evaluation may be able to
contact only those stakeholders logistically easiest to reach, the program may receive an inac-
curate impression of its operation and effects.

Note that column one documents possible effects on the evaluation while column two describes
the resulting ethical implications for the program. This difference in wording is very deliberate,
and we think the distinction is important. Certainly the underfunding limits the conduct of the
evaluation, so the wording of column one seems clear. However, column two reflects that these
ethical implications are particularly a hindrance for the program, not the evaluation, because the
program’s future operations will be affected. Of course, there may be times when a program
deliberately underfunds an evaluation (or at least capitalizes on underfunding) to skew the
results of an evaluation, but even the most conscientious and open-minded program managers
will be affected when underfunded evaluations provide less than valid and useful information.

553
554 American Journal of Evaluation 31(4)

The Ethical Implications of Underfunding, at Three Different Levels

Level #1. Table 1 and its contents can be viewed at three different levels. At its basic level,
columns one and two show 10 common effects of underfunding on a development evaluation and
8 common resulting ethical implications for a program. In our opinion, each of these 10 effects and
eight implications is individually worthy of serious review and consideration, and we hope readers
will take the time to review these carefully.

Level #2. At the next higher conceptual level, we can aggregate and summarize the eight ethical
implications shown in column two of Table 1 into two separate, overarching ethical risks. First, a
development program that underfunds an evaluation might easily not learn important findings about
its operations and effects. These missed findings might be issues not recognized as important at the
beginning of the evaluation and not able to be followed-up during or after field work; unexpected
side effects (either positive or negative) that are not uncovered due to limitations of sampling or data
collection; important contextual aspects about the program’s role and impacts within its physical and
social environment (Rog, 2009); or other information gaps impossible to predict in advance. To us, it
is an ethical issue when an evaluation fails to provide such essential information.
The second overall ethical risk is that a development program that underfunds an evaluation
might easily receive an overly positive report of its operations and impacts. In this case, a program
is likely to hear inaccurately positive findings and recommendations, whereas at the same time not
hear about the harm it has caused or inequities it has created. We believe this misrepresentation is an
especially important ethical issue, because the program may be causing genuine harm to some per-
sons (Bamberger, 2009).

Level #3. At an even higher conceptual level, and extending our analysis one more step, we can
also ask, ‘‘What are the further ethical implications of these two overall ethical risks? That is, what
might result if development programs miss important findings and hear overly positive news?’’ We
believe there are several additional ethical implications at this level that come to bear on both pro-
grams and evaluations:

 programs may receive continued funding, even though they are not as effective as stakeholders
believe;
 as a result, funds may not be available for testing other, potentially more effective programs;
 because funders are told that their programs are working well, they may be less likely to chal-
lenge weak (and often invalid) methodologies than they might be if findings were more negative;
 development evaluators may learn that, because weak methodologies are not challenged, they
have no incentive to propose more rigorous evaluations in the future;
 the lack of rigorous, comprehensive evaluations may perpetuate the status quo or program tin-
kering but rarely a fundamental program review.

Two Important Observations

To put these ethical implications in perspective, we should add two important observations.
First, although this article is about the evaluation of international development programs, many
of our points also apply to domestic evaluations within the United States. In fact, one could
plausibly argue that each point we make applies equally to domestic and international evalua-
tions, and if so, our analysis might be relevant and useful for all evaluators, no matter where
they practice.

554
Hendricks and Bamberger 555

However, we do believe that development evaluations generally run a greater risk of encounter-
ing these ethical issues, largely because of the extremely limited time on-site. Some development
evaluators call some of their work ‘‘evaluation flybys,’’ not entirely inaccurately, due to the typical
shortage of time for in-country planning, coordinating, data collection, analysis, and follow-up.
Although domestic evaluators frequently face time and budget pressures, they often have more time
on-site and a greater ability to follow-up afterward if the need arises.
Our second observation is that more funding for development evaluations does not automatically
resolve the ethical implications we have discussed. That is, even though the absence of sufficient
funds may create many ethical implications, the presence of those funds will not necessarily avoid
those implications. In addition to money, it requires political will, managerial openness, and evalua-
tor creativity to avoid many of the issues we have identified. For example, if the main data collection
instrument used in an evaluation of a microcredit program targeting low-income women entrepre-
neurs is a structured survey, it is very likely that this structured survey would not capture information
on increased domestic violence from husbands who feel humiliated that their wife is now the main
breadwinner. In such cases, even doubling the evaluation budget would not address this ethical issue
unless the evaluation methodology was changed.
For that reason, it is difficult to specify how well-funded a development evaluation ought to be, or
even how to know if one is underfunded. In our opinion, it may not be productive to declare an abso-
lute dollar amount or even a percentage of a program’s budget as the ‘‘proper’’ amount for a devel-
opment evaluation. Ultimately, defining underfunding may be somewhat like U.S. Supreme Court
Justice Potter Stewart’s famous definition of pornography: ‘‘I can’t define it, but I know it when I see
it.’’ If so, then one possible way to ‘‘see’’ underfunding may be to determine whether the proposed
evaluation would face any of the limitations listed in column one of Table 1. If so, then that partic-
ular development evaluation may not have sufficient funds to avoid the resulting ethical implications
of column two.

What Can be Done?

Ending on a practical note, can anything be done to ameliorate this situation? Are these limitations
and ethical implications inevitable, or might these situations be improved? In our opinion, much can
be done, and column three of Table 1 lists several suggestions for each of the three different actors.
Others can no doubt add to this list, but in our opinion:

 Internal or external development evaluators might analyze upfront the budget’s limitations on
the evaluation, remain aware of ethical implications throughout the evaluation, and describe
in the final report both the limitations and the implications.
 Funding agencies might pay close attention to their RFPs and TORs, increase use of local con-
sultants, require evaluators to make explicit the ethical implications, and ensure the report and
feedback mechanisms are accessible to all stakeholders.
 The global evaluation community might raise the visibility of this issue, gather case studies of
underfunding leading to ethical implications, declare the unacceptability of such situations, and
provide guidelines for ameliorating these situations in the future.

Conclusion
Many of the logistical, methodological, and political implications of underfunding development eva-
luations have been known for some time. In this article, we have tried to extend the discussion in
seven different ways, by (1) raising the question of whether underfunding also creates ethical

555
556 American Journal of Evaluation 31(4)

implications, (2) suggesting four dimensions along which development evaluations ought to be espe-
cially ethical, (3) proposing that the most objective way to identify ethical implications is to trace
them from specific limiting effects on the evaluation itself, (4) suggesting that although underfund-
ing affects the conduct of the evaluation, the ethical implications of that underfunding are felt pri-
marily by the program, not the evaluation, (5) aggregating and summarizing the eight most common
ethical implications into two overall ethical risks, (6) considering the likely effects of these two ethi-
cal risks, and (7) suggesting possible actions for three different sets of actors. We hope these are
useful contributions, and we also hope that others will improve and extend our work.

Acknowledgments
We thank Richard Blue, Leslie Goodyear, Leanne Hogie, Michael Morris, and Craig Russon for
their helpful comments on an earlier version of this article.

Declaration of Conflicting Interests

The author(s) declared no conflicts of interest with respect to the authorship and/or publication of
this article.

Funding
The author(s) received no financial support for the research and/or authorship of this article.

References
American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved from: http://www.eval.
org/Publications/GuidingPrinciples.asp.
Bamberger, M. (2009). Why do many evaluations have a positive bias? Should we worry? Evaluation Journal
of Australasia, 9.
Bamberger, M., Rugh, J., & Mabry, L. (2006). Realworld evaluation: Working under budget, time, data, and
political constraints. Thousand Oaks, CA: Sage.
Ethics. (n.d.). Merriam-Webster Dictionary online. Retrieved from http://www.merriam-webster.com/
dictionary
Hendricks, M. (2009). Evaluating the International Labor Organization’s (ILO’s) five-year country program in
Indonesia. Presentation to the Annual Conference of the American Evaluation Association, Orlando, FL,
USA, November 12, 2009.
Rog, D. (2009). Toward context-sensitive evaluation practice. Presidential address to the Annual Conference of
the American Evaluation Association.

556