You are on page 1of 40

Evaluation with an Impact

BSILI- June 2012

Christina A. Christie, Ph.D.

 

University of California, Los Angeles

Michael Harnar, Ph.D.

 

310-825-0432

Goals for this Session

  • Ground evaluation as a practice in varying definitions of evaluation

  • Understand what it means for an evaluation to have impact and consider the various ways evaluation can influence decisions and activities

  • Identify strategies for promoting and increasing evaluation impact

What is Evaluation? (anyway!)

Research vs. Evaluation

Evaluation and Research have many similar characteristics; however, they are very different in the following ways:

Evaluation

Research

  • Intended for:

    • Program decision making

    • Rendering judgments

  • Stakeholders set the agenda

  • Primary audience for the study:

    • Program staff &

  • stakeholders

    • Intended for:

      • Adding to the existing

    knowledge base

    • Researcher sets the agenda

    • Primary audience for the study:

      • Scientific/academic community

    • Findings are:

    • Findings are:

      • Program & context specific

      • Shared on an ongoing basis

      • Intended to be broadly applicable or generalizable

      • Shared at the end of the

    Research vs. Evaluation Evaluation and Research have many similar characteristics; however, they are very different in

    Evaluation Defined

    • Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs. (Rossi, Lipsey, Freeman, 2004, p. 28)

    • Evaluation refers to the process of determining the merit, worth, or value of something, or the product of that process. (Scriven, 1991, p. 139)

    • The evaluation of educational and social programs aspires to be an institution for democratizing public decisions by making programs and policies more open to public scrutiny and deliberation. (House, 1993, p. 1)

    • Program evaluation is the systematic collection of information

    about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program

    effectiveness, and/or inform decisions about future
    5 programming. (Patton, 1997, p. 23)

    Evaluation Defined  Program evaluation is the use of social research methods to systematically investigate

    What Can Evaluation Help Us Know

    • Know-about problems: knowledge about health, wealth and social inequities

    • Know what-works: policies, programs, strategies that bring about desired outcomes at acceptable costs and with relatively few unwanted consequences

    • Know-how (to put into practice): effective program implementation

    • Know-who (to involve): estimates of clients needs as well as information on key stakeholders necessary for potential solutions

    • Know-why: knowledge about why an action is required e.g., the relationship between values and

    What Can Evaluation Help Us Know  Know-about problems : knowledge about health, wealth and social
    • 6 policy decisions

    Why Evaluate?

    • People (stakeholders) naturally make evaluative

    judgments about programs & policies, often based

    on limited information and are susceptible to biases

    • Evaluators use a set of “tools” (research designs, methods) and “roadmaps” (evaluation theories) that

    offer stakeholders’ understanding of and action in

    relation to programs and policies

    How Does One Evaluate?

    • There are many ways to approach an evaluation of a program or policy

    • Roadmaps are also called “theories” (but are really “models” or “approaches” or

    “frameworks” ) about how best to conduct an

    evaluation

    study

    • There are many evaluation theories and there is no one “right” theory or way of doing an evaluation

    • An evaluation theory offer evaluators a conceptual framework that helps to organize the procedures

    • Theories are distinguished by what is believed to be the primary purpose of evaluation

    Impact as a Motivator for Evaluation

    The organizational contexts in which stakeholder’s

    use information about programs & policies vary, so the

    demands on and approaches to evaluation should vary accordingly.

    CDC Evaluation Model

    CDC Evaluation Model <a href=www.cdc.gov/eval " id="pdf-obj-9-5" src="pdf-obj-9-5.jpg">

    What Does it Mean for an Evaluation to Have Impact?

    Weiss (2012) Says:

    • Good

    • Cheap

    • Fast

      • Evaluation can not be all three

    Evaluations with Impact

    Evaluation

    Program Activities

    Process & Findings

    Evaluations with Impact Evaluation Program Activities Process & Findings Improvements in programs that are intended to

    Improvements in programs that are intended to promote social betterment

    Evaluation Use

    The connection between the work of the evaluation and the

    program/policy activities that promote social betterment

    Evaluation Impact: A Trade Off

    • Challenge the Status Quo- potential for more fundamental change

    • Action Orientation- explicit guidance and clear direction for feasible reform

    Impact? What Do We Mean? Use

    Enlightenment/Conceptual

    • More general learning that takes place as a result of evaluation- systematic evidence makes its way into a person’s knowledge-base that is acquired informally overtime

    Instrumental

    • Evaluation findings lead to specific actions such as program continuation, expansion, revision or termination

    Process Use

    • Beyondfindings use

    • What happensto people & organizations as a result of participating in evaluation activities

    Evaluation Capacity Building

    • The primary purpose of the evaluation - intentional

    Examples of Use

    • Please describe some instances when you experience use?

      • What were the conditions?

      • What was the impact?

      • What was done to help promote use? By whom? How?

    Strategies for Promoting and Increasing Evaluation Impact

    Messy, Subtle, Complex

    • Studies have shown that evaluation use is not a straightforward, direct application of information by program leaders, practitioners, or policy makers.

    • Nutley, Walter & Davies characterize use as a complex process that is “difficult to trace and resulting in equally subtle and complex outcomes” (p. 33).

    In Search of Information? Not Very Often (Decision Accretion)

    • People tend to make decisions based on what they already know (know about)

    • People value information that comes naturally

    • People seek evidence when:

      • New issues & an “orientation” to the issue is needed

      • Important (or expensive) consequences

      • Feel underprepared

      • Seek authoritative support (position may be challenged)

    Discussion Example

    • In 1998, Mathematica was commissioned to conduct a congressionally mandated evaluation of the

    effectiveness of abstinence education programs.

    Programs receiving these funds taught abstinence from sexual activity outside of marriage as the expected standard for school-age children and could

    not endorse or promote contraceptive use.

    Discussion Example

    • What were the nature and underlying theories of the abstinence education programs supported with Section 510 funding?

    • What were the implementation and operational experiences of local communities and schools that received abstinence education funding?

    • What were the impacts of abstinence education

    programs?

    • How successful were they in changing the knowledge, attitudes, and intentions of youth?

    • How successful were they in reducing teen sexual activity among youth?

    • How did they change the risk of pregnancy and STDs?

    Discussion Example  What were the nature and underlying theories of the abstinence education programs supported

    22

    Discussion Example

    • The multi-year evaluation used an experimental design to estimate impacts on youth attitudes and

    behaviors

    • Although the evaluation produced numerous findings, two key findings were:

      • Youth in the program group were no more likely than those in the control group to have abstained from sex and, among those who reported having had sex, they had similar numbers of sexual partners.

      • Contrary to concerns raised by critics of abstinence education, youth in the program group were no more likely to have unprotected sex than youth in the control group.

    Discussion Example

    • Health bill restores $250 million in abstinence- education funds

    • By Rob Stein

    • Washington Post Staff Writer

    • Saturday, March 27, 2010

    Discussion Example

    • “But the effort came under mounting criticism when independent evaluations concluded that the approach was ineffective, and evidence began to emerge that the long decline in teen pregnancies was reversing.”

    • “During the health legislation debate in the Senate Finance Committee, Sen. Orrin G. Hatch (R-Utah) added $50 million in annual funding for five years to states for abstinence programs -- a provision that survived the tumultuous process that ensued.”

    Weiss’s I-I-I Analysis

    • Ideology- principles, values, political orientation

    • Interest- self interest

    • Information- systematic evidence + “the mess” & the least important

    • Information can be influential when:

      • people hold conflicting ideologies & interests

      • new information clarifies interests & ideologies

      • interests & ideologies are at a stalemate

    How Do We Foster Evaluation Use?

    Research on Use

    Factors identified that lead to use:

    Relevance

    Credible Evidence

    Political Factors

    Engage Stakeholders: Relevance

    • Fostering input, participation, and power-sharing among

    those persons who have an investment in the conduct of

    the evaluation and the findings; it is especially important to engage primary users of the evaluation.

    • Helps increase chances that the evaluation will be useful; can improve the evaluations credibility, clarify roles and responsibilities, enhance cultural competence, help protect human subjects, and avoid real or perceived conflicts of interest.

    Identifying Potential Stakeholder Audiences

    • Primary Audiences

      • Major decision makers, funders

      • Program staff, supervisors, managers, external constituents

  • Secondary Audiences

    • May have little or no daily contact with program but may have some level of responsibility for the program; may use results in some decision making situations (e.g., program participants or their supervisors or managers)

  • Tertiary Audiences

    • More distanced from programs inner workings; may be interested in the results (e.g., future program participants, general public, special interest groups)

  • Critical Activities

    • Consult insiders (e.g., leaders, staff, clients, and program funding sources) and outsiders (e.g., skeptics)

    • Take special effort to promote the inclusion of less powerful groups or individuals

    • Coordinate stakeholder input throughout the process of evaluation design, operation, and use; and

    • Avoid excessive stakeholder identification, which

    might prevent progress of your work

    Credible Evidence

    • Information that stakeholders perceive as trustworthy and relevant

    • Persons involved in an evaluation should strive to collect information that will convey a well-rounded picture of the program and be seen as credible by the

    evaluations primary users.

    • When stakeholders find evaluation data to be credible, they are more likely to accept the findings and to act on its recommendations.

    Credible Evidence

    • How evaluation questions are posed

    • Beliefs about truth, knowledge and knowing

    • Sources of information

    • Conditions of data collection, reliability of measurement, validity of interpretations, and quality control procedures

    • These may vary from context to context, user to user

    Credible Evidence

    Indicators

    • How will general concepts regarding the program, its context, and its expected effects be translated into specific measures that can be interpreted? Will the chosen indicators provide systematic data that is valid and reliable for the intended uses?

    Quality

    • Is the information trustworthy (i.e., reliable, valid, and informative for the intended uses)?

    Quantity

    • What amount of information is sufficient? What level of

    confidence or precision is possible? Is there adequate power to detect effects? Is the respondent burden reasonable?

    Fair Justification of Conclusions

    • Evaluation conclusions are justified when they are linked to the evidence gathered and judged against

    agreed-upon values or standards set by the

    stakeholders.

    • Stakeholders must agree that conclusions are

    justified before they will use the evaluation results

    with confidence.

    Critical Activities

    • Use culturally and methodologically appropriate methods of analysis and synthesis to summarize findings;

    • Interpret the significance of results for deciding what the

    findings mean;

    • Make judgments according to clearly stated values that classify a result (e.g., as positive or negative and high or low);

    • Consider alternative ways to compare results (e.g., compared with program objectives, a comparison group, national norms, past performance, or needs);

    • Generate alternative explanations for findings and indicating why these explanations should be discounted;

    • Recommend actions or decisions that are consistent with the conclusions;

    • Limit conclusions to situations, time periods, persons, contexts, and purposes for which the findings are applicable.

    Sharing Findings

    • An evaluation, however technically proficient it may be, is of little value if the findings are not shared in ways that connect with the natural sensemaking processes of relevant parties. This has implications not only for the dissemination and presentation of evaluation findings, but also for the potential array of evaluator roles related to evaluation influence. (Mark & Henry, 2012)

    Mechanisms for Use

    • Dissemination- present information in a manner that is accessible to potential users

    • Interaction- develop stronger links between decision- makers and evaluators

    • Social Influence- relying on influential others (experts & peers) to inform people about the study & its value

    • Facilitation- enabling the use of findings through technical, financial & organizational support

    Mechanisms for Use

    • Don’t assume that lessons learned in the course of evaluation will automatically translate into informed

    decision-making and appropriate action

    • Deliberate effort is needed to ensure that the evaluation processes and findings are used and disseminated

    appropriately.

    • Preparing for use involves strategic thinking and continued vigilance, both of which begin in the earliest stages of stakeholder engagement and continue throughout the evaluation process.

    Critical Activities

    • Prepare stakeholders for eventual use by rehearsing throughout the project how different kinds of conclusions would affect program operations and/or decisions

    • Provide continuous feedback to stakeholders regarding interim findings, provisional interpretations, and decisions to be made that might affect likelihood of use;

    • Schedule follow-up meetings with stakeholders to facilitate the transfer of evaluation conclusions into appropriate actions or decisions; and

    • Disseminate both the procedures used and the lessons learned from the evaluation to stakeholders, using tailored communications strategies that meet their particular needs.

    Thank you for your time and attention!