The Policy Analysis Process

Goals of Policy Analysis

Reducing uncertainty and providing information for decision makers in the public arena As systematic evaluation of the technical and economic feasibility and political viability of alternative policies, strategies for implementation, and consequences for policy adoption

Policy Analysis Process

Approach to problems that is logical, structured, valid, and replicable Generation of feasible courses of action A search for information and evidence of benefits and other consequences of courses of action In order to help policy makers choose the most advantageous policy action

Evaluating Alternative Policies or Programs

Policy Analysis Process

Ex-Ante
(rational model)

Ex-Post
(program evaluation)

Evaluating Alternative Policies or Programs

Ex-ante

evaluation
future states

projecting

with and without policy

or program

Evaluating Alternative Policies or Programs

Ex-post
Did

evaluation

policy or program achieve policy or program

its objective?
Was

efficient, equitable, and

politically acceptable?

Using the Scientific Method in Social Science Policy Analysis Framework

will involve a RESEARCH PLAN

From Inspiration to Legislation

Research is more an art than a science. Discovering the important facts, sorting out superfluous information, and assembling the best sources requires knowledge, creativity, and a bit of luck. Although its impossible to create a simple how-to list to the steps to faultless policy research, certain core principles guide the most effective legislative research. Black, p. 14

RESEARCH PLAN

Problem definition

motivation for the analysis

defining the policy problem defining the analysts problem

RESEARCH PLAN

Background research

legislative history
political environment previous policy efforts other relevant information

RESEARCH PLAN

Research design

specification of research objectives outline of informational needs specifications of methods to be used

RESEARCH PLAN

Research implementation

data collection data tabulation results presentation analysis of results

Conclusions and recommendations

Step 1:

Problem Definition

From Inspiration to Legislation

Often ideas for legislation begin with a legislators general sense that a problem exist though he or she doesnt have much background information that points to a solution. Some people do try to create solutions without defining problems first, but it makes far more sense to identify the nature and extent of the policy concern that needs to be addressed before designing a potential solution.

Problem Definition

Policy problem:

a meaningful definition of the problem and proposed policy solution so that it is amenable to systematic research

Analysts problem:

recasting the problem situation into practical, manageable, and actionable framework for analysis

Problem Definition Issues

Complex problems with many facets Definition changes as you investigate the problem

Stated objective not clear and highly ambiguous

Policy Problem: EXAMPLE

Background

Violent crime involving weapons on public school property has been increasing at an alarming rate during the past 10 years.
Would installing metal detectors at school entrances contribute to reducing violent crime on school property?

Policy problem

Analysts Problem: EXAMPLE

What are the characteristics of school crime? What have other school districts done to address the problem? How do the teachers and parents view the proposed policy? What will the policy cost? What alternative policies exist to address the problem?

Developing an Analysts Problem Statement:

Think about the problem

Delineate the boundaries of the problem

Develop a fact base Lists goals and objectives Identify policy parameters Develop problem statements

A Good Analysts Problem Statement

Expresses the problem in a meaningful way Eliminates irrelevant issues and information

Focuses on critical, actionable aspects of the problem

Step 2:
Background Information

From Inspiration to Legislation

Any piece of legislation is only as powerful and effective as the depth of understanding that inspires the bill. Without a clear knowledge of the past and present legislation on related issues and without systematic research that uncovers the depth and complexity of the policy at hand, no relevant legislation can be developed. Black p. 12

Legislative History

Local
County

State
Federal

Courts

Political Environment

Taking into consideration information about Policy Actors or people and groups in the Policy Space during the design

and evaluation of a policy

such that the policy will achieve the requisite political support.

Political Environment An Evaluation of

Actors/action groups

Motivations
Beliefs

Resources
Location

From Inspiration to Legislation

An interest group is any group that is based on one or more shared attitudes and makes certain claims upon other groups or organizations in society for the establishment, maintenance, or enhancement of forms of behavior that are implied by the shared attitudes.

Political Environment An Evaluation of ..

Evaluate the policy from both the expanders and detractors point of view in term of:

Incidence/severity Causality Characteristic of the problem population

Values underlying the issue How they frame the solution

Step 3:

Research Design

A. Objectives of the Research

Clear

and concise statement of intent of the analysis following the outline in the analysts problem statement of evaluative criteria for judging results of the analysis

Establishment

Research Objectives: Example Violent Crime in Schools

What have the trends been in violent crimes on school property during the past decade? Where are these crimes committed geographically? What type of crimes are committed? Who commits these crimes? When are they committed? What type of weapons are used on school property? How are the weapons entering schools? How have other school districts dealt with the problem?

Establish Evaluative Criteria

Measures, rules, and standards of judgement used to guide decision making Criteria: specific statements about dimensions of the problem that will be used to evaluate alternative policies or programs

Establish Evaluative Criteria

Costs of policy

Benefit in terms of crime reduction

Long-term effectiveness in reducing crime Externalities or policy Administrative ease of implementation

B. Outline informational needs

What

data do you need to answer the analysts questions? Where will you obtain this information?

C. Methods to be used
How will the data be structured? What type of comparisons will be made? How will you evaluate your results? How will results be quantified and compared? What type of conclusions will be drawn?

Remember to always.
Identify existing alternative policies or programs

Remember to always.
Evaluate alternative policies or programs in the same way you evaluate the policy problem

Step 4:
Research Results and Analysis

A. Reporting Results
structured logical clear factually

according to research objectives

correct

Transforming Data into Useable Information

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

90 80 70 60 50 40 30 20 10 0 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr East West North

B. Analysis of your data

well

reasoned on data presented

based

logically

& critically evaluated

clearly delineated

alternatives

C. Cost-benefit analysis

Identification of the benefits

individual & social, measurable and not

Identification of the costs

individual & social, measurable and not

Quantification of measurable costs and

benefits Comparison of measurable costs and benefits

taking into account time value of money

Conclusion of the cost-benefit analysis

D. Policy Externalities and Implementation Problems

Unintended side effects

Property devaluation in school district Students skip school to avoid detection/because they cant protect themselves

Unintended offsetting behavior

Implementation problems

Location of detectors, maintenance, other exits from school property, etc.

Step 5. Conclusions

Conclusions
Regarding

proposed policy or program Regarding alternative policies and programs

Step 6:
Recommendations

Practical example
California Ignition Interlock Pilot Program

Step 1:

Problem Definition

Background information: DUI in California

1982 350,000 arrests for DUI or

about 1,000 per day

1986 state had > 2,500 alcohol-

related traffic fatalities

1/3 convicted DUI drivers have

previous convictions

Ignition Interlock System (II System)

High-tech device installed in a car which prevents operation of the car if person driving has a breath alcohol level of a certain specified level

Step 1: Policy Problem

Policy

Problem

Would

the II System be an effective sentencing option in reducing recidivism in the case of DUI convictions?

Step 1: Analysts Problem

What are the characteristics of recidivists? What are the statistics on recidivism in CA? How effective is the II System in reducing recidivism? How much does it cost to implement? How easy would it be to implement as a sentencing option? How would judges react to the new policy? How would DUI offenders react to this policy?

Step 2: Background Information

Background
Farr-Davis

Safety Act 1986

Californias

II Pilot Program

(ex-ante demonstration project)

Farr-Davis Driver Safety Act of 1986

The Farr-Davis Driver Safety Act of 1986 made California the first state in the nation to pass legislation authorizing the use of ignition interlock devices as a condition of DUI probation. The law also authorized the California Office of Traffic Safety to plan and administer a pilot program in four representative counties to test the effectiveness of the devices in reducing DUI recidivism, and to provide information on its implementation.

Step 2: Political environment

State DMV Courts, judges, probation officers Manufacturers of device Installers MADD/SADD Person convicted ?

Step 3: Research Design

Objectives
To design an experiment to test the effectiveness of the II System as a sentencing option in reducing recidivism To conduct background research on the phenomenon of recidivism (time, occurrence) To undertake interviews with stakeholders to assess their perceptions of the sentencing option

Step 3: Research Design

DUI Convictions
Sentenced by judge to experimental groups Sentenced by judge to control groups

Sentenced to II program

Other sentencing option

Recidivates

No further offences

Recidivates

No further offences

Step 3: Research Design

Evaluative Criteria

Effectiveness Cost Ease of implementation

Informational Needs

Characteristics of the samples (demographic, arrest records, blood alcohol levels at conviction, etc) Data from II Systems device Court data Interviews with stakeholders (judges, convicted individuals, etc.)

Step 4.
Research Results and Analysis

Step 4. Research Results and Analysis

Experimental Group County Recidivism Rate Control Group Recidivism Rate

-----------------------------------------------------------------Alameda San Diego Sonoma 17.7% 11.2% 15.4% 8.2% 18.4% 27.7%

Santa Clara 14.6% 13.1% ------------------------------------------------------------------Total 13.6% 16.7%

Projected vs. Actual Reconvictions

100 90 80 70 60 50 40 30 20 10 0 97 79

46

40 22 25 22 12

Projected Actual

14 6 All Alameda counties San Diego Santa Clara

Sonoma

Policy Externalities and Implementation Problems

Judicial bias in sentencing Resistance by judges Affordability of devices Ownership of cars Bypassing II devices Borrowing the cars of others

Step 5. Conclusions

Step 6. Recommendations

Actual Policy Outcome

California passed legislation extending authority for using the II System device throughout the state as a DUI probation option

Another Practical Example

Impacts of Abstinence Education on Teen Sexual Activity

Step 1:

Problem Definition

Background Information:

Approximately half of all high school students report having sex More than one in five students report having had four or more sexual partners by the

time they complete high school

One quarter of sexually active adolescents nationwide have an STD Many STDs are lifelong viral infections with no cure

Personal Responsibility and Work Opportunity Reconciliation Act of 1996

Title V Section 510 Authorized Abstinence Education Programs as one of its legislative centerpieces 1998 Congress authorized a rigorous evaluation of the education programs Ex-post policy analysis (evaluation)

Funded programs had to meet certain guidelines

Four major programs formed the target of the evaluation

My

choice, My Future!
VA

Powhatan, Miami,

ReCapturing

the Vision

FL

Families

United to Prevent Teen Pregnancy (FUPTP)

Milwaukee,

WI MS

Teens

in Control Group

Clarksdale,

Step 1: Policy Problem

Policy
Do

Problem

abstinence education programs reduce adolescent sexual behavior and promote healthy behavior among teens?

Step 1: Analysts Problem

Do the programs have an impact on teen abstinence? Do the programs have an impact on teen pregnancy rates? Do the programs reduce teen risky sexual behavior? Do the programs reduce the risk of STDs? How do the programs impact knowledge of risky behavior and STDs? Are abstinence education programs more effective than comprehensive sexual education programs in reducing these behaviors?

Step 2: Background Information

Background
PRWORA

1996

Title

V, Section 510

(ex-post evaluation project)

Step 2: Political Environment

School officials and teachers Parents Religious groups Legislators (welfare reform) ? ?
Politically/religiously charged program

Step 3: Research Design

Objectives

Conduct a multiyear, rigorous, experimentally-based impact evaluation of four abstinence education programs around the country. Estimate the effect of abstinence education programs on:

Teen sexual abstinence Teen pregnancy Age at first intercourse Rates of unprotected sex Number of sexual partners Related knowledge outcomes

Step 3: Research Design

AY=1999-2000, AY=2000-2001, AY=2001-2002

Grade and middle school children

Random assignment

Received abstinence education program (N=1,209)

Control group: Comprehensive sexual education program (N=848)

Four surveys: baseline/program start, three follow-up surveys spanning 42 -78 months

Outcomes

Outcomes

Step 3: Research Design

Evaluative

Criteria
in terms of

Effectiveness

outcomes

Informational

Needs

Characteristics

of sample Data from four surveys

Step 4.
Research Results and Analysis

Abstinence

Number of sexual partners if not abstinent

Knowledge of STDs and Risky Behaviors

Unprotected Sex and Outcomes

Step 5. Conclusions

Conclusions

None of the programs had statistically significant impacts on the rate of sexual abstinence No differences in the number of sexual partners once sexual activity was initiated No affect on age at which sexual activity was initiated No difference in rate of unprotected sex No difference in rate of teen pregnancy Statistically significant difference in ability to correctly identify true STDs

Step 6. Recommendations