The Politics of Policy Evaluation

Introduction

This essay is based on a book chapter by M. Bovens, P. 'T hart and S. Kuipers
which is titled as The politics of policy evaluation. It is part of a book titled,
The Oxford Handbook of Public Policy. This chapter of the book focuses on
the evaluation part of the policy process and has a special focus on the
political side of the evaluation process. The authors explain an important
aspect of public policy that is policy evaluation. It basically talks about how
two different approaches i.e. the rational or positivist and the argumentative
approaches explain the public policy evaluation process. The chapter
explains the view of the two competing school of thoughts and along with
strengths and weaknesses of their argument. The study of different
approaches to policy evaluation gives us an understanding of how different
researchers have used different approaches of policy evaluation in different
periods of time and how effective they have been. In the first section of the
paper, Ill give a summary of the chapter; in the second section, there will be
a discussion of the approaches and arguments used by the authors and
finally the last part will have the critical evaluation of the text.

Policy Evaluation is defined by (Knill & Tosun, 2012) as comparing the

intended and actual effects of public policies and which can give insights
regarding policy outputs/outcomes/impacts. This is the most generic way of
defining policy evaluation in which the goal which had to be achieved are
compared to the actual results or output/impacts of the public policy under
evaluation. Policy Evaluation, in this chapter according to Bovens et al. is the
ex-post assessment of the strengths and weaknesses of public programs
and projects. Ex-post refers to what comes after so the Ex-post assessment
refers to the public policy evaluation that comes after the policy
implementation. The need for evaluation has always been there as it
establishes accountability, enhances legitimacy of the policy, helps
acceptance and more trust in the government but the process of evaluation
is not as simple as it seems. The chapter explains why the public policy
evaluation is much more than just the comparison of the intended and actual
outcomes of the policy. In the chapter, the author is of the opinion that policy
evaluation is mostly just political judgement which, according to the author
means that better understanding and information about policy matters can
just make it more informed but other than that, absolute scholarly knowledge
about policy evaluation cannot be gained. When public policies are adopted
and the programs are implemented, that is not the end of policy making as
the controversies over nature of problems exist even after implementation.
The need of evaluation can arise due to a number of reason like prescription
by law or by political processes. A public policy might have become highly
controversial among the public. Apart from these, policy evaluation can
provide or take away legitimacy to a certain policy. This works the same way
for accountability and acceptance of a certain policy by the public. So
keeping in view all these reasons, evaluation may be initiated to go for or
against a certain policy by the politicians. That is explained by certain
theorists like Bovens. Bovens et al. claims that a number of times,
evaluations are initiated only for political reasons. (Bovens, 'T hart, &
Kuipers, 2006) This means that the needs of the political evaluation can
sometimes only be to shut down a policy, make it controversial or to give it
more legitimacy. This is the political side of policy evaluation. The author
moves on with the difference in the ideal and real world situation of political
evaluation. Ideally speaking, policy evaluation is used as a tool for
improvement after getting all the feedback and learning. But in reality, it is
very hard to find something which actually contributes to improvement of
governance and instead of improvement, sometimes, it can make the
matters worse by obstructing the good governance process. Moreover, the
authors stress that politics of policy making is complex because it goes
beyond just adoption of policy and implementation of policy programs.
Discussion

The ideal-typical structure of an evaluation body initiates an evaluation with

a certain scope, employs some criteria, gathers information, draws
conclusions and presents its findings but a lot of variations exist behind the
scenes. (Vedung, 2009). In reality it can be a lot more different. Bovens et al.
move on with explaining the ideal world and real world of policy analysis.
Thats where two different approaches to policy analysis kick in. They talk
about two traditions and different approaches regarding policy evaluation are
seen in terms of these approaches.

Rational Policy Evaluation

The first one being the rationalistic or positivist tradition which has a focus
on value neutrality and objective assessments of policy performance. It
advocates an explicit separation of facts and values. Moreover, it ignores all
the political pressures on the policy evaluation process. For the positivists,
the information from policy analysis is assumed to be politically and morally
neutral as information. According to this perspective, information must be
collected that is free from cultural, linguistic and psychological context. Rossi
et al. (1999) also argue that evaluation should provide value-neutral
information to political decision makers. (Rossi, Freeman, & Lipsey, 1999) But
this is their idea of how the information should be. How it actually is can be
in a stark contrast. William Dunn has explained the rationalistic policy
approach as it yields policy-relevant information about the distance of factual
and actual policy performance. According to him, rational methods can be
used to build theoretical policy optimums and these are compared with
actual policy outcomes. (Dunn, 2004) In a rational-analytic approach, policy
evaluation is concerned with a thorough investigation into the
implementation and effects of public policy (Dunn, 2004). In simple words,
the aim of the positivists is to maintain a high degree of objectivity in search
of accurate information and the ultimate truth. This may sound perfect in
theory but the reality might differ as this approach has been criticized by the
argumentative approach followers as an ideal and non-practical approach
which focus more on what should be instead of what it actually is.

Argumentative Policy Evaluation

Other researchers differ from the rationalists take on policy evaluation.

Knaap (2001) explains the problem with the analytic approach being that it
assumes a lot of things to be rational and ideal which practically is not the
case. Knapp gives the example that rationalists would assume that during
the whole public policy process, the government itself will remain stable and
rational which is not the case in reality. (Knaap, 2001). Moreover, the
argumentative approach followers are critical of the rationalists on just
assuming that the evaluating process will be free from all sorts of bias and
pressure from different sources, especially the political pressure. Bovens et
al. call policy evaluation as the continuation of politics by other means which
refers to the political use of policy evaluation. (Bovens, 'T hart, & Kuipers,
2006) Which in other words, just mean that policy evaluation is just part of
political process. He argues that this case specially holds true for policies
that have become highly controversial. A number of times, its not the failed
results of the policies but certain policies being highly contested, inefficient,
having problematic implementation or simply being costly can be criticized
and evaluated for that. And the process of the policy evaluation is not just for
evaluation as such but for deeper political gains. Bovens and t Hart are of
the opinion that evaluation of controversial policies is not a politically neutral
activity. ('t Hart & Bovens, 1998) Moreover, once a policy is viewed as
controversial or failure, the responsibility has to be on someone and
someone has to be held accountable for the damage, if any is caused.
Sometimes, just in order to lessen the controversy and blame-game, policy
evaluation is done. While evaluating public policies, another problem is that
different evaluation bodies may be competing with each other for evaluation
which may include government, opposition, different stakeholders,
parliament, chambers of parliament, legal bodies etc. Form the outlook and
in an ideal world of rationalists, this multiplicity might be very useful as it will
result in more discussion, bring on more facts and more issues to discuss on
the table. But in reality, different competing evaluations tend to be diverging
widely and are non-cumulative in nature. This makes it really hard for a
number of bodies, working on a policy evaluation for different reasons to
reach a single consensual judgement.

The Political aspect:

Most of the chapter is about the difference between the two major
approaches to evaluation research and whats positive and negative about
these approaches but the focus is on the political aspect in evaluation and
how it affects the process of policy evaluation. According to Bovens et al.,
the very decision to have evaluation for a program or incidence may be part
of a political strategy. It is possible that the process of policy evaluation was
initiated in order to make sure the policy goes on or to change, terminate it.
Marston and Watts argue that there is a risk that evidence-based policy can
be misused by political elites to increase their strategic control over the
nature of problems in policy making (Marston & Watts, 2003). The implication
of this can be seen in the example presented in this chapter. Bovens et al.
use the example of Great societys Head start program to explain how the
post positivists see evaluation. The program was meant for better cognitive
development of the kids who were socially deprived. The first evaluation of
the program found out that the program had failed to achieve one of the
targets. It was just one of the several program and the impact of the
program had to be evaluated after several years not just after one year. But
the Nixon administration ignored the long term impacts of the policy and
used the evaluation as an excuse to discontinue it. Nixon welcomed the
evaluation, cheery picked the program that hadnt shown favorable results
and discontinued the program. This is a good example of how policy
evaluation can be politically motivated or influenced and how it can be used
to change the fate of the policy under consideration. For the rationalists, it
may sound fair as the intended and actual outcomes differed but they fail to
realize the fact that for that specific program, most of the impacts had to be
apparent in the long run and based on the evidence out of just one program
evaluation, the whole policy should not be changed or shifted.

The critics of the rationalist approach argue that the positivist approach to
policy evaluation is distorted by the separation of facts from values. For any
political and social phenomenon, policy evaluation will always be normative
value-laden activity which gives space to a biased evaluation. The
argumentative tradition explicitly incorporates roles of politics in the policy
performances ex-post analysis. (Bovens, 'T hart, & Kuipers, 2006). This
negates the positivist idea that policy evaluations may provide policy makers
with value-neutral objective information for policy improvements. According
to the argumentative perspective, the process of evaluation can never be
value or bias free in practice. This is because of the political side of the
evaluation process. The authors mention another approach which is called
frame-reflection approach. The author doesnt stress on this approach to be
used as a solution to the problems faced by the rationalist and
argumentative approach but mentions it to deal with the bias and value
laden judgement problem in policy evaluation. Schon and Rein mention a
frame-reflection approach to policy evaluation in which the analysts need to
reflect and reassess how they view the world. The key players in the political
arena need to communicate with the analysts using a different set of
assumptions. Reflective policy analysts may strive for a position as a
systematic, well-informed and fair minded provider of inputs to political
process of debate and argumentation that are part of controversial policy
episodes ( Schn & Rein, Frame Reflection: Toward the Resolution of
Intractable Policy Controversies, 1994).

Critical Evaluation
Although benefit of having contrasting programmatic and political evaluation
modes is that it shows disparities between a policy making entitys political
and programmatic performance. But having different evaluation bodies
competing with each other can also be problematic as theyll be having
diverging and contrasting results. His problem can be huge as it can open
Pandora boxes of more controversies instead of offering an improvement or
reform to the problems faced. The chapter although seemingly in favor of the
argumentative approach, doesnt offer a solution to this problem. Bovens et
al. has explained appropriately the differences between the two major
approaches to policy evaluation. The example of Nixon administrations use
of policy evaluation for political purposes is also appropriate in
understanding the argumentative school of thought. The authors have also
been successful in explaining the political aspect of the policy evaluation and
how it cannot be ignored as the policies always have social and political
aspects. In the chapter, the authors briefly explain the frame-reflection
approach. I think recommending a frame reflection approach as a simple
answer to the complexities of the policy evaluation is not an answer. The
frame-reflection approach has also been criticized by many like Schon and
Rein, who are of the opinion that frame-reflection approach does not always
lead to reframing of issues and the reframing does not always lead to
resolution of issues. ( Schn & Rein, Frame Reflection: Toward the Resolution
of, 1994) It is also debatable if a policy evaluator is able to develop a bias-
free frame for reflection or not as the frame reflection approach talks what
should an analyst ideally do, it is debatable if it is actually possible in real
world situations or not.

Moreover, the author seems to support the case of the argumentative at

times but he doesnt offer any solution when the result of a policy evaluation
is more confusion and blame game. If the result of policy evaluation causes
hindrance in good governance instead of improving it, as the author
mentions, how do you deal with that? I agree with the argument of the
argumentative approach that the nature of socially and politically
constructed assessments is important and relevant but he does not offer any
solution to the lack of clear or widely accepted evaluation principles and
tools for capturing this dimension of policy evaluation. Besides the problem
of lack of solutions to the problems discusses, the chapter was a good read
and despite the relatively difficult style of writing, I could make sense of the
concepts discussed. Overall, the text is well written and to the point and has
appropriate explanation for the concepts mentioned. In my opinion, the
author could use some models or figures to explain the different approaches.
Additionally, the author while explaining the different approaches does not
explain the time periods in which what approach was followed the most. In
conclusion, the chapter is an informative read on the whole political side of
public policy evaluation.
References
Schn, D., & Rein, M. (1994). Frame Reflection: Toward the Resolution of.
New York: Basic
Bovens, M., 'T hart, P., & Kuipers, S. (2006). The Oxford Handbook of Public
Policy. Oxford University Press.
Dunn, W. N. (2004). Public Policy Analysis: An Introduction. Pearson Prentice
Hall.
Knaap, P. v. (2001). Policy evaluation and outcome-oriented management.
eft.
Knill, C., & Tosun, J. (2012). Public Policy: An introduction. Palgrave
Macmillan.
Marston, G., & Watts, R. (2003). Tampering with the evidence: A critical
appraisal of evidence-based policy making. RMIT University.
Rossi, P., Freeman, H., & Lipsey, M. (1999). Evaluation: A systematic
Approach. SAGE Publications.
't Hart, P., & Bovens, M. (1998). Understanding policy fiascoes. Mark Bovens,
Paul 't Hart: Transaction publishers.
Vedung, E. (2009). Public Policy and Program Evaluation. Transaction
Publishers.
Wright, J., & Rossi, P. (1984). Evaluation Research: An Assessment. Annual
Reviews.