Evaluation and Appraisal Guidance

CONTENTS
Foreword Chapter 1: Introduction ................................................................................................................................................................. 1
Designing effective policy ............................................................................................................................................... 1 Who is this guidance for? ................................................................................................................................................. 1 Whats the point? ................................................................................................................................................................ 1

Chapter 2: Overview of Appraisal and Evaluation ............................................................................ 2

Appraisal and evaluation in the policy-making cycle ........................................................................................... 2 Who is responsible for appraisal and evaluation in the DfES? ........................................................................... 3 When should it be done? ................................................................................................................................................. 3 How much resource should be devoted to it? ......................................................................................................... 3

Chapter 3: Policy Rationale and Objectives ................................................................................................. 4

What is the rationale for government intervention? ............................................................................................. 4 Objectives and outcomes ................................................................................................................................................ 4

Chapter 4: Appraising the Options ............................................................................................................................. 6

Introduction to appraisal ................................................................................................................................................. 6 Creating and short-listing options ............................................................................................................................... 6 Identifying the costs and benefits of short-listed options .................................................................................. 7 Valuing the costs and benefits of short-listed options ......................................................................................... 8 Discounting........................................................................................................................................................................... 8 Net Present Value and Rate of Return ......................................................................................................................... 9 Equity and distribution of costs and benefits .......................................................................................................... 9 Considering risks and uncertainties.......................................................................................................................... 11 Regulatory Impact Assessment .................................................................................................................................. 11 Selecting the best option ............................................................................................................................................. 12

Chapter 5: Evaluating the Impact of our Policies ............................................................................ 13

Introduction to evaluation ........................................................................................................................................... 13 What is to be evaluated? ............................................................................................................................................... 13 How it is to be evaluated .............................................................................................................................................. 14 Contracting Out Evaluation ......................................................................................................................................... 14 Good practice in managing an evaluation ............................................................................................................. 15 Feedback and Publication ............................................................................................................................................ 15

Annex A: The Rationale for Government Intervention .......................................................... 17 Annex B: Some Additional Guidance on Valuing Costs and Benefits ............ 18 Annex C: Where Can I Get Help? ................................................................................................................................... 19 Glossary ................................................................................................................................................................................................................. 20

Foreword by the Permanent Secretary

Education, Skills, and Childrens Services make up one of the largest areas of government expenditure, and our decisions affect the lives of millions of people. In July 2004, the Department set out its Five Year Strategy for children and learners. To successfully deliver that strategy, it is important that our actions are based on a clear understanding of what works, what represents good value for money, and what does not. This requires that first, we appraise our proposals to assess which will offer the best value. And second, we evaluate our policies and strategies in order to enhance our understanding of what is working, for whom, why and in what circumstances. Indeed, we have some evaluation evidence that several Departmental policies have made a tangible and measurable difference to peoples lives. However, the challenge must always be to make a difference to many more. I commend this guidance to you as an excellent introduction to how appraisal and evaluation can help us maximise the impact of our work.

David Normington Permanent Secretary Department for Education and Skills

Chapter 1: INTRODUCTION
Designing Effective Policy
Appraisal and evaluation are vital components of sound policy making. They ensure that our actions are well thought through and are framed by evidence on what works and what does not. When we consider developing new proposals, we first need to be very clear why we are considering doing something. What is wrong with the status quo? What is the rationale for government intervention? We then need to be equally clear about what we want to achieve, specifying some very precise and measurable objectives. But what are appraisal and evaluation? Appraisal is the process of examining options for meeting policy objectives and weighing up the costs, benefits, risks and uncertainties of these options before a decision is made. Evaluation is a retrospective analysis of a policy to assess how successful or otherwise it has been and what lessons can be learnt for the future. For both appraisal and evaluation, what counts when judging the success of a policy is the difference it made compared to what would have happened instead. Establishing what would have happened instead is often the hardest bit, and the use of pilots, and particularly the use of control groups1, can be very important. This guidance will help steer you through the key considerations you must make when developing policy. The information is based on HM Treasurys Green Book which constitutes binding guidance for government departments.

Who is this Guidance For?

This guidance is important to all those of us in the Department responsible for policy development and delivery, capital projects, the use or disposal of assets, the specification of regulations or any major procurement decision. So it is important to most of us! This is general guidance on appraisal and evaluation. It is not a substitute for establishing and maintaining good working relations with your specialist colleagues as you appraise and evaluate your policies. It is crucial to consult specialists as early as possible in the policy development process. This will maximise the opportunity to add real analytical value to policy development by, for example, drawing on the best available evidence, analytical techniques and data.

What is the Point?

Ministers are responsible for deciding the priorities for public policy. We have a duty to give Ministers the best advice we can on the relative effectiveness and pros and cons of different policy options. What counts is what works. What works will be well thought out and well implemented policies, programmes, projects and regulations that deliver the desired outcomes. This really means us maximising three things:

Effectiveness having the largest impact possible on the objectives set Efficiency achieving the best outcome or maximum output in relation to the resources used Economy ensuring that the resources used are of best value

Put simply, appraisal and evaluation help us ensure that the Departments policies meet our commitment to get the best results possible from public money.
1. Similar individuals or people who are not offered the policy or programme

Chapter 2: OVERVIEW OF APPRAISAL AND EVALUATION

Appraisal and Evaluation in the Policy Making Cycle
However policy proposals come about, it is important that we are able to robustly answer some key questions. These include:

Why do we need to do anything about this? What is the justification for us acting or spending public money? What outcome are we hoping to achieve? What are the different ways we can achieve that outcome, and which is the best way? Have we sufficiently considered the risks of things going wrong? What are going to be the impacts and costs on different groups within society? How are we going to know whether we are being successful? What are we going to do to learn from this?

The Departments standard approach is to consider these questions in the ROAMEF framework, which is illustrated in Figure 1. Figure 1: The ROAMEF Policy Making Cycle

Rationale

Objectives Feedback

Appraisal

Evaluation

EVALUATION PLANNING IMPLEMENTATION Monitoring

Considering the ROAMEF framework, effective policy making starts with a clear, evidence-based statement of the Rationale for DfES intervention; what is the problem our intervention aims to solve? Are we sure that DfES intervention is the best way to solve the problem? Out of this clear understanding of the problem should flow specific and measurable Objectives that the DfES intervention aims to achieve. We use Appraisal to consider the various options of how to meet these identified policy objectives. This is a key aid to selecting the best course of action. Policy implementation follows, and during this stage we Monitor the progress and effectiveness of the intervention.

Policies should be subject to an Evaluation, the planning for which should precede policy implementation. The evaluation results should offer Feedback on the success of the policy which can inform subsequent policy decisions, and add to the DfESs knowledge base on what works and what does not.

Who is Responsible for Appraisal and Evaluation in DfES?

The primary responsibility for initiating and owning the appraisal of new policy, and the evaluation of current policy, lies with policy directorates. Ordinarily, specialists in your directorate will be on hand to assist in conducting appraisals and to advise on evaluation methodology. The department has structures in place to help ensure that the right appraisals and evaluations take place, that they are used in the development of policy, and that they address the strategic questions which need to be answered for policy to be well designed. Further details are available from the contacts listed in Annex C.

When Should It Be Done?

Neither evaluation nor appraisal is an add-on layer of bureaucracy. They should be done or developed as an integral part of the development of the proposal itself, and begin as early as possible. Appraisal should be done in time to fully inform proposals as they are considered by Ministers. Certainly, we should be clear before committing spending or other resources that the project or policy is economically justified and is the best option. You should also by that stage have clear plans for how evaluation will be carried out. So, in terms of the ROAMEF framework, before the policy gets the go ahead, you should be clear about the Rationale and Objectives and have completed an Appraisal, and have plans as to how any Monitoring, Evaluation and Feedback will take place.

How Much Resource Should Be Devoted To It?

Appraisal and evaluation will of necessity require time and financial resources from policy directorates. However because they help to improve the effectiveness and cost-effectiveness of policy, the benefits of appraisal and evaluation will normally outweigh the cost of the resources devoted to them. The Treasury asks for a comprehensive but proportionate assessment of all new policies and projects. The amount of time and resource invested in an appraisal or evaluation should be lesser or greater depending on the following factors:

How central the initiative is to the Departments objectives and broader Government policy aims The overall scale of expenditure on the policy Whether the appraisal or evaluation results will add significantly to our existing knowledge of what works Whether the evaluations results will be used to redesign or change the policy or project

You should also consider whether there is scope for joining up evaluations of policies across your directorate or indeed across the department.

Chapter 3: RATIONALE AND OBJECTIVES

What is the Rationale for Government Intervention?
The first step in the policy cycle is to establish whether intervention is necessary. We need to step back from the initial problem and to think why we need to do anything. Why is the status quo not acceptable? For example, in the absence of intervention would there be:

Insufficient investment in certain types of education or skills to secure economic prosperity, international competitiveness or societys well-being? Some would call this economic inefficiency or market failure. For example, is there evidence that the economy or society is worse off due to a lack of particular skills in the workforce? Is there evidence that our attainment or participation levels lag behind those of our international competitors? A loss of fairness, or unequal access to some types of education? For example, is there evidence that because some individuals are born into poor households, they miss out on the good start that nursery education affords?

If the answer to either or any similar question is yes, there may be justification for intervention, provided that the likely benefits of doing something exceed the costs. Rationale for policy should be evidence-based. The departments Research Portal2 is an excellent tool for marshalling pertinent research evidence and you can also seek advice from specialists in your directorate. Annex A offers more general background on the rationale for government intervention.

Objectives
The next step is to define our objectives. Understanding WHY we want to intervene (i.e. establishing the rationale) helps us understand WHAT it is that we are trying to achieve by that intervention. It helps us to clarify exactly what difference to outcomes any policy or project is intended to make, over and above what would have happened if we were not to intervene. In specifying policy objectives, it is instructive to think of a hierarchy of ultimate outcomes (for society), outputs (of the education system or from childrens social services) and inputs (into the system) as shown in Figure 2. There will usually be processes that turn inputs into outputs whilst outputs will ultimately translate into outcomes for society. Figure 2: Outcomes, Outputs and Inputs

Inputs include funding for more early years education places; increased teachers pay; connecting every school to the internet; improving support for foster carers

Processes include better teaching; better availability to information for young people and parents; better incentives to learn or stay on

Outputs include increased pupil knowledge; better early cognitive ability; better adult literacy and numeracy; greater post-16 participation; better workforce skills; more adult learners

Outcomes include reaised economic growth; higher employment; greater fairness or equality; better health or lower crime

2. The research Portal is available through Pathfinder on PRISM. The weblink is http://ntweb1/researchportal/Homepage.asp

HM Treasury will want us to justify our policies in economic terms, which means having a view of the outcomes3 which education and skills policy ultimately contributes to. It is often difficult to specify policy objectives in terms of outcomes, so we often specify them in terms of outputs. Policy objectives should not be specified solely in terms of inputs or processes (although it will often be necessary to monitor these to ensure resources are being used properly or that political commitments are being met). We are ultimately judged on the outputs we produce, and the outcomes these lead to, not the inputs we put into the system or the processes that lead to outputs. In all cases, objectives should be defined in such a way that it can be easily established after the event whether, and to what extent, they were met. Good objectives should be SMART: Specific Measurable Agreed Realistic Time-bound Finally, policy objectives should relate to the aims and objectives in the DfESs Five Year Strategy document published in July 2004, and to Public Service Agreement (PSA) targets. Indeed, your rationale and objectives statement (see below) should make reference to the DfES objectives or PSA targets your policy is furthering. See Annex C for web links to the DfES Five Year Strategy and PSA targets.

The Rationale and Objectives Statement

The Rationale and Objectives Statement is a brief document, typically 2-4 pages long, which clearly sets out

the rationale for the policy, citing relevant evidence, and the objectives for the policy, which should be SMART

All policies should have a Rationale and Objectives statement drawn up at the outset by the responsible policy team. Your analytical contacts will be able to advise you as you draft this document. The Research Portal is an excellent aide in marshalling evidence for the policy rationale, and is available through Pathfinder on PRISM. The Rationale and Objectives statement is a helpful tool in the early stages of the policy development process. It is an essential starting point for developing policy options, appraising these options and evaluating a policy.

3. See the DfES publication Education and Skills: The Economic Benefit, pages 13 to 22 for a graphical introduction to the evidence on the contribution of education and skills to ultimate outcomes. The publication is available at www.dfes.gov.uk/economicbenefit

Chapter 4: APPRAISING THE OPTIONS

Introduction to Appraisal
Appraisal is also sometimes known as economic appraisal, option appraisal, investment appraisal and even cost benefit analysis. Good appraisal helps us to decide whether intervention is necessary and, if so, the best form for that intervention to take. The purpose of appraisal is to help develop the most effective and best value for money way to meet our policy objectives. Thinking of and testing options also helps to identify and understand the range of potential actions that exist. The checklist below details the main steps you have to follow to complete an appraisal. The rest of the chapter gives more detailed guidance on how to complete the various steps. A Checklist for Completing an Appraisal

Check

Reference

1. Complete or Review Policy Rationale and Objectives Statement

Pages 4-5

2. Identify and Value Costs and Benefits of Each Policy Option

Pages 7-8

3. Complete NPV and IRR Calculations

Pages 9-10

4. Make Adjustments for Distribution of Costs and Benefits

Page 9

5. Analyse Potential impact of key risks and uncertainties

Page 11

6. Present the Appraisal and Select the Best Option

Pages 11-12

Creating and Short-listing Options

There is usually a variety of ways to achieve any given objective. For example, the objective of raised pupil attainment could potentially be achieved through, amongst other things, good financial incentives for the best teachers to remain in the school workforce, reducing non-teaching burdens on teachers, curricular reform, more collaboration amongst schools, or consultants working with teachers to improve their skills.

In some cases it might appear that there is only one policy option and that appraising this option or considering alternatives is not going to add anything useful to the foregone decision. This approach should be avoided. Appraisal can often reveal that there are better alternatives, or serious risks to a policy. At the very least, there is always the do nothing option, which must be considered as seriously as options to act or spend public funds. Even where time is short, it is better to start with a full range of options. In drawing up options, you should consider the evidence on how various policy levers or options might lead us towards the objectives. You should also consult as widely as possible in drawing up the range of options. Each option should be assessed against the do nothing option in terms of: impact on objectives acceptability and practicality likely costs and benefits Out of this outline assessment a shortlist of options may be created in order to keep the appraisal process manageable. However be aware of the risk that a process of short-listing may eliminate the optimal solution before it is given full consideration. Try to ensure that your shortlist still covers a wide range of potential action and make sure you record the reasons for rejecting an option.

Making Use of Pilots in Policy Development

Where there is little or no hard evidence about whether or not one or more options would achieve the policy objectives, think seriously about using pilots to generate this knowledge. Whilst we may want rapid national roll out for any number of reasons, we should weigh this against the fact that piloting new initiatives can help avert expensive - or indeed embarrassing - policy mistakes. The Education Maintenance Allowance provides an excellent example of a DfES policy whose pilot was used to inform the decision to roll it out nationally. A pilot allows comparison of those in receipt of a policy against others, which enables better identification of the impact of a policy. If a policy is rolled out nationally from the outset, it is difficult to know whether it is having the intended impact. Pilots can also be used to test different variants of a policy. For example, the EMA piloted a core EMA variant, and others which paid young people more or less to stay in education post-16. In addition, pilots can be used to test different ways of implementing a policy and can expose pitfalls. Finally, pilots are also an opportunity to bring important stakeholders, such as teachers, on board before national roll out.

Identifying the Costs and Benefits of Short-listed Options

To move towards a full appraisal of the short-listed options, you need to identify what the costs and benefits of the approach are likely to be over time. You should consider costs and benefits to society, not just government. Those most frequently encountered in appraisals of DfES proposals are shown in Table 1. Table 1: A Checklist of Most Common Benefits and Costs in DfES Appraisals

Benefits

Costs
Direct Costs (one-off ) e.g. policy implementation costs; cost of any buildings or equipment Direct Costs (ongoing) e.g. tuition costs; policy admin costs Indirect Costs e.g. earnings foregone by students whilst participating in post-16 study

Higher early cognitive ability Raised school attainment Better skilled adults Improved behaviour Improved child outcomes e.g. health or crime Reduced child abuse Higher employment rates More productive workers

It is important to isolate the additional benefits and additional costs. The word additional refers to an impact over and above that which would have happened under the do nothing option. In other words, the additional benefits of an option are its benefits less the benefits of the do nothing option; additional costs is similarly defined. Note that the benefits and costs of the do nothing option are not generally nothing and are not necessarily those that were occurring before the policy started. Using control groups or taking background factors into account are important in isolating additional benefits. Specialists in your directorate will be able to advise on how to best estimate additional benefits and costs in the case of your policy.

Valuing the Costs and Benefits of Short-listed Options

Once you have identified the additional benefits and additional costs, you should quantify them and then value them in money terms. The purpose of valuing costs and benefits is to consider whether an options additional benefits are worth its additional costs and to allow alternative options to be systematically compared. Actual or estimated market prices provide the first idea as to the value of benefits. For instance, the value of gaining a qualification (e.g. 5+ good GCSEs, a degree) can be estimated by comparing the earnings of people who have achieved that qualification with those who have not. Costs and benefits should be expressed at the general price level applying when the appraisal is carried out. This ensures that all future costs and benefits are in real terms i.e. they ignore the impact of inflation on prices. If you have good reason to forecast that different costs and benefits will be subject to different rates of inflation, you may have reason to adjust your benefit and cost streams accordingly. Where benefits and costs can be identified but cannot be quantified, they should be noted and mentioned qualitatively in the appraisal write-up. Valuation of benefits is an area where it would be sensible to seek advice from specialists in your directorate. Some additional guidance is also provided in Annex B.

Discounting
Most appraisals will have to compare costs and benefits which occur in different time periods (typically measured in years). Normally people prefer to receive benefits such as goods and services sooner rather than later and to pay bills or costs later rather than sooner. In other words they value 1,000 more if they receive it today than if they receive it in 10 years time. So when conducting appraisals, to make all costs and benefits comparable regardless of the year in which they occur, they need to be converted to present values Converting into present values is also called . discounting, and is done by multiplying costs and benefits by a discount factor (see next paragraph). Discounting should not be confused with any adjustment for the impact of inflation. The discount factor is calculated from the discount rate, which is defined as the rate at which the value of a pound held today falls away over time. The Treasury requires that a discount rate of 3.5% per annum is used in appraisals. At a discount rate of 3.5% or 0.035, the discount factor for a given year t, is given by the formula: Discount factor in year t = 1/(1+0.035)t So the discount factor in year 5 with a discount rate of 3.5% is 1/(1+0.035)5 which equals 0.842. The box below gives an example of how to discount a stream of benefits to give their present values.

Discounting: A worked example

It is estimated that a hypothetical policy to raise teaching standards will deliver additional benefits of 1,000 a year (after inflation) for each of its first 6 years4. These additional benefits5 in real terms are shown in the first row of Table 2 below. To express these benefits in present values or as discounted benefits, we need to multiply the actual benefits by the discount factor. The discount factor, calculated from the formula above, is shown in the second row of Table 2 for years 0 to 5. The discounted benefits are shown in the third row. Table 2: Discounting Example

Costs
Additional Benefits () Discount Factor Discounted Benefits ()

0
1,000 1 1,000

1
1,000 0.966 966

2
1,000 0.934 934

3
1,000 0.902 902

4
1,000 0.871 871

5
1,000 0.842 842

The discounted benefits can be interpreted as follows. Because we prefer receiving benefits sooner rather than later, we would be equally happy receiving a smaller benefit of 842 today as one of 1,000 in 5 years time. So, 1,000 of benefits in 5 years time is valued the same as 842 of benefit today, and 1,000 of benefits in 3 years time is valued the same as 902 of benefits today, and so forth.

Net Present Value and the Rate of Return

Net benefits at any given time are simply the benefits in that time period minus the costs in that period. Both benefits and costs need to be additional (see page 8). The net present value (NPV) of an option is the cumulative total of net benefits after they have been discounted. HM Treasury require that a discount rate of 3.5% per annum is used to calculate NPV6. In general, as the discount rate used increases, the NPV of an option falls. The Internal Rate of Return or simply the rate of return of an option is the discount rate at which the NPV falls to zero. NPV and rate of return are the primary criteria for deciding whether government action is economically justified. These also help to choose between proposed options. Investment of public money is typically justified if either of the following equivalent statements is true: 1) the NPV is greater than zero, using HM Treasurys suggested 3.5% discount rate 2) the (internal) rate of return is greater than 3.5% per annum The box on the following page gives an example of the NPV and rate of return for two options being calculated.

Equity and Distribution of Costs and Benefits

Expenditure or other policy proposals often result in both gainers and losers. Once you have calculated the NPV or rate of return for the various options, considering how the costs and benefits will be distributed across different groups for each option is very important. This is the case even if greater equity is not one of the original objectives of the programme.

4. Note that by convention, the year in which a policy starts is called year 0, and subsequent years are year1, tear 2 etc. 5. Real benefits are benefits adjusted for the impact of inflation 6. For costs and benefits more than 30 years into the future, the Green Book (pages 98-99) says lower discount rates should be used. A link to the green Book is at Annex C.

The options may have varying impacts on people of different incomes, ages, locations, ethnic origins or genders. The Departments intention to mainstream equal opportunities means that the distributional impact of proposed policies needs to be seriously considered as part of your appraisal7. Even if two options have exactly the same net benefits, if one creates a more equitable outcome, it will normally be justifiable to prefer it on that basis. The Treasury asks that a distributional adjustment is factored into the NPV calculation itself if (a) the proposal has significant distributional impacts and (b) it is possible to sensibly estimate the impact on different outcome groups. It is recommended that you seek specialist advice before attempting this.

Net Present Value (NPV) and Internal Rate of Return (IPP) Calculations: A worked example
Assume we have a policy objective to raise GCSE attainment. Two policy options are being considered, both of which cover a five year period. Option 1, a policy to raise teaching standards, has initial costs of 500 in years 0 and 1, and produces additional benefits of 300 in years 1 and 2 and of 500 in years 3 and 4. Option 2, a policy to reduce truancy, has initial costs of 700 in year 0 and 500 in year 1. It produces additional benefits of 400 a year from year 1 to year 4. The costs, benefits, net benefits and discounted net benefits of each option are shown in Table 3 Table 3: NPV Calculation example (All figures in s)

Year
Option 1 Benefits Costs Net Benefits Discounted Net Benefits Option2 Benefits Costs Net Benefits Discounted Net Benefits

NPV

IRR

0 500 -500 -500

300 500 -200 -193

300 0 300 280

500 0 500 451

500 0 500 436

474

20%

0 700 -700 -700

400 500 -100 -97

400 0 400 373

400 0 400 361

400 0 400 349

286

11%

The net benefits row is calculated as benefits less costs. The discounted net benefits row is calculated as the net benefits for a given year multiplied by the discount factor for that year8. NPV is the sum of all the discounted net benefits. As shown in Table 3, Option 1 has an NPV of 474 and Option 2 has an NPV of 286. The (internal) rates of return for each option are 20% and 11% respectively. As both NPVs are positive or equivalently both rates of return are above 3.5% - both options meet the Treasurys criterion for investment of public funds. However, since Option 1 has a greater NPV than Option 2, it would be preferred to Option 2, all things equal. The expression all things equal is important. Where options involve different costs to government, NPV estimates may need to be adjusted to reflect this. Analysts in your directorate can advise here. On a practical note, Microsoft Excel is a good programme to use to calculate NPVs and Internal Rates of Return (IRR), since it has functions that will compute these two measures for a row or column of numbers. To access the NPV or IRR functions click on the f* button in Excel and type NPV or IRR in the search box.
7. You will find a lot of useful questions to consider and advice on the Fairway site http://ntweb1/newmainstreamingwebtool/default.htm 8. Discount Factors are shown in Table 2

10

Considering Risks and Uncertainties

As much of your appraisal will be based on estimates of costs and benefits and assumptions about the links between policy and outputs, there is always likely to be some difference between what is expected and what will eventually happen. Piloting (see page 7) is one tactic to clarify the extent of costs and benefits. In any case, any appraisal should include a robust judgement of the risks and uncertainties associated with the proposals. Sensitivity analysis shows how changes in particular assumptions made in the appraisal affect costs and benefits, and therefore the NPV and rate of return. Sensitivity analyses needs to be well designed and well presented. The alternative assumptions need to be chosen carefully, to focus especially on those uncertainties which are most important to the outcome of the project. The discipline of considering alternative outcomes in a systematic way through sensitivity analysis will often feed back into new ideas and judgements about both risks and options. You should also explicitly consider whether you are being optimistic about the anticipated costs and benefits. HM Treasury require that appraisals consider such optimism bias in their sensitivity analysis. For example, any belief that past problems, or problems like them, will not reoccur should be robustly justified, and sensitivity analyses should include NPV estimates of a worst case scenario. In appraisals of capital projects, HM Treasury require that explicit adjustments are made in NPV calculations to account for optimism bias (see Annex C for references to further guidance).

Regulatory Impact Assessment and Environmental Considerations

All proposals for regulation that are likely to have an impact on businesses, charities or voluntary organisations, should have a Regulatory Impact Assessment (RIA) carried out, to help inform Ministerial decisions. Like appraisal, RIA considers the benefits, costs and risk of various policy options, so it is often worth carrying them out side-by-side. Where a proposal for regulation is likely to have an impact only on the public sector, a Public Services Threshold Test (PSTT) should first be carried out to determine whether a full RIA is necessary. The purpose of the PSTT is to ensure that full RIAs are only completed for those policy proposals likely to either impose relatively large burdens on public services or draw large political or media interest. See Annex C for links to further guidance on completing RIAs and the PSTT. All new policy proposals have to be screened for environmental impacts. DfES needs to show evidence that an initial screening on environmental issues was carried out on these policy proposals. Where this initial screening suggests that the impacts are likely to be significant, then you need to undertake an environmental impact assessment. Links to further guidance on environmental appraisals are at Annex C.

Presenting the Results of an Appraisal

The results of your appraisal should be presented in a document summarising: Policy background, making reference to the policys rationale and objectives The options considered to meet the objectives, including the do nothing case The categories of anticipated costs and benefits for each option References to available evidence justifying the anticipated costs and benefits Overall Net Present Values (NPVs) and rates of return for each option NPV or rate of return estimates after any adjustments for distribution Sensitivity analysis of the effects on NPV and rates of return of changing key assumptions Any additional costs and benefits not included in the NPV calculation Practicality and acceptability of each option The precise audience for this document will vary from case to case. Sometimes Treasury will require the information as a condition for funding a programme. Ministers should also be informed of the expected costs and benefits underlying any decision they make. The appraisal may also be published as part of a Regulatory Impact Assessment (see above).It may be sensible to bring together all this information in a matrix so that NPVs, risks, unvalued benefits and other impacts can be easily compared between options.

11

Selecting the Best Option

The analysis and presentation of the results will sometimes point to a clear-cut recommendation about whether to intervene and, if so, which option to follow. But often this will not be possible. While the NPV is the key criterion, the risks and uncertainties may bring into question how robust the NPV is. There may also be other key benefits which you have not been able to value and so are not within the NPV calculation. The selected option is likely to require further refinement. It is sensible to consider whether the better elements of other options can be grafted on to the selected option. Finally, it is important to remember that even the most justifiable and financially affordable proposals are of no value if, realistically, they cannot be implemented.

12

Chapter 5: EVALUATING THE IMPACT OF OUR POLICIES

Introduction to Evaluation
Evaluation is common sense. It is good practice. It is about assessing how successful, or otherwise, a project or policy has been. Evaluation results inform decisions as to whether existing policies or strategies should be continued or modified. On a longer-term horizon, they provide lessons to improve subsequent rounds of expenditure and policy making. Evaluation is very similar to appraisal. But rather than being based on informed predictions and forecasts like appraisal, evaluation is carried out once we know what actually happened. To effectively conduct evaluation, it should be thought about and planned before a programme is implemented. The checklist below details the main steps involved in an evaluation of a policy. Detailed guidance on these steps is either in this chapter or earlier in this guidance. A Checklist for Evaluation

Check

Reference

1. Complete or Review Policy Rationale and Objectives Statement

Pages 4-5

2. Write Evaluation Plan to Clarify What and How to Evaluate

Pages 13-14

3. Conduct Evaluation (Through Contractor or Internally)

Page 14

4. Manage Evaluation Through Steering Group

Page 15

5. Publish and Feed Back Evaluation Findings

Pages 15-16

What is to be Evaluated?
You need to clearly specify the questions the evaluation aims to answer. Ideally these should be focused on identifying the outcomes of the policy, and explaining how they came about. Evaluation focused on the impact of the project on outcomes is called summative evaluation. Summative evaluation should consider additional impact at all times (see additional benefits on page 8). For example, summative evaluation would consider whether a policy such as Specialist Schools had an additional impact on pupils attainment and attendance. It would quantify the magnitude of any such impacts. Formative evaluation can be used to determine why, how or under what conditions a policy can be best directed or implemented, in order to shape its future direction. Formative evaluation may consider, for example, whether and why the collaboration element of Specialist Schools was pivotal in instances where

13

Specialist Schools were successful in raising attainment, or in what type of schools collaboration had its greatest effect. Summative and formative evaluation should be complementary. However, as a minimum, all evaluations should include a summative assessment of the overall additional impact of the policy on outcomes. Good practice in performance management means we should continually monitor the progress of a policy towards meeting its objectives. This should involve the systematic collection and analysis of data relating to both the inputs into the policy and early indicators of its outputs. This data is sometimes called monitoring information. Whilst monitoring information can be distinct from data collected for evaluation, policy makers should try to minimise duplication in data collection for monitoring and evaluation purposes.

How it is to be Evaluated
Just as with appraisal, you should identify and measure the direct and indirect benefits and costs of the intervention over and above what would have happened in the absence of the policy. In other words, evaluation should seek to establish the additional impacts of the policy. Where possible the evaluation should include a control group to whom the activity was not applied as this will help clarify the net or additional impact of the project. You should consider how outputs and activities might be measured as part of the evaluation. Some of the data needed will be available through any monitoring information that is being collected, or through existing DfES data sources such as the National Pupil Database. Some additional data or evidence may need to be specifically collected for the evaluation.

The Evaluation Plan

The Evaluation Plan (EP) is a fairly brief document that should set out, upfront:

The questions which the evaluation seeks to answer The evaluations likely data requirements Potential evaluation methodologies Outputs expected from the evaluation The timescale within which the evaluation should report Evaluation costs

The first bullet above will typically include questions concerning the extent to which the policy has met the policy objectives as set out in the Rationale and Objectives statement (see pages 4-5). The EP is a good basis for any subsequent Invitations to Tender. You should seek advice from specialists in your directorate in drawing up the EP.

Contracting out Evaluations

In practical terms, evaluation is often contracted out to external organisations. Sometimes, evaluations are conducted by a consortium of 2 or 3 external organisations, so that the differing expertise of different organisations can be brought to bear. Evaluations can last from a few months to several years depending, amongst other things, on the policy outputs or processes being investigated. A process of competitive tender should always be used to select the organisation(s) to contract out the evaluation to. Advice on the whole of the tendering process is available from the contact listed at Annex C. When tendering, do your best to ensure that you receive at least 3-5 good quality bids. Informally contacting researchers to gauge their interest in the work before sending out the tenders, taking care to follow guidelines on such contact, is a good way to do this. Once a contract has been agreed, you should conduct regular steering group meetings attended by the policy team managing the contract, relevant specialists in your directorate, appropriate stakeholders and the contractors.

14

Good Practice in Managing an Evaluation

An evaluation can run over several years, often makes a significant call on the departments financial and human resources and can be a technical undertaking. It frequently requires the co-operation of various stakeholders within the department, in the academic research community and elsewhere. As with all major undertakings, good management is important, and as a minimum, you should consider the risks to the ultimate success of your evaluation and how to manage them. These risks will include anything that could prevent your evaluation from successfully producing credible results of use to policy makers, within a useful timescale. You will want to consider actions to manage any such risks, and you may find that a risk register9 is an appropriate tool for doing this. An invaluable resource in advising you in managing your evaluation is the analysts in your directorates, many of whom work on evaluations nearly full-time. You should seek their advice early, ideally as soon as you know a policy is being considered or agreed. You should maintain contact with your analysts throughout the life of the evaluation. At the very least, you should seek advice when the following are being considered:

drafting Evaluation Plans, and any Invitations to Tender deciding the evaluations methodology and data requirements tendering and drawing up evaluation contracts commenting on interim reports variations to contract over the course of the research publication and presentation of results

Feedback and Publication

At the very least, evaluation reports should summarise:

How effective the activity was in achieving its objectives, and why Whether the activity has been more beneficial for some groups than others, and why Suggestions for how the policy could be improved to make it more effective The cost effectiveness of the activity if practicable, a brief discussion of cost-effectiveness relative to other ways of achieving the objectives is useful Why the outturn may have differed from that foreseen in the appraisal What the results imply for future management or policy decisions

The results should generally lead to recommendations for the future. These may include changes in the way the policy is delivered, its cessation or its replacement. An action plan should be developed and executed so that the evaluations recommendations are put into practice. Under the Freedom of Information Act 2000, the Department should publish findings from all the research and evaluation it funds (including policy-funded projects). Many of the departments evaluations are published at www.dfes.gov.uk/research and currently all DfES research and evaluation is published on the last Thursday of every month. For a link to further guidance on DfES publication arrangements, see Annex C. You should make efforts to disseminate evaluation findings widely inside and outside the Department as appropriate. To help you gauge whether you are obtaining good value for money from your evaluation, you should assess, perhaps 6-12 months after publication, the impact your evaluations results have had on policy development in your area and beyond.
9. This is a well-known tool for prioritising risks according to (i) each risks likelihood and (ii) the scale of the adverse impact should each risk happen.

15

To end this chapter, the box below provides a case study of a well-executed evaluation and pilot.

Case Study of the Evaluation of the Education Maintenance Allowance Pilot

Pilot and Policy Background
The Education Maintenance Allowance (EMA) pays 16 to 19 year olds who meet the eligibility requirements up to 30 per week to stay in further education. It was first piloted in 15 LEAs in September 1999, and expanded to a further 41 areas in September 2000; this brought the EMA pilots coverage to 30% of the country at that time. The purpose of the pilot and its evaluation, was to test the effectiveness of offering a financial incentive to young people to stay on in education and to test which particular variant of the eight piloted was most effective.

Evaluation Background and Methodology

The DfES commissioned an evaluation from a consortium of research organisations. The statistical evaluation design was a longitudinal cohort study involving large random sample surveys of young people (and their parents) in 10 EMA pilot areas and 11 control areas. The use of a control group was central to estimating EMAs additionality. The fact that the methodology was well-designed and robust meant that reliable quantitative estimates of the EMAs additional impact on participation could be established.

Evaluation Results
The EMA evaluation results estimated that the overall impact of EMA across England would be to raise year 12 participation in full time education by 5.9 percentage points for eligible young people (and 3.8 pp for whole cohort), and year 13 participation by 6.9 percentage points for eligible young people (4.1pp for whole cohort). The design of the evaluation as far as data collected has meant that these results can be disaggregated to see whether EMAs impact has been greater for high versus low attainers, or for high versus low social class individuals. Other breakdowns are possible. The results were made publicly available some evaluation reports are on the DfES research website at www.dfes.gov.uk/research. They can be accessed by typing EMA into the free text search box on this site.

Feedback of Evaluation Results

The evaluation results have been used by policy teams in the policys development. Following favourable results from the EMA evaluation, the extension of EMA to the rest of the country from 2004 was announced. The press notice of the Secretary of States announcement suggested that the pilot and the evaluation results were important elements of the decision to roll out nationally. The press notice is at http://www.dfes.gov.uk/pns/DisplayPN.cgi?pn_id=2004_0053

16

ANNEX A: The Rationale for Government Intervention

A well-established result from economic theory is that a free market results in an outcome that maximises societys welfare, given certain conditions. However, breach of these conditions is not uncommon and is called market failure. Market failure, along with considerations of equality, is a major rationale for government intervention or expenditure in any area including education. You should consider whether the main market failures that pertain to education relate to your policy idea. Your Rationale and Objectives statement should reflect one or more of these. The market failures most important in education are: 1) When the benefits of an activity, in this case education, extend beyond the individual taking part in the activity. We call the market failure in this case positive externalities. Some economists believe, for example, that education not only makes the individual more economically productive, but also makes those they work with more productive. A parents education level is also an important determinant of their childs educational aspirations and ultimate level. As positive externalities are associated so many potential educational investments, we should consider the relative size of our policys externality compared to other potential policy candidates. Basic skills policy and primary schools expenditure, for example, are partly justified by the positive externalities of individuals being numerate and literate. 2) When highly able, but poor, individuals cannot afford educational investments. The market failure here is that few private lenders will offer loans to individuals wishing to invest in their education, in the way they would for other investments such as home improvements. The result is that society as a whole has a sub-optimal level of education. The market failure here is one of missing credit markets. Policies such as universal free school education, subsidies in early years and post-16, and student loans have this as part of their justification. 3) When individuals or their parents are not well-informed about the benefits of education. The result is that the individuals may drop out at the earliest possible opportunity or not try hard enough in school to get good grades. Some parents may not send their children through early years education. The market failure in this case is sometimes referred to as imperfect information. Policies such as Connexions and Aim Higher have this market failure as part of their justification. Economist colleagues in your directorates will be able to help you think through the links between the Rationale for your policy and the market failures it may be trying to tackle.

17

Annex B: Some Additional Guidance on Valuing Costs and Benefits

Labour Costs - Costs relating to labour or peoples time require particular care. Firstly, note that wages and salaries typically rise at a greater rate than the general inflation rate, and this should be taken into account when forecasting costs and benefits. Secondly, measures of staff and student time should include not just wages or foregone earnings but all other employment costs such as pensions and National Insurance. The department presently adds 25% to gross earnings to account for these non-wage labour costs . Transfer Payments These are transfers of money from one party to another, without the latter offering a good or service in exchange. Examples are the payment of benefits and grants from government to individuals. When appraising the effectiveness of economic resources, transfer payments should not be included in the NPV calculation, but should be reported separately in the appraisal documents. Non-marketed benefits - Where costs (and benefits) are non-marketed that is do not normally have a , readily quoted price, it is still necessary to consider whether it is possible to value these in real money terms. For example, there is no market for peoples time and therefore no readily available price for it. However, the monetary value of an hour of an individuals time could be estimated by considering the gross hourly wage that they could be expected to command in the labour market.

18

ANNEX C: Where can I get help?

Help on appraisal and evaluation can be obtained from specialist staff and economists in your directorates. The Analytical Divisional Managers in your directorates should be able to point you in the right direction if you are in doubt. General queries on appraisal and evaluation of policy or this Guide should be addressed to ea.guidance@dfes.gsi.gov.uk . Further written guidance on appraisal and evaluation, is at www.hm-treasury.gov.uk/greenbook For a detailed note on the rationale for government intervention in education, contact ea.guidance@dfes.gsi.gov.uk Additional guidance on evaluation is available in The Magenta Book, available at http://www.policyhub.gov.uk/evalpolicy/magenta/guidance-notes.asp This source will help users to determine what constitutes high quality work in the field of policy evaluation and analysis. For information on the ultimate outcomes that education and skills policy helps to influence, see Education and Skills: The Economic Benefit, pages 13-22, at www.dfes.gov.uk/economicbenefit If you are considering a pilot of your initiative you should consult the Cabinet Office publication The Role of Pilots in Policy-Making available at www.number-10.gov.uk/su/gsr/html/index.htm If you need to correct your costs and benefits for inflation, you can refer to HM-Treasurys GDP Deflator time series which is available at: www.hm-treasury.gov.uk/Economic_Data_and_Tools/GDP_Deflators/data_gdp_index.cfm For the DfES Five Year Strategy document published in July 2004, see www.dfes.gov.uk/5yearplan On adjustments for optimism bias in appraisal of capital projects, see pages 30 and 85-6 of the Green Book, available at www.hm-treasury.gov.uk/greenbook For the full guidance on Regulatory Impact Assessment see the departments RIA internet site which can be found at: http://www.dfes.gov.uk/ria/ The site includes copies of RIAs currently out for consultation, plus an archive of previous RIAs. The site also includes links for guidance on completing RIAs and the Public Sector Threshold Test. On Environmental Appraisal see www.sustainable-development.gov.uk/sdig If you require guidance on tendering and contracting evaluation see http://ntweb1/eorp/comm_guide/GUIDE/intro%20mk2.htm For guidance on DfES publication arrangements see http://ntweb1/eorp/comm_guide/GUIDE/printing/newarrangements.htm

19

GLOSSARY
Additionality Appraisal
An impact arising from an intervention is additional if it would not have occurred in the absence of the intervention The process of defining objectives, examining policy options and weighing up the costs and benefits, risks and uncertainties of those options before a decision is made Data, typically on outcomes, collected immediately before the commencement of a policy, to enable comparison with data collected after the policy has had time to have an effect. A group of individuals who do not receive a policy, but have similar characteristics to those individuals who do receive the policy

Baseline Data

Control Group

Cost Benefit Analysis Comparison of costs and benefits to see if the benefits justify the costs. Cost
Benefit Analysis will typically result in a net present value or a rate of return

Deadweight Discounting Discount Rate Displacement Evaluation Leakage Net Present Value Optimism Bias Regulatory Impact Assessment Sensitivity Analysis Substitution

Expenditure to promote a desired outcome which would have occurred anyway, without the expenditure A method used to convert future costs or benefits to present values using a discount rate The annual percentage rate at which the present value of a future pound is assumed to fall away over time The degree to which increases in the desired outcome are offset by consequential reductions elsewhere Retrospective analysis of a project or policy to assess how successful or otherwise it has been, and what lessons can be learnt for the future Benefits to a policy which accrue to people outside the spatial area or group which was the intended beneficaiary. The discounted value of a stream of future costs and benefits The systematic tendency for appraisers to be optimistic about key project parameters, including capital and operating costs, and benefits delivery An assessment of the impact that regulatory proposals are likely to have on business, charities and the voluntary sector. RIAs consider all available options including non-regulatory alternatives. Analysis of the effects on an appraisals results of varying the projected values of important variables Where a policy creates the desired impact on person A, but at the expense of removing that desired impact from person B

20