Please cite this paper as:

OECD (2009), “Evaluation of Programmes

Concerning Education for Entrepreneurship”,
report by the OECD Working Party on SMEs and
Entrepreneurship, OECD.

Evaluation of Programmes
Concerning Education for
Entrepreneurship

OECD Centre for Entrepreneurship,

SMEs and Local Development (CFE)
 FOREWORD

This report presents the output of work on the Evaluation of Entrepreneurship Education Programmes
undertaken in 2007-2008 by the OECD Working Party on SMEs and Entrepreneurship (WPSMEE), with
the support of the Spanish authorities.

The document was prepared by Malcom Maguire (United Kingdom) in co-operation with Mariarosa
Lunati, OECD Centre for Entrepreneurship, SMEs and Local Development (CFE), under the direction of
Marie-Florence Estimé, CFE Deputy-Director. An informal Steering Group of WPSMEE delegates
[Denmark, New Zealand (Co-chair), Spain (Co-chair) and United Kingdom] provided guidance to this
work.

2
 TABLE OF CONTENTS

EXECUTIVE SUMMARY .............................................................................................................................4

I. INTRODUCTION ..................................................................................................................................10
II. AIMS AND OBJECTIVES OF THE STUDY ...................................................................................10
III. DESIGN OF THE STUDY ................................................................................................................11
IV. EDUCATION FOR ENTREPRENEURSHIP PROGRAMMES.......................................................13
V. EVALUATION ..................................................................................................................................22
VI. IMPACT AND EFFECTIVENESS OF EVALUATION ...................................................................25
VII. EVALUATION MATRIX..................................................................................................................26
VIII. LIST OF REFERENCES ....................................................................................................................29
IX. RESOURCES TOOLKIT ...................................................................................................................51
X. CONCLUSIONS AND RECOMMENDATIONS .............................................................................52
REFERENCES ..............................................................................................................................................54
ANNEX. EVALUATION OF ENTREPRENEURSHIP EDUCATION PROGRAMMES: RESOURCES
TOOLKIT ......................................................................................................................................................58

3
EVALUATION OF PROGRAMMES CONCERNING EDUCATION FOR ENTREPRENEURSHIP

EXECUTIVE SUMMARY

Although the importance of education for entrepreneurship, or enterprising behaviour, has been
widely acknowledged, this has not necessarily been reflected in the systematic application of evaluation of
education for entrepreneurship programmes.

The overall aim of the study, carried out by the OECD Working Party on SMEs and Entrepreneurship
(WPSMEE), was “to strengthen the culture of evaluation within entrepreneurship education and to provide
guidelines for evaluating in order to gain a better understanding of how to promote entrepreneurship
education”. The objectives accompanying this aim were:

 to emphasise the importance of the role of evaluation

 to strengthen the case for supporting education for entrepreneurship activity

 to enable those concerned with the quality and impact of provision to be better able to carry out
evaluations

 to deepen the understanding of the impact of education for entrepreneurship programmes

 to align with the OECD Framework for the Evaluation of SME and Entrepreneurship Policies and
Programmes

 to encourage the development of commonality and comparability in the evaluation of education

for entrepreneurship programmes

The study comprised two phases:

Phase 1 focused on an overview of research methods by which education for entrepreneurship

programmes might be evaluated, and involved an analytical literature review and the
development of an evaluation matrix.

Phase 2 saw the production of practical tools and materials, notably a resource toolkit and a list
of evaluation studies, along with a further developed evaluation matrix.

4
Education for entrepreneurship programmes

Education for entrepreneurship is concerned with the inculcation of a range of skills and attributes,
including the ability to think creatively, to work in teams, to manage risk and handle uncertainty. This is
underpinned by the recognition that changing „mindsets‟ is fundamental. It can be described as being “part
of the entrepreneurial pipeline which starts in education, runs through research to business”.

As such, the effects or impact of the programme may not become apparent until well after the
completion of participation in the programme, and therefore may require a longitudinal element in any
evaluation of impact.

Education for entrepreneurship programmes can be delivered at different levels of the education
system, as well as by private and voluntary organizations. Also, it is important to recognize that
entrepreneurial skills and attitudes can be applied in: work organizations of all types; non-work settings,
such as voluntary work or organizing clubs and societies; and domestic or social activities.

There is a multiplicity of activities which can be subsumed within the category of education for
entrepreneurship. Broadly, the aims and objectives can be divided into the following categories, which are
not mutually exclusive;

 The acquisition of key (or core) skills;

 The development of personal and social skills; and

 Skills relating to business start-up or financial literacy.

Clearly, the aims and objectives of programmes determine the outcomes which are sought, and which,
in turn, provide indicators which an evaluation should seek to measure or assess. They can be delivered in
a variety of forms within different countries, often ranging from programmes delivered nationally, to
individual, one-off events delivered locally. A broad distinction can be made in the forms in which
programmes are delivered between:

 Education-based programmes, wherein education for entrepreneurship may be taught as a distinct

subject, may be integrated more widely into the curriculum, or may involve the setting up of
mini-enterprises;

 Award schemes; and

 Partnership schemes

Education-based programmes are often targeted at young people in secondary school, or at university
undergraduates. There are, however, some examples of programmes for pupils at primary school level,
such as those run by Junior Achievement in the United States, and the New Zealand Primary Enterprise
Programme (PrEP).

Courses and programmes associated with entrepreneurship, many of which contain a strong business
focus, are delivered at university level. For example, the Berger Entrepreneurship Program at the
University of Arizona, which was originally introduced in 1983, draws on staff to teach the programme
from the departments of finance, economics, marketing, and management. The core courses of the
curriculum include competitive advantage, venture finance, market research and business plans
development.

5
 The idea of education for entrepreneurship being embedded within the broader school curriculum has
been recognised and adopted widely. For example, programmes which have entrepreneurship as a
curriculum goal have been launched in New Zealand. This approach is often predicated on the need for a
„whole school‟ design to be adopted. In this case, education for entrepreneurship provides “an underlying
basis for delivering education across all areas of the curriculum”.

Programmes frequently have a „hands-on‟ approach, where students set up and run „mini-companies‟.
A report on the use of mini-companies in secondary education (European Commission, 2005) stated that
“the objective of a student company is to develop either a real economic activity on a small scale or a
realistic simulation of an economic activity”.

An example of an award scheme is the ASDAN awards scheme in the UK, where students receive
credits for completing challenges in a number of areas, including Science and Technology, Work Related
Activities, and Information Transmission. In Australia, the Plan Your Own Enterprise Competition is
designed to raise students‟ awareness of small business management and planning principles and practices.
It is targeted at Year 11 students and consists of a business planning competition.

An example of a partnership scheme is Young Achievement Australia, which offers programmes

which provide a link between education and industry. A Business Skills Programme, which is facilitated
by mentors from the business world, is delivered outside school to groups of 15 to 25 young people for two
hours a week over 16 to 24 weeks. They are required to “create, manufacture and market a product or
service in a competitive environment”, thereby going through the stages of a business cycle “and take
responsibility for all essential business processes, from selling shares and raising capital to liquidating the
company”.

Evaluation

Although this study took account of, and was aligned with, the OECD Framework for the Evaluation
of SME and Entrepreneurship Policies and Programmes, the nature of education for entrepreneurship
programmes, with the emphasis on education, differentiates the focus of evaluation of these programmes
from those of other SME related activities and processes.

Given that the aim of programmes is to generate a shift in attitudes towards entrepreneurship, it
becomes difficult to ascribe quantifiable measures, so that, instead of „hard‟ outcome evidence (such as the
numbers initiating a business start-up), an attempt has to be made to gauge „softer‟ outcomes (such as
changes in attitude).

A key starting point is the belief that evaluation should be viewed as vital in order to ensure that
optimum benefits are derived from a programme. It is important to understand that the lessons learned
through the evaluation process, including why things did not work, and what were the unanticipated
consequences or side effects, are as valuable as the identification of exemplary practices. It should also be
acknowledged that different sets of participants will have different sets of goals and expectations from the
programme.

There is no single approach to the evaluation of education for entrepreneurship programmes, and
therefore no single model which can be applied in all situations. A combination of qualitative and
quantitative approaches may be used. For example, quantitative measures of success, akin to traditional
output-related performance indicators, may be used in conjunction with qualitative assessments of
attitudinal or perception shifts.

Evaluation can have both backward and forward looking purposes. It can be designed to tell us what
outputs and outcomes were generated by a project/programme (what is called summative evaluation). But

6
it can also explain how, why, and under what conditions a policy intervention worked, or failed to work
(i.e. formative evaluation). Formative evaluations are important for determining the reasons for effective
implementation and delivery of policies, programmes or projects. For example, the findings of a formative
evaluation process can provide valuable feedback on issues such as the planning and design of courses, as
well as aspects of the content and the teaching methods employed.

A number of methodologies may be considered when undertaking an evaluation of education for

entrepreneurship programmes. These include:

 The experimental approach - where people are randomly assigned to either a „treatment‟ group,
composed of those participating in some form of education for entrepreneurship activity, or a
„control‟ group composed of individuals who do not participate in the activity;

 Non-experimental methodology – which often involves a before and after comparison of the same
individuals; or a comparison of independent groups of people, one of whose members are
exposed to a treatment, to a similar group not exposed to the treatment;

 Matching – which attempts to pair each individual in the treatment group to a member of the
control group who has similar characteristics;

 Propensity Score Matching (PSM) – which makes it possible to match along a single measure
(the propensity score), which summarises these differences;

 The difference-in-difference approach – which combines matching with before and after
treatment comparison.

When considering methods to evaluate the long-term impact of these programmes, the difficulties of
establishing causality should not be underestimated.

A principal requirement for evaluating an impact is establishing that a difference in one, or more,
relevant outcomes is caused by a particular treatment. This may require the establishment of a
counterfactual.

Impact and effectiveness of evaluation

Examples can be found of positive impacts emanating from education for entrepreneurship
programmes. The evaluation of the Berger Entrepreneurship Program, delivered at the University of
Arizona, found that participation in the programme had a positive impact in terms of: risk-taking and the
formation of new ventures; increasing the likelihood of becoming self-employed; income; the growth of
firms; promoting the transfer of technology from the university to the private sector; and, less strongly, job
satisfaction.

Two studies conducted in Ireland, one of which had a considerable longitudinal element, also
concluded that, over time, benefits can be derived from education for entrepreneurship programmes.
However, the ability of education for entrepreneurship programmes to elicit positive outcomes will be
highly dependent on the quality and appropriateness of the programme delivered.

7
Evaluation matrix

The evaluation matrix provides a guide for those intending to undertake evaluations of education for
entrepreneurship activities. In particular, it indicates appropriate methodologies to adopt and affords
reference to examples of evaluations.

The two main axes for the matrix comprise:

Types of programmes. This axis details the variety of programmes which can be subsumed under the
three broad categories of:
 Learning entrepreneurial skills and attitudes
 Enhancing business start-up and entrepreneurial behaviour
 Development of business start-up and entrepreneurial skills

Evaluation requirements. Against each type of programme, the matrix identifies:

 Key objectives – the objectives of the programme
 Evaluation questions – the main issues which the evaluation is seeking to address
 Required information – the evidence which the evaluation will need to obtain to answer the
questions
 Evaluation methods – the most appropriate methods to achieve the objectives of the evaluation.

List of references

The list of references is intended to provide examples of a variety of evaluations which have been
conducted on education for entrepreneurship programmes, covering a range of programme types,
evaluation methodologies and cultural contexts.

Each of the studies included in the list of references has an identification number which can be used to
refer the reader to examples from evaluations which are most relevant. One of the features of this list is
that it is capable of being updated, as and when new evaluations are identified. It should therefore be
regarded as a continually evolving resource.

Resources toolkit

The resources toolkit provides analytical frameworks for evaluating education for entrepreneurship in
different forms of education. It has the practical aim of assisting the decision-making of those seeking to
evaluate education for entrepreneurship programmes, and in particular those who have little or no prior
experience of conducting an evaluation. It seeks to generate a deeper understanding of the objectives of
evaluation, as well as providing practical tools to enable the evaluation process to be carried out in an
appropriate and rigorous manner. As such, the toolkit takes into account the requirement to provide
templates and ideas which are appropriate for the different levels at which education for entrepreneurship
programmes may be delivered: primary education; secondary education; post-secondary/vocational
education; and higher education.

Crucially, in order to assist newcomers to evaluation or research methodology, references to examples

of questionnaires and other research instruments from evaluations of education for entrepreneurship
programmes are included in the Toolkit.

8
 The sections which make up the toolkit are under the following headings:

 About evaluation
 Why evaluate?
 Purpose of the evaluation
 Types of evaluation
 Who should undertake the evaluation?
 Approach to evaluation
 Targeting and sampling
 Data collection
 Analysing data
 Reporting and interpreting findings

Conclusions and recommendations

Although there has been a dramatic increase in, and a growing emphasis on education for
entrepreneurship programmes in recent years, there is a relative scarcity of robust evaluation data on which
to make a compelling case for funding and supporting such programmes. This may partly be attributable
to the variability in the objectives, content and delivery of education for entrepreneurship programmes,
which militates against reliable data-gathering techniques. In particular, there is a lack of data which
measures the outcomes and impact of programmes. Methodologically, the majority of these studies are
limited, because, they may lack „before and after‟ comparisons, there are frequently no control groups and
very few have a longitudinal dimension.

There is clearly a need to generate a greater understanding of, and attachment to, the evaluation
process, and the variety of methods which may be used. The following recommendations can be made:

 the inclusion of an evaluation strategy should be a pre-requisite of any submission for funding of
an education for entrepreneurship programme;

 that evaluation strategy should be a key component of the design stage of any programme;

 when initiating any form of substantive evaluation of education for entrepreneurship

programmes, it is essential, at the outset, to ensure that there is an agreed operational definition of
education for entrepreneurship, and the outcomes/impact which the study is seeking to measure;

 the approach, format, degree of sophistication and timescale of the evaluation should be
determined by the resources available and by the complexity and scope of the programme;

 where resources allow, externally provided experts in evaluation methodologies and techniques
should be charged with conducting the evaluation;

 evaluation findings, even where they indicate a lack of impact, should inform future decision-
making about the effectiveness, sustainability and format of programmes;

 effective and widespread dissemination of evaluation findings should be encouraged and

supported, in order to enhance the development of a culture of evaluation.

9
I. INTRODUCTION

Recent years have seen increasing emphasis being placed, by governments across a range of
countries, on stimulating greater commitment to enterprise and entrepreneurial activity, as part of broader
economic goals. In addition to a pronounced growth of activity within schools to enhance students'
awareness of enterprise opportunities, there has been an escalation of enterprise education and experience
programmes. This was recognised at the second OECD Conference of Ministers responsible for Small and
medium-sized enterprises (SMEs), where it was stated that:

“Developing an entrepreneurial culture and fostering entrepreneurial attitudes and values has
moved high on government agendas. Education and training (including lifelong training) in
entrepreneurship and creativity are the preferred instruments for encouraging entrepreneurial
behaviour in societies, and evidence suggests that such programmes can have an impact on
entrepreneurial activity and enterprise performance” (OECD, 2004, p.10).

A key theme of the conference, which was held in Istanbul in June 2004 and was entitled “Promoting
Entrepreneurship and Innovative SMEs in a Global Economy”, was „fostering entrepreneurship‟. Within
this theme, the importance of evaluating programmes was asserted:

“Regular evaluation is essential to identify ways of improving those programmes that should be
retained and to provide a basis for reallocating funds where they should not” (OECD, 2004,
p.37).

Fundamental to this emphasis on the need for the evaluation of programmes was the recognition that
there was a lack of authoritative evaluation studies of education for entrepreneurship programmes.
Therefore, a recommendation of the conference was for the development of “an „evaluation culture‟ by
making evaluation of programmes central to the policy process”.

Although the importance of education for entrepreneurship, or enterprising behaviour, has been
widely acknowledged, this has not necessarily been reflected in the systematic application of evaluation of
education for entrepreneurship programmes, which could provide evidence of the emergence of such a
culture.

II. AIMS AND OBJECTIVES OF THE STUDY

In line with the recommendations of the Istanbul conference, the OECD Working Party on SMEs and
Entrepreneurship (WPSMEE) developed a proposal for a Study on the Evaluation of Programmes
Concerning Education for Entrepreneurship. The overall aim of the study was “to strengthen the culture of
evaluation within entrepreneurship education and to provide guidelines for evaluating in order to gain a
better understanding of how to promote entrepreneurship education”. The fundamental premise of the
project was that the diversity, and relative scarcity, of approaches to the evaluation of education for
entrepreneurship, pointed to a need for encouraging greater attachment to the evaluation of programmes.

10
Crucially, it was also recognised that examples and suggestions of good practice can both enhance the
robustness of individual evaluations, and generate a degree of comparability across programmes and
national boundaries which has hitherto been lacking

The stated objectives of the study relating to issues and perceived gaps in relation to the evaluation of
education for entrepreneurship activities were:

i. to emphasise the importance of the role of evaluation and thereby encourage its
assimilation/integration more widely into the delivery of education for entrepreneurship
programmes;

ii. to strengthen the case for supporting education for entrepreneurship activity, by identifying
examples of evaluations which are able to provide evidence of the beneficial effects of
participation;

iii. to enable those concerned with the quality and impact of provision to be better able to carry
out evaluations;

iv. to deepen the understanding of the impact of education for entrepreneurship programmes and
the needs of individuals, and to provide hard and persuasive evidence to support the case for
investment in such programmes, by identifying and describing examples of „good practice‟ in
undertaking evaluations;

v. to build on and align with the OECD Framework for the Evaluation of SME and
Entrepreneurship Policies and Programmes;

vi. to encourage the development of a degree of commonality and comparability in the evaluation
of education for entrepreneurship programmes across different national contexts – with a view
to establishing a robust and compelling body of evidence to support the funding to sustain such
programmes.

III. DESIGN OF THE STUDY

The study comprised two phases. Phase 1 focused on an overview of research methods by which
education for entrepreneurship programmes might be evaluated. The purpose of Phase 1 was to review
analytically a wide range of relevant literature, with a particular focus on the methods adopted and
approaches taken to the evaluation of programmes, especially where a measurement of the impact of such
programmes has been sought. The key components of the phase were:

i. undertaking an analytical review of the literature, with a view to identifying and assessing the
key issues and approaches on the evaluation of programmes concerning education for
entrepreneurship; and

ii. developing a matrix, with axes for types of programmes and evaluation requirements, which
provides a starting point for those considering conducting an evaluation.

11
 The review focused on material which provides some evidence or description of:

 The stated aims and objectives of particular programmes;

 The methods adopted for delivering the programmes;

 The intended outcomes of such programmes;

 The development of methodological approaches and mechanisms for generating data relating to
the effectiveness of programmes;

 The introduction in the programme design of monitoring or data-gathering instruments which

offer the possibility of obtaining information relating to the measurement of impact;

 Examples of where evaluation of the impact of education for entrepreneurship programmes has
been carried out and their results;

 Examples of good practice in terms of the structure, format and effectiveness of education for
entrepreneurship programmes.

The overriding aim of Phase 2 was to produce a range of practical tools and materials which will
enable the application of appropriate methods to be applied to the evaluation of education for
entrepreneurship programmes. Accordingly, the key components of Phase 2 were:

i. the production of a resources toolkit which provides analytical frameworks for different forms
of education for entrepreneurship programmes and suggested methodologies which might be
used. It also sets out the advantages and disadvantages of different methods;

ii. a list of evaluation studies of education for entrepreneurship programmes which have been
undertaken; and

iii. an evaluation matrix, whose structure was set out in the Phase I report.

These three components are designed to enable those intending to carry out evaluations to understand
the range of potential approaches and mechanisms for conducting the evaluation and will be an invaluable
resource. The following diagram illustrates how they are mutually supportive.

For those with relatively little experience or knowledge of evaluation, the matrix will be the starting
point, as it provides an overview of the factors which need to be considered when designing an evaluation.
Thereafter, the toolkit becomes the essential resource for determining which approaches to adopt and
which research instruments may be most appropriate.

12
 Figure 1. Education for Entrepreneurs: Key Components of Phase 2

Evaluation Matrix

Types of programmes
Key objectives
Evaluation questions
Required information
Evaluation methods

Types of evaluation Details and location

Approaches to evaluation
Targeting and sampling
Methods – surveys, case
studies etc
Gathering data
Analysing data
Reporting findings
Type and objective of
programme
Target group/participants
Type of evaluation
Methodology and scope
Main findings

Evaluation
Evaluation
references
Toolkit

IV. EDUCATION FOR ENTREPRENEURSHIP PROGRAMMES

Definition

Attempts to provide a precise definition of education for entrepreneurship tend to stress that it relates
to the development of one or more of a combination of attitudes, personal qualities, and formal knowledge
and skills. Certainly, the strength of opinion in the literature is that education for entrepreneurship is
concerned with the inculcation of a range of skills and attributes, including the ability to think creatively, to
work in teams, to manage risk and handle uncertainty. Lewis (2002) couches her definition of „education
for enterprise‟ in terms of an earlier OECD description of enterprise, by referring to it as relating to the
development of “a group of qualities and competencies that enable individuals, organisations, communities
to be flexible, creative and adaptable in the face of rapid social and economic change” (Lewis, 2002, p. 1).

However, underpinning this broad categorisation is the contention that changing „mindsets‟ is
fundamental. Thus, an overarching goal becomes that of fostering the development of a mindset which is
conducive to entrepreneurship and to entrepreneurial behaviour.

13
 “Entrepreneurship education is important as a crucial determinant of the supply of
entrepreneurship by forming (potential) entrepreneurs as well as contributing to a positive
entrepreneurship culture. In this respect, entrepreneurship education should not only focus on
narrow defined tools (e.g. how to start up a business, financial and human resources
management) but also to broader attitudes (like creativity, risk taking, etc.) especially on the
lower and secondary level” (OECD, 2008, p. 112)”

Thus, the notion of education for entrepreneurship as essentially about inculcating a positive
„mindset‟, or attitude to entrepreneurship is fundamental to this report. An apt description was provided by
a discussant of an earlier draft of the report – “it is part of the entrepreneurial pipeline which starts in
education, runs through research to business”1. Another quote from a European Commission report points
up that, if changing mindsets is required, then, almost inevitably, this becomes a drawn out process rather
than an immediate conversion.

“Entrepreneurship is a dynamic and social process where individuals, alone or in collaboration,

identify opportunities for innovation and act upon these by transforming ideas into practical and
targeted activities, whether in a social, cultural or economic context” (European Commission,
2006, p 20)

This recognition of education for entrepreneurship programmes being part of a process which may
take some time to reach fruition and may produce outcomes across a range of contexts, has fundamental
implications for the evaluation of such programmes. Firstly, it needs to be acknowledged that the effects
or impact of the programme may not become apparent until well after the completion of participation in the
programme. This lends itself to the notion that longitudinal, or certainly longer-term evaluation studies are
required.

Moreover, as it can be argued that the development of the appropriate mindset is a process which,
over time, is dependent on a number of interventions, possibly of different types, it therefore becomes
increasingly difficult to assess, or quantify the impact of a single intervention or programme on this
process.

The point about the application of entrepreneurial skills and attitudes in different contexts is also
important, as it broadens our perspective on what „entrepreneurial‟ behaviour is about. It is not restricted
to starting a business or running one‟s own enterprise. Individuals can act entrepreneurially within a wide
range of roles in work organisations, large or small. Moreover, they can do so outside the working
environment – in non-work activities, such as voluntary work or the organisation of sports clubs, and in the
domestic and social spheres (see Iredale, 2002).

It is also necessary to be clear about the scope of education for entrepreneurship programmes. They
can be delivered in a wide range of settings. At the geographical or administrative level, programmes may
be available nationally, regionally, locally or at an institution level. The scope of this study will encompass
provision from different levels of education, as well as that which is delivered by private sector or
charitable organisations, or as part of national campaigns.

Aims and objectives of programmes

There is a multiplicity of activities which can be subsumed within the category of education for
entrepreneurship. For example, some will seek to provide skills which are fundamental to supporting
business start-up and self-employment, while others will seek to imbue individuals with the appropriate

1
This quotation is taken from a valuable contribution by Sanne Tonneijck, former Dutch delegate to the WPSMEE.

14
mindsets/perspectives and confidence to operate in an entrepreneurial manner in the course of their work.
While there may be similarities between the aims and the content of different programmes, it is important
to recognise that these differences exist, and that an understanding of the aims and objectives is essential
when undertaking an evaluation. Broadly, the aims and objectives can be divided into the following
categories, which are not mutually exclusive;

 The acquisition of key (or core) skills: these may relate to literacy, numeracy, communications,
ICT and problem solving. They represent the fundamental requirements for operating effectively
in a working environment, and for career planning and the process of identifying and accessing
appropriate work opportunities;

 The development of personal and social skills: a whole raft of skill areas or personal attributes
may be subsumed within this category, including: team working; self-confidence; self-awareness;
risk taking; problem solving; creativity; and the desire to innovate;

 Skills relating to business start-up or financial literacy, such as drafting business plans,
marketing, financial management, sales, and human resource management. Participants often
undertake an exercise in setting up and running their own company. In some programmes, the
inclusion of a financial element enables participants to develop the ability to plan personal and
family budgets.

Clearly, the aims and objectives of programmes determine the outcomes which are sought, and which,
in turn, provide indicators which an evaluation should seek to measure or assess. For example, outcomes
may include:

 Greater confidence to work independently or to operate in an organisational environment;

 Enhanced employability; or

 Increases in business start-up.

15
 Box 1. Example of an Approach to Establishing Objectives

In the United Kingdom, the Qualifications and Curriculum Authority (QCA) provides guidance on the statutory
1
requirement for work-related learning . Emphasis is placed on three main components of employability:
enterprise capability; financial literacy; and economic and business understanding. This approach is derived
1
from the findings of the Davies Review , which defines these components, and the knowledge, skills and
attitudes which they require as follows:
Enterprise capability: the capability to handle uncertainty and respond positively to change, to create and
implement new ideas and new ways of doing things, to make reasonable risk / reward assessments and act
upon them in one‟s personal and working life.
• Knowledge and understanding of concepts - organisation, innovation, risk, change;
• Skills - decision-making (particularly under conditions of uncertainty), personal and social, leadership,
risk management, presentational;
• Attitudes - self-reliance, open-mindedness, respect for evidence, pragmatism, commitment to making
a difference;
• Qualities - adaptability, perseverance, determination, flexibility, creativeness, improvisation,
confidence, initiative, self-confidence, autonomy, action-orientation.
Financial literacy: the knowledge, skills and attitudes necessary to become a questioning and informed
consumer of financial services and the ability to manage one‟s finances effectively.
• Knowledge and understanding - familiarity with a range of concepts such as money, credit and
investment;
• Skills and competence - budgeting, financial planning and personal risk management;
• Attitudes - taking responsibility for the wider impact and implications of money and financial decisions
on individuals, business and the community.
Economic and business understanding: a process of enquiry, focused on the context of business, central
to which is the idea that resources are scarce so that choices have to be made between alternative uses.
• Knowledge and understanding - familiarity with a range of economic concepts such as the market,
competition, price, efficiency and economic growth
• Skills - the ability to take decisions and make judgements on issues with an economic dimension,
investigate simple hypotheses and apply theoretical understanding to practical situations
• Attitudes - an interest and concern in: economic affairs, responsible use of resources, challenges of
business and its importance to society, responsibility of employers to the community and the environment
1
QCA (2003) Work-related learning for all at key stage 4: Guidance for implementing the statutory requirement
from 2004. Qualifications and Curriculum Authority.
1
Davies, H. (2002) A Review of Enterprise and the Economy in Education. HM Treasury

Types of programmes

Education for entrepreneurship can be delivered in a variety of forms within different countries, often
ranging from programmes delivered nationally, to individual, one-off events delivered locally. For
example, in the United States, a number of education and enterprise programmes, such as Junior
Achievement and NFTE, are nationally available, whilst others are delivered in one state (or area), such as
the IN2BIZ Entrepreneur summer camp which is held in Oregon. An Australian example of a programme
available nationally is Young Achievement Australia (YAA), whose programmes link education with
business and aim to provide appropriate skills and knowledge to help young people in their working lives.

16
 In addition to the various objectives of programmes identified earlier, the method of delivery and the
scope of the programme (e.g. whether national or local) will have a bearing on the approach taken in any
evaluation, as these will determine from whom data may be obtained (teachers, participants etc), the
timescale over which data collection is possible or desirable, and the sample sizes which can be derived.

A broad distinction can be made in the forms in which programmes are delivered between:

 Education-based programmes wherein education for entrepreneurship may be taught as a distinct

subject, may be integrated more widely into the curriculum, or may involve the setting up of
mini-enterprises;
 Award schemes; and
 Partnership schemes

Education-based programmes

As indicated above, education-based programmes may have a variety of methods of delivery.

However, in line with the aims and objectives of programmes outlined in the previous section, they range
from what Australia‟s Ministerial Council on Education, Employment, Training and Youth Affairs
(MCEETYA) National Goals for Schooling in the Twenty-First Century policy (1999)2 termed:

 “narrow views of education about enterprise (i.e. a knowledge of the world of work and
business), or education for enterprise (i.e. focusing on the knowledge and attitudes necessary for
self-employment or small business commencement), to

 the broad view of enterprise as an approach to teaching and learning. The broad view focuses on
teaching and learning that is student-driven and authentic (usually through ways that integrate the
curriculum and are delivered in authentic contexts, often in partnerships with the community)”

Programmes delivered by Enterprise New Zealand have been designed to promote a culture of
enterprise education at different school levels. At the primary level, the Primary Enterprise Programme
(PrEP) enables students to begin “learning to live” by setting up their own businesses. A key element of
the programme is the need to solve problems and they are encouraged to take risks and resolve any
challenges which may arise. The Enterprise Studies Programme (ESP) targets young people in Year 10
and is “an innovative, action based programme targeting the economics, social studies and technology
curriculum areas”. An experiential approach, whereby students develop their own projects, is adopted.
The emphasis is on self-discovery, team work and motivation. Thereafter, the Young Enterprise Scheme
(YES) is “a learning experience for young people, teachers and business people in which senior secondary
school students form a company; become directors; develop products and services, which they market and
sell. YES is school based and teaches skills in budgeting, planning, interpersonal relations, decision
making, reporting, communications; risk management and teamwork” 3.

Similarly, in Scotland, Determined to Succeed has been introduced across all school levels, with a
view to developing an enterprise ethos and effectively changing the culture within schools. The four main
strands of the programme are: enterprising teaching and learning; entrepreneurial learning; work-based
vocational learning; and career education (Scottish Executive, 2007).

2
The new 2009 MCEETYA National Goals do not make explicit mention of enterprise education.
3
Enterprise New Zealand Trust website: www.enzt.co.nz

17
 Although education-based programmes are often targeted at young people in secondary school, or at
university undergraduates, there are examples of programmes for pupils at primary school level, such as
those run by Junior Achievement in the United States, and the New Zealand Primary Enterprise
Programme (PrEP).

Box 2. Examples of Education-based Programmes

Junior Achievement’s elementary school programmes have six sequential themes, each with five hands-on activities,
as well as an after-school and capstone experience, work to change students‟ lives by helping them understand
business and economics:
®
Ourselves uses stories read aloud by the volunteer, along with hands-on activities to demonstrate helping, working,
earning, and saving.
®
Our Families emphasizes the roles people play in the local economy and engages students with activities about
needs, wants, jobs, tools and skills, and interdependence.
®
Our Community explores the interdependent roles of workers in a community, the work they perform, and how
communities work.
®
Our City studies careers, the skills people need to work in specific careers, and how businesses contribute to a city.

JA More than Money™ teaches students about earning, spending, sharing, and saving money, and businesses they
can start or jobs they can perform to earn money.
®
Our Region introduces the relationship between the natural, human, and capital resources found in different regions
and explores regional businesses that produce goods and services for consumers.
®
Our Nation provides practical information about businesses' need for individuals who can meet the demands of the
job market, including high-growth, high-demand jobs.

JA BizTown™ provides a simulated community where students assume the roles of workers and consumers.

American examples of programmes for students at both primary and high school level include
programmes for young people in elementary school - Bizworld, and Kids and the Power of Work
(KAPOW), whereas YoungBiz covers a wider age range, having programmes for young people aged 8 to
18 years.

The Dynamo Programme, which operates in Wales, aims to develop entrepreneurial attitudes and
skills among primary and secondary school students, through setting up role model networks and providing
curriculum materials.

Examples of programmes targeted at secondary school students include:

 „Let‟s Make a Company‟ (Kaisha wo tsukrou), which is delivered in Japan and is an activity for
Junior High School students. The programme, which was developed in 2003, was run in two
schools in Mie Prefecture in 2003/04. It has a business focus, wherein students in teams of 5 to
10 set up and run a mini-company competing for the highest profits;

 The Young Entrepreneurs Scheme (YES) is a national programme running in Irish schools which
is aimed at young people aged between 12 and 18. The scheme aims to stimulate enterprise and
innovative activity by operating businesses in schools;

18
  In Germany, the “Go! To School” project, run by the Institute of German Economy, in Cologne,
gives students an overview about entrepreneurship and encourages them to consider starting their
own businesses;

 The Txingu Programme, in Spain, targets students undertaking vocational training, in order to
enhance their understanding of business and promote an entrepreneurial culture;

 Junior Achievement Young Enterprise, which is delivered throughout Europe and many other
countries, is targeted at 15-19 year old school students, to enable them to gain experience of how
business works.

Courses and programmes associated with entrepreneurship, many of which contain a strong business
focus, are delivered at university level. The following examples give a flavour of what is provided:

 The „Dare to be Entrepreneurial‟ (Ousar Empreender) project, which was launched in Portugal
for university students in 1999;

 The INNOVA programme at the Polytechnic University of Catalonia, in Spain, seeks to bring
about cultural change relating to innovation, risk-taking and a commitment to enterprise;

 The Norwegian University of Science and Technology runs a series of courses on

entrepreneurship and innovation, for both undergraduate and those studying doctorates;

 Also in Norway, Entreprenørskap og foretaksutvikling (Entrepreneurship and enterprise

development) is a programme delivered at the University College of Østfold, with a remit to
encourage business start-up;

 In Ireland, the four-year degree in Finance, Computing and Enterprise, based in Dublin, has a
major component in Enterprise Development;

 For European Junior Enterprises, university students are offered the opportunity of running their
own company, to provide practical experience which complements their academic studies;

 The University of Ljubljana, in Slovenia, runs a Masters programme in Entrepreneurship;

 A post-graduate university course in Bulgaria which focuses on business start-up;

 In the United States, the University of Wisconsin‟s Weinert Center for Entrepreneurship offers
both undergraduate and postgraduate programmes on entrepreneurial management and enterprise
development;

 In Austria, the UNIUN project (University Graduates and Students Develop Businesses) seeks to
develop entrepreneurial attitudes and business start-up skills among university students.

The Berger Entrepreneurship Program at the University of Arizona was originally introduced in 1983,
and draws on staff to teach the programme from the departments of finance, economics, marketing, and
management. The core courses of the curriculum include competitive advantage, venture finance, market
research and business plans development. In addition, courses in management information systems,
management, finance, and marketing may be taken. Students work in pairs on a business plans
competition, for which substantial cash prizes are awarded

19
 The idea of education for entrepreneurship being embedded within the broader school curriculum has
been recognised and adopted widely. For example, programmes which have entrepreneurship as a
curriculum goal are being launched in New Zealand. There, the Regional Enterprise Clusters seek to
embed an enterprise culture across the curriculum (Ref. 006). In Australia, the framework for vocational
education in schools proposed in the 1999 National Goals for Schooling in the Twenty-First Century the
Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA) asserted that
“Enterprise Education is a way of teaching and learning, not another addition to the curriculum or a subject
or an alternative curriculum. It is about developing enterprising attitudes in students. The focus is not on
discrete activities, but on providing an underlying basis for delivering education across all areas of the
curriculum”. It further advocated the adoption of a “whole school approach to the development of a culture
of enterprise”, with its inclusion in the strategic planning process.

This approach is similar to that in the United Kingdom, where the Teachernet website, in describing
the role of education for entrepreneurship, contends that “the most effective schools take an inclusive
approach: providing in-school training and awareness-raising for staff, who then devise different ways to
enhance enterprise capability through changes to 'usual' lessons and teaching style, as well as through
specific enterprise activity”.

Mini-companies

Programmes frequently have a „hands-on‟ approach where students set up and run „mini-companies‟.
For example, the Europrise project, which emanated from the Leonardo Da Vinci programme in 1995
(European coverage), had the objective of increasing entrepreneurial competence in young people by
combining group based learning with learning through observation and running a company.

Other examples include Young Entrepreneurship; a study programme for young people aged between
15 and 20 years old which is nationally available in Finland. In groups of 5 to 10, students manage and run
an enterprise for a few hours a week over an academic year. The JUNIOR project in Germany (which is a
member of „Young Enterprise Europe‟) includes pupils in the 4th year of secondary school setting up mini-
businesses.

A report on the use of mini-companies in secondary education (European Commission, 2005) stated
that “the objective of a student company is to develop either a real economic activity on a small scale or a
realistic simulation of an economic activity”. The Managing Firm initiative in Poland was cited as an
example of the latter, which tend to be run on computers, rather than through the establishment of „real‟
products and services.

The Young Achievement Australia Business Skills Program is delivered over a 24 week period to
groups of Year 9-11 and tertiary students. Within the programme, they “operate their own company, raise
share capital; design, manufacture and sell a product or provide a service; and then wind up the company,
paying a dividend to shareholders. Students work with mentors from the business community”.

Similarly, in the UK, Young Enterprise offers a range of programmes, which are based on the
principle of Learning by Doing. A key element of the programme is the introduction of volunteers from
business into the classroom to work with teachers and students. Some programmes require students to
work together to run their own company, while others make use of “games, hands on activities and role
play to develop skills and capabilities for business and enterprise”.

The IG Students programme, in Italy, enables secondary school students to gain practical experience
of managing a company, thereby learning more about team working and the skills needed to become
entrepreneurs.

20
 The JUNIOR project in Germany (which is a member of „Young Enterprise Europe‟) includes pupils
in the 4th year of secondary school setting up mini-businesses.

Award schemes

An example of an award scheme is the ASDAN awards scheme in the UK, where students receive
credits for completing challenges in a number of areas, including Science and Technology, Work Related
Activities, and Information Transmission. Students must demonstrate evidence of action planning
(including recording and reviewing achievements), as well as competence in the Wider Key Skills
(opportunities are available to develop the other Key Skills).

For the Barclays New Futures Programme, schools and colleges develop and run programmes with a
social goal (e.g. refugee after school support club, leisure sports clubs for special needs young people, etc.)
to provide students with an opportunity to develop the skills necessary to plan, organise and set up a
community enterprise. The scheme aims to encourage young people to act positively on issues affecting
their local communities. Awards go to projects which enable students to learn new skills for life, work and
citizenship through making a contribution to the community.

In Australia, the Plan Your Own Enterprise Competition is designed to raise students‟ awareness of
small business management and planning principles and practices. It is targeted at Year 11 students and
consists of a business planning competition.

Partnership schemes

Schemes promoting partnerships between schools and enterprises are commonly found. For instance,
as part of the Junior Achievement programme in Hungary, 10,000 students in 50 schools were given the
opportunity to meet and learn from entrepreneurs.

Another example is Young Entrepreneurship; a study programme for young people aged between 15
and 20 years old which is nationally available in Finland. In groups of 5 to 10, students manage and run an
enterprise for a few hours a week over an academic year.

Young Achievement Australia offers programmes which provide a link between education and
industry. A Business Skills Programme, which is facilitated by mentors from the business world, is
delivered outside school to groups of 15 to 25 young people for two hours a week over 16 to 24 weeks.
They are required to “create, manufacture and market a product or service in a competitive environment”,
thereby going through the stages of a business cycle “and take responsibility for all essential business
processes, from selling shares and raising capital to liquidating the company”. Additional support is
provided through seminars and workshops, and annual awards are presented for regional, state and national
winners.

A programme in four African countries (Benin, Togo, Burkina-Faso, and Cote d‟Ivoire) which has
reached more than 30,000 students since its launch in 1993 comprises the Junior and Senior Achievement
programmes. The programme‟s aim was to develop an enterprising culture amongst young people,
through partnerships between education and the private sector.

21
V. EVALUATION

Approaches to evaluation

In addressing the evaluation of education for entrepreneurship programmes, it was important that this
study took account of, and was aligned with, the OECD Framework for the Evaluation of SME and
Entrepreneurship Policies and Programmes (2007), which has as its four objectives:

 To increase the awareness of politicians and public officials of the benefits from having an
evaluation culture;
 To disseminate examples of good micro evaluation practice at national and sub national levels;
 To highlight key evaluation debates: Who does evaluation? What procedures and methods should
be used? When to do evaluation? What about the dissemination of findings? Should all policies
be disseminated in the same way?
 To make a clear distinction between policies that operate at the micro level, i.e SME and
entrepreneurship specific policies, and those that operate at the macro level, i.e. mainstream
policies that nonetheless influence SMEs and entrepreneurship.

However, the nature of education for entrepreneurship programmes, with the emphasis on education,
differentiates the focus of evaluation of these programmes from those of other SME related activities and
processes. As already identified, the aim of many of these programmes, as well as informing participants
about entrepreneurial activity and behaviour, is to generate a shift in attitudes towards entrepreneurship.
This makes it more difficult to ascribe quantifiable measures, so that, instead of „hard‟ outcome evidence
(such as the numbers initiating a business start-up), an attempt has to be made to gauge „softer‟ outcomes
(such as changes in attitude). It may be argued that this level of complexity, and the difficulties of
measuring this change, is one of the reasons for the relative scarcity of robust findings from rigorously
applied evaluations of education for entrepreneurship programmes.

The limitations of evaluating any individual education for entrepreneurship programme are
highlighted by Lewis (2002), who asserts that “all enterprise education programmes have different
characteristics and are delivered in a variety of different contexts (cultural and educational)” (Lewis, 2002,
p 21), thereby severely constraining any attempt to generalize on the basis of findings from a single study.
Nonetheless, for this report, the WPSMEE sought to concentrate on the implementation of evaluation
which seeks to address these complex issues, across a range of aims, objectives and delivery mechanisms
of programmes, and to draw out issues which are specific to the evaluation of education for
entrepreneurship programmes, rather than to the evaluation of SME and entrepreneurship programmes
more generally.

A key starting point is the belief that evaluation should be viewed as vital in order to ensure that
optimum benefits are derived from a programme. It is important to understand that the lessons learned
through the evaluation process, including why things did not work, and what were the unanticipated
consequences or side effects, are as valuable as the identification of exemplary practices. It should also be
acknowledged that different sets of participants will have different sets of goals and expectations from the
programme.

There is no single approach to the evaluation of education for entrepreneurship programmes, and
therefore no single model which can be applied in all situations. It is more appropriate to think of
evaluation in terms of a range of options or tools which can be selected and used according to the measure

22
or programme being assessed. This toolkit will comprise a combination of qualitative and quantitative
options. For example, quantitative measures of success, akin to traditional output-related performance
indicators, may be used in conjunction with qualitative assessments of attitudinal or perception shifts.

Evaluation can have both backward and forward looking purposes. It can be designed to tell us what
outputs and outcomes were generated by a project/programme (what is called summative evaluation). But
it can also explain how, why, and under what conditions a policy intervention worked, or failed to work
(i.e. formative evaluation). Formative evaluations are important for determining the reasons for effective
implementation and delivery of policies, programmes or projects. For example, the findings of a formative
evaluation process can provide valuable feedback on issues such as the planning and design of courses, as
well as aspects of the content and the teaching methods employed.

At the outset, evaluation should be considered as a means for assessing the extent to which the
objectives of the initiative are being met efficiently, effectively and economically. The notion of „value for
money‟, which inevitably arises when new initiatives are being assessed, is commonly accompanied by
consideration of the following effects:

 substitution – where one form of subsidised education for entrepreneurship programme is

preferred to an unsubsidised one;

 deadweight – where programmes have simply compensated recipients for actions they would
have taken even without such programme assistance;

 displacement – where participation in a programme has had the effect of forcing or displacing
others from engaging in such activity;

 leakage – where a programme has an effect on those who are not participating (e.g. members of
the control group).

Methodological options

A number of methodologies may be considered when undertaking an evaluation of education for

entrepreneurship programmes. Firstly, there is the experimental approach to evaluation, where people are
randomly assigned to either:

 a „treatment‟ group, composed of those participating in some form of education for

entrepreneurship activity, or

 a „control‟ group composed of individuals who do not participate in the activity.

This enables an assessment of the effect of the activity to be made. However, experiments are
difficult to carry out in practice, and assume a „common effect‟ across the whole treatment or control
group. Also, the effects may only apply within the particular setting (e.g. educational level, geographical
area) of the programme being evaluated, so that it may not be possible to say what would happen when a
programme is generalised in a different setting. It is therefore advisable to complement this approach with
non-experimental methods.

Non-experimental methodology subsumes a wide range of techniques. A common theme is either a

before and after comparison of the same individuals; or a comparison of independent groups of people, one
whose members are exposed to a treatment, to a similar group not exposed to the treatment.

23
 Matching attempts to pair each individual in the treatment group to a member of the control group
who has similar characteristics (e.g. age, gender, socio-economic group, qualification attainment). Matches
are selected on the basis of similarities in observed characteristics.

Propensity Score Matching (PSM), a derivation of matching, makes it possible to match along a single
measure (the propensity score), which summarises these differences. Technically, the method has a
number of difficulties, and depends on having a good understanding of the determinants of the outcome
variables, a rich suite of variables measuring these determinants, and pre-intervention measures of these
determinants.

The difference-in-difference approach combines matching with before and after treatment comparison.
A treatment group and a control group are selected and data collected both before and after participation
(or non-participation) in the education for entrepreneurship programme. A calculation is then made of the
difference in the average outcome in the treatment group before and after treatment, minus the difference
in the average outcome in the control group before and after treatment.

Overall, non-experimental methods can be powerful evaluation tools, particularly if the process
underlying the outcome variables is well known, and data exist on the relevant measures. In such cases,
matching offers a suitable method of evaluation. If it can be combined with a difference-in-difference
approach, then certain unobserved measures can also be controlled for, which may improve the quality of
the study.

When considering methods to evaluate the long-term impact of these programmes, the difficulties of
establishing causality should not be underestimated. A key issue here is the time period over which data
collection needs to take place before meaningful findings can be produced. This is particularly important,
as it could be argued that longer-term outcomes, for the individual, the organisation, and society as a
whole, are likely to reflect the true benefits of education for entrepreneurship. Thus, while changes in the
propensity of programme participants to set up their own business could be apparent over the medium
term, depending upon the age of the students, evidence of significant attitudinal shifts related to an
attachment to entrepreneurship may take several years to emerge. It may be possible to detect attitudinal
shifts from ongoing national household surveys, although attributing any change to a specific education for
entrepreneurship programme would be extremely difficult.

Measuring impact

A principal requirement for evaluating an impact is establishing that a difference in one, or more,
relevant outcomes is caused by a particular treatment. Typically, this is achieved by exposing members of
one group to a treatment and preventing members of another group (the 'control' group) from receiving this
treatment. The evaluation problem is to establish and quantify the impact of a treatment, and estimate what
would have happened to the treatment group in the absence of the treatment. This has come to be known
as the „counterfactual‟.

Purdon(2002) lists four pertinent steps in defining the counterfactual:

i. decide what the impact of the treatment is to be compared against;

ii. identify what constitutes success for the programme in order to determine what outcomes will
be measured;

iii. specify the target population;

iv. decide when to measure the impact.

24
 In attempting to determine whether any of the numerous interventions that could be classified as
education for entrepreneurship activity have been influential, it is first necessary to decide what counts as a
treatment. It is then necessary to define and measure the counterfactual. This entails deciding upon the
outcome measures; and having data on outcomes measured at relevant time points.

The use of Propensity Score Matching techniques would require a rich and relevant suite of pre-
intervention data in order to undertake the matching. If selection modelling were to be used, this would
require an instrument that enabled selection into the programme to be modelled, with the instrument being
conditionally independent of the outcomes. Not only would the data be required for the treatment group,
but, more crucially, corresponding data would also be required for the control group.

What is important is to have a clear idea of what is meant by a treatment; what are the outcomes,
when are they likely to occur; how should the counterfactual be defined; which effect do we want to
measure; and what is the target population of interest? Once these questions have been addressed, it
becomes possible to consider an evaluation method.

VI. IMPACT AND EFFECTIVENESS OF EVALUATION

An important aspect of the whole Study on the Evaluation of Programmes Concerning Education for
Entrepreneurship is the need to provide evidence of the benefits of education for entrepreneurship
programmes, in order to elicit ongoing support and funding for these activities. One of the major
difficulties for making claims about the impact of education for entrepreneurship programmes, especially
when attitudinal shifts are concerned, is that of establishing causality. While shifts in attitude or behaviour
may be identified, uncertainty surrounds the attribution of that change, or part of that change, to a single
factor. This may be one of the reasons for the scarcity of robust evidence of the positive impact of
education for entrepreneurship.

A recent study conducted in the Netherlands (Oosterbeek, van Praag and Ijsselstein, 2008. Ref. 004)
sought to identify the impact of a mini-company programme, whose participants were vocational college
students, on the entrepreneurial skills and competences of those students. In a methodologically rigorous
piece of research, which involved a difference-in-difference approach, with a control group, the authors
concluded that the overall effect of the programme on entrepreneurial skills was insignificant. Moreover,
the impact on the students‟ intentions to become an entrepreneur was “significantly negative”.

Nonetheless, examples can be found of positive impacts. A study which provides some of the most
powerful evidence of the benefits to be derived from education for entrepreneurship programmes was that
undertaken by Alberta Charney and Gary D Libecap (Charney and Libecap, 2000. Ref. 001) of the Berger
Entrepreneurship Program, which had been running at the University of Arizona since 1983. The
evaluation compared graduates of the programme between 1985 and 1998 with a matched sample of non-
entrepreneurship business graduates from the same university). A response rate of 21 per cent yielded a
sample of 105 programme participants and 406 non-participants. The findings suggested that participation
in the programme had a positive impact in terms of: risk-taking and the formation of new ventures;
increasing the likelihood of becoming self-employed; income; the growth of firms; promoting the transfer
of technology from the university to the private sector; and, less strongly, job satisfaction. In addition, a
survey of deans, department heads and development officers at the university pointed to there having been
a beneficial impact on curriculum development.

25
 A recent study by McHugh and O‟Gorman (2006) was initiated on the premise that, while the rapid
growth in the number of education for entrepreneurship programmes in recent years had led to a significant
increase in evaluation activity, there still remained a dearth of evidence of impact. Accordingly, they set
out to discover whether there were differences in the attitudes and performance of entrepreneurs who had
undertaken some form of education for entrepreneurship programme, and those who had not. Their study,
which was conducted in the South-East of Ireland, sought to identify differences between those who had
been on a twelve month full-time education for entrepreneurship programme and those who had been on a
part-time programme, or had not received any entrepreneurship education at all. The research found
pronounced differences between these two groups, in terms of the number of problems they were required
to address, and, importantly, in the manner in which they were handled. They concluded that the findings
of the study did provide solid evidence of a positive link between participation in education for
entrepreneurship programmes and subsequent behaviour. Crucially, however, they placed emphasis on the
fact that the type of programme they had undertaken was also a key determinant of their ability to operate
entrepreneurially.

An earlier study conducted in Ireland (Fleming, 1996; Fleming and Owusu-Ansah, 2001. Ref. 018) is
also important, because of its longitudinal nature, and the recognition that evidence of a predilection to
become an entrepreneur or to engage in entrepreneurial activity, as a result of participation in an education
for entrepreneurship programme, may not become apparent for a number of years. The conclusions
reached by Fleming, which are, importantly, founded on soundly based and rigorously executed research,
point to the benefits which, over time, can be derived from education for entrepreneurship programmes.

“From the evidence of this longitudinal study it appears that creating an awareness of the
entrepreneurship process and developing and transferring knowledge about business formation
during higher education can indeed stimulate graduate entrepreneurship” (Fleming, 1996, p 116).

It is important to be aware of the point made by Fleming concerning the significance of the type of
education for entrepreneurship in determining outcomes. While the examples cited above provide
compelling evidence of the positive impact which education for entrepreneurship programmes can have, a
cautionary note must be sounded about the dangers of assuming that whatever is delivered under the rubric
of education for entrepreneurship will have similar beneficial effects. Clearly, the ability of education for
entrepreneurship programmes to elicit positive outcomes will be highly dependent on the quality and
appropriateness of the programme delivered.

VII. EVALUATION MATRIX

The matrix was developed on the basis of the findings of Phase 1, and is an attempt to accommodate
the range of objectives and delivery methods of education for entrepreneurship programmes, as well as the
various approaches to evaluation and instruments for gathering and analysing data which are available. As
such, it seeks to represent, in a simple form, the complexities involved. The matrix provides a guide for
those intending to undertake evaluations of education for entrepreneurship activities. In particular, it
indicates appropriate methodologies to adopt.

The two main axes for the matrix comprise:

26
Types of programmes

This axis indicates the programmes which can be subsumed under the three broad categories of:

 Learning entrepreneurial skills and attitudes

 Enhancing business start-up and entrepreneurial behaviour

 Development of business start-up and entrepreneurial skills

Evaluation requirements

Against each type of programme, the matrix identifies:

 Key objectives – the objectives of the programme

 Evaluation questions – the main issues which the evaluation is seeking to address

 Required information – the evidence which the evaluation will need to obtain to answer the
questions

 Evaluation methods – the most appropriate methods to achieve the objectives of the evaluation.

The matrix structure is based on the discussion within the Phase 1 report about the different types of
programme, their objectives, the issues which should be addressed in any evaluation and the most apt
methods through which the evaluation should be conducted.

Table 1. Evaluation requirements

Overall purpose of Key Evaluation Required Evaluation methods

programmes objectives questions information

LEARNING ENTREPRENEURIAL SKILLS AND ATTITUDES

Programmes delivered at Acquisition of What knowledge Participants‟ Surveys of course

all levels of education – key skills was learned by levels of graduates
primary/secondary/furthe participants? satisfaction
r and higher Development with the Observations of course
of What skills were programme delivery
Modules embedded in entrepreneurial developed?
work related curriculum attitudes Quantitative Focus groups of
How useful will data on participants
Education/industry these skills be in acquisition of
partnerships future careers? skills and Interviews with
knowledge programme deliverers
Mentoring schemes covered by
the
Programmes delivered programme
by external agencies
Perceptions
Award schemes of usefulness

27
 of
Campaigns to promote programmes
entrepreneurship

Development What has been Participant Surveys of course

of personal, the impact of the perceptions of graduates
social skills programme and benefits
the benefits derived Interviews/questionnaire
derived by s with course tutors and
participants? participants

Encouragemen To what extent Qualitative Interviews/questionnaire

t of has the assessments s with course tutors and
entrepreneurial programme of changes in participants
attitudes and affected attitudes
behaviour – attitudes to Surveys of course
promoting an entrepreneurshi graduates
enterprise p or
culture employment Pre-test and post-test
more generally? design

Longitudinal studies

Mini-enterprises To give How far has Assessments Surveys of schools

students involvement in of those
experience of mini-enterprise running the Focus groups of
the world of initiatives programmes participants
work impacted on
students‟ Participant
To provide perceptions of perceptions of
experience of „enterprise‟? benefits
taking derived
responsibility
and being
accountable

ENHANCING BUSINESS START-UP AND ENTREPRENEURIAL BEHAVIOUR

Initiatives targeted at Encouraging To what extent Tendency of Analysis of output

enhancing business business start- has the participants to performance data –
start-up up programme led start up own proportion of participants
to business businesses starting up own business
Provision of start-up?
support and Participants‟ Surveys of course
mentoring, and What are the perceptions of graduates
the characteristics usefulness of
development of associated with the Longitudinal studies of
business skills entrepreneurial programme course participants
behaviour?
Analysis of Matched comparisons
What factors the

28
 help individuals characteristic Case studies of
make the s of business start-ups
transition to self- „successful‟
employment? entrepreneurs

What are the

chances of
business
survival?

DEVELOPMENT OF BUSINESS START-UP AND ENTREPRENEURIAL SKILLS

Business awareness Development How appropriate Participants‟ Surveys of course

courses of business are the skills perceptions of graduates
start-up skills which are usefulness of
(e.g., finance, delivered? the Longitudinal studies of
legal, sales & programme course participants
marketing) To what extent
have they been Application of Matched comparisons
Providing applied by knowledge
knowledge of participants? and skills Case studies
the technical acquired to
process of What has been the
setting up a the impact of the work/busines
business programmes on s context
the sustainability
of start-up Propensity of
businesses? participants to
start their own
business

VIII. LIST OF REFERENCES

The following list of references is intended to provide examples of a variety of evaluations which
have been conducted on education for entrepreneurship programmes, covering a range of programme
types, evaluation methodology and cultural contexts.4

Each of the studies included in the list of references has an identification number which can be used to
refer the reader to examples from evaluations which are most relevant. One of the features of this list is
that it is capable of being updated, as and when new evaluations are identified. It should therefore be
regarded as a continually evolving resource.

4
As well as the programmes identified during the course of the research, this list includes some detailed in the
ENTREDU website.

29
 Table 2. Evaluations references

Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

001 Type Type Matched samples of Participation in the This is an excellent

programme programme had a example of an
Berger Learning Summative evaluation
participants and non- positive impact in evaluation which is
Entrepreneurship entrepreneurial skills to measure the effect
entrepreneurship terms of: risk-taking founded on a
Program and attitudes of the programme on
business graduates, and the formation of methodology
graduates
University of Arizona, Target group controlling for socio- new ventures; incorporating matched
1985-1999 Aims economic increasing the samples, to achieve a
University-based
characteristics of the likelihood of becoming counterfactual, and
programme with a To evaluate “the effect
individuals involved so self-employed; income; extended over a
dedicated curriculum of the programme on
Charney, A, and that the marginal effect the growth of firms; number of cohorts.
technology transfer
Libecap, G. D., (2000), Undergraduates, MBA of entrepreneurship promoting the transfer
from the university to It is also one of the
The Impact of students and graduate education can be of technology from the
the private sector, the most powerful
Entrepreneurship students from the observed, holding university to the private
effect of the program evaluations of
Education: An colleges of individual factors sector; and, less
on private giving to the education for
Evaluation of the engineering, medicine, constant. strongly, job
business college, and entrepreneurship
Berger science, agriculture satisfaction. In
the pedagogical effect programmes, in
Entrepreneurship Interviews with addition, the survey of
Objectives of the entrepreneurship establishing the
Program at the samples of department deans, department
curriculum on other programme‟s positive
University of Arizona, To provide business heads and other heads and
disciplines in the impact.
1985-1999 and entrepreneurial college administrators, development officers at
college”.
skills the college Dean the university pointed
development officer to there having been a
and other officials, as beneficial impact on
well as those involved curriculum
in technology transfer. development.

002 Type Type Mixed method NET was found to have A good example of a
approach with been very successful in formative evaluation
Evaluation of the Professional Formative evaluation
quantitative and stimulating and which sought the
Northland Enterprising development
Aims qualitative aspects. supporting change at perspectives of those
Teachers (NET) programme.
the school planning responsible for
initiative To assess changes in
Case studies and level in its first two delivering the
pedagogy resulting
30
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Target group from the schools‟ surveys within four years. programme.
involvement in NET NET schools; and
New Zealand Secondary teachers The combination of
and to which education The impact of
and school leaders methods (surveys,
for entrepreneurship Postal surveys sent to education for
case studies etc.) to
Objectives had been integrated NET teachers/school entrepreneurship was
Bolstad, R. (2006), derive data from a
into school planning leaders in the variable, depending on
Evaluation of the To help secondary range of respondents
and infrastructure. remaining 14 NET the enthusiasm of the
Northland Enterprising teachers and school provides an example of
schools teachers, but had not
Teachers (NET) leaders develop an how to select the
become embedded in
initiative. New Zealand “enterprising” approach appropriate method for
the whole curriculum or
Council for Educational to teaching and different data sources.
culture of the schools.
Research. learning in their
Appendix A of the
subjects, and across
report, which contains
their school curriculum.
http://www.nzcer.org.n the questionnaire used
z/pdfs/15059.pdf for the survey of school
principals, is a valuable
resource.

003 Type Type Strategic interviews The local authorities This is a thorough
with policy makers and had made great efforts evaluation which
Determined to Learning Qualitative
key stakeholders to implement the provides a good
Succeed. entrepreneurial skills
Aims strategy. example of how to
and attitudes
Scotland Desk study to review conduct stakeholder
To gather and analyse Considerable
National strategy planning and delivery interviews and school
the views of key awareness had been
across 32 local case studies.
Target group stakeholders in order generated.
National Foundation for authorities.
to assess the impact It also provides
Educational Research Primary and Some progress had
on the infrastructure of examples of
(2006) National Secondary school Interviews with 32 local been made in
Enterprise in Education questionnaires.
Evaluation of pupils authority co-ordinators. embedding Enterprise
and the extent to which
Determined to in Education into the
Objectives inter-agency working
succeed. Scottish Case studies in schools curriculum.
had been successful.
Executive To prepare young
All secondary schools
people for the world of
had developed links
work through
with local colleges.
enterprising and

31
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

entrepreneurial Most teachers had

learning, work-based received appropriate
vocational learning and training.
career education.

004 Type Type Difference-in-difference The programme was This study is extremely
study, comparing found not to have had interesting for the
Junior Achievement Development of Impact study
students participating the intended effects, analytical techniques
Student Mini-Company business start-up skills
Aims in the programme and and there was no employed in a rigorous
Program.
Setting up and a comparable group in significant difference evaluation.
To assess the impact
The Netherlands operation of a mini- a college without the between the two
of the programme on Also, importantly, it
company programme. groups in the
students‟ concludes that the
development of their
Target group entrepreneurial programme had had no
Oosterbeek, H., van Instrumental variables entrepreneurial skills.
competencies and significant impact on
Praag, M. and Vocational college approach
intentions participants.
Ijsselstein, A. (2008) students
The impact of Regression analysis
Objectives
entrepreneurship
education on To teach students to
entrepreneurship put theory into practice
competencies and and to understand
intentions: An what entrepreneurship
evaluation of the Junior is all about.
Achievement Student
Mini-Company
Program. Jena
Economic Research
Papers.

005 Type Type Interviews with The fund for enterprise This is a good example
providers, participants activities was found to of an evaluation which
Promoting a Business Learning to become Impact evaluation.
and stakeholders. be having a positive makes concrete
and Enterprise Culture entrepreneurial,
Aims effect, but the other recommendations
Programme through:
Desk research of components had not about future policy
To examine the
1. Research into New documents and reports been successful. It direction.
New Zealand implementation,
32
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Zealanders' attitudes delivery and outcomes was recommended that

and development of of the programme the communications
Evaluation of the
core values; campaign and the
Promoting a Business
initiative as a whole
and Enterprise Culture 2. A communications
should be discontinued
Programme campaign;
http://www.med.govt.nz
3. A fund for enterprise
/templates/MultipageD
activities in schools
ocumentPage____245
and new businesses.
23.aspx
Target group
1. General public
2. School and
university students
Objectives
“to encourage a more
supportive culture
toward entrepreneurial
activities within New
Zealand, and
enterprising skills and
mindsets among
current and potential
entrepreneurs”

006 Type Type Mixed method Ongoing 2007-2009 An extremely well-

quantitative/qualitative conducted evaluation
Regional Enterprise Learning Process evaluation To date, a series of
which provides an
Clusters entrepreneurial skills Tracking studies of mini-reports have been
Aims example of how to
and attitudes students and teachers, produced. The 2009
New Zealand conduct a tracking
To “examine the report on the 2008
Target group study
processes by which the student survey states
Schools based clusters establish and that there was “a
Roberts, J., Bolstad, pursue education for generally positive

33
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

R., McDowall, S., Objectives entrepreneurship and perspective on the

Cooper, G. and Gilbert the outcomes that are experience of students”
Embedding an
J. (2007) Evaluation of achieved (in relation to
enterprise culture
the regional education both local and national
across the curriculum –
for enterprise (E4E) objectives)”
enterprise skills,
clusters (2007-2009).
attributes and
New Zealand Council
knowledge
for Educational
Research.

www.med.govt.nz/templates/
multipagedocumentpage___
_24523.aspx

http://www.nzcer.org.n
z/pdfs/16847.pdf

007 Type Type Review of the literature High proportions of This is particularly
both current students interesting for its use of
European Junior Development of Mixed method
Qualitative survey of and alumni had an online survey. For
Enterprises business start-up skills
Aims JE members and started, or planned to cost considerations,
Target group alumni – personal and start their own speed of data-
To assess the
email companies. gathering and enabling
Jesenovec, J. and University students “development of
respondents to form
Redien-Collot, R. entrepreneurial spirit
Objectives Online quantitative Women tended to start considered responses,
(2005) Research on among JEs' members
survey of JE members, companies on their online methods have
the Entrepreneurial To provide practical and its present and
alumni and customers own and were over- great attraction, and
mind-set of Junior experience of running long-term effects on
represented among may be utilised to a
Entrepreneurs. Jade a company to students and alumni.”
entrepreneurial leaders much greater extent in
(European complement the
future.
Confederation of Junior theoretical knowledge
Enterprises). derived from their
www.jadenet.org/downl studies
oad/publications/entrep
34
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

reneurial_spirit/2005/2
005_JADE_Research_
Entrepreneurial_Spirit_
Executive_Summary.p
df

008 Type Type On-line survey of The results showed This highlights the
young people that the idea of benefits of an online
The Make Your Mark Learning Impact evaluation –
“running your own survey method in
Campaign entrepreneurial skills tracking study
business” was still generating responses
and attitudes
United Kingdom Aims popular amongst most from large samples.
National campaign respondents, and that
To track changes over
enterprise had “entered
Target group time in young people‟s
GfK NOP (2007) the mainstream market
attitudes towards
Enterprise Campaign Young people for young people”
enterprise
Research Tracker
Objectives
2006. Report of Key Those who had
findings prepared for: To “inspire young participated in an
The Make Your Mark people to be enterprise event in the
Campaign. May. enterprising in the previous 12 months
broadest sense – were more confident
through business start- about starting a
ups, social and business.
community activism
and as enterprising The engagement of
employee” women was identified
as an area for attention

009 Type Type Survey of There was an increase Given the increasing
headteachers and in “total enterprise” at attachment to the
Whole School Learning Impact evaluation
other teachers, using a the primary level. The notion of the whole
Approach to Enterprise entrepreneurial skills
Aims questionnaire programme had school approach, this
in Education. and attitudes.
“increased awareness study, with its use of a
“To determine the
Scotland Target group On-line survey of of the Determined to combination of
impact of the
35
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Primary and secondary programme between participating students Succeed strategy and methods, illustrates the
schools January 2004 and and control group Enterprise in Education way such activities
Wiggins, A., Cowie, M.
March 2006 and to in both secondary and may be evaluated.
and Tymms, P. (2007) Objectives
consider how it might Focus groups with primary schools, and
Evaluation of the
Promoting and be more widely teachers helped raise the
Whole School
developing enterprise applied.” Enterprise in Education
Approach to Enterprise
in education in schools. Interviews , both face- skills of some of the
in Education: May
to-face and by teachers”.
2004 – January 2006.
telephone with
Scottish Executive
headteachers and
Social Research.
stakeholders

Observation in
classroom

School visits

Attendance at events
010 Type Type Mixed method – mainly The policy had not, at The timing of this
quantitative, that time, had a direct evaluation, in order to
Determined to Learning Impact evaluation
comprising: impact on “pupil skills, assess early impact,
Succeed entrepreneurial skills
Aims attitudes and indicates how
and attitudes
Scotland - desk research behaviours”. evaluations can be
To measure “the
Target group built up over time to
effectiveness and Schools were very
- consultations with show a developing
All pupils in full-time success of Determined positive about the
York Consulting local authority pattern of responses.
education in Scotland to Succeed, to inform initiative.
Limited (2007) National representatives
policy, delivery and
Evaluation of Objectives The Index of
future commitment to
Determined to - quantitative surveys Enterprising Attitudes
To prepare young the strategy”. This was
Succeed – Phase 2: of teachers, parents and Behaviour
people for the world of done by:
Early Impact Across and pupils (IOEAB) was positive
work and encourage
Scotland. Scottish - assessing changes in for parents, pupils and
ambition, through
Executive Social views, attitudes and - telephone interviews teachers.
entrepreneurship, and
Research. perceptions of young with headteachers
to prepare them for life.

36
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

people - case studies of

schools
- identifying changes in
behaviour
- assessing
implementation and its
impact on the
effectiveness of the
policy

011 Type Type Mixed method The evaluation Although this study has
qualitative and considered that Xlerate quantitative elements,
Xlerate Learning Summative evaluation
quantitative evaluation, was “an excellent and it is the qualitative
entrepreneurial skills
Scotland Aims involving: very effective material which is of
and attitudes
programme”, which most interest.
To make an
Target group - Visits to meet had had a very positive
assessment of the
Wiggins, A., Jones, K. students and advisers impact on the young
School pupils at S3 or whole programme
and Tymms, P. (2006), - Focus groups with the people, helping them to
S4 stage (aged 15 or focussing on process
National Evaluation of co-ordinators and become more
16) who are not and outcomes
Xlerate with XL: advisers enterprising.
achieving their full
January 2005 – - Telephone interviews
potential.
January 2006. with headteachers
Scottish Executive Objectives - Meetings and
Social Research. ongoing contact with
To promote personal
key people
development,
- On-line surveys of
citizenship and
students
enterprise in schools.
The programme aims
to increase pupils‟
knowledge of
enterprises and
enterprising behaviour.

37
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

012 Type Type Mixed method design The Education for As well as being a
with qualitative and Enterprise (E4E) commendable study in
Regional Education for Learning Mixture of formative,
quantitative elements, programme was its own right, this report
Enterprise Clusters entrepreneurial skills process and impact
including: perceived to offer is extremely valuable
and attitudes evaluation
New Zealand considerable potential for the descriptions of
st
Target group Aims Consultations with as a vehicle for “21 the various methods
programme co- century learning”. employed and for the
Secondary schools To inform the
Roberts, J., McDowall, ordinators examples of
pupils development of the
S. and Cooper, G. questionnaires and
Education for
(2008) Enterprising Objectives Surveys of school other research
Enterprise programme
and future focused? principals and students instruments in the
To develop young through an
The first report from appendix.
people‟s competencies examination of the
the Regional Education Focus groups of
in innovation and processes which are in
for Enterprise Clusters students
managing and setting place to implement the
Evaluation. Report
up businesses programme, and an
prepared for Ministry of Case studies of
assessment of the
Education and New schools
clusters‟ impact in
Zealand Trade and
creating sustainable
Enterprise Surveys of
education for
community/business
enterprise
partners
013 Type Type Quantitative and “Overall program Methodologically, this
Qualitative satisfaction is high is a relatively
Junior Achievement Development of Summative
among teachers, unsophisticated study
(JA) Titan Programme business start-up skills
Aims Review of background volunteers, JA member which nonetheless
Worldwide Target group documents and site staff, and students. points to considerable
“To assess the extent
literature positive impacts
High school students to which the program
Benefits are emanating from the
attained desired
Lodestar (2007) Junior Objectives Pre and post-student particularly high in the programme.
outcomes in student
Achievement Titan assessment areas of core
“To help students knowledge, attitudes,
Program: 2007 workforce readiness
develop the behaviours and skills.
Program Evaluation. Training session for skills, communication
knowledge, skills and
Lodestar member site staff and teamwork.
attitudes to become To obtain the reactions
Management/Research
productive citizens and of diverse stakeholders
Telephone interviews Computer-based
38
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Inc. workers.” about the with staff simulation is an ideal

implementation and learning tool
quality of the various Online survey with
program elements. teachers, volunteers, The program is
and staff effective in facilitating
To identify specific the understanding of
strategies and Focus groups with business and
practices for program teachers, volunteers economics concepts
improvement.” and increasing student
engagement

The program has had a

positive impact on
student attitudes
towards business”.

014 Type Type The findings from a Young people who This report provides a
series of evaluations participated in the synthesis of the
Junior Achievement Learning Summative
conducted across programme were findings from a series
Young Enterprise (JA- entrepreneurial skills
Aims different countries significantly more likely of evaluations. It offers
YE). The Company and attitudes
to become a good example of how
Programme To assess the
Target group entrepreneurs than these can be brought
cumulative effects and
Europe those who did not together to provide
15-19 year old school long-term impact of
participate succinct and
students students‟ participation
persuasive
in JA programs
"Enterprise in Objectives dissemination of
Education – Evaluation results.
To promote
JA-YE Company
entrepreneurship and
Programme"
entrepreneurial
(Document from Junior
behaviour among
Achievement Young
young people.
Enterprise (JA-YE)
Norway, Ungt
To enable young
Entreprenørskap).
people to experience
39
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

how business works

and to establish
education/industry links

015 Type Type Mixed method Positive aspects were: This is an example of a
approach involving: - examples of pupils thorough study of data
Enterprise learning at Learning Formative evaluation
being motivated by generated through the
Key Stage 4 entrepreneurial skills
Aims - visits to schools enterprise learning and inspection process,
and attitudes
United Kingdom - surveys of schools developing relevant and indicates how this
To identify and
Target group - observation of skills can contribute to the
recommend good
enterprise classes - importance of improvement of
Secondary school practice in the
Ofsted (2004) Learning - interviews with commitment from programme delivery.
pupils aged 14-16 implementation of
to be enterprising: An headteachers and senior managers
enterprise education
evaluation of enterprise Objectives providers of enterprise - importance of clearly
learning at Key Stage programmes defined aims and
Developing
4. Office for Standards objectives, pupils
entrepreneurial
in Education. HMI taking responsibility
attitudes and skills
2148. August and being able to
through pathfinder
operate autonomously
projects incorporating a
variety of approaches,
Problems identified
including curriculum
included:
based study, enterprise
- lack of a clear
days, involvement in
definition of
work related learning
enterprise learning
etc provided by
- inadequate
organisations offering
assessment of pupils‟
enterprise programmes
enterprise learning,
- inadequate
monitoring and
evaluation of the
implementation of
enterprise learning
- little evidence of

40
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

work-related learning
being part of a
coherent curriculum

016 Type Type The following methods Participating schools‟ An excellent example
were used: knowledge of business of a mixed method
Enterprise Education in Learning Mixed method –
– a national telephone and enterprise had approach, which
Schools entrepreneurial skills combination of
survey of 647 school increased. gathers data from a
and attitudes qualitative and
Australia principals; range of sources.
quantitative
Target group – in-depth interviews Students attained a
with 60 schools, sense of achievement,
Primary and secondary
Keys Young (1999) including 40 that had gained a greater
school students Aims
Evaluation of the participated in EES knowledge of their own
Enterprise Education in Objectives – to assess the funded projects; strengths and
Schools (EES) effectiveness of the – face-to-face weaknesses, improved
“To achieve a learning
Element of the School projects funded in interviews with school their ability to work
culture which will result
to Work Programme: achieving its three key principals and cooperatively,
in greater numbers of
Final Report. goals; teachers, students, increased their
students equipped and
Department of – to identify barriers to parents, business/ willingness to take the
enthused to identify,
Education, Training the take-up of EES industry and initiative and exercise
create, initiate and
and Youth Affairs, activities by primary community responsibility, and in
successfully manage
Australia. and secondary representatives; some cases improved
personal, business,
schools; – in-depth interviews their performance.
work and community
– to assess the level of with 20 key informants;
opportunities”
awareness and – consultations with
There was
appreciation of the project managers of
considerable support
value of being key EES funded
for enterprise
enterprising by school initiatives, and where
education in schools
students, their teachers appropriate, their
and parents, school evaluators.
principals and careers The „hands on‟
advisors, local experience of
businesses and involvement in
community based enterprising activities
clearly had the greatest
41
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

organisations; impact on individual

– to suggest strategies principals, teachers,
for effectively and students who
promoting and participated.
supporting enterprise
education in Australian
schools.

017 Type Type Qualitative methods, The programme fitted This is an objective
comprising: strategically with policy study which
Dynamo Programme Learning Summative evaluation
direction at European exemplifies the positive
entrepreneurial skills
Wales Aims Review of literature and Welsh levels role which independent
and attitudes
and documentation evaluators can play in
To assess: Training of Role
Target group enhancing the quality
Models was adequate
Arad Consulting (2007) - awareness of, and Case studies of 11 of provision.
Primary and secondary
Evaluation of demand for and use Secondary schools, 2 The programme was
school students
Entrepreneurship of the programme Primary schools and 2 well promoted in
Action Plan Culture Objectives colleges of further schools across Wales
- value of the
Projects: Evaluation of education
Developing programme to Management of the
the Dynamo
entrepreneurial schools and programme was
Programme. Final Interviews and group
attitudes and skills colleges consistent across
Report to the Welsh discussions with key
through Role Model Wales
Assembly Government. - impact on target stakeholders
Networks and
October. audiences The approaches
Curriculum Materials
adopted by Role
- scale and selection
Models were more
of target audiences
appropriate for younger
- integration with the age groups
curriculum
Curriculum materials
- effectiveness of were of high quality
delivery models and user-friendly
- quality of training Dissemination of
for teachers and information within

42
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Role Models schools was

haphazard
- strategic fit with
national policy Use of Dynamo
areas materials was limited in
Primary schools
- value for money
analysis Retention of Role
Models was high
The impact of the
programme, and the
penetration across
schools were below
anticipated levels
The level of in-school
support required was
intense
More work is required
to support the
dissemination and
promotion of
information within
schools.

018 Type Type Longitudinal The enterprise initiative The overall scarcity of
had an effect on the longitudinal studies of
Entrepreneurship Learning Impact study
Initial 1991 postal cohort‟s subsequent education for
Education in Ireland entrepreneurial skills
Aims survey of 838 career aspirations. entrepreneurship
and attitudes
Fleming, P. (1996) graduates – 419 in the programmes serves to
“To evaluate the ways “As graduates mature,
„Entrepreneurship Target group treatment group and underline the
in which graduates‟ the proportion entering
Education in Ireland: A 419 in the control importance of this
Higher Education attitudes and business ownership
Longitudinal Study‟ in group study. Not only is it
students behaviours relate over increases”.
Academy of commendable for the
time to new venture
Entrepreneurship Objectives Follow-up 1996 survey rigour of the
creation following Half of those in
Journal, European tracking the members methodology, but the
43
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Edition, vol.2, 1: 94 – To promote an exposure to of the treatment group employment indicated ongoing series of
118. awareness of self- entrepreneurship conducted an interest in or sweeps of respondents
Fleming, P. and employment as a concepts and practical expectation of setting means that its
Owusu-Ansah, W. career option and assignments while at up their own business importance will
(2001) The Impact of motivate young people college/university.” at some time in the continue to grow.
Entrepreneurship to begin equipping future.
Education on Business themselves with the
Venturing: A skills, knowledge and
Longitudinal Survey of experience required for
Irish graduates. Paper effective business
presented at the ownership.
Internationalising
Entrepreneurship
Education and Training
Conference.

019 Type Type Quantitative Providing support in This is a good example

the design and of a study targeted at
Local development Learning to become an Formative evaluation
A survey of local planning of the project programme providers
Agents entrepreneur
Aims development agents – were the most which can be achieved
Sanchís, J R and Target group achieved sample of important aspects within a limited budget.
To analyse the way in
Cantarero, S (2001) 216
Those in adult and which local agents
Factores de éxito en la
further education manage the
gestión de proyectos
programmes
en el marco del Objectives
desarrollo local
To provide support in
(Success factors in the
setting up a business
management of
programmes in the
frame of local
development)

020 Type Type Postal survey of The scheme was found In conjunction with the

44
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

IDA Student Enterprise Learning Impact assessment students and a control to have a positive study described in Ref.
Award Scheme entrepreneurial skills group effect in stimulating the 018, this report
Aims
and attitudes level of graduate provides valuable
Ireland
To assess the impact, entrepreneurship evidence of the impact
Target group
Fleming, P. (1994) The effectiveness and of programmes.
Role of Structured Higher education and usefulness of
Interventions in adult education education for
Shaping Graduate students entrepreneurship
Entrepreneurship. Irish programmes in
Objectives
Business and enabling or
Administrative To teach students encouraging graduates
Research, Vol. 15, No. about entrepreneurship start their own
8, p.146. businesses

021 Type Type Quantitative Students from the This is an example of a

Economics faculty quantitative survey
Enterprise education in Learning Impact assessment
Survey of a quota were more likely than which can achieve
three faculties at the entrepreneurial skills
Aims sample of 469 students those from other interesting results
Johannes Kepler and attitudes
faculties to start up within a limited budget.
University Linz. To assess the start-up
Target group their own business
propensity of the
Austria
University students Johannes Kepler The most important
Richter, J. (2001) University's students, barriers to start-up
Objectives
StudentInnen und and the barriers they were perceived to be:
Unternehmensgründun To develop an perceived lack of equity capital,
g. Eine empirische awareness of administrative barriers,
Studie an der entrepreneurship high financial risk, lack
Johannes Kepler of debt capital, and
Universität Linz lack of a start-up idea.
(University Students
and Start-ups. An
empirical study at the
Johannes Kepler
University Linz.

45
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

022 Type Type Quantitative The majority of This is a good example

students were not of quantitative survey
UNIUN project (UNIUN Learning Impact assessment
Questionnaires were considering starting a techniques.
- University Graduates entrepreneurial skills
Aims completed by 837 business on
and Students Develop and attitudes
students across 9 graduation.
Businesses) To assess the
Target group universities
emergence of Those from business
University students entrepreneurial faculties showed the
Austria
attitudes and the greatest likelihood
Objectives
Frank, H., Korunka, C. likelihood of business
and Lueger, M. (2002), To develop start-up
Entrepreneurial Spirit. entrepreneurial
Unternehmerische attitudes and business
Orientierung und start-up skills
Gründungsneigung von
Studierenden.
(Entrepreneurial Spirit.
Entrepreneurial
Orientation and Start-
Up Propensity of
Higher Education
Students). Department
of Small Business
Management, Vienna
University of
Economics and
Business
Administration.

023 Type Type Multi-method approach There was a high level This study provides a
incorporating of satisfaction among good example of how
Begabungsförderungs Learning Formative
qualitative and students and their empirical research can
modell Schumpeter- entrepreneurial skills
quantitative elements be informed and
46
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Handelsakademie and attitudes Aims parents. supplemented by desk

(Pilot project of an Interviews research.
Target group To assess the The coaching element
educational model for
appropriateness of the of the programme was
the promotion of gifted Secondary school Surveys
curriculum and the rated especially highly
students called students
teaching and learning
'Schumpeter- Observations
Objectives methods used
Handelsakademie')
To combine Document analysis
Austria comprehensive
enterprise education Data derived from
with an educational participants, teachers,
Aff, J., Magyar, T. and
model for the funders, parents and a
Probst, H. (2001)
promotion of gifted control group of
Wissenschaftliche
students participants
Begleitforschung des
Begabungsförderungs
modells Schumpeter-
Handelsakademie
(Accompanying
Scientific Research of
the Pilot Project for the
Promotion of Gifted
'Schumpeter-Classes').

024 Type Type Combination of Male students This is an interesting

qualitative and expressed a greater study which combines
Junior classes and Learning Impact assessment
quantitative likelihood of starting a qualitative and
'ordinary classes' at entrepreneurial skills
Aims business quantitative research
different types of and attitudes
Workshops with methods.
advanced-level To determine: The school‟s attitudes
Target group teachers
secondary schools towards
1. The extent to which
Secondary school entrepreneurialism and
different types of Survey of students –
students active learning
Austria advanced-level achieved sample of
methods were key
Objectives secondary schools in 890
Frank, H., Korunka, C. factors in students‟
Austria contribute to
47
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

and Lueger, M (2002) To develop promote their students' likelihood of starting a

Unternehmerorientieru entrepreneurial entrepreneurial business
ng und attitudes through a attitudes
Gründungsneigung. variety of initiatives,
Eine such as mini- 2. The extent to which
Bestandsaufnahme bei companies the teaching of start-up
SchülerInnen related and
Allgemeinbildender entrepreneurial
und Berufsbildender knowledge promote the
Höherer Schulen in development of
Österreich. positive attitudes
(Entrepreneurial towards
Orientation and Start- entrepreneurship and
up Propensity. A venture creation as a
Survey at Advanced- future career option by
Level Secondary students
Vocational Schools
and at Academic
Secondary Schools).

025 Type Type Mixed method Students have high This is an interesting
regard for the theory- study which assesses
Entrepreneurship Learning Impact assessment
Quantitative – survey based programmes but the impact of the
studies in 14 different entrepreneurial skills
Aims of students – achieved feel that more programme on
polytechnics and attitudes
sample of 739 education for participants‟ attitudes.
To assess the extent to
Finland Target group entrepreneurship
which teaching
Qualitative – analysis should be provided.
Mansio, H. (1997) Polytechnic students entrepreneurship in
of study guides
Opetuksen polytechnics However, the courses
Objectives
vaikuttavuus yrittäjäksi encourages students to were not felt to
ryhtymiseen To develop become entrepreneurs encourage students to
ammattikorkeakouluiss entrepreneurial skills become entrepreneurs
a (The effectiveness of
education in terms of
entrepreneurial activity
48
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

in polytechnics).
Helsinki School of
Economics and
Business
Administration

026 Type Type Surveys of There was found to be This study exemplifies
stakeholders and a wide variation in the the ability of evaluation
Ten entrepreneurship Learning Formative
programme quality of provision to draw comparisons
programmes in entrepreneurial skills
Aims participants between different
Southwest Finland and attitudes
programmes.
To evaluate ten
Finland
entrepreneurship
Puhakka, V (1999) Target group programmes
Yrittäjyyskoulutusohjel
Adults
mien monitahoarviointi
II. Tarkastelussa Objectives
kymmenen
To develop
varsinaissuomalaista
entrepreneurial skills
koulutusohjelmaa.
(Evaluation of
Entrepreneurship
Education II: the
programmes of
Southwest Finland
under examination).

027 Type Type Mixed method – YES provided students This is an excellent
quantitative and with skills and example of a well-
Young Enterprise Learning A „responsive‟
qualitative strategies which would researched evaluation
Scheme (YES) entrepreneurial skills evaluation
benefit them in the which is presented in a
and attitudes
New Zealand Aims Literature review short and long term.” succinct and thoughtful
Target group manner.
Lewis, K. (2002) An To determine the
Observation of student
Enterprising Future: Secondary school „concerns and issues Students felt that
activities The findings are placed
49
Title of programme and Description of programme Evaluation Methods Main findings and Comment
reference recommendations

Evaluating the Young students of stakeholding teachers were in a wider context

Enterprise Scheme. audiences‟ Group interviews with delivering the YES with through a discussion of
Objectives
Enterprise New students a narrow focus on the broader aims of
Zealand Trust. To give students the profit making and education for
http://sme- opportunity to run their Survey of schools, with winning. entrepreneurship
centre.massey.ac.nz/fil own company within questionnaires for programmes.
es/Complete_final_EN the school environment teachers and students Mentors were more
ZT_report.pdf effective when they The report
had empathy with the acknowledges the
students limitations of the study,
notably the difficulties
Being innovative and of generalising on the
having commitment basis of the findings,
were the key elements and the timeframe (one
for „being enterprising‟ year) which prevents
longer-term impact
The opportunity to being assessed.
interact with new
people in the context of
a team was the primary
positive impact of the
YES on the
participants

Programmes need to
be predominantly
learner driven
YES made students
more employable by
giving them a taste of
the benefits of hard
work and a degree of
risk-taking

50
IX. RESOURCES TOOLKIT

The resources toolkit has been designed in order to provide analytical frameworks for evaluating
education for entrepreneurship in different forms of education. It has the practical aim of assisting the
decision-making of those seeking to evaluate education for entrepreneurship programmes, and in particular
those who have little or no prior experience of conducting an evaluation. It seeks to generate a deeper
understanding of the objectives of evaluation, together with practical tools to enable the evaluation process
to be carried out in an appropriate and rigorous manner. As such, the toolkit takes into account the
requirement to provide templates and ideas which are appropriate for the different levels at which
education for entrepreneurship programmes may be delivered: primary education; secondary education;
post-secondary/vocational education; and higher education. It is also able to accommodate the evaluation
of activities which fall outside the traditional delivery of education for entrepreneurship programmes, such
as campaigns which are targeted at the general population in order to create a more responsive attitude to
entrepreneurship.

As resources for evaluation may be limited, the toolkit recognises that there will be considerable
variations in the scale and scope of evaluations, as demonstrated by the OECD WPSMEE Framework for
the Evaluation of SME & Entrepreneurship Policies and Programmes (OECD, 2008). Also, when
describing the elements of, for example, particular methods of investigation, it elaborates on the
advantages and disadvantages of the method or approach, and the contexts for which they are particularly
appropriate.

The Resources Toolkit is presented in the Annex.

51
X. CONCLUSIONS AND RECOMMENDATIONS

Recent years have seen a dramatic increase, throughout many countries, in initiatives designed to
introduce a greater degree of entrepreneurial expertise, in order to support the achievement of broader
economic goals. In particular, there has been a growing emphasis on education for entrepreneurship within
education systems. This has resulted in a plethora of activities which can be described as education for
entrepreneurship programmes.

At the same time, there has been a relative dearth of evaluation data on which to construct a robust
case for allocating resources to education for entrepreneurship activity. Although some evaluation studies
have identified the positive impact of such activity on participants‟ subsequent behaviour, there is no
overwhelmingly conclusive body of evidence to support the case for its continuing funding.

There are substantial difficulties in undertaking evaluations which provide findings to underpin a
compelling case for education for entrepreneurship programmes. These include: the variety of
programmes, with different objectives, which are on offer; the timescale which may be required to observe
the long-term impact of programmes; the problems of assigning causality for subsequent behaviour to a
single intervention; and the complexities of measuring „soft‟ outcomes, such as increased self-confidence
or attitudinal shifts. Thus, the variability in the objectives, content and delivery of education for
entrepreneurship programmes militates against reliable data-gathering techniques.

The evaluations which have been carried out to date have varied greatly, in terms of both the methods
used and the outcomes measured. A wide range of designs have been employed, from cross-sectional
surveys and pre-test and post-test designs, to a small number of studies which have incorporated a
longitudinal element. Methodologically, however, the majority of these studies are limited, because, they
may lack pre-test and post-test, there are frequently no control groups and very few have a longitudinal
dimension.

For many of the studies, the evaluation only lasted the length of the initiative, thus providing a one-off
snapshot, carried out immediately after programme completion, rather than any attempt to track subjects
over successive years. Overall, the number of evaluations carried out which have attempted to measure the
impact of programmes appears limited. The most common focus of those evaluations has been on the rate
of business start-up as an impact measure.

It follows from this that there is a need to generate a greater understanding of, and attachment to, the
evaluation process, and the variety of methods which may be used. A principal aim of this WPSMEE
study was to enhance such understanding and attachment, with the combination of the evaluation matrix,
list of references and resources toolkit providing a foundation for those embarking on evaluation activity.

The following recommendations can be made on the basis of the foregoing analysis:

 the inclusion of an evaluation strategy should be a pre-requisite of any submission for funding of
an education for entrepreneurship programme;

 that evaluation strategy should be a key component of the design stage of any programme;

52
 when initiating any form of substantive evaluation of education for entrepreneurship
programmes, it is essential, at the outset, to ensure that there is an agreed operational definition of
education for entrepreneurship, and the outcomes/impact which the study is seeking to measure;

 the approach, format, degree of sophistication and timescale of the evaluation should be
determined by the resources available and by the complexity and scope of the programme;

 where resources allow, externally provided experts in evaluation methodologies and techniques
should be charged with conducting the evaluation;

 evaluation findings, even where they indicate a lack of impact, should inform future decision-
making about the effectiveness, sustainability and format of programmes

 effective and widespread dissemination of evaluation findings should be encouraged and

supported, in order to enhance the development of a culture of evaluation.

53
REFERENCES

Angrist, J., and Imbens, G. W. Two-Stage Least Squares Estimation of Average Causal Effects in Models
with Variable Treatment Intensity', Journal of the American Statistical Association, 90(430), 431-
442.

Angrist, J., Imbens, G. W. and Rubin, D. B. (1996) 'Identification of Causal Effects using Instrumental
Variables', Journal of the American Statistical Association, 91 (434), 444-455.

Blundell, R. and Costa Dias, M. (2000) 'Evaluation Methods for Non-experimental Data', Fiscal Studies,
21(4), 427-468.

Bolstad, R. (2006) Evaluation of the Northland Enterprising Teachers (NET) Initiative. New Zealand
Council for Educational Research. http://www.nzcer.org.nz/pdfs/15059.pdf

Brown, R. (1995) „The Graduate Enterprise Programme: Attempts to Measure the Effectiveness of Small
Business Training‟ in British Journal of Education and Work, vol.8, 1: 27-38.

Bryson, A., Dorsett, R. and Purdon, S. (2002) “The Use of Propensity Score Matching in the Evaluation of
Active Labour Market Policies”, Department of Work and Pensions...

Burniston, S., Rodger, J. and Brass, J. (1999) Enterprise in Higher Education – Changing the Mindset,
DfEE Research Report RR117.

Burtless, G. (1995) 'The Case for Randomized Field Trials in Economic and Policy Research', Journal of
Economic Perspectives, 9(2), 63-84.

Charney, A. and Libecap, G. D. (2000) The Impact of Entrepreneurship Education: An Evaluation of the
Berger Entrepreneurship Program at the University of Arizona, 1985-1999. Final report to The
Kauffman Center for Entrepreneurial Leadership.

CEI (2002) Learning About Enterprise and Entrepreneurship: Conference Report. Report of conference
held at Warwick University of Warwick, June 28th, 2002.

Darby, G. (2004) The Young Entrepreneurs – Revisited, BusinessDynamics Report.

DTI (2002) Independent Research into Learning for Enterprise and Entrepreneurship.

European Commission (2006), “Entrepreneurship Education in Europe: Fostering Entrepreneurial

Mindsets through Education and Learning”. Final Proceedings of conference held in Oslo, 26-27
October.

European Commission (2005) “Mini-Companies in Secondary Education. Best Procedure Project: Final
Report of the Expert Group”.

54
Fleming, P. (1996), “Entrepreneurship Education in Ireland: A Longitudinal Study” in Academy of
Entrepreneurship Journal, European Edition, vol.2, 1: 94 –118.

Fleming, P. and Owusu-Ansah, W. (2001) The Impact of Entrepreneurship Education on Business

Venturing: A longitudinal Survey of Irish graduates. Paper presented at the Internationalising
Entrepreneurship Education and Training Conference.

Fletcher, M. (1999), „Promoting Entrepreneurship as a Career Option – The Graduate Enterprise

Programme‟ in Journal of European Industrial Training, vol. 23, 3: 127-139.

Gibb, A. (2002), „In Pursuit of a new „Enterprise‟ and „Entrepreneurship‟ Paradigm for Learning: creative
destruction, new values, new ways of doing things and new combinations of knowledge‟ in
International Journal of Management reviews, vol4, 3: 233-269.

Hales, J. et al. (2000), 'Evaluation of the New Deal for Lone Parents: Early lessons from the Phase One
Prototype – Synthesis Report', Department of work and Pensions Research Report 108, London:
CDS.

Harris, A. (1989), „Teachers‟ Evaluation of Enterprise Education in the Secondary School‟ in British
Journal of Education and Work, vol. 3, 1:. 83-92.

Harris, A. (1996), „Teaching Approaches in Enterprise Education – A Classroom Observation Study‟ in

British Journal of Education and Work, vol.8, 1: 49-58.

Hatten, T. S. and Ruhland, S. K. (1995) „Student Attitude toward Entrepreneurship as affected by

Participation in an SBI Program‟ in Journal of Education for Business, vol. 70, 4: 224-227.

Heckman, J .J. (1996) “Randomisation as an Instrumental Variable”, The Review of Economics and
Statsitics,78(2), 336-341.

Heckman, J .J. (1996) 'Randomisation as an Instrumental Variable', The Review of Economics and
Statsitics,78(2), 336-341.

Heckman, J. J. (1990) 'Selectivity Bias: New Developments, Varieties of Selection Bias', The American
Economic Review, 80(2), 313-318.

Heckman, J. J., Ichimura, H. and Todd, P. (1998) 'Matching as an Econometric Evaluation Estimator',
Review of Economic Studies, 65,261-294.

Hisrich, R., Langan-Fox, J. And Grant, S. (2007) ‚Entrepreneurship Research and Practice: A Call to
Action for Psychology‟, in American Psychologist, September, pp 575-589.

Holland, P. W. (1986) 'Statistics and Causal Inference' (with discussion), Journal of the American
Statistical Association', 81, 945-970.

Hytti, U. and Kuopisjarvi, P. (2004) Evaluating and Measuring Entrepreneurship and Enterprise
Education: Methods, Tools and Practices Small Business Institute, Business Research and
Development Centre, Turku School of Economics and Business Administration,

Hytti, U. (ed), (2003) State-of-Art of Enterprise Education in Europe, Results from the ENTREDU project.
Small Business Institute, Business Research and Development Centre, University of Turku.

55
Imbens, G. W. and Angrist, J. (1994) 'Identification and Estimation of Local Average Treatment Effects',
Econometrica, 62(2), 467-475.

Iredale, N. (2002) Enterprise Education: A Briefing Paper, Foundation for Small and Medium Enterprise
Development, University of Durham, UK.

Jesenovec, M. and Redien-Collot, R. (2005) Research on the entrepreneurial mind-set of Junior

Entrepreneurs. JADE, European Confederation of Junior Enterprises, September.
www.jadenet.org/download/publications/entrepreneurial_spirit/2005/2005_JADE_Research_Entrepr
eneurial_Spirit_Executive_Summary.pdf

Kemple, J. J. and Snipes, J. C. (2000) Career Academies – Impacts on Students‟ Engagement and
Performance in High School, MRDC Research Report.

Lewis, K. (2002) An Enterprising Future: Evaluating the Young Enterprise Scheme. Enterprise New
Zealand Trust. http://sme-centre.massey.ac.nz/files/Complete_final_ENZT_report.pdf

Lissenburgh, S. (2001) 'New Deal for the Long-term Unemployed Pilots: A Quantitative Evaluation Using
Stage 2 Survey', Employment Service Research Report ESR81.

Low, M. B. and MacMillan, I., C. (1988) „Entrepreneurship: Past and Future Challenges‟ in Journal of
Management, vol.14, 2: 139-161.

McHugh, M. and O‟Gorman, B. (2006) Enterprise Education Does Make a Difference! Centre for
Entrepreneurship, Waterford Institute of Technology.

Meager, N., Bates, P. and Cowling, M. (2003) Business Start-Up Support for Young Adults Delivered by
the Prince‟s Trust: A Comparative Study of Labour Market Outcomes, DWP Research Report No.
184.

Moran, M. T. and Cooney, T. M. (2003) To What Extent can the Impact of Enterprise Education be Truly
Evaluated by State Development Agencies, Conference Paper RENT XVII 2003, Lodz, November
19-21.

NICEC/CEI (2005) Methodological Study: Measuring the Long-term Impact of Enterprise Education and
Experience Programmes. Report submitted to Small Business Service

OECD (2008), “Entrepreneurship Review of Denmark”, p. 112.

OECD (2007), OECD Framework for the Evaluation of SME and Entrepreneurship Policies and
Programmes, Paris.

OECD (2004) “Promoting Entrepreneurship and Innovative SMEs in a Global Economy: Towards a More
Responsible and Inclusive Globalisation”, 2nd OECD Conference on SMEs, Executive Summary of
the Background Reports. .

Ofsted (2004) Learning to be Enterprising: An Evaluation of Enterprise Learning at Key Stage 4, August
HMI 2148.

Owusu-Ansah, W and Fleming, P (2001) The Impact of Entrepreneurship Education on Business

Venturing: A Longitudinal Survey of Irish Graduates, Conference Paper 11th Global IntEnt
Conference, July 2-4.

56
Patton, M. Q. (1997) Utilization-Focused Evaluation: The New Century Text. Edition 3. Sage.

Peterman, N. E. and Kennedy, J. (2003) „Enterprise Education: Influencing Students‟ Perceptions of

Entrepreneurship‟ in Entrepreneurship – Theory and Practice, vol.28, 2: 129-144.

Pfatteicher, S. K. A., Bowcock, D. and Kushner, J. (1998) Program Assessment Toolkit: A Guide to
Conducting Interviews and Surveys. LEAD Center, University of Wisconsin-Madison.

Purdon, S. (2002) 'Estimating the Impact of Labour Market Programmes', Department for Work and
Pensions Working Paper No. 3.

QCA (2003) Work-related learning for all at Key Stage 4: guidance for implementing the statutory
requirements from 2004, London, DfES.

Roberts, J., McDowall, S. and Cooper, G. (2008) Enterprising and future focused? The first report from
the Regional Education for Enterprise Clusters Evaluation. Report prepared for Ministry of
Education and New Zealand Trade and Enterprise

Rubin, D. B. (2005) “Causal Inference using Potential Outcomes”, Journal of the American Statistical
Association, 100(1), 322-331.

Saunders, L., Storey, S. and Weston, P. (1997) „The Impact of Work Related Curriculum on 14-16 Year
Olds‟ in Journal of Education and Work, vol.10, 2: 150-164.

Scottish Executive (2007) Determined to Succeed: Three years on … Investing in Scotland‟s Future.
Scottish Executive.

Scriven, M. (1991) Evaluation Thesaurus. Fourth Edition, Sage.

Shutt, J., Clarke, T., Sutherland, J. and Koutsoukos, S. (2001) Alive with Opportunities: Evaluating Young
Business Start-Up Success 1994-1999, Princes Trust, Leeds Business School.

Storey, D. J. (2000) “Six steps to heaven: evaluating the impact of public policies to support small
businesses in developed countries”, in Landström, H. and Sexton, D. L. (eds) Handbook of
Entrepreneurship, Blackwells, Oxford, pp 176-194.

57
 ANNEX. EVALUATION OF ENTREPRENEURSHIP EDUCATION PROGRAMMES:
RESOURCES TOOLKIT

This toolkit has been designed in order to assist those who are involved in undertaking evaluations of
education for entrepreneurship programmes. It is intended to provide a user-friendly guide to the different
approaches and methods which may be adopted, dependent on the objectives of the evaluation.

About evaluation

Evaluation should be viewed as vital in order to ensure that optimum benefits are derived from the
Education for Entrepreneurship programme. It is important to understand that the lessons learned through
the evaluation process, including why things did not work, and what were the unanticipated consequences
or side effects, are as valuable as the identification of exemplary practices. It should also be acknowledged
that different sets of participants will have different sets of goals and expectations from the project.
Furthermore, in order that all relevant data is collected from the outset, it is vital that the objectives of the
evaluation, and the manner in which it will be conducted, are included in the design of the programme, and
prior to its implementation.

What is evaluation?

Programme evaluation has been described as “the systematic collection of information about the
activities, characteristics, and outcomes of programs to make judgements about the program, improve
program effectiveness, and/or inform decisions about future programming” (Patton, 1997).

Another definition of evaluation is “the process of determining the merit, worth, or value of
something, or the product of that process” (Scriven, 1991).

Thus, evaluation should be seen as an essential component of any education for entrepreneurship
programme, because of its capability to enable a greater understanding of:

a. the impact of the programme, by considering the outcomes of the programme; and/or

b. how the programme may be improved, on the basis of its effectiveness and efficiency. For this
to happen, there needs to be a process of reflection about the programme and how it has
worked. This will include lessons about what has not worked well, as these can be valuable in
avoiding difficulties which may inhibit the effectiveness of the operation.

Key stages in the evaluation process are:

 Establishing the rationale for the evaluation – why is it being conducted?

 Deciding what type of evaluation to carry out

 Setting out the aims and objectives of the evaluation

 Selecting appropriate individuals/organisations to carry out the evaluation

58
  Securing and allocating resources

 Identifying an appropriate approach, timescale, and data requirements

 Choosing the most appropriate methods for collecting and analysing data

 Deciding on methods of dissemination and follow-up

The following sections address these requirements.

Why evaluate?

Evaluation is necessary in order that we can determine to what extent a programme is being
implemented appropriately and/or is achieving its stated objectives. In terms of education for
entrepreneurship programmes, evaluation enables judgements to be made about the effectiveness,
efficiency or impact of the programme. In particular, it seeks to provide information and robust evidence
on which decisions about the future of a programme can be made, and to address questions such as:

 is it achieving its objectives?

 what impact is it having on its participants?

 what improvements can be made?

Depending on the questions to be explored, this often entails seeking the views of one or more of:

 programme participants

 are sufficient numbers of the target group aware of and/or participating in the programme?

 is there evidence of shifts in attitudes or behaviour as a result of the programme?

 those delivering the programme

 are levels of take-up satisfactory?

 what is working well on the programme?

 what is not working well on the programme?

 what improvements could be made to the content and delivery of the programme?

 providers of funding

 does the programme offer „value for money‟?

 is it achieving the desired outcomes, in terms of, for examples, individuals acquiring
entrepreneurial skills and attitudes, setting up businesses etc?

59
Purpose of the evaluation

At the outset, it is necessary to define and clarify what the evaluation is seeking to achieve.

a. Why is it being carried out?

b. How was this need identified?

c. What are the specific aims and objectives?

d. Who will be the main recipients of the information and conclusions which emerge from the
evaluation?

e. How will this information be used?

f. What resources are available? - what or who should be the sources of that information. For
example, for process evaluations, it is usually necessary to obtain the perspectives of both
those responsible for delivering the programme and those participating in the programme.

Types of evaluation

The model of evaluation to be undertaken will depend on:

 the type of programme which is under investigation – e.g. is it primarily concerned with
embedding broad entrepreneurial skills, as in many school-based programmes, or is it offering
practical advice and skills training related to business start-up.

 What do we want to know about the programme? – e.g. do we want to assess the implementation
and delivery of the programme or do we want to measure its impact?

Two types of evaluation are commonly used in the evaluation of education for entrepreneurship
programmes:

 Summative evaluation focuses on the outputs or outcomes achieved by the programme, and is
sometimes called impact evaluation. This may be done through assessing the post-programme
destinations or behaviour of participants or the cost-effectiveness of programme. It often requires
an assessment of what would have happened if the programme had not been in place.

Evaluation of Berger Entrepreneurship Program (Ref 001)

The Berger Entrepreneurship Program was launched by the University of Arizona in 1983. The evaluation of this
programme, which was conducted in 1999, was concerned with assessing its overall effect on those who had
graduated. In addition, the effects on technology transfer from the university to the private sector, and of the
entrepreneurship curriculum on other disciplines, were measured.

Evaluation of Junior Achievement Student Mini-Company Program (Ref 004)

This Netherlands-based programme seeks to develop an understanding of entrepreneurship by enabling vocational

college students to set up and operate a mini-company. The evaluation focused on the programme in 2005-06, and
assessed the outcomes in terms of the impact on the students‟ entrepreneurial skills.

60
  Formative evaluation, or process evaluation, assesses the process through which the education
for entrepreneurship programmes have been designed and implemented, in order to identify ways
in which improvements to how the programme is delivered can be introduced. It can explain
how, why, and under what conditions a programme worked, or failed to work. The findings of a
formative evaluation process can provide valuable feedback on issues such as the planning and
design of courses, as well as aspects of the content and the teaching methods employed. For
example, the evaluation may ask whether the programme is fulfilling its objectives and reaching
those who were originally identified as intended participants.

Enterprise Learning at Key Stage 4 (Ref 015)

This formative evaluation considered a variety of secondary school based programmes in the United Kingdom which
aimed to develop entrepreneurial attitudes and skills. The purpose of the evaluation was to identify and recommend
good practice in the implementation of relevant programmes.

Northland Enterprising Teachers Initiative (Ref 002)

The Northland Enterprising Teachers Initiative is a professional development programme in New Zealand which seeks
to encourage an „enterprising‟ approach to teaching across the curriculum. The evaluation is a key element of this
process as it assesses changes in pedagogy and school planning emanating from schools‟ involvement in the
programme.

Importantly, evaluations can be designed to capture data which will fulfil the requirements for both
summative and formative evaluations, as the following examples indicate:

Evaluation of Xlerate Programme (Ref 011)

Xlerate was introduced in Scotland with the aim of targeting young people in school who were perceived to be under-
achieving, possibly due to their resistance to a formal school curriculum. The evaluation, which was conducted
throughout 2005, sought to assess the whole programme, with a focus on both process and outcomes.

Regional Education for Enterprise Clusters (Ref 012)

This is a secondary school based programme in New Zealand, which is designed to develop young people‟s
competences in innovation and managing and setting up a business. The first report of a two year evaluation
investigates processes and outcomes, provides formative feedback and assesses conceptual viability and progress
towards sustainability.

Longitudinal evaluation

The underlying premise of education for entrepreneurship programmes is that they fundamentally
change attitudes and mindsets among their participants. However, the timescale involved in such changes
cannot be predetermined. For some, it may occur relatively quickly, while for others, the lessons learned
may not become apparent for some time. In terms of the evaluation of programmes, this can present a
problem as far as the timing of the fieldwork for the evaluation is concerned. Moreover, observable
manifestations of impact, as evidenced by entrepreneurial behaviour, may also take a considerable period
of time to emerge.

For those who are exposed to education for entrepreneurship programmes during secondary
education, and continue into tertiary education before entering the labour market, the time lag between
participation in the programme and making career decisions will make it difficult to assess the relevance of
inputs from their time at school. Also, the timescales required for the emergence of meaningful data vary
considerably between different types of measures.

61
 An acknowledged deficiency in the evidence base for the impact of education for entrepreneurship
programmes is the scarcity of longitudinal research. Notwithstanding, these difficulties, it is clear that,
wherever resources and timescales permit, longitudinal evaluation offers the prospect of deriving robust
findings on which policy-makers may base recommendations and decisions with confidence. Moreover, a
longitudinal study enables variables such as commitment and attitudes to entrepreneurship to be measured
at the outset, with shifts in these attitudes being measured over time.

Enterprise Insight Hub Evaluation Study (Ref 008)

This evaluation consisted of a tracking study to assess changes over time in young people‟s attitudes towards
enterprise, following the launch of the Make Your Mark Campaign.

Enterprise Education in Ireland (Ref 018)

This longitudinal study of a group of participants in education for entrepreneurship programmes in higher education
included a control group in the first survey. The participants were followed up five years after the initial survey and
their attitudes to entrepreneurship and business start-up explored.

Post-course evaluations

Quantitative data are most frequently collected and evaluations are often carried out in the form of a
post programme or activity evaluation form. These typically seek information on:

 participant‟s level of satisfaction with aspects of the programme;

 changes in attitudes;

 knowledge about the areas covered in the programme; and

 perceptions about skills development.

This may be derived from: those designing the programme; teachers or providers; and participants.

In some cases, evaluation of education for entrepreneurship programmes is conducted solely by

assessing the views and experience of the teachers or business partners involved in the programme and
does not ask students about their experience.

Who should undertake the evaluation?

An important consideration when designing or commissioning a programme evaluation is whether

this should be conducted by individuals from within the institution or agency providing the programme, or
whether it should be carried out by external specialists.

 an external evaluator offers greater independence and objectivity, as well as the possibility of
introducing a fresh perspective or ideas;

 an internal evaluator may offer a greater depth of understanding of the objectives and
implementation of the programme, as well as getting better co-operation from those delivering
the programme, and having the ability to effect change on policy and practice to a greater degree.

However, in most circumstances, external evaluation should be the preferred option, because of the
greater likelihood of objectivity and for the subsequent credibility of the findings. This need not

62
necessarily be people who are external to the institution at which the programme is being delivered, but
should certainly be those who are external to the programme itself.

In some circumstances peer evaluation, involving a similar programme which is undergoing

evaluation, may be used.

Approach to evaluation

There is no single approach to the evaluation of education for entrepreneurship programmes, and
therefore no single model which can be applied in all situations. It is more appropriate to think of
evaluation in terms of a range of options or tools which can be selected and used according to the measure
or programme being assessed. Clearly, factors such as the level of resources, both financial and
professional time, which can be allocated to the evaluation, and the timescale within which results are
required, need to be taken into account in deciding on the most appropriate approach to the evaluation.
However, the type of programme being evaluated (e.g. whether to encourage the development of
entrepreneurial attitudes, or to provide specific „training‟ in the skills needed for business start-up) will be
crucial, as will its scope (is it local or national?), its method of delivery, and whether the programme is
voluntary within an institution or compulsory (in which case, it may not be possible to identify a control
group).

A broad distinction can be made between quantitative and qualitative evaluation.

Quantitative evaluation

Quantitative evaluation entails collecting data from large samples of respondents often through a post
-programme or activity evaluation form, or a large-scale survey. For example, programme participants may
be targeted, rather than those delivering the programme, whose numbers would be smaller – although for a
national programme which was delivered in the education system, it may be appropriate to survey teachers
to ascertain their views on the delivery of the programme. They may be used in conjunction with
qualitative assessments of attitudinal or perception shifts in a mixed method approach.

Determined to Succeed (Ref 010)

The largely quantitative evaluation of the Scottish Executive‟s Enterprise in Education strategy comprised telephone
interviews with 469 teachers and 500 parents of primary and secondary school pupils, as well as 3,700 self-completion
questionnaires from pupils.

Enterprise Learning at Key Stage 4 (Ref 015)

This formative evaluation considers a variety of secondary school based programmes in the United Kingdom which
aim to develop entrepreneurial attitudes and skills. The purpose of the evaluation is to identify and recommend good
practice in the implementation of relevant programmes.

Qualitative evaluation

Qualitative evaluation, and the methods associated with it, enables issues to be explored in greater
depth by seeking to gain a better understanding of the perceptions and attitudes of programme participants
and/or those delivering the programme. In particular, it may be used to identify shifts over time in
attitudes to entrepreneurship. It is also highly appropriate for process evaluation, through gaining the views
of participants and deliverers on how the programme has been implemented, and how improvements can
be made.

63
 The Dynamo Programme (Ref 017)

This programme is targeted at primary and secondary school pupils, using role model networks and curriculum
materials. The evaluation had three qualitative strands: a review of relevant literature and documentation; case studies
of schools and colleges of further education; and interviews and group discussions with key stakeholders.

Determined to Succeed (Ref 003)

The qualitative evaluation of Determined to Succeed sought to assess the impact of the programme on enterprise
education in Scotland more generally by gathering the views of stakeholders. The methodology incorporated a
combination of desk research, interviews with policy makers, key stakeholders, and local programme co-ordinators,
and case studies of its operation in schools

Mixed methods

Many of the larger-scale evaluations of education for entrepreneurship programmes adopt a mixed
method approach, incorporating elements of both quantitative and qualitative evaluation. Thus, the former
may focus on establishing the characteristics and post-programme destinations of participants, while the
latter may concentrate on perceptions of the usefulness of various components of the programme.

Northland Enterprising Teachers Initiative (Ref 002)

The methodology adopted for this evaluation combined the qualitative elements of case studies and surveys within a
small number of selected schools, and a quantitative strand where a postal survey of teachers in a range of schools
was administered.

Enterprise Learning at Key Stage 4 (Ref 015)

A mixed method approach was adopted for this evaluation, with qualitative elements of: observation of enterprise
classes; visits to schools; and interviews with headteachers and providers of enterprise programmes. In addition, a
quantitative survey of schools was undertaken.

Impact evaluations

A key element of evaluation approaches which intend to assess the impact of a particular programme
is the counterfactual. This commonly involves the setting up of two groups from which to gather data – a
„treatment‟ group, composed of those participating in the education for entrepreneurship activity, and a
„control‟ group, composed of individuals with comparable characteristics, but who do not participate in the
activity.

Evaluation of Junior Achievement Student Mini-Company Program (Ref 004)

The evaluation of this programme used a difference-in-difference approach with a control group of students (achieved
sample of 220 students) who did not participate in the programme as a comparison to the participating group (achieved
sample of 189 students).

Whole School Approach (Ref 009)

The evaluation of the Whole School Approach to Enterprise in Education initiative in Scotland examined the impact of
the programme on „total enterprise‟, and how the programme might be applied more widely.

However, experiments of this type are difficult to carry out in practice, and assume a „common effect‟
across the whole treatment or control group. Also, the effects may only apply within the particular setting
(e.g. educational level, geographical area) of the programme being evaluated, so that it may not be possible

64
to say what would happen when a programme is generalised in a different setting. It is therefore advisable
to complement this approach with non-experimental methods.

Other variants of this approach may be applied. For example, „matching‟ allocates each programme
participant a comparable member of the control group of non-participants. This matching is usually made
in terms of characteristics such as age, gender, socio-economic group, and level of qualifications achieved.
For example, the evaluation of the Berger Entrepreneurship Programme at the University of Arizona (Ref
001), which surveyed a „treatment‟ group of 460 entrepreneurship graduates (achieved sample of 105) and
a control group of 2,024 non-entrepreneurship graduates (achieved sample of 406) controlled for year of
birth, gender, ethnicity, high school graduation year, and educational and employment history.

A derivative of this approach is Propensity Score Matching (PSM), which uses a single measure (the
propensity score) as an indicator of the likelihood of participation on the programme. There is therefore an
assumption that the participants and their matched non-participants had an equal likelihood of embarking
on the programme. This being the case, differences in, for example, attitudes, outcomes and activities can
be attributed to the effects of the programme.

An approach which is favoured in a number of the more sophisticated evaluations of education for
entrepreneurship programmes is that of difference-in-difference, whereby comparisons of the attitudes and
experiences of the participant group and the non-participant control group are made both before and after
the attitudes and experiences of the period of the programme. This enables a more robust measure of the
impact of the programme to be derived. The study of the impact of entrepreneurship education on
entrepreneurship competencies and intentions, conducted by the University of Amsterdam and the
Tinbergen Institute in the Netherlands (Ref 004), adopted just such a framework. Outcome variables were
measured before the start of the programme and at the end of the programme to produce “unbiased
estimates of the program‟s impact”.

Non-experimental methods can be powerful evaluation tools, particularly if the process underlying the
outcome variables is well known, and data exist on the relevant measures. In such cases, matching offers a
suitable method of evaluation. If it can be combined with a difference-in-difference approach, then certain
unobserved measures can also be controlled for, which may improve the quality of the study.

Targeting and sampling

Data may be collected from a range of people, depending on the nature of the programme and the
objectives of the evaluation. Typically, the key respondents from whom this data would be obtained in the
course of evaluations of education for entrepreneurship programmes are participants, non-participants,
teachers/lecturers, other staff from providing institutions and stakeholders, such as those responsible for
sources of funding, employers etc.

Many evaluations require data to be collected from a sample of the total number of
participants/stakeholders/programme deliverers etc. This may require:

 random sampling, where each member of the population has an equal chance of being selected,
and which can ensure equal probability. A table of random numbers is used to select the sample.

 systematic sampling, which is very similar to random sampling, the difference being that,
instead of a table of random numbers, the total population (N) is divided by the size of the sample
required (n). If n is 100 and N is 1,000, the sampling fraction is 1/10 and therefore every 10th
person in the sampling frame will be selected for the sample.

65
Data collection

The methods of data collection to be employed for an evaluation will be partly determined by:

 the resources available – both financial and in terms of staff time

 the timescale of the evaluation – i.e. when the results are required

 what type of information is being sought

 who will be targeted to provide the data (e.g. programme participants, delivery staff etc)

A variety of methods may be considered. Here, we set out those most commonly used, with the
advantages and disadvantages of the methods outlined. Clearly, they are not mutually exclusive, and an
evaluation may comprise a combination of different methods.

Quantitative surveys

Quantitative surveys enable data to be collected from large numbers of respondents. In order for the
results to be valid and credible, a pre-determined set of questions is administered in the same way to all
respondents. This tends to make them more appropriate for impact evaluations than for formative
evaluations, where greater depth is required.

The questionnaires which are used to collect the data may be administered in a number of ways,
principally:
 postal, for self-completion
 examples of postal or self-completion questionnaires can be found in the appendix to the
evaluation of the Northland Enterprising Teachers (NET) initiative (Ref. 002), and the
appendix to the national evaluation of Determined to Succeed report (Ref. 010). Other
studies which used these research instruments are: Whole School Approach to Enterprise in
Education (Ref. 009), Regional Education for Enterprise Clusters (Ref. 012) and Enterprise
Learning at Key Stage 4 (Ref. 015)
 telephone interview
 examples of evaluations where telephone interviews have been used include: evaluation of
Enterprise Education in Schools (Ref. 016), with the questionnaire used for the interviews
with schools representatives in an appendix; the national evaluation of Determined to
Succeed report (Ref. 010); Whole School Approach to Enterprise in Education (Ref. 009);
and the evaluation of Xlerate (Ref. 011).
 face-to-face interview
 the evaluations of Enterprise Education in Schools (Ref. 016), and the Whole School
Approach to Enterprise in Education (Ref. 009) provide examples of face-to-face interview
schedules.
 online self-completion
 online self-completion questionnaires were used in: the Make Your Mark Campaign
evaluation (Ref. 008); the Whole School Approach to Enterprise in Education (Ref. 009); and
the evaluation of Xlerate (Ref. 011).

66
 The postal and online self-completion methods are considerably cheaper than telephone or face-to-
face interviewing, can be delivered quickly to large numbers, and do not run the risk of interviewer bias
affecting the responses which are recorded. However, they require the questions to be relatively simple
and unambiguous, as there is no control over whether the respondent has understood the question in the
way in which it was intended to be understood.

Questionnaires

The advantages and disadvantages of self-completion questionnaires can be summarised as follows:

Advantages

 Low cost
 Can be administered quickly to large numbers
 Avoid interviewer bias
 Standardised questions make all responses comparable
 Relatively easy to analyse (online questionnaires can be pre-coded)
 Respondents have time to think about their answers

Disadvantages

 Low response rate

 Self-selected sample, which may be biased
 Require simple, unambiguous questions to avoid misunderstandings by respondents
 Questions are not necessarily answered in the same order
 Difficult to explore complex issues in depth

Telephone interviews, while being more expensive and time-consuming than self-completion
methods, are also relatively inexpensive. However, there are drawbacks in terms of difficulties in reaching
some target respondents, and time restrictions – the length of time which individuals can be expected to
give to a telephone interview limits the degree of depth which can be explored.

Online questionnaires are extremely cost-effective, but the sample obtained may be skewed due to
some potential respondents not feeling comfortable or confident in responding in this way.

Questionnaire design

The design of the questionnaire can be critical, not only in encouraging an adequate response rate, but
also in ensuring that the data obtained meets the objectives of the survey. It is therefore essential that all
questions are rigorously assessed in terms of whether they will be likely to provide information which is
relevant to the purpose of the study.

A choice can then be made about whether the questions should be „open‟ or „closed‟. Closed
questions are those which restrict the respondent to a selection of pre-coded answers. They have the
advantages of being easier to analyse, because the range of possible responses is known in advance, and a
coding frame can be formulated before the questionnaire is administered. For evaluations of education for
entrepreneurship programmes, they may be used to determine the characteristics of participants and the
programme in which they were engaged, as well as providing evidence of the impact of the programme, for
example in determining the post-course activities of participants. However, by restricting the potential
range of answers in this way, they may inadvertently force responses into categories which are not wholly
representative of the respondent‟s views.

67
 Open-ended questions allow the respondent to answer in their own words. Importantly, they also
allow the respondent to answer in whatever way they wish. While this may generate a wider range of
responses, and may more accurately represent the perspectives of those participating in or delivering the
programmes, it does make analysis of the responses from the whole sample more difficult and time-
consuming. There can also be a problem if the respondent strays from the main issues being addressed.

Whichever type of question is used (and many questionnaires comprise a combination of closed and
open-ended questions), the layout of the questionnaires can be vital in eliciting a good response rate. The
key points to remember are:

 keep the length of the questionnaire within reasonable limits, so as to avoid putting off the
respondent;

 ensure that the layout is consistent, and easy to navigate;

 allow sufficient space for open-ended questions;

 be clear and precise in instructions for completion, especially in terms of the routing of questions

Qualitative research

Qualitative methods enable issues to be investigated in greater depth. This is especially important
when conducting formative evaluations, where perceptions of the way in which an education for
entrepreneurship programme was delivered, and its subsequent usefulness and value need to be examined
in detail. The methods used may include in-depth face-to-face interviews, focus groups, case studies and
observation. For example, the evaluation of Determined to Succeed in Scotland (Ref. 003) incorporates:
interviews with policy-makers and local authority co-ordinators, as well as case studies, while the
evaluation of the Dynamo Programme in Wales (Ref.017) has a combination of a literature review, school
and college case studies, interviews and group discussions.

Interviewing

Face-to-face interviews may be structured in different ways, ranging from those which follow a fully
structured questionnaire, to those which are semi-structured or unstructured, using topic guides which
identify the broad themes or topics to be discussed, rather than specific, tightly-defined questions. In all
cases, they enable the investigation of views, experiences, outcomes etc to be conducted in greater depth.
This is particularly important where complex issues, such as the degree to which a particular programme
has impacted on an individual‟s acquisition of entrepreneurial skills, or on their preparedness to
countenance setting up their own business, are concerned.

The advantages and disadvantages of interviews can be summarised as:

Advantages

 High response rates can be achieved

 Complex issues can be explored in depth
 Respondents can be asked to clarify or expand upon the answer they have given
 The data obtained is often rich in terms of detail and insights
 Sensitive issues can be probed
 Verbatim quotation of responses can provide powerful evidence when reporting

68
Disadvantages

 Time-consuming to set up and carry out

 Relatively expensive
 More difficult and lengthy analysis phase
 Susceptible to interview bias or misinterpretation
 Require considerable skill on part of the interviewer

Focus groups

Focus groups composed of samples of programme participants or course teachers, enable people to
share experiences and draw out a range of views or perceptions, e.g. about the usefulness of a particular
education for entrepreneurship programme or the way in which it is delivered. They can be particularly
useful when conducting formative evaluations of programmes, as they can encourage participants in the
groups to develop ideas for addressing shortcomings or problems and for improving the programme.
Examples of the use of focus groups include the evaluations of the Whole School Approach to Enterprise
in Education (Ref. 009), Regional Education for Enterprise Clusters (Ref. 012) and Xlerate (Ref. 011).

The advantages and disadvantages of focus groups can be summarised as:

Advantages

 Cheaper than individual interviews

 Highlight a range of perspectives
 Generate fresh insights into issues
 May develop solutions or improvements to current practice

Disadvantages

 Time-consuming to set up and carry out

 Relatively expensive
 Require considerable skill on the part of the interviewer
 Difficult to analyse and attribute weight to points made
 Discussion may be dominated by a few individuals in the group

Case studies

Education for entrepreneurship programmes lend themselves to different types of case studies. For
examples, case studies of individuals could track their experiences of a programme, and identify
difficulties which had occurred and benefits which had accrued. The individual who was the subject of the
case study would be interviewed at regular intervals and possibly asked to record events and their views
about them in a diary.

In contrast, a case study could be undertaken of the implementation and delivery of a programme
within an institution. This may involve regular collection of information, by any of the methods described
above, from a range of key stakeholders, notably programme teachers, the management of the institution,
funding bodies etc.

The evaluation of the Northland Enterprising Teachers (NET) initiative in New Zealand provides a
good example of the case study approach. In order to assess the effect which the programme had had on
the schools, and especially on the curricula, the case studies involved:

69
  a structured interview with the principal;

 a structured interview with the enterprise co-ordinator;

 collection of any school documentation which might include reference to Education for
Enterprise (E4E) (school policy documents, planning and reporting frameworks, curriculum
plans, etc.);

 (where possible) a structured interview with another teacher involved in E4E; and

 focus group interviews with a sample of students involved in E4E.

Observation

Observation is another qualitative method of collecting data, especially about the delivery of courses
and the responses or activities of programme participants. The evaluation of Enterprise Learning at Key
Stage 4 in the United Kingdom (Ref. 015) included observations of enterprise classes.

Evaluation of Young Enterprise Scheme (Ref 027)

Members of the evaluation team observed the activities (e.g. decision-making about products, marketing and individual
roles) of a group of scheme participants over a seven month period. This entailed being present at 15 team meetings,
as well as the group‟s presentation at a Trade Fair. Other groups were observed on a less frequent basis. The
observations enabled the evaluation team to formulate their survey research instruments.

The advantages and disadvantages of observation are:

Advantages

 Provides greater insight of programme delivery and participant behaviour than is possible
through individuals‟ accounts
 The data collected is rich

Disadvantages

 Difficult to gain permission for access

 High cost
 Time-consuming
 Potential for bias or misinterpretation on the part of the observer
 Those who are being observed, and are aware of it, may inadvertently behave differently to
their normal behaviour.

Analysing data

An essential aspect of the process of planning an evaluation is to think through, at the outset, how the
data collected will be analysed. It is definitely not something which can be left until after the data has been
collected, as it requires adequate time, resources and expertise to be allocated to it. Moreover, the
capability and capacity for undertaking analysis should be a key factor in deciding the nature and content
of the instruments to be employed in data collection.

70
 The nature of the data collected, whether it is quantitative or qualitative, will determine the type of
analysis to be carried out. At a basic level, decisions about how the data is to be analysed will be
dependent on:

a. What the evaluator wants to know; and

b. What the „findings‟ will be used for

As far as a) is concerned, a choice can be made between:

 Univariate analysis, which focuses on one characteristic of the sample (e.g. the proportion of a
course cohort who successfully completed the course). This is essentially descriptive, with the
results often presented in the form of a frequency table. An important consideration when
calculating percentages is whether to include those in the sample who did not respond to a
particular question. If they are excluded, then this must be made clear.

 Bivariate analysis, which focuses on two characteristics, or variables, and attempts to establish a
relationship between the two. The data is presented in crosstabulations, which enable an
association between the two variables to be identified. For example, there could be a relationship
between participants‟ prior educational achievement (or their social class background) and their
level of attainment on a course.

 Multivariate analysis enables examinations of more than two variables to be carried out.

Reporting and interpreting findings

When undertaking the reporting and interpretation of the findings from an evaluation, it is important
to re-visit the objectives which were established at the outset of the evaluation. This will then lead on to
identifying two crucial components of the reporting process:

1. The target audience for the findings of the study. There is likely to be a number of individuals and
groups who will be interested in at least some aspect of the report. For example, national policy-
makers may use the findings on the impact of a programme to decide on its future sustainability,
especially where government funding is provided. Evidence of a positive impact could be used to
support the case for future funding. They may also need to gain information about the
implementation and delivery of the programme, so that any lessons learned can be taken on board
when deliberating on the appropriate design for a more widespread roll-out of the programme.

At the same time, provider institutions (e.g. schools, colleges, higher education institutions) may
be interested in those aspects of programme content or delivery mechanisms which have been
well-received by participants, or which have been regarded as problematic, in order that curricula
and administrative arrangements can be amended appropriately. Participants in the study, and
students more generally, will be interested, especially if the evaluation throws light on the possible
outcomes of their engagement with an education for entrepreneurship programme.

2. What are the key themes and issues emerging from the evaluation which will deepen our
understanding of the processes involved and help to shape future formulation and delivery of
education for entrepreneurship programmes? The presentation of findings should reflect the
relative importance of key themes and issues which emerged. This should be apparent in the space
and emphasis which they are accorded, so that the balance of the report is in accordance with the
weight of the data.

71
 Different formats and methods can be used for disseminating the findings of an evaluation, including
seminars, conference papers, journal articles, press releases, Powerpoint presentations etc. Some will be
more appropriate for different audiences than others. However, the core output of an evaluation is likely to
be a written report. The structure of that report will depend, among other things, on the nature of
evaluation and the number of data-gathering strands which were used. Pfatteicher et al (1998) suggest that
the main components of any report should be:

 Introduction – this provides an appraisal of the purpose and objectives of the evaluation,
including the key themes and issues pertaining to the focus of the study, and the contextual
background, which may refer to the policy context in which the programme under evaluation has
been developed. It will also present the reader with a signpost for what will be addressed in each
of the following sections of the report

 Methods – the methods for identifying and selecting the sample, and gathering and analysing the
data will be discussed, along with a description of the characteristics of the sample

 Findings/Themes – this represents the core of the report, with a description and interpretation of
the findings. Tables, charts and quotations from respondents may be used to emphasise the main
findings

 Conclusions – this section may be the most significant for policy-makers, as it provides the
opportunity to draw together the main points arising from the study and to discuss their
implications. It may also be used to spell out some recommendations for future policy and
practice

Appendices – these may be used to locate copies of the research instruments and other material, such
as detailed analytical tables, which may be referred to by the reader to substantiate and gain a deeper
understanding of the basis on which the finding

72