Impact Study Methodology PDF

IST Impact Study
Microelectronics & Microsystems, Healthcare,

Mobile communications
Methodology
Databank Consulting
empirica
Idate
Teleport Sachsen Anhalt
Wise Guys Ltd.
IST Impact Study Methodology
Foreword
This Report outlines the methodological framework of the IST Impact Study, as well as the
methodological approach applied in, on the one hand, the questionnaire survey and, on the
other hand, the stakeholder interviews.
This study was undertaken by Databank Consulting in collaboration with the tender partners
empirica GmbH, Idate and Teleport Sachsen Anhalt in 2004.
Annexes to the report are the Interview Guide (Annex 1) and a word version of the (electronic)
Survey Questionnaire (Annex 2).
Executive Summary
The Impact Study had the following main goals:
To develop a pragmatic methodology for the systematic impact assessment of IST RTD
activities;
To implement the methodology on the RTD projects supported by FP4 and FP5 in three
domains: Health applications, Mobile communications and systems, microelectronics and
Microsystems, analysing their socio-economic impacts;
To assess the scope and scale of impacts in such a way as to enhance assessments of
programme performance and suggest recommendations for the future.
The studys methodological framework built on the basic model of causality which considers
that RTD projects have downstream consequences that can be described in terms of a range of
project outputs (publications, new tools and techniques etc.) and a range of resultant outcomes
affecting different sets of actors, often described as the research teams themselves (first circle
actors); the organisations to which these teams belong (second circle actors); the immediate
users of project results or outputs (third circle actors etc.) and numerous actors even further
downstream (fourth circle, fifth circle actors etc.), who are often given the collective label of
society at large.
Causality and attribution become more difficult to establish the further downstream one goes
along the impact chains, and the study attempts to solve these difficulties to assess socioeconomic impacts on the society at large by
Applying a longitudinal (survey) and horizontal (interviews) approach
Assessing the socio-economic impacts on the immediate user communities
Assessing the relative impacts of the projects regarding impacts on the society at large.
The Impact Study assessed the performance of the projects and programmes focusing on the
following evaluation issues:
Effectiveness ! have the goals been attained?
Efficiency ! what is the return of investment?
Impacts ! what has happened as a consequence of the project outputs?
Additionality ! what has happened which wouldnt have happened otherwise?
Relevance ! how important are these impacts for all parties and audiences concerned?
Progress factors ! which factors have affected the progress of the projects?
The elaboration was carried out according to the following main dimensions:
Type of result
Type of Project
Class of stakeholder /typology of organisation
Project cluster/action line
Domain
Programme (IST vs FW4)
A key feature of the methodology is the analysis of results by type of stakeholders.
Data analysis techniques included multivariate techniques capable of identifying groups of

results and type of stakeholders/projects (cluster analysis) and searching for correlations
between them. Correlation analysis was preceded by variable reduction techniques such as
Principal Component Factor analysis.
The statistics of results were further elaborated to produce synthetic performance indicators by
type of result at Domain/ Programme level, to allow a synthetic assessment of achievements.
The following main categories of indicators have been included in the questionnaire:
Effectiveness indicators;
Efficiency indicators;
Additionality indicators;
Progress factors
Input indicators
Output indicators
Outcome indicators
Socio-economic impact indicators
The main goal of the qualitative interviews, instead, was to characterise each domain and gain
strategic perspectives on the role of the Commission in the creation of sustainable innovation
within these domains.
Their objectives were therefore twofold:
To reach an overall view on the strategic role of the Commission in the various market
sectors, provided by key opinion leaders/stakeholders that are having a long and deep
experience of research projects and of the domain as a whole. This view will provide a
'landscape' for the strategic impact of the Commission action in each of the three
domains.
To reach understanding of the key common and differentiating factors in the different
domains
Among the issues to address during these interviews were:
The drivers for the technical/commercial development in the domain
The relevance of the instruments and the effects of the EC support
The trend in the industry RTD
The strategic objectives of the domains
From a methodological perspective, the stakeholder interviews had the objective to provide the
study with needed qualitative information in order to allow for a correct interpretation of the
statistical data, setting these into the societal context. Furthermore, they were to provide a
qualitative view on exactly those impacts downstream the impacts chains that are difficult to
assess using the questionnaires tools.
The choice of the study to interview key opinion leaders in the three different stakeholder
communities was fundamental from a methodological point of view as it marked the line of
continuity between the two assessment tools.
Table of Contents
1
CONCEPTUAL FRAMEWORK ....................................................................................8

1.1
1.2
1.3
1.4
OBJECTIVES OF THE STUDY .................................................................................................................. 8

OVERALL METHODOLOGICAL APPROACH ........................................................................................... 8
1.2.1
The limitations of impact assessments ..................................................................... 8
1.2.2
The relevance of the stakeholders analysis ........................................................10
METHODOLOGICAL APPROACH TO THE QUESTIONNAIRE SURVEY .............................................11
1.3.1 Goals ..........................................................................................................................11
1.3.2 Scope of the questionnaire survey .................................................................11
1.3.3 Conceptual issues .................................................................................................12
1.3.4
Evaluation focus........................................................................................................17
1.3.5 Models, assumptions and hypotheses ...........................................................17
1.3.6 Definition of the variables ...................................................................................17
1.3.7 Data elaboration ...................................................................................................18
METHODOLOGICAL APPROACH TO THE STAKEHOLDER INTERVIEWS ..........................................19
1.4.1 Scope of the selected interviews ....................................................................19
1.4.2 Quantitative versus qualitative data..............................................................20
THE QUESTIONNAIRE SURVEY ................................................................................22

2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
2.10
2.11
STRUCTURE OF THE QUESTIONNAIRE ...................................................................................................22

CATEGORIES OF INDICATORS ...........................................................................................................22
TERMS OF MEASUREMENT .................................................................................................................22
EFFECTIVENESS INDICATORS ..............................................................................................................23
EFFICIENCY INDICATORS ..................................................................................................................23
ADDITIONALITY INDICATORS .............................................................................................................24
PROGRESS FACTORS .......................................................................................................................25
INPUT INDICATORS ...........................................................................................................................26
2.8.1
The project input indicators ....................................................................................26
2.8.2
The organisation typologies ....................................................................................28
OUTPUT INDICATORS .......................................................................................................................28
OUTCOME INDICATORS ...................................................................................................................29
SOCIO-ECONOMIC IMPACT INDICATORS ...........................................................................................31
2.11.1 Definition of the user communities.........................................................................31
2.11.2 Impacts on the immediate user communities.....................................................31
2.11.3 Impacts on society at large ....................................................................................32
IN-DEPTH INTERVIEWS.............................................................................................34
3.1
3.2
OBJECTIVES ...................................................................................................................................34
INTERVIEW PROCESS ........................................................................................................................34
Annex 1: Interview guide

Annex 2: Survey questionnaire for the RTD projects (word version)
Annex 3: Survey questionnaire for the Support Measure projects (word version)
List of Exhibits
Table 1 Projects in the targeted domains ...............................................................................................11
Table 2 Action types included in the projects database.....................................................................14
Table 3 Action types groupings ................................................................................................................16
Table 4 Mapping of the variables ............................................................................................................17
Table 5 Effectiveness - indicators & terms of measurement ................................................................23
Table 6 Cost/benefits ratio - terms of measurement.............................................................................23
Table 7 Leverage effect indicators & terms of measurement..........................................................23
Table 8 Attribution indicator & terms of measurement .....................................................................24
Table 9 Additionality - indicators & terms of measurement.................................................................24
Table 10 Progress factors indicators & terms of measurement ........................................................25
Table 11 Project type indicators & terms of measurement..................................................................26
Table 12 Projects in sequence indicators & terms of measurement ..............................................27
Table 13 Network effects in projects in sequence - indicators & terms of measurement ...........27
Table 14 User-orientation of the projects indicators & terms of measurement .............................27
Table 15 Organisation typology indicators .............................................................................................28
Table 16 Output indicators & terms of measurement...........................................................................28
Table 17 Outcome indicators & terms of measurement ......................................................................29
Table 18 Industry-science interaction indicator & terms of measurement ....................................30
Table 19 Impacts on the workforce indicators & terms of measurement ......................................30
Table 20 Implementation/Dissemination indicators & terms of measurement .............................30
Table 21 User communities identification indicators & terms of measurement ............................31
Table 22 Impacts on the immediate user communities indicators & terms of measurement....32
Table 23 Impacts on society at large indicators & terms of measurement ...................................33
Figure 1 Overall methodological approach ...........................................................................................10

Figure 2 Overall Performance Indicator development ............... Errore. Il segnalibro non definito.
Figure 3 Quantitative versus qualitative data .........................................................................................20
Conceptual framework
1.1
Objectives of the study
The Impact Study had the following main goals:

To develop a pragmatic methodology for the systematic impact assessment of IST RTD
activities;
To implement the methodology on the RTD projects supported by FP4 and FP5 in three
domains: Health applications, Mobile communications and systems, microelectronics and
Microsystems, analysing their socio-economic impacts;
To assess the scope and scale of impacts in such a way as to enhance assessments of programme
performance and suggest recommendations for the future.
1.2
Overall methodological approach
The methodology proposed was based on a combination of quantitative and qualitative

methodologies that take into account:
The micro level, i.e. the project participant/ single project level;
The meso level, i.e. the Domain level and the Programme level (ACTS, TAP, ESPRIT, IST);
The macro level, i.e. the EU social and economic system
The field research was based on:
A structured questionnaire survey to projects participants, followed by telephone interviews
aimed at validating data;
Selected interviews with key stakeholders and opinion leaders in the Healthcare, Mobile and
Microelectronics domains, to investigate the impacts of the Community Framework
Programmes investments on the macroeconomic and social environment.
The survey addressed data gathering at micro level; the aggregation and elaboration of survey data
were to provide results at meso level; the qualitative interviews were intended to produce results at
the macro level (see Exhibit 1).
1.2.1
The limitations of impact assessments
The basic model of causality underpinning impact assessment is that RTD projects have downstream
consequences that can be described in terms of a range of project outputs (publications, new tools and
techniques etc.) and a range of resultant outcomes affecting different sets of actors, often described
as the research teams themselves (first circle actors); the organisations to which these teams belong
(second circle actors); the immediate users of project results or outputs (third circle actors etc.) and
numerous actors even further downstream (fourth circle, fifth circle actors etc.), who are often given
the collective label of society at large.
The basic problems associated with the assessment of socio-economic impacts along these causal
chains are as follows:
Events (projects) are often associated with multiple consequences (outputs, outcomes, impacts
etc.), not all of which were intended and not all of which may be identified;
Causality has to be demonstrated, i.e. consequences have to be attributed to particular events,
and since many outcomes (e.g. new products, processes etc.) are a consequence of many stimuli
(multiple RTD projects, the availability of investment capital etc.), attribution in anything other
than a trivial sense (RTD project X contributed to the development of product Y) is difficult to
demonstrate or assess in any meaningful way;
Causality and attribution become more difficult to establish the further downstream one goes
along the impact chains. A research project is likely to have noticeable effects on the teams
involved in projects, but very much smaller almost infinitesimal impacts on society at
large;
Causality and attribution are also difficult to establish further downstream because of positive
and negative feedback loops over time.
Exhibit 1
Methodological approach to the causality & attribution issue
Questionnaire survey
Stakeholders interview
Society at large
Immediate target audiences/users

Parent organisations
Longitudinal analysis
RTD teams
involved in the
projects
Horizontal analysis
involved in the projects

Market sectors
Source: Impact Study, 2004

All these problems of complexity, feedback and scale complicate the efforts to assess the socioeconomic impacts of individual RTD projects. Attempts to overcome these problems typically take
two forms:
Longitudinal analyses which focus in the first instance on hard assessments of the more easily
identifiable first circle impacts, before making progressively softer assessments of second,
third, fourth circle impacts etc along the chains identified for each project;
Horizontal analyses which concentrate in the first instance on describing trends in various socioeconomic domains (equivalent to segments of downstream circles corresponding to society at
large) before attempting to associate these with the project portfolios in question via the use of
qualitative techniques (interviews, focus groups) or macroeconomic modelling techniques.
Both longitudinal and horizontal routes have been taken in this study.
1.2.2
The relevance of the stakeholders analysis
A key feature of the methodology was the analysis of results by type of stakeholders. RTD projects in
the analysed domains typically include several different kinds of organisations, such as:
ICT suppliers,
Private companies, research centres, universities,
Government agencies and organisations providing services,
Associations representing end-users (such as the disabled) and often end-users group for trials
and demonstrations.
They play different roles in the projects (project manager, partner producing research results, partner
testing results in a trial, exc.) and have different goals and achievements.
Evaluation studies, over and again, have proven that the nature of stakeholders creates a fundamental
difference in the impacts created/expected, their engagement in the project, the likeliness to follow
through with further investments after the project is ended.
The most relevant differences were related to the segmentation used in this study:
Suppliers, who mainly invest to develop products and services for future commercialisation;
Research institutions, who generally look for knowledge improvements and technological
advances;
Client/ User organisations, who wish to test innovation and eventually implement it.
These considerations have also led to the decision to conduct the interviews with key opinion leaders
as key stakeholders in the targeted domain. The analysis of results by stakeholder type at the
questionnaire level was to be complemented by the qualitative interviews providing more general
views and visions. This logical continuity would help to smoothly connect the two parts of the
study, the quantitative and qualitative, which is normally difficult to do.
Exhibit 2
Overall methodological approach
Project participants
Key stakeholders & opinion leaders
Interviews
Organisations
Micro level
Stakeholders
Project
Programme
Meso level
Domain
Market
sector
Macro level
EU social & economic
system
10
1.3
Methodological approach to the questionnaire survey
1.3.1
Goals
The present methodological framework has been developed in order to allow assessment of the
outputs, outcomes and impacts of the projects taking into account the multiple factors influencing
impacts, such as the attribution of effects to interventions, the heterogeneous interests and
expectations of actors, the variety of policy rationales and purposes, and the multiple levels at which
policies are pursued.
The following definitions have been adopted:
Outputs: the technical results of the projects, such as software tools and management
techniques;
Outcomes: the direct effects of the projects, such as new jobs created, increased productivity,
measurable increases in workplace safety etc.;
Impacts: the wider effects of the programme on society, e.g. faster diffusion of technology,
increased service sector competitiveness.
1.3.2
Scope of the questionnaire survey
The scope of the study concerned research projects supported by the Community Framework
Programme in three IST domains: Health Applications Mobile Communications and Systems and
Microelectronics and Microsystems.
This included all projects that ended before 31 December 2003 and were supported by:
Framework Programme 4:
TAP: Telematics Applications for Healthcare
ACTS: Domain 4 Mobility and Personal Communication Networks
ESPRIT IV: Domain II Technologies for Components and Subsystems
Framework Programme 5:
IST Key Action I.1 and I.2 eHealth
IST Key Action IV.5 Mobile and Personal Communication and systems including satellite
systems and services
IST- Key Action IV.7 Subsystems and Microsystems and IV.8 Micro- and Opto-electronics
Based on Commission information, the total number of projects funded in these domains corresponds
to approximately 700.
Exhibit 3
Projects in the targeted domains
Domains
Time frame
FP4 (1994-98)
Nr. Projects
FP5 (1998-2002)
(circa)
Nr. Projects Total

(circa)
Domain
Microelectronics, subsystems and

microsystems
ESPRIT
261
KA IV.7 / 8
136
397
Healthcare applications
TAP
130
K.A. I.1/2
80
210
Mobile & Personal

communications and systems
ACTS
37
KA IV.5
60
97
276
704
TOTAL
428
11
by
1.3.3
Conceptual issues
The assessment of social, policy & macro-economic impacts

As has been outlined in chapter 2.2.1, in terms of establishment of causality and attribution,
assessment of the socio-economic impacts of individual RTD projects is increasingly challenging the
further downstream the assessment goes along the impacts chain.
As an attempt to define innovative ways to overcome these problems, the study has adopted the
following methodological approach for the assessment of the social, macroeconomic and policy
impacts:
Assessment of the possible impacts in terms of impacts for the immediate users (third circle
actors), see chapter 3.12.1 ;
Assessment of the relative impact of the projects on society at large (fourth circle), i.e.
compared to other projects (EU, national, in-house) the participants had been active in (see
chapter 3.12.2).
It was the assumption that, compared to the non-EU projects, EU funded projects (because of
their cooperative and pre-competitive nature, as well as due to the ) will show higher relative
impacts in socialoriented as well as knowledge and infrastructure creation areas and lower
relative impacts in economy-oriented ones.
Exhibit 4 Assessment approach
Outputs
Society at large
Impacts
Immediate target audiences/users

Parent organisations
RTD teams
involved in the
projects
involved in the projects

Market sectors
Outcomes
Relative impacts
Source: Impact study, 2004
12
The domains and frameworks

The Study covered projects that were active in three different domains and two different frameworks,
representing three different stages of R&D and specific Commission objectives, as well as two
different eras in terms of societal and technological developments and Commission strategic policy
objectives.
The definition of project typologies through the use of specific input indicators and the introduction
of goal importance and goal attainment indicators throughout the questionnaire, as well as the
correlations between the input and the output indicators provided the study team with the
necessary means to capture these diversities and reach a correct evaluation of the effectiveness and
typology of impacts for the various domains.
In contrast to its original plans, the study team has not foreseen domain-specific indicators for the
assessment of the impacts at the broader societal level.
This issue is closely linked with the socio-economic assessment challenges explained above.
Focusing, in fact, on the assessment of the possible impacts for the immediate users, the expected
possible impacts for the three domains could be brought back to broader categories of impacts,
recognizable to all. The domain specific aspect in this case lies therefore in the (tacit) definition of
the immediate user by the respondent.
13
The action types

The Framework programmes and the different domains are characterised by a vast range of activities,
ranging from fundamental research to experimental developments and accompanying measures.
The official Commission documentation describes as follows the different action types included in the
projects database.
Exhibit 5
Action Type
Action types included in the projects database

Explanation
Access Action (take- Access actions are designed to provide co-ordinated access to advanced, emerging technologies and
up measure)
services, knowledge and competence.
Accompanying
Measure
Actions contributing to the implementation of a Specific Programme or the preparation of future

activities of the programme. They will also seek to prepare for or to support other indirect RTD actions
(financial participation: maximum of 100 % of total eligible costs).
Assessment Action
(take-up measure)
Assessment actions (by users and suppliers) promote the use of innovative equipment and materials in
industrial and service environments through evaluation of innovative products against user
requirements and specifications.
Best Practice Action

(take-up measure)
Best Practice actions, (for users) promote improvements in the practices, processes and operations in
industry and services through the take-up of well-founded, mature and established - but insufficiently
deployed - methods and technologies, so as to achieve greater efficiency, higher quality and greater
economy (in the user organisation).
Projects combining the RTD projects and demonstration actions (financial participation: 35 to 50 % of
total eligible costs)
Combined RTD &

Demo Project
CRAFT Project
Cooperative research (CRAFT) projects are designed to enable small and meduim-sized enterprises
(SME) not able to do research work themselves to either entrust the resolution of their common
technological problems to third legal entities with appropriate research capacities or to jointly try to
resolve them.
Demonstration
Action / Project
Projects designed to prove the viability of new technologies offering potential economic advantage but
which cannot be commercialised directly (financial participation: 35 % of total eligible costs)
Dissemination and
Awareness
These actions aim to stimulate and promote the rapid take-up of RTD results in industry and to enhance
the awareness of RTD activities and results, the IT programme itself and the exploitation of its results.
ESD Cooperative
Research
ESD Cooperative Research is an action to develop and to make available design and test solutions (tools
and methodologies) to users through independent providers. The objectives are to increase the portfolio
of available electronics design and test solutions relevant for European users; and to make these solutions
readily available to the European users with professional support and maintenance.
First User Action
First User Action (FUSE) Stimulating industrial enterprises to incorporate electronic technologies into
their products in the context of an action for first users.
First User
Application
During the Application Experiment (AE), the enterprise, termed First User (FU), will use new
technologies to conduct the specification, design, prototype, manufacture and testing of a component or
system which is relevant to the improvement of its manufactured products or manufacturing processes.
The AE is conducted by the FU in collaboration with subcontractors to ensure that sufficient knowledge
is transferred to the enterprise and a sustained increment in its technological capabilities is achieved.
Network of
Excellence
A network of excellence brings together industry, users, universities and research centres with a common
RTD objective. Networks of excellence can be particularly beneficial for groups and institutions in
outlying regions through the channel they provide for training, technology transfer, and access to
expertise and resources.
RTD Project /
Industrial R&D
Project
Projects obtaining new knowledge intended to develop or improve products, processes or services
and/or to meet the needs of Community policies (financial participation: 50 % of total eligible costs)
14
Action Type
Explanation
SME Exploratory
Award
Feasibility studies, validation, partner search (support of 75 % of total eligible costs for an exploratory
phase of a project of up to 12 months)
Subvention
Does not belong to the official "types of actions supported" according to the work programme; these
projects will be treated as RTD projects
TCS-ESD
Technology for Components and Subsystems - Electronic Systems Design: ESD stimulates the wider
exploitation of state-of-the-art design technology by European industry, thus helping it to stay at the
competitive edge by optimal use of advanced electronics in their systems.
TCS-SEA
Technology for Components and Subsystems - Equipment Assessment (SEA) Assessment of advanced
prototype equipment for the manufacture of electronic components and subsystems.
Technology Support Does not belong to the official "types of actions supported" according to the work programme; there are
Centres and
only 2 projects of this action type (EP21101 Europractice; 29964 Euraccess) which will be treated as
Networks
support measures
Thematic Network
Training networks for promoting training-through-research especially of researchers at pre-doctoral and

at post-doctoral level - and thematic networks for bringing together e.g. manufacturers, users,
universities, research centres around a given S&T objective. Support will cover maximum 100 % of the
eligible costs necessary for setting up and maintaining such networks.
Trial Action (take-up Trials (for users and suppliers) aiming at the adaptation and introduction of leading edge technology
measure)
(promising but not yet fully established) in industrial/service applications and its joint evaluation (by
supplier and user).
User Group &
Working group
User Groups and Working Groups: User groups are intended to co-ordinate specification of user
requirements related to one or more ongoing projects. Working groups aim at improving the systematic
exchange of information and the forging of links between teams which carry out research around a
common theme, through short scientific visits and the organisation of seminars, workshops or
conferences.

This vast variety in action types sets in the first place the requirement to a certain degree of
flexibility in the applied methodological tool in order to be sufficiently relevant to all project
participants, thus avoiding evaluation fatigue and adding on to the user-friendliness of the
questionnaire.
The Impact Study team made the reflection that for a determined group of projects and typically
those who are not involved in any stage of RTD - the project output indicators needed to be heavily
adapted. A different type of questionnaire will therefore be developed for these projects.
For a correct definition of these projects, the project Action Types have been subdivided into two main
groups, RTD and Support measures, according to the following criteria:
RTD projects whose main goal is the development of technologies, applications or services
that are or will be of use to the communities outside of the Commission environment.
Support measures, i.e. those projects whose main goal is not technical RTD, i.e. including
accompanying measures, take-up measures etc.
15
Exhibit 6
Action types groupings
Action Type
Support measures
Access Action
Accompanying Measure
Assessment Action
Best Practice Action
RTD
Combined RTD & Demo Project
CRAFT Project
Demonstration Action / Demonstration Project
Dissemination and Awareness
ESD Cooperative Research
First User Action
First User Application
Network of Excellence
RTD Project / Industrial R&D Project
SME Exploratory Award
no
no
Subvention
TCS-ESD
TCS-SEA
Technology Support Centres and Networks
Thematic Network
Trial Action
User Group & Working group

The Impact Study team decided to not include the SME Exploratory Award in the sample of projects
to be assessed by this study. These Action Types, in fact, were not truly projects but aimed at
providing financial support to SMEs during the exploratory phase of projects.
16
1.3.4
Evaluation focus
The Impact Study assessed the performance of the projects and programmes focusing on the following
evaluation issues:
Effectiveness ! have the goals been attained?
Efficiency ! what is the return of investment?
Impacts ! what has happened as a consequence of the project outputs?
Sustainability ! are the impacts likely to continue into the future in the absence of assistance?
Additionality ! what has happened which wouldnt have happened otherwise?
Relevance ! how important are these impacts for all parties and audiences concerned?
Progress factors ! which factors have affected the progress of the projects?
1.3.5
Models, assumptions and hypotheses
The implicit assumption was that policy interventions in the form of support for collaborative R&D
programmes and projects have a high probability of producing downstream effects that can be
classified in terms of specific outputs, various outcomes for the organisations involved in projects, and
a whole range of impacts on a host of affected audiences and socio-economic spheres. A related
assumption was that a variety of progress factors affect the probability of these downstream effects
occurring.
The hypotheses associated with such a model were that relationships exist between input variables
characterising the structure, organisation, scale and content of projects, and output variables or
performance variables characterising project outputs, outcomes and impacts.
Other hypotheses assumed relationships between exogenous variables (e.g. technological and
commercial developments external to a project) and project progress and performance.
1.3.6
Definition of the variables
In line with the above-mentioned assumptions and hypotheses, the following sets of key variables
have been identified:
Exhibit 7
Category
Input
variables
Mapping of the variables

Variable
Project resources
Nature of work in projects
Typology of the partner organisations
Output
variables
Outcome &
Impact
variables
Progress variables related to the project

characteristics
Expected and Actual project output variables
Expected and Actual first circle impacts
Expected and Actual second circle impacts
Expected and Actual third circle impacts
Expected and Actual fourth circle impacts

Exogenous
variables
Progress variables
environment
related
to
the
external
17
Example
Financial size of project, Commission contribution,
number of project partners etc.
Fundamental/applied research, high/low technical
complexity etc.
Market sector, core activity, size, role in the project,
geographical location etc.
Technical and managerial competence of the
partners, clarity of project goals, etc.
Publications, new tools and techniques etc.
Impacts on knowledge base, scope of RTD, etc.
Impacts on the commercial prospects, strategic
alliances, etc.
Impacts on the quality and efficiency of processes,
technologies, services, knowledge management,
information management etc.
Relative impacts on economic welfare, social
welfare, policy & regulatory environments etc.
Additional support schemes, external technological
and commercial events etc.
1.3.7
Data elaboration
The assessment of project and programme performance is intrinsically associated with the
determination of the (causal) relationships between the above input and performance variables.
The elaboration was carried out according to the following main dimensions:
Type of result
Type of Project
Class of stakeholder /Typology of organisation
Action line/Project cluster
Domain
Programme (IST vs FW4)
The level of segmentation of the elaboration depended on the characteristics of the sample. In most
cases the sample of respondents was not large enough to allow segmentation by type of stakeholders
and type of project at the same time. The most articulate reading of results was done at Domain level.
Data analysis techniques included multivariate techniques capable of identifying groups of results and
type of stakeholders/projects (cluster analysis) and searching for correlations between them.
Correlation analysis was preceded by variable reduction techniques such as Principal Component
Factor analysis.
Overall Performance Indicators

The statistics of results were further elaborated to produce synthetic performance indicators by type of
result at Domain/ Programme level, to allow a synthetic assessment of achievements. These
quantitative indicators, however, would not be able to measure the macroeconomic and policy level
impacts.
As is illustrated in the figure below, Overall Output/Impact, Overall Efficiency and Overall
Effectiveness indicators led to two compound indicators: an Overall Output Performance Indicator;
and an Overall Impact Performance indicator.
An Overall Performance compound indicator finally combined the two previous ones.
18
Exhibit 8
Overall Performance Indicator development
First Circle
Achievement Score Goal Attainment
First Circle
Achievement Score Impact at Project End
First Circle Performance

Indicator
First Circle
Achievement Score Impact after 3 Years
Second Circle
Achievement Score goal attainment
Second Circle
Second Circle
Performance Indicator
Overall Programme
Performance Indicator
Second Circle
Third Circle
Achievement Score goal attainment
Third Circle
Third Circle Performance

Indicator
Third Circle
1.4
Methodological approach to the stakeholder interviews
1.4.1
Scope of the selected interviews
The main goal of the qualitative interviews was to characterise each domain and gain strategic
perspectives on the role of the Commission in fostering innovation in these domains. Their objectives
were therefore twofold :
To provide a general view by key stakeholders, having a long and deep experience of research
projects and of the domain as a whole. This view will provide a 'landscape' for the strategic
impact of the Commission action in each of the three domains. It may help in the interpretation
of the quantitative results from the questionnaire survey.
To allow to understand the key common and differentiating factors in the different domains.
The three selected domains for the Impact Study give a coverage of very different domains in a
sense that Microelectronics is dealing with hardware with only one professional market,
Healthcare is dealing with market-oriented and public activities and Mobile services address
immaterial services for both consumers and professionals. In that perspective it will contribute
to the structuration of a common methodology usable for all domains in applying a common
scheme for qualitative assessment.
Among the issues to address during these interviews were:
The drivers for the technical/commercial development in the domain
The relevance of the instruments and the effects of the EC support
The trend in the industry RTD
The strategic objectives of the domains
19
The three domains selected for the Impact study were:

Microelectronics, dealing with hardware with only one professional market
Mobile services, addressing immaterial services for both consumers and professionals
Healthcare, dealing with market-oriented and public service activities.
1.4.2
Quantitative versus qualitative data
From a methodological perspective, the stakeholder interviews had the objective to provide the study
with needed qualitative information in order to allow for a correct interpretation of the statistical data,
setting these into the societal context. Furthermore, they were to provide a qualitative view on
exactly those impacts downstream the impacts chains that are difficult to assess using the
questionnaires tools.
Figure 1 Quantitative versus qualitative data
Domain trends, drivers and strategic

objectives
Benefits for organisation
Qualitative interviews
Socio-economic impacts
Outputs/outcomes, succes factors
FP vs other RTD public/private

supports
Impact assessment from a

methodological perspective
Qualitative interviews were also to be used as an input for recommendations on a methodology for a
broader and more systematic impacts assessment system, which is one of the objectives of the Impact
Study. The issues of the right balance between quantitative based on questionnaire survey and
qualitative analysis, the complementarities of the analysis have been addressed and evaluated during
the Impact study.
The choice of the study to interview key opinion leaders in the three different stakeholder
communities was fundamental from a methodological point of view as it marks the line of continuity
between the two assessment tools.
It was expected, in fact, that the interviewees each from his/her own perspective - would provide
the study team with strategic information on the importance of the outputs and outcomes created by
the projects.
20
Suppliers were considered to be able to provide a more general view on the market value of lines of
products created or on the true dimension of the contribution to European competitiveness, thus
providing objective information to impacts declared by respondents. Researchers would give their
feedback on the knowledge stock and innovation networks present in their sector, and users would
provide a precious input to the interpretation of the value of the overall project outputs from a user
point of view.
21
The questionnaire survey
2.1
Structure of the questionnaire
The study team developed an electronic questionnaire that was e-mailed to the participants of the
assessed projects.
The questionnaire was composed of 4 worksheets:
1) Introduction, containing the presentation of the study, name and acronym of the project, name of
the organization, name and contact details of the respondent, function and role of the organization
in the project, the typology of organization and the market sector;
2) Impacts on the organisation, containing the questions regarding the impacts on the project teams
and the organization at large
3) Impacts on society, containing the questions regarding the impacts on the immediate users
communities and the society at large.
4) The project, requesting specific information on the project characteristics and the progress factors
Two different questionnaires have been made up by the project team: one tailored to the
characteristics of RTD projects and one tailored to the characteristics of Support Measures (see ch.
2.3.3), based on the consideration whether specific indicators or indicator categories were relevant or
not to the different project typologies. In the chapters below listing the various indicators, references
have been made to which indicators/indicator categories were not included in (especially) the
Support Measure questionnaire.
Following the input from the Steering Committee and the testing of the questionnaires, further
improvements have been made to both of the questionnaires.
2.2
Categories of indicators
The following main categories of indicators have been included in the questionnaires:
Effectiveness indicators;
Efficiency indicators;
Additionality indicators;
Progress factors;
Input indicators;
Output indicators;
Outcome indicators;
Socio-economic impact indicators.
2.3
Terms of measurement
The following terms of measurement have been applied in the questionnaire:

Ordinal data, range 1-5
Ratio data
Interval data
Nominal data (yes/no)
In the chapters below describing the various indicator categories, terms of measurement were
indicated for each indicator.
Furthermore, at the end of each subsection of the questionnaires, space has been foreseen for any
additional comments from the respondents.
22
2.4
Effectiveness indicators
The effectiveness of a project (and programme) was defined in terms of goal attainment, i.e. the
difference between expected and actual outputs, outcomes and impacts.
Assessment of the effectiveness of a project or programme is often interpreted as the definition of the
level of project or programme success.
The inclusion of effectiveness indicators in this study was even more important when taking into
account the wide range of activities (from R&D to Take-up actions) covered by the Framework
programmes as well as the different strategic objectives of the various assessed domains and
programmes.
Effectiveness indicators have been introduced for all the indicators regarding outputs and outcomes
(impacts on R&D team and organisations), as well as for the indicators regarding the socio-economic
impacts on the immediate user communities.
Exhibit 9
Effectiveness - indicators & terms of measurement
Ref. Sections 1A, 1B & 2B of the questionnaires

Indicators
Importance as a goal
Achievement compared to expectations

Idem
2.5
Efficiency indicators
The indicators included in this study assessed efficiency

at microlevel, i.e. from the perception of the participating organizations; and
at mesolevel, i.e. from the perception of the European Commission.
The perception of efficiency for participating organisations was measured through the ratio between
costs and benefits for the participating organisations.
Exhibit 10
Cost/benefits ratio - terms of measurement
Ref. Section 1B of the questionnaires

Indicator
Balance costs and benefits for organisation

Within the limits of the current study, the evaluation of Return of Investment was focused on the
definition of the leverage effect of EU funding, i.e. the ratio between the funding received by the
organisation and the investments made during and/or after the end of the project (for further research
or commercialisation/implementation of the project results).
The indicators regarding the leverage effect of EU funding were not included in the questionnaire for the
Support Measures because considered not to be relevant due to the characteristics of these projects.
Exhibit 11
Leverage effect indicators & terms of measurement
Ref. Section 1B of the RTD questionnaire

Indicator
Value of EU Commission grant

Expenditure during the project
Post-project investment for further research related to the project results
Post-project investment for the commercialisation or implementation of the results
23
Ratio data
Interval data
Interval data
Interval data
In the questionnaire for the RTD projects, the study team furthermore introduced a single question on
the issue of attribution, in order to identify how much an organisations current performance is due to
involvement in a particular project .
Table 1 Attribution indicator & terms of measurement
Ref. Section 1B of the RTD questionnaire

Indicator
Attribution of current performance of unit/organisation to participation in the project
2.6
Additionality indicators
Additionality is at the heart of justification of policy intervention, in this case the Commission
investment. It is therefore a critical factor for evaluation. It was posed in terms of the questions of
what difference is made by intervention and whether the difference justifies the intervention.
Additionality can be defined in terms of the extent to which input and output variables are related:
Additional input variables, i.e. the funding induces an activity which would not have been
carried out in its absence
Additional output variables, i.e. the counterfactual of whether the same outputs would have
been obtained without policy action
Behavioural additionality, articulated in scale additionality (the activity is larger than it would
have been without funding), scope additionality (the coverage includes a wider range of
applications or markets) and acceleration additionality (the activity is significantly brought
forward in time)
The project team decided not to include the additionality indicators for the Support Measures.
Exhibit 12
Additionality - indicators & terms of measurement
Ref. Section 3A of the RTD questionnaire

Indicator category
Input additionality
Output additionality
Process or
behavioural
additionality
Indicator
The project would not have been undertaken without EU funding
The project would have been undertaken without EU funding but the
responding organisation would not have participated in it
The project would have been carried out with internal funds replacing
EU funds
The project would have been carried out with funds from other sources
replacing EU funds
The project would have been carried out with reduced funds
The organisation would have participated but the project would have
produced less satisfactory outputs
The organisation would have participated but the overall competence of
the organisation would not have been enhanced to the same extent
The organisation would have participated but the overall benefits to the
organisation would have been less
had less ambitious objectives
The organisation would have participated but the project consortium
would have had fewer partners
The organisation would have participated but the project consortium
would have had national rather than European partners
taken longer to complete
24
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
2.7
Progress factors
The study introduced a range of factor indicators in order to assess the influence of a range of different
obstacles or success factors on the progress of a project.
Due to the fact that all factors can have a good or bad dimension as well as a positive or
negative influence, the study has developed a two-step approach:
In a first instance, the assessment regards the good or bad dimension of the factor within
the project (e.g. weak or strong project management).
In a second instance, the influence of the factor on the progress of the project is assessed
(positive or negative influence).
The terms of measurement used for these dimension and level of influence assessments were two
five-point scales. The answers to these questions were then combined to produce a five-category
classification of answers that can be used in the statistical analyses.
The progress factors can be subdivided into two major categories: internal to the project (project and
project consortium characteristics) and external to the project.
Some of these progress factor indicators (indicated below in slash) were not included in the Support
Measure questionnaires due to their lack of relevance.
Exhibit 13
Progress factors indicators & terms of measurement

Progress
factors
Internal
Progress
factors
External
progress
factors
Factor indicators
The overall commitment of project partners
The professional competence of project partners
The quality of project management
The clarity of the project goals
The ambitiousness of the project goals
Structural changes or strategic shifts by project partners
External technological developments (e.g. breakthroughs in rival technologies)
External commercial developments (e.g. the dot.com boom/bust)
The availability of funding for project completion or commercialisation
The availability of skilled personnel for project completion
commercialisation
Interaction with Commission officials
Interaction with national representatives and agencies
Interaction with project reviewers
Attempts to cluster with other EU projects
Availability of other EU support measures
25
Description - Influence
idem
idem
idem
or
idem
idem
idem
idem
idem
idem
idem
idem
idem
idem
2.8
Input indicators
The study introduced various categories of input indicators as legitimate variables to correlate with
the performance indicators.
These input indicators were related to:
The project itself: basic project characteristics, project resources and project types
The organizations participating in the project: organization typologies
2.8.1
The project input indicators

Basic project characteristics
Domain
Action line
Timeframe
Duration
Contract type
Project resources
Budget
Commission contribution
Size of consortium
Whereas the above-mentioned project input indicators derived as much as possible - from
information provided by the Commission, the project input indicators describing the characteristics of
the projects, thus allowing the definition of project types, were introduced in the survey questionnaire.
Once again, some indicators (indicated below in slash) were not included in the Support Measure
questionnaires because of non-relevance.
Exhibit 14
Project type indicators & terms of measurement
Ref. Section 3A of the questionnaires

Indicator
Cost
Technical risk orientation (low/high)
Idem
Commercial risk orientation (low/high)
Idem
Technical complexity (low/high complexity)
Idem
Technical horizon (short/long term orientation)
Idem
Technical orientation (basic/applied)
Idem
Technical interest (routine/exciting)
Idem
Technical need (a luxury/a necessity)
Idem
Strategic importance (low/high strategic importance)
Idem
Technical centrality (core/peripheral technology area)
Idem

It needs to be mentioned here that also the expected benefit indicators for the project outputs and the
additionality indicators allowed for further definition of project types.
26
In relation to the short or long term orientation of the projects, the study has also introduced
indicators to assess the phenomenon of the projects in sequence. (in slash indicators not included in
the Support Measure projects)
Exhibit 15
Projects in sequence indicators & terms of measurement

Indicators
The project was based upon work conducted in an earlier project
The previous project received funds from external sources
From the EU under FP3
From local/national government
From private sector sources
From other sources
Idem
Idem
Idem
Idem
Idem
Idem
Follow-up projects have been planned or undertaken

They involve funds from external sources
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem

From local/national government
From private sector sources
From other sources
Focus on further R&D
Implementation of the project results
Commercialisation of the project results

For both cases (projects based on previous projects/projects having a follow-up), indicators were also
introduced to identify the level of ongoing collaboration effect among the project partners.
Exhibit 16
Network effects in projects in sequence - indicators & terms of measurement

Indicators
Involvement of partners in previous projects (all, some, or none)
Involvement of partners in follow-up projects (all, some, or none)
Idem
Another issue related to project typologies constituting a variable for correlation with performance
regards the user-orientation of the projects (not included in the Support Measure questionnaires).
Exhibit 17
User-orientation of the projects indicators & terms of measurement

Indicators
Involvement of the final users in the design of the project
Implementation of trials involving final users
Dimension of involvement of final users in trials
Idem
Numerical data
(1-50; 51-100; >100)
27
2.8.2
The organisation typologies
The identification of the organisation typologies was strictly linked to the implementation of the
analysis at stakeholders level (see 2.2)
Exhibit 18
Organisation typology indicators
Indicator category
Basic organisation indicators
Indicators
Type of organisation
Description
Public administration, university or public
research, private company, etc.
Software producer, telecom network operator,
health/medical services etc.
Market sector
Related to the participation in the

project
Country
Function in the project
Role in the project
Coordinator, partner, associated partner etc.

Research, supplier, client/user
2.9
Output indicators
Project outputs are the actual results of an RTD project. They can be classified in three categories:
Immediate outputs, such as publications;
Intermediate outputs, such as patents;
Final outputs, such as new products or services.
Apart of the measurement of achievement (at project end/3 years later), for all of these output
indicators also the levels of importance as a goal and of achievement versus expectations were being
asked (see Effectiveness Indicators, 3.5). Some of these indicators (in slash below) were not included in
the Support Measure questionnaires.
Exhibit 19
Output indicators & terms of measurement
Ref. Section 1A, 1B, & 2B of the questionnaires

Indicator
category
Immediate
Intermediate
Final
Indicator
Publications
The development and use of new tools and techniques
Models/simulations
Prototypes, demonstrators or pilots
Patent applications
Copyrights, trademarks, registered designs, etc.
Enhanced skills of staff
New or improved products
New or improved processes
New or improved services
Licence incomes
New or improved standards at European level
New or improved standards at national level
28
Achievement end of project/3 yrs later

Terms of Measurement
Ordinal data (range 1 to 5)
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
Idem
2.10
Outcome indicators
Outcome indicators assess the direct effects of the projects on the research teams involved in the
projects (first circle impacts) as well as on the organizations to which these teams belong (second circle
impacts).
These outcomes typically include
Knowledge-oriented impacts
Network-oriented impacts
Strategic management impacts
Exploitation oriented or commercial impacts
Impacts on the workforce
Apart of the measurement of achievement (at project end/3 years later), for all of these output
indicators also the levels of importance as a goal and of achievement versus expectations were asked
(see Effectiveness Indicators, 3.5). Terms of measurement always had a range of 1 to 5. Indicators not
included in the Support Measure questionnaires are indicated in slash.
Exhibit 20
Outcome indicators & terms of measurement
Ref. Section 1A, 1B, 2B, 3A of the questionnaires

Impact category
Achievement project end/3 yrs later

Ordinal data (range 1 to 5)
Outcome indicators
An enhanced knowledge base
Idem
Knowledge-oriented Establishment of critical mass of research
Network-oriented
Strategic
management
oriented
Reorientation of the organisations R&D strategy
Idem
Access to complementary expertise
Idem
The formation of new R&D partnerships and networks
Idem
Improved university-industry cooperation
Idem
Formation of new, longer-term business alliances
Idem
Access to additional funding
Idem
Reduced or shared technical risks
Idem
Eventual access to external funding for future R&D
Idem
Eventual access to external funding for implementation
Idem
Eventual access to external funding for commercialisation
Idem
Use of project results within your organisation
Idem
Reduced or shared project costs
Idem
Increased turnover
Idem
Increased profitability
Idem
Enhanced productivity
Idem
Enhanced competitiveness
Idem
Implementation/
Exploitation-oriented Improved market shares
Idem
Access to new markets
Idem
Creation of a spin-off company
Idem
Reorientation of the organisations commercial strategy
Idem
Enhanced reputation and image
Idem
Reduced commercial risks
Idem
29
Additional indicators to measure the outcomes for the participating organisations have been included
by the study team regarding
-
the industry-science interaction in terms of tacit know-how transfer (RTD projects only)
Exhibit 21
Industry-science interaction indicator & terms of measurement

Indicators
Transfer of RTD personnel
Numerical data
(1, 2-5, 6-10, more than 10)

-
their workforce
Exhibit 22
Impacts on the workforce indicators & terms of measurement

Indicator
Maintain employment levels
Increase employment levels
Idem
Rationalise/decrease employment levels
Idem

-
the implementation/dissemination of the project results
Exhibit 23
Implementation/Dissemination indicators & terms of measurement
Ref. Section 1B & 2B of the questionnaires

Indicator
Use of project results within your own organisation
Use of project results by project partners
Idem
Use of project results by select users and commercial customers
Idem
Open access to project results
Idem
30
2.11
Socio-economic impact indicators
2.11.1
Definition of the user communities
A first step in the process of assessment of socio-economic impacts requires identification of the user
communities for the results and outputs from the projects.
A distinguish between immediate users and final users needs to be made here.
Respondents were therefore asked to identify the most important Immediate users and Final users
communities for their project results and outputs.
Exhibit 24
User communities identification indicators & terms of measurement

Indicator
Your own organisation
Project partners
Idem
The research community
Idem
The manufacturing sector
Idem
Other industrial sectors, e.g. software
Idem
The private service sector, e.g. telecom services, private healthcare, etc.
Idem
The public service sector, e.g. public healthcare, transport, etc.
Idem
Local, national or EU administrations
Idem
Professionals and their organisations, e.g. physicians
Idem
Citizens in general
Idem
Other
Idem
2.11.2
Impacts on the immediate user communities
Apart of the measurement of achievement (at project end/3 years later), for all of the impact indicators
listed below also the levels of importance as a goal and of achievement versus expectations were
asked (see Effectiveness Indicators, 3.5).
Regarding the questionnaires for the Support Measure projects, some of the impact indicators were
considered as non-relevant and have not been included (indicated in slash), while other additional
indicators have been developed, related to the specific characteristics of these projects. These
additional impact indicators are listed below separately.
31
Exhibit 25
Impacts on the immediate user communities indicators & terms of measurement

Indicators
Achievement project end/3 yrs later

Ordinal data
(range 1 to 5)
Idem
Enhanced knowledge management within user communities

Improved information transfer or exchange
Idem
Improved productivity within user communities
Idem
Enhanced competitiveness within user communities
Idem
Improved security of technologies
Idem
Improved functionality and/or reliability of technologies
Idem
Improved compatibility and/or interoperability of technologies
Idem
Improved policy development

Improved access to information and/or services
Idem
Improved public services
Idem
Support Measure projects only:

Ordinal data
(range 1 to 5)
Idem
Improved tools and platforms for awareness raising

Enhanced networking of , e.g., policy makers, researchers, etc.
Enhanced skills through improved education & training
Improved tools for socio-economic research and analysis, e.g. indicators and
benchmarking tools
Enhanced information for decision making on policies, e.g. future
scenarios/forecasting, policy recommendations, future RTD roadmaps, etc.
Idem
Idem
Idem
2.11.3
Impacts on society at large
In this chapter the study introduced relative impact indicators, i.e. the assessment of the impacts of the
project in comparison to impacts achieved in other projects in which the organisation has been
involved (other EU projects, other projects in national programmes, and other in-house projects).
Terms of measurement were always a scoring 1-5 (much less much more)
32
Exhibit 26
Impacts on society at large indicators & terms of measurement
Ref. Section 2C of the questionnaires

Indicator
Ordinal data
(range 1 to 5)
Enhanced levels of scientific and technological knowledge

Enhanced science and technology infrastructures
Idem
Economic welfare at a regional level
Idem
Economic welfare at a national level
Idem
Economic welfare at an EU or global level
Idem
More sustainable development

Enhanced environmental well-being
Idem
Enhanced social welfare, e.g. improvements in the overall quality of life
Idem
Enhanced cohesion, e.g. fewer disparities between different social and economic
groups
Enhanced social inclusion, e.g. greater involvement of all groups in societal affairs
Impacts on policy and regulatory environments at national level
Idem
Idem
Impacts on policy and regulatory environments at European level
Idem
Impacts on policy and regulatory environments globally
Idem
33
In-depth Interviews
3.1
Objectives
The main goal of the in-depth interviews was to characterise the domain and gain strategic
perspectives on the role of the Commission. It was meant to allow for some general views on the
factors that the consortium intends use for measuring the impact of the EC/FP programme.
The interview was divided in two parts into two parts :
-
3.2
The first part was dedicated to a general overview of the domain, and will address the three
points listed in the inception report D1
Drivers and barriers for technical and commercial development in the domain
Trend of the industry RTD
Strategic objectives of the domain

The second part dealt with the relevance of instruments and effects of the support. It addressed
also methodological issues, such as the possible measurements that could be used for the
various impacts (increase of knowledge, dissemination, support of cooperative R&D,..). One
issue was to differentiate the impact of IST programme from other RTD public supports to the
domain, (national RDT funding, RTD tax reduction,) or supporting measures to the domain
not addressing directly RTD (public action for welfare or health, market or industry support,)
Interview process
Interview preliminary preparation

The interviewer was expected to have a good knowledge of the domain and of IST programme. Main
players and main general issues of the domain were not to be the subject of the interview, unless the
interviewee wanted to insist on a specific point.
This was meant to allow concentrating during the interview on the key issues addressed which are the
relevance of instruments and effects of the support.
Interview management
The interviewees were informed prior to the interview about the objectives of the study and the scope
of the qualitative assessment. The interview was to be conducted face-to-face, with an expected
duration of the interview of around two hours.
Interview report
The interview report was to use a common format. As the work done was to give birth to a general
methodology dealing with all domains, it was considered important to have a common format as
general as possible, while allowing to express the main specifics of the domain. So we suggest that the
report will use the same structure as the interview guide.
34
Annex 1: Interview Guide

Annex 2: Survey Questionnaire for the RTD projects
Annex 3: Survey Questionnaire for the Support Measure projects
35
Annex 1: Interview Guide
Interview guide
Part 1: general overview of the domain
Understanding the interviewee domain
As the defined domains are very wide, it seems interesting to have a short notion of the precise
domain where the interviewees operates or has operated.
Issues to be addressed
Action and expertise related to the domain.
professional background
formal position
fields of activity
Drivers for technical and commercial development in the domain

The drivers and barriers are to be found on the technological side : probability of existence of new
technologies, either in the continuation of the existing one or disruptive ones. They could be found on
the market side : expected acceptance from the consumers or users or in other cases existing needs for
new development. They could also come from general public policy : increasing the population health
or economic strength of Europe.
Drivers and barriers could also be found at the market structure level : level of competition, size of the
market, and strategic importance of the domain to consolidate a position in other domains.
-
Current economic situation in the sector

Main drivers and barriers: technical, market, public polic
Importance of ICTs (level of usage) in the domain
Importance of the regulatory environment
Importance of institutions (and their role) for the domain.
Strategic objectives of the domain

One may consider three strategic objectives for domains actors. For private activities, strategic
objective include maximising the revenues and increasing the market share. For public activities one
objective is maximising the service provided to the public. There is a third objective at the geographic
level (national or European), which is the need to be (or to become) a large enough player in the
domain in order to support national (European) interests.
This translates into more questions, concerning the key conditions in order to satisfy these two
requirements.
Issues to be adressed
-
Strategic objectives in terms of public service

Key factors in this domain for achieving these strategic objectives
Strategic objectives at an industrial level
Key factors in this domain in order to achieve these strategic objectives
Trend of the industry RTD

RTD has very different aspects : it may be widely concentrated on one generally accepted
technological roadmap : that is the case for CMOS microelectronics. In other cases, it may be dispersed
in many directions : that is being the case for micro-systems, which may cover a wider range of
technologies and domains, and include much more uncertainties. The level of uncertainty of the future
"technological winners" is probably one characteristic of a domain. Another could be the level of
investment needed for doing significant RTD, which vary from domain to domain. The importance of
standardisation action inside the RTD domain may be addressed, as well as the level and the forms of
R&D internationalisation.
-
Current state of RTD : most important fields, neglected fields in the opinion of the interviewee,
trends, challenges
Existence and evolution of public and private research (at the largest possible level : worldwide
if possible)
Existence and evolution of public support of provate research (at the largest possible level :
worldwide if possible)
Level of uncertainty of the RTD evolution : existence and possibility of one or multiple
roadmaps
Level of investment needed for RTD : how can you define it?
Importance and evolution of international cooperation for RTD
Importance and evolution of RTD outsourcing outside Europe
Evolution and main problems expected for the size of RTD effort
Other key characteristics (typical from your domain or not) for RTD evolution
Part 2: relevance of instruments and effects of the support

Some points may be considered :
Our main goal is to establish the impact of IST programme, in three domains, and also to define
a general methodology. The interview should therefore concentrate on all methods to measure
EC/FP efficiency. We may submit to the interviewees a preliminary list of types of impact, in
order to obtain their comments on them.
There are instruments affecting domains outside direct research that have an important impact.
They usually differ from one domain to the other. As an example, industrial players may
consider support for investment as more important, and more effective than support for RTD.
There is even some (perhaps not so subtle) overlap, since building and operating a semiconductor plant can be considered as RTD as far as industrial yields are not reached
As far as direct support to R&D is concerned, there are many types of support, going from tax
exemption for RTD, training support, decreased taxes for import aimed at RTD, creation of
specific zones with a large amount of various supports. The Taiwanese case provides a full list
of measures.
-
View and opinions as a whole on the impact of IST programme

Comment on the proposed list of effects
Effective results related to this effect

Possible ways to qualify and/or quantify the effect
Your view on its effect
European Commission support

National support (through purely national programmes, Eureka programmes)
Others : private foundations
Funding R&D (grant or loan), whether cooperative or not

Support for training
Support for research dissemination
R&D tax break
Other
Funding R&D (grant or loan), whether cooperative or not

Support for training
Support for research dissemination
R&D tax break
Other
Importance of EU support relative to other funding
Views and opinions on the main and most relevant policy instruments used
What are the main tools used?
Possible distinction between the effects of these various tools and measurement of their
impacts. Views and opinions on relevant existing methods and possible ones to be used.
Any other specific support measures, not addressing directly R&D, but with a more significant
impact than direct aid
Other comments on relevance of instruments and effects of the support in your specific domain.
Annex 2: The survey questionnaire for the RTD projects (word version)
IST Impact Study

Survey
Questionnaire
RTD projects
Databank Consulting
empirica
Idate
Wiseguys
IST Impact Study
INTRODUCTION
This questionnaire is part of a socio-economic assessment study conducted on behalf of the European
Commission -DG INFSO. The study covers all projects in the Microelectronics & Microsystems,
Healthcare and Mobile Communications sectors that were financed during the Fourth (1994-1998) and
the Fifth (1998-2002) Framework Programmes.
Please use a separate questionnaire for each project in which you were involved.
In addition, please answer for your organisation's participation in each project, and not for your
participation in Commission projects in general.
The questionnaire is divided into the following sections:
Details on the organisation
Section 1, which looks at the impacts of the project on your project team and on your
organisation as a whole;
Section 2, which assesses the impacts of the project on immediate user communities and on
society at large;
Section 3, which gathers information on the project itself and on the factors affecting project
progress and outcomes.
IST Impact Study

Please fill in or tick the appropriate boxes.
Name of organisation
Name of respondent
Email:
Telephone:
Project Acronym:
Project Name
Approximately how many organisations were part of this project?
yes
Was your organisation the leading coordinator of the project?
no
What was the nature of involvement of your organisation in the project?

Research
mainly active in the research process for the definition of applications/technologies
Supplier
mainly active in the development process as technology supplier or application provider
User
involved as user or representant of the users, active in trials, demonstrations etc.
To which of these types of organisations does your organisation belong?

Public
administration
Private research organisation
University
Representative Organisations/
NGO
Large
private
employees)
Small
private
employees)
Public
research
organisation
Public/private or semi-state
Other
company
(250+
company
(250-
Which market sector describes best the activities of the department/unit active in the project?
ICT equipment manufacturer
Transport services
Finance
Software engineering
Health/medical services
Cultural
IT service provider
Agriculture, forestry, fishing
Research
Telecoms network operator
Wholesale or retail
Education and training
Other
utility
Telecoms)
Tourism and travel
Public/government
administration
(excluding
Other
IST Impact Study

SECTION 1: IMPACTS ON THE ORGANISATION
A
IMPACTS ON THE PROJECT TEAM
Please consider your participation in the project in terms of the following questions, providing individual scores in a scaling from 1 to 5 as indicated below:
Nil/Very low/much less
Low/ less
Moderate
High/more
Very high/much more
Did your project have the following goals? If so, how important were they? Compared to your initial expectations, were these goals achieved?
Did achievements in these areas (even if they were unintended) have high or low impacts on your project team by the end of the project?
Were these impacts high or low three years after the end of the project, or are they expected to be high or low after three years?
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

Publications
The development and use of new tools and techniques
Models/simulations
Prototypes, demonstrators and pilots
Reorientation of the organisation's R&D strategy
The establishment of a critical mass of research
Improved university-industry cooperation
Reduced or shared project costs
Reduced or shared technical risks
Tick the appropriate boxes

Did the project lead to the transfer of RTD personnel from your organisation?
If yes, please estimate the number
No
Yes
2
6-10
More than 10
IST Impact Study

Please feel free to offer any other comments on these impacts and results of your project.
IMPACTS ON YOUR ORGANISATION
Low/ less
Moderate
High/more
Very high/much more
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

Patent applications
Copyrights, trademarks, registered designs, etc.
New or improved products
Licence incomes
Increased turnover
Increased profitability
Enhanced productivity
Improved market shares
IST Impact Study

Low/ less
Moderate
High/more
Very high/much more
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

The creation of a spin-off company
Reorientation of the organisation's commercial strategy
The formation of new, longer-term business alliances

Not at all/very
low
Low
Moderate
High
Very
high
To what extent is the current performance of your unit/organisation attributable to your participation in this project?
Less than the EC

contribution
Equal
Between 1 and 2
times
Between 2 and 10 More than 10 times

times
Compared with the financial contribution from the Commission, how much did your
organisation contribute over the course of the project?
How much did your organisation invest after the end of the project
- in further research related to the project results?
- in the commercialisation or implementation of the results?
Costs much greater than Costs greater than
benefits
benefits
Costs equal to
benefits
Benefits greater
than costs
Benefits much
greater than costs
In your opinion, how did the costs of participation compare with the eventual benefits
of involvement for your organisation?
IST Impact Study

SECTION 2: IMPACTS ON SOCIETY

A
USER COMMUNITIES
Please identify the most important user communities for the results and outputs from your project.
Immediate
users
Final
users

Project partners
The research community
The private service sector, e.g. telecom services, private healthcare etc.
Citizens in general
Other
Please feel free to offer any other comments on the user communities affected by your project.
IST Impact Study

B
IMPACTS ON IMMEDIATE USER COMMUNITIES
Low/ less
Moderate
High/more
Very high/much more
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

Improved security of technologies
Improved functionality and/or reliability of technologies
Improved compatibility and/or interoperability of technologies
New or improved standards at European level
New or improved standards at national level
Improved public policy development
Improved access to information and/or services
IST Impact Study

IMPACTS ON SOCIETY AT LARGE
Much less
Less
Moderate
More
Much more
To what extent did your project impact on the following compared to:
(a) other EU projects in which you have been involved?

(b) other projects in national programmes in which you have been involved?
(c) other in-house projects in which you have been involved?
Other EU
projects
Other national
projects
Other in-house
projects

Enhanced mobility of scientific and technological personnel
Enhanced economic welfare at a regional level
Enhanced economic welfare at a national level
Enhanced economic welfare at an EU or global level
Enhanced cohesion, e.g. fewer disparities between different social and economic groups
Impact on policy and regulatory environments at national level
Impact on policy and regulatory environments at European level
Impact on policy and regulatory environments globally
IST Impact Study

SECTION 3: THE PROJECT

A
INFORMATION ON THE PROJECT
Please consider your participation in the project in terms of the following questions, providing individual scores to describe the characteristics of the project in a scaling from 1 to 5 as indicated below:
Nil/Very low
Low
Moderate
High
Very high
5
Score
Cost
Technical risk
Commercial risk
Technical complexity
Technical horizon
Technical orientation
Technical interest
Technical need
Strategic importance
Technical centrality
(1-low cost; 5-high cost)

(1-low risk; 5-high risk)
(1-low risk; 5-high risk)
(1-low complexity; 5-high complexity)
(1-short term; 5-long term)
(1-basic; 5-applied)
(1-routine; 5-exciting)
(1-a luxury; 5-a necessity)
(1-low strategic importance; 5-high strategic importance)
(1-in a peripheral technology area; 5-in a core technology area for your organisation)
10
IST Impact Study

Was the project based upon work conducted in an earlier project?
If yes, did the previous project receive funds from
external sources?
No
No, from in-house sources only

Yes, from the EU under FP3
Were all or some of the same partners already involved?
Have any follow-up projects been planned or undertaken?

If yes, will they or do they involve funds from
external sources?
Yes
Yes, from local/national government

Yes, from private sector sources
Yes, from other sources
All
Some
Yes
No
None

Will they or do they involve all or some of the same partners?
All

Some
None
Are they geared towards further R&D, the implementation of project results or commercialisation?
Were final users involved in the design of the project?
Yes
No
Did your project carry out trials involving final users?
Yes
No
If yes, were these final users individuals or organisations?

Approximately how many final users were involved in these trials?
Individuals
1-50
Further R&D
Implementation
Commercialisation
Organisations
51-100
100-1000
more than 1000
11
IST Impact Study

If the project had not received EU funding, would it have been undertaken anyway?
If yes, would your organisation have participated in it?
How would that project have been funded?
Yes
Yes
No
No
with internal funds replacing EU funds

with funds from other sources replacing EU funds
with reduced funds
less satisfactory outputs

Would the results or impacts have been different
lower impact on competence levels
for your organisation? If yes, how?
less overall benefits to my organisation
Would the project have been carried out

differently? If yes, how?
less ambitious objectives

fewer partners
national rather than European partners
longer to complete
Please feel free to offer any additional information on these issues
12
IST Impact Study

B
PROGRESS FACTORS
All the factors described below can affect the outcome of projects and some can have a critical influence.
Please use the scales provided to describe each of the factors for your project (first column) and then use the second column to indicate whether any of these factors had a critical influence
on project progress and outcomes.
Nil/Very low/very negative
Weak/Negative
Moderate/Not critical High/Positive Very high/Very positive

3
Description
Influence

External technological developments, e.g. breakthroughs in rival technologies
External commercial developments, e.g. the dot.com boom/bust
The availability of skilled personnel for project completion or commercialisation
Please feel free to offer any other comments useful to understand the impacts and results of your project
Thank you very much for your collaboration
13
IST Impact Study

D2 (revised) - Methodology
Annex 2: The survey questionnaire for the RTD projects (word version)
IST Impact Study

D2 (revised) - Methodology
IST Impact Study

Survey
Questionnaire
Support Measures
Databank Consulting
empirica
Idate
Wiseguys
IST Impact Study
INTRODUCTION
This questionnaire is part of a socio-economic assessment study conducted on behalf of the European
Commission -DG INFSO. The study covers all projects in the Microelectronics & Microsystems,
Healthcare and Mobile Communications sectors that were financed during the Fourth (1994-1998) and
the Fifth (1998-2002) Framework Programmes.
Please use a separate questionnaire for each project in which you were involved.
In addition, please answer for your organisation's participation in each project, and not for your
participation in Commission projects in general.
The questionnaire is divided into the following sections:
-

Section 1, which looks at the impacts of the project on your project team and on your
organisation as a whole;
Section 2, which assesses the impacts of the project on immediate user communities and on
society at large;
Section 3, which gathers information on the project itself and on the factors affecting project
progress and outcomes.
IST Impact Study

Please fill in or tick the appropriate boxes.
Name of organisation
Name of respondent
Email:
Telephone:
Project Acronym:
Project Name
Approximately how many organisations were part of this project?
yes
Was your organisation the leading coordinator of the project?
no
What was the nature of involvement of your organisation in the project?

Research
mainly active in the research process for the definition of applications/technologies
Supplier
mainly active in the development process as technology supplier or application provider
User
involved as user or representant of the users, active in trials, demonstrations etc.
To which of these types of organisations does your organisation belong?

Public
administration
Private research organisation
University
Representative Organisations/
NGO
Large
private
employees)
Small
private
employees)
Public
research
organisation
Public/private or semi-state
Other
company
(250+
company
(250-
Which market sector describes best the activities of the department/unit active in the project?
ICT equipment manufacturer
Transport services
Finance
Software engineering
Health/medical services
Cultural
IT service provider
Agriculture, forestry, fishing
Research
Telecoms network operator
Wholesale or retail
Education and training
Other
utility
Telecoms)
Tourism and travel
Public/government
administration
(excluding
Other
IST Impact Study

SECTION 1: IMPACTS ON THE ORGANISATION
A
IMPACTS ON THE PROJECT TEAM
Low/ less
Moderate
High/more
Very high/much more
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

Publications
The establishment of a critical mass of research
IST Impact Study

B
IMPACTS ON YOUR ORGANISATION
Low/ less
Moderate
High/more
Very high/much more
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

Improved market shares
The creation of a spin-off company
Reorientation of the organisation's commercial strategy
The formation of new, longer-term business alliances
IST Impact Study

Costs much greater than Costs greater than
benefits
benefits
Costs equal to
benefits
Benefits greater
than costs
Benefits much
greater than costs
In your opinion, how did the costs of participation compare with the eventual benefits
of involvement for your organisation?
SECTION 2: IMPACTS ON SOCIETY

A
USER COMMUNITIES
Please identify the most important user communities for the results and outputs from your project.
Immediate
users
Final
users

Project partners
Other EU funded projects
The research community in general
The private service sector, e.g. telecom services, private healthcare etc.
Citizens in general
Other
IST Impact Study

Please feel free to offer any other comments on the user communities affected by your project.
IMPACTS ON IMMEDIATE USER COMMUNITIES
Low/ less
Moderate
High/more
Very high/much more
Importance of
goals
Achievement versus
expectation
Impacts end of
project
Impacts after 3
years

Improved tools and platforms for awareness raising
Enhanced networking of, e.g., policy makers, researchers, etc.
Enhanced skills through improved education & training
Improved tools for socio-economic research and analysis, e.g. indicators and
benchmarking tools
Enhanced information for decision making on policies, e.g. future scenarios/forecasting,
policy recommendations, future RTD roadmaps, etc.
IST Impact Study

IMPACTS ON SOCIETY AT LARGE
Much less
Less
Moderate
More
Much more
To what extent did your project impact on the following compared to:
(a) other EU projects in which you have been involved?

(b) other projects in national programmes in which you have been involved?
(c) other in-house projects in which you have been involved?
Other EU
projects
Other national
projects
Other in-house
projects

Enhanced economic welfare at a regional level
Enhanced economic welfare at a national level
Enhanced economic welfare at an EU or global level
Enhanced cohesion, e.g. fewer disparities between different social and economic groups
Impact on policy and regulatory environments at national level
Impact on policy and regulatory environments at European level
Impact on policy and regulatory environments globally
IST Impact Study

SECTION 3: THE PROJECT

A
INFORMATION ON THE PROJECT
Please consider your participation in the project in terms of the following questions, providing individual scores to describe the characteristics of the project in a scaling from 1 to 5 as indicated below:
Nil/Very low
Low
Moderate
High
Very high
5
Score
Cost
Technical interest
Technical need
Strategic importance
Technical centrality
(1-low cost; 5-high cost)

(1-routine; 5-exciting)
(1-a luxury; 5-a necessity)
(1-low strategic importance; 5-high strategic importance)
(1-in a peripheral technology area; 5-in a core technology area for your organisation)

Was the project based upon work conducted in an earlier project?
If yes, did the previous project receive funds from
external sources?
Yes
No

Were all or some of the same partners already involved?
All

Some
None
IST Impact Study

Have any follow-up projects been planned or undertaken?
If yes, will they or do they involve funds from
external sources?
Yes
No

Will they or do they involve all or some of the same partners?
All

Some
None
Please feel free to offer any additional information on these issues
PROGRESS FACTORS
All the factors desribed below can affect the outcome of projects and some can have a critical influence.
Please use the scales provided to describe each of the factors for your project (first column) and then use the second column to indicate whether any of these factors had a critical influence on project progress
and outcomes.
Weak/Negative

3
4
Description
5
Influence

The availability of skilled personnel for project completion or commercialisation
Weak/Negative

3
10
IST Impact Study

Description
Influence

Please feel free to offer any other comments useful to understand the impacts and results of your project
Thank you very much for your collaboration
11

Impact Study Methodology PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Impact Study Methodology PDF

Uploaded by

Copyright:

Available Formats

IST Impact Study

Microelectronics & Microsystems, Healthcare,

IST Impact Study Methodology

IST Impact Study Methodology

IST Impact Study Methodology

Data analysis techniques included multivariate techniques capable of identifying groups of

IST Impact Study Methodology

CONCEPTUAL FRAMEWORK ....................................................................................8

OBJECTIVES OF THE STUDY .................................................................................................................. 8

THE QUESTIONNAIRE SURVEY ................................................................................22

STRUCTURE OF THE QUESTIONNAIRE ...................................................................................................22

Annex 1: Interview guide

IST Impact Study Methodology

Figure 1 Overall methodological approach ...........................................................................................10

IST Impact Study Methodology

Objectives of the study

The Impact Study had the following main goals:

Overall methodological approach

The methodology proposed was based on a combination of quantitative and qualitative

The limitations of impact assessments

IST Impact Study Methodology

Methodological approach to the causality & attribution issue

Immediate target audiences/users

involved in the projects

Source: Impact Study, 2004

IST Impact Study Methodology

The relevance of the stakeholders analysis

Overall methodological approach

Key stakeholders & opinion leaders

IST Impact Study Methodology

Methodological approach to the questionnaire survey

Scope of the questionnaire survey

Projects in the targeted domains

Nr. Projects Total

Microelectronics, subsystems and

Mobile & Personal

Source: Impact Study, 2004

IST Impact Study Methodology

The assessment of social, policy & macro-economic impacts

Immediate target audiences/users

involved in the projects

Source: Impact study, 2004

IST Impact Study Methodology

The domains and frameworks

IST Impact Study Methodology

The action types

Action types included in the projects database

Actions contributing to the implementation of a Specific Programme or the preparation of future

Best Practice Action

Combined RTD &

First User Action

IST Impact Study Methodology

Training networks for promoting training-through-research especially of researchers at pre-doctoral and

Source: Impact Study, 2004

IST Impact Study Methodology

Action types groupings

Best Practice Action

Combined RTD & Demo Project

Demonstration Action / Demonstration Project

Dissemination and Awareness

ESD Cooperative Research

First User Action

First User Application