What Bangs for your Bucks?

Assessing the design and impact of transformative policy

Matthijs J. Janssen
Dialogic Innovatie en Interactie.
Hooghiemstraplein 33-36, 3514 AX, Utrecht, The Netherlands. Janssen@dialogic.nl
Center for International Development, Harvard Kennedy School, Harvard University. 79 JFK Street,
Cambridge, MA 02138, USA. Matthijs_janssen@hks.harvard.edu.

ABSTRACT
After an era of generic support for economic development and innovation, narrowly targeted
transformation policy is back on the table (OECD, 2014). Recent advances in the fields of new
industrial policy (Rodrik, 2004; Hausmann & Rodrik, 2006) and transition thinking (Weber &
Rohracher, 2012) converge on the idea that achieving structural change requires governments to take
an active role in overcoming inertia. Rather than just leveraging R&D investments and setting
framework conditions, policy makers are urged to participate in the development of socio-economic
systems around particular technologies (Bergek et al., 2008). Associated policy support typically
involves a diverse portfolio of system-specific interventions (Flanagan et al., 2011).
The rise of transformative policy, in this paper characterized by being selective, processoriented and multi-instrumental, poses severe challenges to rising standards of public accountability
(Warwick & Nolan, 2014). Evaluation methods for calculating the Bang for the Buck of R&Dleveraging measures are ill-suited when policy mixes are supposed to enact collectively driven
economic transformation (Edler, 2008). While econometric techniques do get more sophisticated, they
are unable to account for the indirect but significant influence of evolving socio-technical systems,
directionality of change, and the combinatory effect of narrowly-targeted interventions. Thus, in order
to make sense of transformative policys overall role in economic transformation, a novel assessment
approach is urgently needed.
To come up with an assessment scheme for transformative policy, we draw upon different
literatures concerned with governing structural transformation. Whereas studies on new industrial
policy contains clues on how to orient and organize public support for economic diversification - a
first aspect worth to be scrutinized -, the innovation and in particular TIS literature helps to explain the
evolution of socio-technical systems in which new economic activities are embedded (Hekkert et al.,
2007). The key insight proposed in this paper is that policy contributions to the building of
technological innovation systems are in fact the bangs auditors should be looking for. Surprisingly,
as it is primarily focused on governance and performance of technological innovation systems, the TIS
literature itself has rarely raised the issue of impact assessment.
The starting point of our assessment approach is a simple model for determining how policy
matters for the strengthening of systemic functions (Hekkert et al., 2007). This model is essentially an
adaptation of the one used for reform analytics (Hausmann et al., 2005). Instead of determining which
lacking input is holding back growth, we apply it to express which system function would benefit most
from policy intervention. Apart from taking into account interdependencies between the functions, we
also introduce stringency with respect to the actual need for policy involvement.

P Public inputs

Functions i {0k}

Increase capabilities
(training)

R&D funding

1-j

ji

Knowledge
development

R&D facilities

Policy mix
for pre-selected
domain x

Entrepreneurial
experimentation

Develop standards

Knowledge
exchange

Coordinate research
and education

Yx =
Evolution of TIS
around pre-selected
domain x

Guiding the
direction of search

Support advocacy
coalition

Market formation

M Other inputs

Resources
mobilization

Industry lobbying
Private R&D

Legitimation /
Countr. resistance

Our model forms the basis for a framework that structures investigation of the extent to which
a multi-instrumental policy approach is effective, decisive and targeted at the system functions most
relevant for advancing the TIS (directly, or through interactions with other functions). As the impact
framework allows us to connect the assessment of the policy design with measures of structural
change, it is the heart of the assessment scheme we propose. To facilitate its use, we provide a set of
guiding questions:

Function

{n1N}

Pj

What did the

government do to
strengthen
function j?

How effective
was policy in
strengthening
function j?
What new firm
capabilities /
which firms
settled?
What research
results? (quality
and topics)
Which new
interactions have
been facilitated?
In which way
have interests
been aligned?
Which signals
given to market
parties?
How much
increase in R&D
funding / fte?
Did policy change
discourse? Less
legal barriers?

How decisive is
policy for
strengthening
function j?

Entrepreneurial
experimentation

(E.g. attracting
foreign firms,
training centers)

F2

Knowledge
development

(E.g. targeting
public science)

F3

Knowledge
exchange

F1

Guiding the
F4 direction of
search
F5

Market
formation

F6

Resource
mobilisation

Legitimation /
F7 counteracting
resistance

(E.g. vouchers,
joint R&D, TTO,
platforms)
(E.g. publish joint
vision / agenda,
studies)
(E.g. public
procurement /
public goods)
(E.g. innovation
funding, R&D
facilities, HR)
(E.g. adapt
legislation, lobby,
do assessments)

Why no market
for training, or for
technology itself?

ji

Does function
How important is
j strengthen
function j for
other
success of TIS?
functions?
Which notable
opportunities are
not yet explored?

In the period
under
Why is interest for What knowledge
evaluation,
certain new
was missing /
which were
technologies low? insufficient?
the most
What mechanisms Which particular
pervasive
were blocking
connections where dependencies
interactions?
weak?
between the
Why did interests Which goals
functions?
use to be
where not shared
divergent?
or recognized?
Did
Why where up- or Which suppliers / strengthening
a function
downstream
demand are
have a
markets missing? crucial?
positive or
Public resources
Which activities
negative
matched by
hampered by
private expenses? lacking resources? effect on other
functions?
What hindered
Which changes
acceptance and
did lobby / laws
compliance?
used to block?

For the sake of illustration, we then test the assessment scheme in its entirety in a tentative
assessment of the Dutch Topsector approach. This exercise highlights what in-depth follow-up
assessments should be pointing extra attention to. The Topsector approach was found to adhere to
most of the design principles for transformative policy organization, while some elements of its policy
orientation are questionable (notably: open architecture, focus on new activities, ensuring spillovers).
Our cross-framework analysis suggests that the design configuration can be linked to observations on
the actual policy impulses that were provided in the Energy Topsector. By filling out the entire impact
framework we are able to construct an overview of how supportive the policy interventions have
jointly been. This underlines that it is paramount to check if a policy mix is decisively contributing to
functions where impulses are truly needed.

Design principles framework

Policy organization
I: Self-organizing
deliberation council
I: Supplier forums /
networks
I: Regional collaborative
inn. Centers
I: Investment advisory
councils
I: Private-public venture
funds
D: Formulating clear
objectives
D: Formulating
measurable targets

Topteams formed bottom-up;

Innovation contracts

Impact framework

Topsectors + TKIs
TKIs, Centers of expertise /
craftmanship
Regional development companies?
Funture Funds?

Overall objective: competitive +

green. Detailed goals per TKI.
Overall target: 2.5% GERD. Detailed
targets/KPIs per TKI.
Annual monitors, Progress Report
D: Close monitoring
BB, AWTI balance,
Evaluation of individual measures.
D: Proper evaluation
Policy experiment?
TKIs programs managed strictly.
D: Well-designed rules
But can Topsectors/TKIs also fail?
A: Pro-active
Progress reports (include rationale),
communication strategy dashboard
A: Visible / responsible
Involvement high-level officials,
leadership
topteam captains
Partially carried out by RVO.nl.
L: Competent leadership
Quality of TS-boards ok?

Policy orientation
Inputs instead of
transfers

TKI-surcharge and HR instead of

subsidies. Also system-specific
interventions? Innovation contracts?

Open architecture

Strongly framed around sectors, but

funding for activities?

Genuine novelty

Focus of support measures is

innovation (and HR). But radical?

Contrive spillovers

Support mostly for public or publicprivate research. But does it

transcend beyond TKI-boundaries?

Function

{n1N}

Pj

Entrepreneurial
experimentation

TKI projects,
MIT

Demand driven
TS-activities spur
innovation

Knowledge
development

Redirecting
NWO +
research to TS
topics

TKI research not

radical, some
topics dropped

Knowledge
exchange

Topteam /
Topsector

Diverse parties
involved (but still
some missing)

TKI vision,
Guiding direction
Innovation
of search
contracts
Do TS-reports
give signals?

Missing: Min. of
I&M, certifiers,
insurance prov.

Resources:
financial capital

TKI-surcharge,
international
funding

Still low finance

availability

Legitimation /
cntr. resistance

j ji

Strong project
focus, inclusion
soc. challenges

Market formation

Resources:
human capital

HR agenda

Better skilled HC
(in the future)

Listening to
firms from
Topsector

Some laws still

hampering; CCS
low acceptation

Finally, we discuss how the frameworks for assessing policy design and impact feed into
analyses concerned with structural change (see figure below). In our conclusions we stress that,
besides facilitating continuous policy learning, our assessment scheme also serves to strengthen policy
makers ability to legitimize the adoption of heterodox economic approaches.

Policy design

Design principles
Discipline
Policy
organization

Information retrieval
Accountability

Changing economic /
knowledge structures

Changing sociotechnical system

Policy contribution to TIS building

TIS function
Entrepreneurial
experimentation
Knowledge
development

Policy
orientation

Leadership competence

Knowledge exchange

Open architecture

Guiding direction of
search

Genuine novelty

Market formation

Inputs, no transfers

Mobilizing resources

Contrive spillovers

Legitimation /
cntr.acting resistance

Pj

Products
X

ji

Publications
X

Patents
X

Professions
X

References
Bergek, A., Jacobsson, S., Carlsson, B., Lindmark, S., & Rickne, A. (2008). Analyzing the functional
dynamics of technological innovation systems: a scheme of analysis. Research Policy, 37,
407429.
Edler, J., Ebersberger, B., & Lo, V. (2008). Improving policy understanding by means of
secondary analyses of policy evaluation. Research Evaluation, 17 (3), 175186.
Flanagan, K., Uyarra, E., & Laranja, M. (2011). Reconceptualising the policy mix for innovation.
Research Policy, 40, 702713.
Hausmann, R., & Rodrik, D. (2006). Doomed to choose: industrial policy as predicament, Blue Sky
Seminar, Center for International Development, John F. Kennedy School of Government,
Harvard University.
Hausmann, R., Rodrik, D. & Velasco, A. (2005). Growth Diagnostics. Center for International
Development, John F. Kennedy School of Government, Harvard University.
Hekkert, M., Suurs, R.A.A., Negro, S., Kuhlmann, S., & Smits, R. (2007). Functions of innovation
systems: A new approach for analysing technological change. Technological Forecasting &
Social Change, 74, 413432.
OECD (2014). New Industrial Policies. OECD Science, Technology and Industry Outlook 2014.
DOI:10.1787/sti_outlook-2014-21-en
Rodrik, D. (2004). Industrial policy for the twenty-first century. CID Working Paper. Center for
International Development, Harvard University, Cambridge, MA.
Warwick, K., & Nolan, A. (2014). Evaluation of industrial policy: methodological issues and policy
lessons. OECD Science, Technology and Industry Policy Papers, 16.
doi:http://dx.doi.org/10.1787/5jz181jh0j5k-en
Weber, K.M., & Rohracher, H. (2012). Legitimizing research, technology and innovation policies for
transformative change: combining insights from innovation systems and multi-level
perspective in a comprehensive failures framework. Research Policy, 41, 10371047.