Research Day

Conference proceedings 2015

OpenLivingLab Days

The Research Day Conference proceedings 2015 reports findings presented during the
OpenLivingLab Days 2015, annual summit of the Living Lab community held in Istanbul
from the 25th to the 28th of August. Now in its third edition (first call for academic
contributions was launched in 2013), this publication is the result of the Call for Papers
launched in December 2014 and tackles some of the numerous Living Lab related
challenges recently investigated by scholars and practitioners around the world.

ISBN (e-book): 9789082102741

2015 ENoLL - European Network of Living Labs

All rights reserved

Table of Contents
Living Lab theory and thematic domains (session I)
Enhancing Co-Creation with Privacy and Security-by-Design methodologies, Francesco
Alberti and Sauro Vicini..7
Living Labs: a systematic literature review, Dimitri Schuurman, Lieven De Marez and
Pieter Ballon.16
Living labs as a mean to spur collaboration and innovation in a tourist destination, Dominic
Lapointe, David Guimont and Alain Svigny.29
Girls Making History Summary Report Communities and culture network funded research
pilot, Penny Evans and Roz Hall..41
Smart City Living Lab city as a place, Maija Bergstrm..47

Living Lab Theory and tools (session II)

The wearable Living Lab: how wearables could support Living Lab projects, Tanguy Coenen,
Lynn Coorevits & Bram Lievens.57
Change Laboratory as a method of innovation management in an Urban Living Lab, Virpi
Lund and Soile Juujrvi.68
Involvement of end-users in innovation process: towards a user-driven approach of innovation A
qualitative analysis of 20 Livings Labs, Perrine VANMEERBEEK, Lara VIGNERON, Pierre
DELVENNE, Benedikt ROSSKAMP, Mlanie ANTOINE.......79
A hypothesis driven tool to structurally embed user and business model research within Living
Lab innovation tracks, Ruben DHauwers, Olivier Rits, Dimitri Schuurman, Pieter
Ballon..87
Living Labs As Innovation Platforms: The Key Constructs, Christ Habib, Westerlund and
Leminen, S98
Getting personal Exploring the usage of persona in order to optimize the involvement of a living
lab panel, Sara Logghe.115

Living Lab cases (session III)

Care Living Labs Flanders: Social and Open Innovation, Mark Leys, Lukas Versteele and
Lien Pots...125
Experimenting with location-based service applications: exploring a new methodology for Living
Labs, Uschi Buchinger, Heritiana Ranaivoson, Pieter Ballon and Karel Verbrugge............137
Organisation of labour, quality of work, and relational coordination in Care Living Labs, Leen
De Kort, Ezra Dessers, Geert Van Hootegem....148
Creating a Smart City Vision in a Living Lab Case Study of Smart Kalasatama Vision-building
Process,Veera Mustonen.156
Use of Living Lab in Innovative Public Procurement: Case Keyless Home Care, Lotta
Haukipuro, Torvinen Hannu, Vinm Satu, Mattila Pasi168
Towards a sustainable panel-based living lab approach in older adult care innovation, Juul
Lemey, Charlotte Brys, Koen Vervoort, Patricia De Vriendt and An Jacobs..........................180

Review panel

Chair
Pieter Ballon

Reviewers
Ana Garcia
Tuija Hirvikoski
Marita Holst
Piotr Krawczyk
Seppo Leminen
Artur Serra
Anna Sthlbrst
James Stewart
Shenja van der Graaf

Living Lab theory and thematic domains

Session I

Enhancing Co-Creation
with Privacy and
Security-by-Design methodologies
Francesco Alberti a, Sauro Vicini a
a

City of the Future Living Lab,

Fondazione Centro San Raffaele, Milan, Italy
{alberti.francesco, vicini.sauro}@hsr.it

Abstract
The definition and the analysis of privacy and security requirements is a major challenge
in the product/service/software lifecycle, given both the complexity of security and privacy
concepts and their being process-dependent. In this paper we present a new requirement
analysis process, SPACE, aiming at involving the end users in the privacy and security
requirement analysis phases. Supported by practical tools arranged in a pipeline, our
process allows tackling privacy and security requirements from the early beginning of the
product/service/software lifecycle. To provide a clear evidence of the applicability and
effectiveness of such methodology, this paper reports the outcome of the application of
SPACE within our City of the Future Living Lab, in the context of developing systems for
handling sensitive data.
Keywords
co-creation, living lab, privacy-by-design, security-by-design.
1 Introduction
The participation of end users to the creative phases of technological innovation is of vital
importance in order to deploy high-quality services meeting their needs (Vicini, Bellini, &
Sanna, User-Driven Service Innovation in a Smarter City Living Lab, 2013). Their role is
even more important when sensitive data come into play. Indeed, in this setting, in
addition to the quality of the services provided, another dimension arises: the one of trust.
It has been widely shown the ultimate importance to adopt Privacy- and Security-byDesign (PSbD) methodologies in order to achieve higher quality requirements leading to
development of more robust products (see, e.g., (Shostack, 2014)). Unfortunately, however,
in such methodologies the elicitation of security and privacy requirements is either
intended mainly as an expert-only task or, in the best cases, it is not supported by any
practical tool favoring the end-users co-design of the system. The outcome is that the final
product might not meet the user requirements in terms of trust, not conveying the
necessary confidence that is required in critical domains where sensitive data are handled.

In this paper we present the SPACE process, where SPACE is an acronym for Security and
PrivAcy CodEsign. SPACE is a new process that is becoming part of the Co-Creation process
of the City of the Future Living Lab (Vicini, Bellini, & Sanna, The City of the Future Living
Lab, 2012). In this paper we introduce the process and discuss the practical tools devised to
apply it for the direct involvement of the end users in the elicitation of privacy and security
requirements. The main strength of SPACE is its being supported by practical tools for the
identification of feared events, based on which privacy and security requirements can be
derived. The tools supporting SPACE have been devised with the goal of engaging the
end-users in their usage during workshops, working groups or brainstorm sessions.
The advantages brought by adopting SPACE in the development of a new software system
are threefold. First, increasing the customers trust in the deployed product. Users trust in
the system plays a central role in achieving confidence and success on the market.
Estimating the needs of the users in terms of trust towards the final product is important
for developing a product achieving high-level of satisfaction from the customers point of
view, especially when the system under deployment will deal with sensitive data in
untrusted environments (Pearson, 2009). Second, the analysis of the feared events and
threats is one of the main ingredients of a PSbD methodology. In this process, undesired
events such as theft or malicious/accidental modification of data, disclosure of data to
untrusted third-parties, etc., are taken into account for a privacy and security initial
assessment that will lead to the definition of privacy and security requirements (Notario, et
al., 2014). Such process strongly benefits from the stakeholders inputs, providing new
points of view on the possible undesired scenarios that might come true once the system
will be deployed and running on the market. Third, privacy and security requirements will
be analyzed very early in the product/service/software lifecycle. The definition and the
analysis of privacy and security requirements is usually addressed after the definition of
functional requirements, because it is generally perceived as a process-dependent task.
The major drawback of addressing security and privacy after having defined the
functionality of the system is that security and privacy requirements most likely influence
the functional part of the system under development, requiring their re-definition.
Iterative methodologies allow achieving high-level quality results, but at the same time
might require several iterations to converge. Our procedure reduces significantly the
number of iterations, and at the same time allows achieving fully satisfactory results in
terms of requirements definition.
As we show in this paper, privacy and security can be addressed in the early stages of the
requirement analysis phase. This helps in setting-up a clear picture, in terms of security
risks and privacy threats, where functional requirements will be developed in light of
preserving security and privacy qualities, which are key needs for applications dealing
with sensitive data and/or untrusted third parties, especially nowadays where distributed
cloud architecture, generally perceived as untrusted environments, are offering an always
more cost-effective environment for deploying end-users services.
2 The SPACE process
The SPACE process can be viewed as part of the Co-Creation methodology of the City of
the Future Living Lab (Vicini, Bellini, & Sanna, User-Driven Service Innovation in a
Smarter City Living Lab, 2013). With SPACE we identified the concrete steps and
participative tools supporting the Scenario Analysis part of the PRIPARE methodology
(Notario, et al., 2014), one of the most advanced privacy-by-design methodology that has
been developed so far. SPACE is structured in four phases: Scenario Definition, Input
Data Identification, Stakeholders Goals and Data of Interest Analysis and Threats and
Feared Events Investigation. Such phases are arranged as shown in following picture,
8

which highlights also the tools/tasks required, and will be described in more details in the
following sections.

Scenario De+inition

Input Data
Identi+ication

Co-Design tools
Interviews
SOTA investigation

List of relevant data-

types handled in the
system

Stakeholders' Goals and

Data of interests
Analysis
Stakeholders' Goals
table (Table 1)

Threats and Feared

Events Investigation
Feared Events table
(Table2)

2.1 Scenario definition

This phase targets the definition of the application scenario. During this phase,
stakeholders will be identified via an incremental process: as the scenario is generated and
enriched of details, new stakeholders may appear. A practical tool widely adopted in the
Co-Creation methodology to achieve this goal is the interview. As widely known, interviews
do not follow strict templates and have to be adapted case by case (see, e.g., (Stickdorn &
Schneider, 2012; Fox, 2009)). In this phase, the interviews have to target the elicitation of
the so-called needs and pain-points of the users, i.e., highlight the missing services and
those services which need a substantial re-design. Despite the interviews are very effective
tools for eliciting the end-users requirements, other co-creation tools (like focus-groups,
workshops, brainstorming sessions, etc.) can be exploited here, if needed, for having a
better picture of the scenario.
The analysis of the state-of-the-art (SOTA) is also required at this step and complements
the aforementioned tools. The output of this phase is a textual description of the scenario.
2.2 Input data identification
The definition of the scenario will naturally allows the identification of the set of input
data of the system, i.e., the data that will be provided by the stakeholders. Notably, once
the scenario with its stakeholders and data into play has been defined and the set of input
data has been identified, a first privacy and security assessment investigation can be
performed, e.g., by identifying which kind of data needs ad-hoc countermeasures because
of their high sensitiveness which is protected by the law. At this stage, inputs from
legal/ethics experts would be beneficial to identify important requirements arising from
the handling of particular data.
2.3 Stakeholders goals and data of interest analysis
The two previous phases generate the elements that are fed into the third phase of the
SPACE process, devised for the analysis of stakeholders goals and their data of interest.
The following table, Table 1, is the template for carrying out such analysis. It is a table
clearly and precisely summarizing the list of stakeholders, their goals and the set of data
they are interested in. Notably, new data-types may be defined here (for a practical
example see the case study presented in Section 3). The importance of filling-in this table is
that it will highlight the data of interest of each stakeholder according to their goals. The
usage of this data without privacy and security countermeasures, however, may violate
some privacy or security principle. It is therefore of vital importance characterize the highlevel goals of the stakeholders and identify the data they need to provide their service, in
order to consider the circumstances in which privacy or security breaches might occur.
9

Stakeholder

Goals

Data of interest

Table 1: The Stakeholders Goals table template, used for analyzing stakeholders goals and data of
interests.

As a side effect, upon completion, this table offers a double-checking tool for identifying
weaknesses of the two previous phases. That is, it allows to double check whether all the
input data that have been defined will be of interest for at least one stakeholder and
whether all the stakeholders will have a clearly defined goal.
2.4 Threats and Feared events investigation
The table with the list of stakeholders, their goals and data of interest is fed into this phase.
This phase is also parameterized by a set of threats that are pre-defined and can be tailored
according the privacy and security facets of greater interest. In our application scenario, we
exploited the privacy and security properties listed in the LINDDUN (Deng, Wuyts,
Scandariato, Preneel, & Joosen, 2011) and STRIDE (Howard & Lipner, 2006) threats
categories.
The LINDDUN threats category, also exploited in the PRIPARE methodology, comprises
seven threats, each associated to a privacy property. The LINDDUN privacy properties are:
1. Unlinkability: hiding the link between two or more actions, identities, and pieces of
information.
2. Anonymity: hiding the link between an identity and an action or a piece of
information.
3. Plausible deniability: ability to deny having performed an action that other parties
can neither confirm nor contradict.
4. Undetectability: hiding the users actvities.
5. Confidentiality: hiding the data!content or controlled release of data content.
6. Content awareness: users consciousness regarding his own data.
7. Policy and consent compliance: data controller to inform the data subject about the
systems privacy policy, or allow the data subject to specify consents in compliance
with legislation.
Each privacy property is associated to a threat: Linkability, Identifiability, Nonrepudiation, Detectability, Disclosure of information, Unawareness, and Non-compliance.
For security properties, we focused STRIDE threats category. The STRIDE privacy
properties are the following:
1. Authentication: establishing the author of an action performed in the system.
2. Integrity: ensuring that data will not be (accidentally or maliciously) modified or
deleted.
3. Non-repudiation: providing unforgeable evidence that a specific action occurred.
4. Confidentiality: avoiding the disclosure to unauthorized third parties of the
information stored in the system or exchanged between two parties within the
system.
5. Availability: the property of being accessible and useable upon demand by an
authorized entity.
6. Authorization: regulating the permits of the system users.

10

The corresponding security threats are: Spoofing, Tampering, Repudiation, (Information)

Disclosure, Denial of Service, and Elevation of Privilege.
The investigation of threats and feared events consists in determining whether each
stakeholder may violate a security or a privacy property by exploiting the associated threat
on some data of their interest. In this phase, the active role of the stakeholders is strongly
beneficial and of vital importance, as each stakeholder may identify possible undesired
events that may become reality. Furthermore, in this table we add a new stakeholder that
is the UIP, Untrusted Infrastructure Provider. The UIP represents an abstract stakeholder,
and is required if the service will run in an untrusted environment, e.g., a cloud service: in
this case, the infrastructure provider has to be considered untrusted to allow the
identification of countermeasures to avoid any malicious behavior from its side. The
outcome of this phase will be a table will fill a table, as the one displayed in Table 2.
Stakeholder Threat

Data
involved

Feared Event

Security/Privacy
requirements

Table 2: The Feared Events table template used for investigating Threats and Feared events.

In this phase the contribution of end-users and the stakeholders plays a central role in
defining the feared events and, by consequence, the security/privacy requirements. This
table reports the security and privacy requirements, i.e., countermeasures that have to be
taken in order to avoid the threats.
3 Case study: A Cloud-based Service for Genomic Data Handling
We applied SPACE in an on-going projects within our City of the Future Living Lab,
which goal is to provide a secure and privacy-preserving service for handling genomic data
in untrusted environments, widely recognized nowadays as key assets in devising costeffective and scalable solutions. The entire requirement elicitation process took only two
months. The compiled Feared Events table constitutes a solid basis for the development
of the future system services. For space reasons we intentionally report here only a
shortened version of the outcome produced with SPACE in this scenario.
3.1 Scenario definition
Genomic data are always playing a more important role in health environments, and will
most likely be one of the fundamental pillars of the future P4 medicine service, i.e., the
Predictive, Preventive, Personalized and Participatory health system (Hood & Flores, 2012).
In particular, the availability of patients genomic data allows achieving a high-quality
Personalized health service to the public. Given the computational complexity of the
algorithms exploited by researchers for advancing the state-of-the-art in their research
fields and support better diagnoses procedures, the adoption of massively parallel
frameworks, like the cloud ones, provides a strategic cost-effective solution to achieve their
goals. Given the intrinsic sensitiveness of the data into play, however, solid security and
privacy-related countermeasures have been taken to avoid (accidental or malicious) loss,
deletion, tampering or theft of data.
The series of interviews we conducted within our City of the Future Living Lab allowed us
to have a clear description of the scenario. The stakeholders we will take into
consideration in this section are the patients, the doctors and the researchers (mainly working
in the bioinformatics field). As stated previously, a default stakeholder (the UIP, Untrusted
11

Infrastructure Provider) will be also considered for eliciting privacy and security
requirements. While patients and doctors are widely known actors, we will not enter into
details explaining their role in the scenario. More words about researchers, instead, are in
order. Within a hospital environment, researchers play several roles, which can be overall
summarized in two main categories: they offer support to the doctors for some clinical
analyses and investigate new solutions to open problems in their research fields. In our
scenario, we will consider only the first set of tasks, i.e., those related to supporting clinical
diagnostic-oriented decisions. In the field of genomic analysis, researchers (mostly in
bioinformatics) analyze some mutations of the patients DNA and inform the doctor on the
likelihood of the patient of developing some disease or their response to some drugs. In
order to do that, they run some state-of-the-art algorithms on the sequenced DNA and
interpret the outcome accordingly. These algorithms are generally very expensive from a
computational point of view, and therefore they would extremely benefit from parallel
architectures like the one offered by cloud infrastructure.
3.2 Input data identification
In our case, since we restricted our scenario to a service providing a secure handling of
genomic data from a clinical point of view, the system will receive as input data the
identification data of the patients (name, surname, date of birth, etc.) and their DNA1.
Researchers will also provide some algorithms and code for the automatic analysis of
DNA. This is the third input data we expect: the code.
3.3 Stakeholders goals and data of interest analysis
The third task of SPACE requires to complete the Stakeholders Goals table. Table 3
reports the outcome of this process for our case study.
Stakeholder
Patients

Doctors
Researchers
UIP

Goals
Take advantage of Genomicbased medical treatments;
know ludic information
about their DNA.
Cure patients; Diagnose
diseases.
Support the hospital staff.
Selling a new service.
Administrating
the
(untrusted) infrastructure on
which the analyses are
carried out.

Data of interest
DNA, Ludic metadata,
metadata.

Clinical

Identification
data,
Clinical
metadata.
DNA
Resource statistics (disk usage,
network traffic, CPU load); System
logs.

Table 3: The Stakeholders Goals table for our case study.

Notably, new data has been defined: Ludic and Clinical metadata and Resource statistics.
These are different data generated by the system itself. Metadata are retrieved by
analyzing DNA either directly from the researchers or indirectly by the system, if it will
embody some data-mining procedure working on genomic data. Clinical metadata can be,
e.g., the probability of developing a disease that can be retrieved from the DNA. Ludic

1

An extension is to consider also more health-related kind of data, e.g., the patients; EHR (Eletronic Health
Record). This can be a viable extension, but will not be taken into account in this paper.
12

metadata refers to non-clinical information (e.g., color of the eyes, etc.). In addition, UIP
will need resource statistics and system logs for administrating the service.
3.4 Threats and Feared events investigation
The forth step of SPACE targets the generation of privacy and security countermeasures.
The outcome of this process is reported in the following table, Table 4. As we stated before,
we are reporting only a small fragment of the actual table that has been compiled in our
case study.
Stakeholder Threat

Data
involved
DNA

Feared Event

Unauthorized
code
modification or
fabrication
(intentional or
unintentional)
with
the
purpose
of
revealing
sensitive data.

UIP

Data
tampering

UIP

(Software)
Tampering

Code

Patients

Disclosure

DNA

Hospital
staff

Detectability

Undesired
addition
or
modification of
the data.

Full disclosure
of
DNA,
affecting
his/her
relatives
privacy.
Clinical
Patient
does
metadata;
not have the
Identification control on who
data
accesses
his/her data.

Security/Privacy
countermeasure
The system must
provide a trusted
access-control
schema
to
control
the
accesses and a
module able to
ensure
the
authenticity and
integrity of data.
The system has
to
provide
means
for
checking
the
authenticity of
the
programs
running in the
Untrusted
Infrastructure
before executing
it, ensuring their
integrity.
The system must
ask the signature
of the informed
consent to the
patients
relatives.
The system must
track logs of all
the access and
allow the user to
know who is
accessing
and
when they are
accessing their
data.

13

Researchers Unawareness

DNA

Unauthorized
research
activity.

Researchers
must sign an
agreement
specifying that
they
cannot
perform
research activity,
if the patient did
not
allowed
third
parties
research
activities.
The system must
provide access to
the
minimum
amount of DNA
that is sufficient
for a specific
task.

Researcher

Disclosure

DNA

Researchers Linkability

DNA

Knowing more
things about
the
patient
than the things
the
patient
expect
the
researcher to
know.
Blood-link
The system must
between
avoid
direct
patients.
access
to
multiple DNA
data that can
lead
to
the
blood-link
discovery.

Table 4: The Feared Events table for our case study.

During our project we observed that such table can be compiled quite easily during
workshops and brainstorming sessions, and allows the real elicitation of the end-users and
stakeholders requirements.
4 Conclusion
We presented SPACE, a new process for the elicitation of security and privacy
requirements. The strength of SPACE is that it allows defining security and privacy
requirements in the very early phases of a software lifecycle, avoiding refinement cycles
between functional requirements and privacy/security ones. An added value of SPACE is
that it also specifies simple and practical tools for engaging the end-users of the system
directly in the requirement elicitation process. Thanks to this intrinsic feature, SPACE is a
valuable extension of the Co-Creation methodology.
SPACE has been evaluated experimentally within the City of the Future Living Lab within
a new project targeting the definition of a new service dealing with genomic data. Such
service badly demands for strong security and privacy requirements given the highly
sensitive nature of the handled data. The evaluation has also been extended, as a proof-ofconcept, to other lighter scenarios, always needing some security and privacy
requirements. In all such setting, SPACE allowed saving time. Furthermore, the cocreative nature of the process allowed achieving requirements that will likely lead to a
high satisfaction of the end-users.

14

5 Acknowledgement
The work described in this document has been conducted within the project WITDOM,
started in January 2015. This project has received funding from the European Union's
Horizon 2020 research and innovation programme (H2020-ICT-2014-1) under grant
agreement No. 64437. This work was supported in part by the Swiss State Secretariat for
Education, Research and Innovation (SERI) under contract No. 15.0098. The opinions
expressed and arguments employed herein do not necessarily reflect the official views of
the European Commission or the Swiss Government.
6 References
Deng, M., Wuyts, K., Scandariato, R., Preneel, B., & Joosen, W. (2011). A privacy threat
analysis framework: supporting the elicitation and fulfillment of privacy requirements.
Requirements Engineering , 16 (1), 3-32.
Fox, N. (2009). Using Interviews in a Research Project. Nottingham: The NIHR RDS for the
East Midlands / Yorkshire & the Humber.
Hood, L., & Flores, M. (2012). A personal view on systems medicine and the emergence of
proactive P4 medicine: predictive, preventive, personalized and participatory. New
Biotechnology , 613-624.
Howard, M., & Lipner, S. (2006). The Security Development Lifecycle. Redmond, WA, USA:
Microsoft Press.
Notario, N., Crespo, A., Kung, A., Kroener, I., Le Mtayer, Troncoso, C., et al. (2014).
PRIPARE: A New Vision on Engineering Privacy and Security by Design. Cyber Security
and Privacy - Third Cyber Security and Privacy EU Forum, CSP Forum 2014, Athens, Greece,
May 21-22, 2014, Revised Selected Papers (p. 65-76). Athens, Greece: Springer.
Pearson, S. (2009). Taking Account of Privacy when Designing Cloud Computing Services.
Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing
(p. 44-52). Washington DC: IEEE Computer Society.
Shostack, A. (2014). Threat Modeling: Designing for Security. Indianapolis, Indiana: John
Wiley & Sons.
Stickdorn, M., & Schneider, J. (2012). This is Service Design Thinking: Basics, Tools, Cases.
Wiley.
Vicini, S., Bellini, S., & Sanna, A. (2012). The City of the Future Living Lab. International
Journal of Automation and Smart Technology , 2 (3).
Vicini, S., Bellini, S., & Sanna, A. (2013). User-Driven Service Innovation in a Smarter City
Living Lab. International Conference on Service Sciences (ICSS) (p. 254 - 259). Shenzhen: IEEE.

15

Living Labs:
a systematic literature review
Dimitri Schuurman a, Lieven De Marez a
& Pieter Ballon a
a

iMinds/ iLab.o, Belgium

dimitri.schuurman@iminds.be

Abstract
This paper presents a systematic literature review of the body of scientific Living Labs
literature. Based on a general review of the Google Scholar and Web of Science databases,
we can conclude that the Living Labs movement in terms of theory and research has taken
off since 2006 in quantity of published papers. However, in terms of quality and impact,
the academic field of Living Labs is still rather insignificant. An analysis of the 45 most
cited papers reveals that the practice-based side is much further developed than the
theoretical side, with only few references to more established innovation theories such as
Open Innovation and User Innovation, despite the fact that concepts from both literature
streams are present in all papers. Strikingly, 18 out of 45 papers refer to no framework at
all, remaining merely descriptive. There is also a lack of empirical, more quantitative and
comparative studies that focus on the added value of Living Labs. This paints the picture of
Living Labs as a research domain in development which calls for a better anchoring within
more established innovation theories in order to advance the field.
Keywords
Living Labs, Open Innovation, User Innovation, Literature review, Innovation
1 Introduction
The official birth of the European Living Labs movement is often situated in 2006, the year
the European Commission officially declared its support by stimulating projects to
advance, coordinate and promote a common European innovation system based on Living
Labs (Dutilleul et al., 2010), and also the year in which the European Network of Living
labs (ENoLL), was established an organization aimed at connecting Living labs for
knowledge exchange, networking purposes and the development of a shared innovation
concept (European Commission, 2013b). Despite the fact that Living Labs have been
around for nearly a decade, in terms of conceptualization, the current literature stream is
still inconsistent and sometimes contradictory. Flstad (2008a) identified nine living lab
characteristics, of which five were diverging, which indicates a large variety of approaches
being labeled as living lab. Moreover, a second literature review two years later by
Dutilleul et al. (2010) revealed five different meanings given to discern living labs in the
papers they studied: 1. an innovation system consisting of organized and structured multidisciplinary networks fostering interaction and collaboration; 2. real-life or in vivo
monitoring of a social setting generally involving experimentation of a technology; 3. an
16

approach for involving users in the product development process; 4. organizations

facilitating the network, maintaining and developing its technological infrastructure and
offering relevant services; 5. the European movement itself. Most recently, Westerlund and
Leminen (2014) even found eight different perspectives on Living Labs. However, to this
date, a structural and systematic analysis of the Living Labs literature is missing. Within
this paper we wish to fill this gap by using a clear methodology for selecting and analyzing
the current body of Living Labs papers and articles. This allows to identify the main
perspectives and viewpoints on Living Labs and how they have been embedded within the
more established innovation theories.
2 Methodology
In order to get an overview of the State-of-the-Art of academic and empirical research into
Living Labs, we conducted an exploratory review of the available literature. Hereto we
constructed a sample of the most cited Living Labs papers. We used the Google Scholar
academic search engine2 and looked for articles by using the search string Living Lab
(end of October 2014). This yielded more than 6.500 results. Subsequently, we narrowed
the number of articles down by only including articles where Living Lab was mentioned
in the title in order to weed out the articles where Living Lab appeared accidentally or
only occurred on a side note. This resulted in 563 articles. From this sample, we chose to
include only journal or conference papers (excluding books, book chapters, theses or other
citations) with a direct link to the abstract and only articles with a citation count of more
than 10. This led to a total sample of 45 articles (see attachments for the full list). In order to
get an overview of the number of Living Labs papers in top ranked journal, we did a
similar exercise in the Web of Science database, looking for all articles that had Living
Lab in the title. This led to 50 articles in total. In the following table we give an overview
of the total number of articles from our three searches, organized per year. In terms of time
intervals, we used 2006 as an anchoring point, as this year marked the establishment of the
European Network of Living Labs and more formal support for Living Labs from the
European Commission. The papers published before 2006 were merged into one category,
while we give an overview of the rest of the sample per year.
Publication
year

Articles in sample Articles in total

(Google Scholar + (Google scholar)
10 citations)

WoS articles

Until 2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Total

4
3
5
7
6
9
5
4
2
0
45

3
0
3
3
8
8
6
7
8
4
50

18
9
15
52
69
74
65
95
92
74
563

Table 1: Sample overview per year (October, 2014)

2

http://scholar.google.be/
17

100
90
80
70
60
50
40
30
20
10
0

Scholar total
Scholar +10 cit
WoS total

Un.l 2006 2007 2008 2009 2010 2011 2012 2013 2014
2005
Figure 1: Living Labs papers evolution

In terms of the total articles, we see a clear explosion of research after the establishment
of ENoLL, somewhat similar to the growth of ENoLL itself in the first years. The years
2009-2011 seemed to mark a stagnation in the number of published papers, whereas 2010
was the top year of new Living Labs entering the network. However, as the number of new
Living Labs started to drop significantly from 2011 onwards, the number of papers started
to increase again in 2012 and seems to have stabilized again. What looks more problematic,
is the evolution of papers that effectively generate impact. In terms of papers with a
citation count of more than 10, no year has yielded more than 10 papers, with a maximum
of 9 papers in 2010 being cited more than 10 times.
In terms of Web of Science-papers, we also get the image of a research field in
development. When we select all articles in the Web of Science database that have living
lab in the title, this results in only 50 papers that have been published in journals (21) or
conferences (29) that are abstracted in this influential database. This is only a fraction of
the almost 600 papers with Living Lab in the title from the Google Scholar-search.
Moreover, when we look at the citation count of these papers, only 2 have more than 10
citations in other WoS-publications: Wolfert et al. (2010) with 24 citations and De Moor et
al. (2010) with 11 citations. The majority of the WoS publications (33) even has no citations
at all. Moreover, the overlap with our Google Scholar most cited sample is rather scant,
with only 8 papers appearing in both list (cf. also the attachments): Budweg et al. (2011), De
Moor et al. (2010), Hlauschek et al. (2009), Liedtke et al. (2012), Schuurman et al. (2011),
Svensson et al. (2010), Wadhwa (2012), and Wolfert et al. (2010).
Therefore, we decided to continue our analysis with the top-cited Google Scholar articles.
For the 45 Google Scholar papers with a citation count higher than 10 the total citation
count is 1943, which means an average of 43 citations per paper. Only 5 papers are cited
more than 100 times: Abowd et al., 2002 135 cit.; Eriksson et al., 2005 176 cit.; Niitamo et
al., 2006 142 cit.; Almirall & Wareham, 2008 124 cit. and Flstad, 2008 182 cit. Note that
none of these papers is also on the WoS.
For Open Innovation, West & Bogers (2013) conducted a similar literature overview which
resulted in 287 papers in SSCI journals (Web of Science papers), with the first 10 papers
being cited at least 500 times, with Chesbroughs book (2003) even cited more than 8000
times, and Chesbrough being (co-) author of most of the top-cited papers. The same is true
when looking for literature with the terms User Innovation and lead user, with von

18

Hippel as a dominant figure and easily more than 10 articles with over 400 citations,
although the Open Innovation literature is clearly dominant in terms of quantity.
Based on these general statistics, we can conclude that the Living Labs movement in terms
of theory and research has taken off since 2006, at least in quantity of published papers.
However, in terms of quality and impact, the academic field of Living Labs is still rather
insignificant. Regarding the authors, 39 papers were authored by European scholars, five
by American scholars and one paper originated from Australia (Third et al., 2011). This is
further proof that the Living Labs field is clearly dominated by Europeans. However, there
is not a single author very dominant as in the Open and User Innovation literature, with
five authors (Almirall, Wareham, Sthlbrst, Eriksson and Feurstein) (co-) authoring 3
papers. This is also a further indication of a scattered research field. We will continue the
rest of our analysis with the 45 Google Scholar 10+ cited papers. We chose to use this
sample as it has some clear advantages. The selection criteria are clear and unambiguous,
which enables later reproduction (e.g. for future comparative studies). Moreover, the
sample size allows to have a more in-depth knowledge of all the papers, while at the same
time representing a fair share of the total amount of papers (8%). However, we also
acknowledge some limitations that come with our selection methodology. Papers that do
not have living lab in the title are excluded (e.g. Ballon et al., 2007), although based on
our knowledge of the literature, this has only a minor impact. Perhaps more impact is
generated by including the criterion of 10+ citations. This tends to limit the inclusion of the
most recent Living Labs papers, as it takes some time to get cited by even newer
publications. However, this would raise the issue on how to measure or assess the quality
of these more recent publications. Therefore, we chose to keep our initial criteria and
propose future research should adhere to these criteria to include more recent literature
that by that time has reached a significant degree of impact.
3 Results & discussion
When going through all the papers, two important issues arise. First, only a small minority
of the papers reports on well-grounded empirical research on Living Labs. The majority of
the papers are descriptive single or multiple case studies, or conceptual papers relying on
desk research, without a rigid methodology being used or explained.
In our sample, 18 out of 45 papers are merely project descriptions with only limited
conceptual value (Abowd et al., 2002; Baida et al., 2007; Schwittay, 2008, Hlauschek et al.,
2009; Krieg-Brckner et al., 2010; Hess & Ogonowski, 2010; Budweg et al., 2011, Schuurman
et al., 2011; Liedtke, 2012; Wadhwa, 2012; Schwartz et al., 2013; Ogonowski et al., 2013) or they
describe a single case study where a Living Lab approach is used, but without Living Labs
themselves being the subject of the research (Haymaker & Chachere, 2006; Scott et al.,
2009; Wolfert et al., 2010; Bliek et al., 2010, Ryu, 2010; Third et al., 2011). Remarkably, all
American papers and the single Australian paper are to be found in this category, which is
another indication that Living Labs are largely a European phenomenon. Also, the Ryu
(2010) paper is the only downright negative paper in the whole sample, as it describes the
power relations a large company can exert in the process of ICT introduction in
developing countries. All other 44 papers approach Living Labs in a neutral or overtly
positive way, which is an indication of the absence of a critical attitude towards Living
Labs as a concept. In the Open and User Innovation literature we also encountered mostly
positive case studies, but in both fields some critical papers have also emerged. To this day,
no real critical Living Labs paper has been published, which is a further proof of the
rather low impact of the field in other literature streams.

19

Paper type
Descriptive papers
State-of-the-Art papers
Conceptual & methodological papers
Empirical paper

Number of papers
18
4
16
7

Table 2: Living Labs paper type

Subsequently, we can discern a category of four papers that contain multiple Living Lab
cases, but merely as high-level descriptions and illustrations. First, we have the oldest
paper from our sample by Markopoulos and Rauterberg (2000) who give an overview of
the American Living Labs that were blossoming at that time, with also examples from this
kind of Living Labs in Europe3. Next, we have the widely cited papers by Eriksson et al.
(2005) and Niitamo et al. (2006) who give an overview of the developing European Living
Labs field, also including some of the American examples. As a fourth paper in this
category, we have Schaffers et al. (2007) who discern the Living Labs for rural
development, which represents a new type of Living Labs that popped up in practice.
Besides these four state-of-the-art papers, we have a rather large sample (16 or just over 1/3
of all papers) that deal with methodological and conceptual contributions to Living Labs,
based on single case studies or purely conceptual papers. Pierson and Lievens (2005),
Kusiak (2007), Flstad (2008b), Levn and Holmstrm (2008), Feurstein et al. (2008),
Schuurman and De Marez (2009), Bergvall-Kareborn et al. (2009a&b), Santoro and Conte
(2009), Pallot et al. (2010) deal with user contribution and project methodologies for Living
Labs. Some papers base themselves on more research data, such as Schumacher and
Feurstein (2007) who report on a Living Labs survey, albeit in a very descriptive way.
Mulder et al. (2007 & 2008, basically two times the same paper) report on a brainstorming
exercise of Living Lab practitioners and map different methods and tools on a
harmonization cube, while Svensson et al. (2010) base themselves on user contribution in
more than 100 user interaction instances in three Living Lab projects to inventarize
different methods. Ponce de Leon et al. (2006) and De Moor et al. (2010) deal with testbeds
in the context of Living Labs, and how to intergrate these, with De Moor et al. (2010)
dealing specifically with Quality of Experience as methodology which can support Living
Labs and vice versa.
However, only seven papers dig deeper into the Living Labs phenomenon with a larger
sample, a more rigid methodological approach or a more in-depth analysis of the cases
studied. First, there are two papers containing literature reviews: the Flstad (2008a) and
Dutilleul et al. (2010), which we both touched upon briefly in the introductory section.
Although their methodology for selecting the papers is not very clear, both articles have
been referred to rather often.
3.1 Theoretical frameworks
We also assessed which theoretical frameworks were used in the papers. Therefore we
examined the theoretical and introductory parts of the paper and looked which
frameworks or paradigms were mentioned as foundations for Living Labs. Building further
on Schuurman et al. (2013), we looked for indications of the Open Innovation and User
Innovation frameworks, and because of the influence of the cooperative design movement
for the early Living Labs (Ballon & Schuurman, 2015), we also looked for indications of
these literature streams. In practice, we looked at the occurrence of the Open Innovation,
User Innovation, user-centered design (UCD) and participatory design (PD)

3

Note that the authors were also European and connected to a Dutch Living Lab.
20

expressions, but also for citations of prominent authors associated to the fields such as
Chesbrough or von Hippel. The table below gives the numbers of articles where the
proposed frameworks are used, together with the number of articles where none of the
theoretical foundations were used.
Paradigm
Open Innovation
User Innovation
UCD / Participatory design
None

N
11
17
19
18

Table 3: Dominant framework

Surprisingly, Open Innovation is only explicitly referred to in 11 papers. This can be

explained by the fact that in a lot of these papers, terms like open collaboration, PublicPrivate-People partnership, or even Open Innovation are used without any referral to
literature from the Open Innovation domain. A lot of the Living Labs papers seem to take
the use of Open Innovation for granted, without reflecting in terms of the Open
Innovation literature base or without apparent knowledge of this literature stream. Papers
like Schuurman and De Marez (2009), Svensson et al. (2010) and Pallot et al. (2010) equal
Open Innovation with user involvement and open collaborative innovation, something
which was also discussed in West & Bogers (2013). In 18 articles, none of these frameworks
was referred to, whereas 17 papers referred to the User Innovation literature. The UCD/PD
framework is the most cited with 19 papers, which is an indication that the cooperative
design predecessor still has a large influence on the current Living Labs movement.
Moreover, the large amount of papers without reference to these frameworks is
remarkable, but also congruent with the previous finding that 18 papers within our sample
are for the largest part descriptive without much attempt at theory building. Among the
earlier papers with a reference to design thinking, we find especially American authors
with references to participatory design and requirements-driven innovation (Abowd et al.,
2002; Haymaker & Chachere, 2006; Kusiak, 2007). In Europe, the Scandinavian authors
have maintained a strong connection between Living Labs and design thinking (Flstad,
2008a&b; Levn & Holmstrm, 2008; Bergvall-Kreborn et al., 2009a).
3.2 Living Labs and Open Innovation
Referring back to the initial goal for the promotion of Living Labs within the wider
European innovation system, which was to help solving the European Paradox, or the
imbalance between knowledge exploration and exploitation, we would also expect Open
Innovation to be more prominent as framework for conceptualizing Living Labs. In order
to solve the European Paradox, Living Labs should be able to facilitate the process of
exploitation. Therefore, we looked at the Living Lab definitions, and more specifically the
goals that were mentioned for the Living Lab activities that were described in the paper.
We coded all papers for the three Open Innovation processes of exploration, exploitation
and retention (Lichtenthaler & Lichtenthaler, 2009; van de Vrande et al., 2009):
Exploration: innovation activities to capture and benefit from external sources of
knowledge to enhance current technological developments
Exploitation: innovation activities to leverage existing knowledge or technological
capabilities outside the boundaries of the organization
Retention: maintaining, storing and reusing knowledge over time outside of an
organizations organizational boundaries
21

Besides the word exploration itself, we considered words such as experimentation, study
(of user behavior), testing, as indicators of exploration goals. For exploitation, we
regarded words and phrases like creating initial demand, adoption, technology transfer,
implement, and business models to refer to an exploitation goal. For retention, indicators
such as knowledge and information sharing, multi-stakeholder communication and
rethinking were used.
Process
Exploration
Exploitation
Retention

N
45
15
7

Table 4: Open Innovation processes

All papers (45) define Living Labs and Living Lab activities as an exploration of new
knowledge, whereas only one out of three (15) mentions exploitation as a motive for Living
Labs. This is a clear mismatch with the original intentions described in the Helsinki
Manifesto (2006) of Living Labs as facilitators of knowledge exploitation. The exploitation
motive of Living Labs is the most common in the more thematic Living Labs (e.g. Baida et
al., 2007; Hlauschek et al., 2009; Wadhwa, 2012) or Living Lab projects where an innovative
infrastructure is rolled out amongst a population (e.g. Schwittay, 2008; Ryu, 2010; Third et
al., 2011; Bliek et al., 2010; Schwartz et al., 2013). The fact that knowledge retention is the
least common is not a real surprise, as this process is also the least studied within the Open
Innovation literature. The seven instances where retention was an explicit goal, were in
thematic Living Lab constellations where stakeholders from a certain sector intend to
collaborate and exchange knowledge regarding future opportunities (Baida et al., 2007;
Wolfert et al., 2010), two projects aimed at sustainable innovation with the creation of user
awareness (Scott et al., 2009; Liedtke et al., 2009), the literature review of Dutilleul et al.
(2010) who refer to the regional knowledge sharing opportunities of Living Labs, and the
two papers by Mulder et al. (2007 & 2008) that incorporate the outcomes of a
brainstorming session of Living Lab practitioners in an attempt to create shared tool and
methodology set for Living Labs. This is an indication of the imbalance in the attention for
the Open Innovation processes in the current Living Labs literature. Moreover, the fact
that only 11 papers explicitly refer to Open Innovation as a defining paradigm, but that in
all papers references to knowledge transfers between actors can be found, suggests that
Living Labs are emanations of Open Innovation. This calls for a better conceptualizing of
Living Labs that allows to frame them in terms of Open Innovation.
3.3 Living Labs and User Innovation
We now turn over towards the appearance of User Innovation within our sample of Living
Labs papers. As within the Living Lab definitions user involvement and user co-creation
are essential characteristics, we looked in our sample for the degree of this user
involvement. As key framework, we chose the categorization of Kaulio (1998), who discerns
innovation/design for, with and by users. Design for denotes an innovation approach where
user involvement is limited to passive user feedback, gathered through Voice of the
Customer-methods or user behavior studies, as were conducted in the American Living
Labs. Design with denotes an innovation approach based on co-creation, as users and
manufacturers work together in an iterative manner, where the locus of innovation can be
seen as shared between both involved actors. Design by refers to an innovation approach
where users innovate themselves, which is in line with the Lead User-approach and the
CAP, as the locus of innovation resides with the user.
22

Design
For users
With
users
By users

N
11
34
0

Table 5: User involvement mode

We looked at all articles and assessed what the dominant mode of user involvement was
for the Living Lab activities that were described in the paper, or in the case of conceptual
papers how the user contribution was defined. Not surprisingly, design with users, or the
co-creation stance, was dominant in the majority of the papers (34). None of the papers
described activities where the innovation by users-mode was dominant, although it was
described in some papers (cf. infra). However, it is remarkable that the majority of the
papers refers to co-creation with end-users, but only 17 papers mention User Innovation as
anchoring paradigm. Apparently, the current Living Labs do not support true User
Innovation, or at least do not see this as the dominant form of user contribution. Design for
users, where the user only plays a passive role in the innovation process, is the dominant
mode in 11 papers, including the American Living Labs and the real-life testbeds with
passive user observation or simple evaluation, and some papers that deal with Living Lab
projects where technologies are rolled out amongst a group of users with technical testing
in real-life as main goal.
Regarding the rest of the papers that dealt with the User Innovation paradigm explicitly,
we would expect that the roles and characteristics of end-users in Living Labs would be
described and researched in greater detail because of the user-centric nature of Living
Labs. However, when going through the literature, this was not really the case. Lead User
methods are mentioned in the context of Living Labs when overviews of methods to be
used are presented (e.g. Pallot et al., 2010; Kusiak, 2007), but how this should exactly be
approached remains unclear. In the works of Almirall et al. (2012), the Lead User concept
also pops up with no clear specification on how to implement this, except for selection of
relevant users (Almirall et al., 2012). The Lead User method is also displayed as separate
from Living Labs, with a slight overlap. The same goes for Pallot et al. (2011), who consider
the Lead User-method as one of the user involvement techniques that are being used in
Living Labs. Interestingly, Almirall and Wareham (2008) consider Lead User
entrepreneurs as an important stakeholder group in Living Labs, something which is also
mentioned by Pallot et al. (2011).
4 Conclusion
Out of this overview of the theoretical state-of-the-art of the field of Living Labs, we have
gathered that the practice-based side is much further developed than the theoretical side.
In terms of empirical research and academic publications, Living Labs have received some
attention, but this attention is virtually absent in top ranked journals. There is also a lack
of empirical, more quantitative and comparative studies that focus on the added value of
Living Labs. In the Living Labs literature, neither Open nor User Innovation is the
dominant paradigm. Referring back to the Living Labs predecessors, it is the User
Centered Design that originated from the Participatory Design movement that is still
dominant. Strikingly, 18 out of 45 papers refer to no framework at all, remaining merely
descriptive. User Innovation occurs more frequently than Open Innovation, but it seems
that in recent papers Open Innovation is more and more adopted within the Living Labs
literature. This is in line with the trend we also discovered in the previous chapter on
23

Living Labs practice, where we noticed the emergence of a new type of Living Lab
constellation, based on multi-stakeholder collaboration and knowledge sharing, rather
than on user involvement.
However, in the Living Labs papers that deal with Open Innovation, for the most part this
is equaled to open collaborative innovation, as it is argued that Open Innovation stresses
user involvement and that Open Innovation takes place in a process of co-creation with
internal and external parties. This ignores Open Innovation processes such as licensing
and buying, which do not involve any form of co-creation at all. For example, this is also
apparent in Westerlund and Leminen (2011) who see Open Innovation as a driver for user
involvement and mention open source and crowdsourcing as alternatives to conventional
in-house development. Based on their research, we proposed five distinct stakeholder roles
within Living Labs: users, utilizers, providers, enablers and researchers. Despite the fact
that Open Innovation is far from the dominant reference framework in Living Labs
literature, we could find references to knowledge transfers between actors in all of the
papers. As we considered this as one of the key characteristics of Open Innovation, we can
conclude that Open Innovation is implicitly present witin Living Labs. Referring to the
European Paradox, or the apparent gap between knowledge exploration and exploitation,
at least in the literature there is also an imbalance in Living Labs. All of the Living Labs
papers refer to knowledge exploration processes, whereas only one out of three papers
mention exploitation processes. At least in terms of the Living Labs theory, there seems to
be an issue with overcoming the European Paradox as there is too much focus on
exploration.
Regarding User Innovation, 17 papers explicitly refer to this paradigm as theoretical
foundation, but in all papers user involvement is a given which also shows that User
Innovation is at least implicitly present in the Living Labs literature. Regarding the degree
of user involvement, one of the key frameworks we identified in the User Innovation
literature, design with users is dominant in the majority of the papers, whereas design for
users, or the classical voice-of-the-customers techniques, is the main user involvement
mode in 11 papers.
However, based on the literature, there is no general methodology towards user
involvement in Living Labs, and the literature from the User Innovation paradigm is rarely
extensively mentioned or implemented in the context of Living Labs. The Lead User
concept pops up from time to time, but no clear method on how to implement this is
provided. The only main difference in user involvement approach between Living Labs
was so-called open user involvement (self-selection) versus closed user involvement
(selecting users with certain characteristics). The most clear definition sees Living Lab
projects as a quasi-experimental approach with a pre and a post assessment of users
with an intervention stage. This adheres to the three principles of Dell'Era and Landoni
(2014), as this allows to capture the use context, the artifact can be seen as the intervention
with the innovation or another stimulus (Proxy Technology Assessment, Prototype,), and
the user is actively involved in multiple stages (triangulization). Our main conclusion is
that in terms of methodology and user characteristics, the Living Labs literature is rather
silent and positions Living Labs too much as an everything is possible concept that
resembles an empty box, in the sense that you can put whatever methodology or research
approach inside. It remains a given that users are involved in Living Labs, but although cocreation was said to be the central process in Living Labs (Levn & Holmstrm, 2012), 11
papers mentioned innovation for users as the dominant interaction mode. For the 34
papers where innovation with users is dominant, no clear co-creation methodology is put
forward. Therefore, within the current Living Labs literature, it remains unclear whether
Living Labs hold value in terms of structuring user involvement according to User
Innovation theory.
24

5 References
Abowd, G. D., Bobick, A. F., Essa, I. A., Mynatt, E. D., & Rogers, W. A. (2002, July). The
aware home: A living laboratory for technologies for successful aging. In Proceedings of
the AAAI-02 Workshop Automation as Caregiver (pp. 1-7).
Almirall, E., & Wareham, J. (2008). Living Labs and open innovation: roles and
applicability. The Electronic Journal for Virtual Organizations and Networks, 10(3), 21-46.
Almirall, E., & Wareham, J. (2011). Living Labs: arbiters of mid-and ground-level
innovation. Technology Analysis & Strategic Management, 23(1), 87-102.
Almirall, E., Lee, M., & Wareham, J. (2012). Mapping living labs in the landscape of
innovation methodologies. Technology Innovation Management Review, (September 2012:
Living Labs).
Baida, Z., Rukanova, B., Liu, J., & Tan, Y. H. (2007). Rethinking eu trade proceduresthe
beer living lab.
Ballon, P., & Schuurman, D. (2015). Editorial introduction: Living Labs-Concepts, Tools
and Cases. info, 17(4).
Bergvall-Kareborn, B. H. M. S. A., Hoist, M., & Stahlbrost, A. (2009). Concept design with a
living lab approach. In System Sciences, 2009. HICSS'09. 42nd Hawaii International
Conference on (pp. 1-10). IEEE.
Bergvall-Kreborn, B., Ihlstrm Eriksson, C., Sthlbrst, A., & Svensson, J. (2009). A milieu
for innovationdefining living labs. In 2nd ISPIM Innovation Symposium, New York (pp.
6-9).
Bliek, F., van den Noort, A., Roossien, B., Kamphuis, R., de Wit, J., van der Velde, J., &
Eijgelaar, M. (2010). PowerMatching City, a living lab smart grid demonstration. In
Innovative Smart Grid Technologies Conference Europe (ISGT Europe), 2010 IEEE PES
(pp. 1-8). IEEE.
Budweg, S., Schaffers, H., Ruland, R., Kristensen, K., & Prinz, W. (2011). Enhancing
collaboration in communities of professionals using a Living Lab approach. Production
Planning & Control, 22(5-6), 594-609.
Campbell, D.T. & Stanley, J.C. (1966). Experimental and quasi-experimental designs for
research. Chicago: Rand McNally College Pub. Co.
Chesbrough, H. W. (2003). Open innovation: The new imperative for creating and profiting
from technology. Harvard Business Press.
Cosgrave, E., Arbuthnot, K., & Tryfonas, T. (2013). Living labs, innovation districts and
information marketplaces: A systems approach for smart cities. Procedia Computer
Science, 16, 668-677.
De Moor, K., Ketyko, I., Joseph, W., Deryckere, T., De Marez, L., Martens, L., & Verleye, G.
(2010). Proposed framework for evaluating quality of experience in a mobile, testbedoriented living lab setting. Mobile Networks and Applications, 15(3), 378-391.
Dell'Era, C., & Landoni, P. (2014). Living Lab: A Methodology between User-Centred
Design and Participatory Design. Creativity and Innovation Management, 23(2), 137-154.
Dutilleul, B., Birrer, F. A., & Mensink, W. (2010). Unpacking european living labs: analysing
innovations social dimensions. Central European Journal of Public Policy, 4(1), 60-85.
Eriksson, M., Niitamo, V. P., & Kulkki, S. (2005). State-of-the-art in utilizing Living Labs
approach to user-centric ICT innovation-a European approach. Lulea: Center for
Distance-spanning Technology. Lulea University of Technology Sweden: Lulea.
Etzkowitz, H. (2008). Triple helix innovation: Industry, university, and government in
action. London and New York: Routledge.

25

Feurstein, K., Hesmer, A., Hribernik, K. A., Thoben, K. D., & Schumacher, J. (2008). Living
Labs: a new development strategy. European Living Labsa new approach for human
centric regional innovation, 1-14.
Flstad, A. (2008a). Towards a living lab for the development of online e community
services. The Electronic Journal for Virtual Organizations and Networks, 10, 47-58.
Flstad, A. (2008b). Living Labs for innovation and development of information and
communication technology: a literature review. Electronic Journal of Virtual
Organisations, 10(Special Issue "Living Labs"), 99-131.
Haymaker, J., & Chachere, J. (2006). Coordinating goals, preferences, options, and analyses
for the Stanford Living Laboratory feasibility study. In Intelligent computing in
engineering and architecture (pp. 320-327). Springer Berlin Heidelberg.
Helsinki
Manifesto
(2006).
Retrieved
at
http://elivinglab.org/files/Helsinki_Manifesto_201106.pdf
Hess, J., & Ogonowski, C. (2010, June). Steps toward a living lab for socialmedia concept
evaluation and continuous user-involvement. In Proceedings of the 8th international
interactive conference on Interactive TV&Video (pp. 171-174). ACM.
Hlauschek, W., Panek, P., & Zagler, W. L. (2009, June). Involvement of elderly citizens as
potential end users of assistive technologies in the Living Lab Schwechat. In Proceedings
of the 2nd International Conference on PErvasive Technologies Related to Assistive
Environments (p. 55). ACM.
Katzy, B., & Turgut, E. (2010, June). Developing an approach to measure innovation
performance in collaborative networks. In 16th International Conference on Concurrent
Enterprising (pp. 21-23).
Kaulio, M. A. (1998). Customer, consumer and user involvement in product development:
A framework and a review of selected methods. Total Quality Management, 9(1), 141-149.
Krieg-Brckner, B., Rfer, T., Shi, H., & Gersdorf, B. (2010). Mobility assistance in the
Bremen Ambient Assisted Living Lab. GeroPsych: The Journal of Gerontopsychology and
Geriatric Psychiatry, 23(2), 121.
Kusiak, A. (2007). Innovation: the living laboratory perspective. Computer-Aided Design
and Applications, 4(6), 863-876.
Leminen, S., Westerlund, M., & Nystrm, A. G. (2012). Living Labs as open-innovation
networks. Technology Innovation Management Review, (September 2012: Living Labs).
Levn, P., & Holmstrm, J. (2008). Consumer co-creation and the ecology of innovation: A
living lab approach. In Public systems in the future: possibilities, challenges and pitfalls.
Lichtenthaler, U., & Lichtenthaler, E. (2009). A Capability-Based Framework for Open
Innovation: Complementing Absorptive Capacity. Journal of Management Studies, 46(8),
1315-1338.
Liedtke, C., Welfens, M. J., Rohn, H., & Nordmann, J. (2012). LIVING LAB: user-driven
innovation for sustainability. International journal of sustainability in higher education,
13(2), 106-118.
Markopoulos, P., & Rauterberg, G. W. M. (2000). LivingLab: A white paper. IPO Annual
Progress Report, 35, 53-65.
Mulder, I., Fahy, C., Hribernik, K., Velthausz, D., Feurstein, K., Garcia, M., ... & Stahlbrost,
A. (2007). Towards harmonized methods and tools for Living Labs. Expanding the
Knowledge Economy: Issues, Applications, Case Studies, 4, 722-729.
Mulder, I., Velthausz, D., & Kriens, M. (2008). The living labs harmonization cube:
Communicating living labs essentials. The Electronic Journal for Virtual Organizations
and Networks, 10, 1-14.
Niitamo, V. P., Kulkki, S., Eriksson, M., & Hribernik, K. A. (2006). State-of-the-art and good
practice in the field of living labs. In Proceedings of the 12th International Conference on

26

Concurrent Enterprising: Innovative Products and Services through Collaborative

Networks. Italy: Milan (pp. 26-28).
Ogonowski, C., Ley, B., Hess, J., Wan, L., & Wulf, V. (2013, April). Designing for the living
room: long-term user involvement in a living lab. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (pp. 1539-1548). ACM.
Pallot, M., Trousse, B., Senach, B., & Scapin, D. (2010, August). Living lab research
landscape: From user centred design and user experience towards user cocreation. In First
European Summer School'Living Labs'.
Perkmann, M., & Walsh, K. (2007). Universityindustry relationships and Open
Innovation: Towards a research agenda. International Journal of Management Reviews,
9(4), 259-280.
Pierson, J., & Lievens, B. (2005, November). Configuring living labs for a
thickunderstanding of innovation. In Ethnographic Praxis in Industry Conference
Proceedings (Vol. 2005, No. 1, pp. 114-127). Blackwell Publishing Ltd.
Ponce de Leon, M., Eriksson, M., Balasubramaniam, S., & Donnelly, W. (2006). Creating a
distributed mobile networking testbed environment-through the living labs approach. In
Testbeds and Research Infrastructures for the Development of Networks
Ryu, K. D., Zhang, X., Ammons, G., Bala, V., Berger, S., Da Silva, D. M., ... & Sabath, M.
(2010, November). RC2-A Living Lab for Cloud Computing. In LISA.
Santoro, R., & Conte, M. (2009). Living labs in open innovation functional regions. In 15th
International Conference on Concurrent Enterprising. Leiden, NL.
Schaffers, H., Cordoba, M. G., Hongisto, P., Kallai, T., Merz, C., & Van Rensburg, J. (2007).
Exploring business models for open innovation in rural living labs.
Schumacher, J., & Feurstein, K. (2007, June). Living Labsthe user as co-creator. In ICE
2007 Proceedings: 13th International Conference on Concurrent Enterprising, Sophia
Antipolis, France. The Free Press.
Schuurman, D., & De Marez, L. (2009, May). User-centered innovation: towards a
conceptual integration of lead users and Living Labs. In Proceedings of COST298conference The Good, The Bad and The Challenging (pp. 13-15).
Schuurman, D., De Moor, K., De Marez, L., & Evens, T. (2011). A Living Lab research
approach for mobile TV. Telematics and Informatics, 28(4), 271-282.
Schuurman, D., De Marez, L., & Ballon, P. (2013). Open Innovation Processes in Living Lab
Innovation Systems: Insights from the LeYLab. Technology Innovation Management
Review, 3(1).
Schwartz, T., Denef, S., Stevens, G., Ramirez, L., & Wulf, V. (2013, April). Cultivating energy
literacy: results from a longitudinal living lab study of a home energy management system.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp.
1193-1202). ACM.
Schwittay, A. (2008). A Living LabCorporate Delivery of ICTs in Rural India. Science
Technology & Society, 13(2), 175-209.
Scott, K., Quist, J., & Bakker, C. (2009, June). Co-design, social practices and sustainable
innovation: involving users in a living lab exploratory study on bathing. In Proceedings of
Paper for the Joint Actions on Climate Change Conference, Aalborg, Denmark (pp. 8-9).
Sthlbrst, A. (2012). A set of key principles to assess the impact of Living Labs.
International Journal of Product Development, 17(1), 60-75.
Svensson, J., Eriksson, C. I., & Ebbesson, E. (2010, January). User contribution in
innovation processes-reflections from a Living Lab perspective. In System Sciences
(HICSS), 2010 43rd Hawaii International Conference on (pp. 1-10). IEEE.
Third, A., Richardson, I., Collings, P., Rahilly, K., & Bolzan, N. (2011). Intergenerational
attitudes towards social networking and cybersafety: A living lab.

27

Van de Vrande, V., De Jong, J. P., Vanhaverbeke, W., & De Rochemont, M. (2009). Open
innovation in SMEs: Trends, motives and management challenges. Technovation, 29(6),
423-437.
Wadhwa, R. S. (2012). Flexibility in manufacturing automation: A living lab case study of
Norwegian metalcasting SMEs. Journal of Manufacturing Systems, 31(4), 444-454.
West, J., & Bogers, M. (2013). Leveraging external sources of innovation: a review of
research on open innovation. Journal of Product Innovation Management. 31, 4 (July 2014),
doi: 10.1111/jpim.12125. Available at SSRN: http://ssrn.com/abstract=2195675
Westerlund, M., & Leminen, S. (2011). Managing the challenges of becoming an open
innovation company: experiences from Living Labs. Technology Innovation Management
Review, (October 2011).
Wolfert, J., Verdouw, C. N., Verloop, C. M., & Beulens, A. J. M. (2010). Organizing
information integration in agri-foodA method based on a service-oriented architecture
and living lab approach. Computers and electronics in agriculture, 70(2), 389-405.

28

Living labs as a mean

to spur collaboration and innovation
in a tourist destination
Dominic Lapointe a, David Guimont a & Alain Svigny a
a

Universit du Qubec, Montral (Canada)

Lapointe.dominic@uqam.ca

Abstract
The fragmented nature of the tourism industry and the ease of copying tourism innovation
limit destinations capacities and motivations to innovate. To counter this fragmentation,
destinations rely on destination management organisations (DMOs). However, DMOs
have had very little or no success in stimulating innovation, hence the importance of
exploring other models, such as living labs (LLs), to boost the innovation capability of
tourism stakeholders. This paper analyses the innovation capability of a group of tourism
stakeholders and users (tourists and tourism providers) involved in action research
through a LL aimed at co-creating an experience enhanced by technology. It argues that
the co-creation processes at work in the LL have raised innovation capability by using a
combination of technology-supported innovation processes and in situ innovation
processes. In this context, collaborative innovation takes place iteratively over a two years
projects.
Keywords
Living lab, open innovation, tourism, Quebec, e-tourism, innovation capability, barriers
Acknowledgement
This research is funded by receive its funding from the Ministre de lducation, de
lEnseignement suprieur et de la Recherche et de la Science du Qubec (Programme
daide la recherche et au transfert Volet innovation sociale)
In todays increasingly technological world, the need to adapt and innovate has become a
major imperative for many industries. Tourism is no stranger to that trend, and is
undergoing a complete technological revolution that has upset the industry and its
businesses and forced them to rethink their operations (Baggio et al. 2014;ii/Gretzel et al.
2006/Buhalis and Law 2008). While the industry is considered a pioneer in the use of
information technology (IT) (Buhalis and Law 2008), innovation has not been widely
discussed in tourism literature (Halkier et al. 2014).
Furthermore, new innovation paradigms are emerging and transforming existing
innovation processes. One such paradigm is open innovation, which proposes that
29

businesses use external knowledge and skills to accelerate the innovation process
(Chesbrough 2006). Strong competition and the risks that innovation entails are driving
many organisations to put the consumer or user at the heart of innovation efforts
(Westerlund and Leminen 2011). However, the tourism industry appears to struggle to
integrate those new approaches (Najda-Janoszka and Kopera 2014).
Although research on tourism innovation is a relatively recent phenomenon, it is possible
to identify drivers, barriers and innovation processes specific to the industry. We will focus
more specifically on the role of destination management organisations (DMOs) in
stimulating tourism innovation. We will look at how implementing a living lab (LL) can
help DMOs to boost tourism innovation at the destination level. Then, we will introduce
the LL concept and discuss the specific LL process implemented as part of our research,
and its impact on innovation among project stakeholders.
1 Innovation in the tourism industry
Innovation in tourism is a complex but relevant topic. The Internet has globalised the
tourism offering (Buhalis and Law 2008), generating fiercer competition between
businesses and destinations and driving many of them to try and innovate in order to
remain competitive (Halkier et al. 2014). This very competitive environment, coupled with
major technological change, has placed innovation at the heart of the tourism
phenomenon. The Internet and mobile phones unleashed a wave of innovation that keeps
transforming the ways of travelling and the tourist experience (Buhalis and Law 2008).
Unsurprisingly, innovation has become a critical issue for many tourism organisations and
businesses, which innovate primarily in an attempt to create value for themselves, for
consumers or for both (Weiermair 2004). Innovation can take place at the offering level
(products and services) or at the process level (Weiermair 2004/Najda-Janoszka and
Kopera 2013). The key factors that drive tourism businesses to innovate include customers,
competition, industry leadership (Weiermair 2004) and technology providers (NajdaJanoszka 2013). But the true catalyst for innovation is the tourists themselves, or customers
(Weiermair 2004/Najda-Janoszka 2013). Constantly searching for satisfactoryeven
unforgettableexperiences, they are eager for novelty.
Although tourists are looking for new experiences, the tourism industry in its current form
is struggling to innovate (Weiermair 2004/Halkier 2014/Najda-Janoszka and Kopera 2014),
faced with a number of barriers. Najda-Janoszka and Kopera (2014) identified the main
barriers to tourism innovation and grouped them into categories: environmental
(external), organisational (internal) and innovation process-related (Table 1).
Environmental
(external)

Organisational (internal)

Innovation
related

process-

Heterogeneity
of
businesses
Size of businesses
Volatility of businesses
Demand fluctuations
Low-trust culture
Inadequate
tourism
policies
Limited legal protection

Small size of businesses

Lack
of
innovation
management, knowledge
management and change
management culture
High personnel turnover
Insufficient IT skills and
resources

Informal, ad hoc and

poorly
understood
innovation process
Inefficient
knowledge
management process
Lack of interest from
businesses

Table 1 - The barriers to innovation in tourism, adapted from Najda-Janoszka and Kopera (2014)

30

These authors argue that several barriers stem from specific features of the tourism
industry: heterogeneity of businesses, large number of small businesses, volatility of
businesses, and vulnerability to demand fluctuations. Low propensity to collaborate on
innovation and inefficient knowledge transfer create a low-trust culture among partners
(Hjalager 2010). Several authors also point out that public institutions, despite their key
role in fostering innovation, are viewed as significant forces of inertia (Halkier 2014/NajdaJanoszka 2013/Weiermair 2004). Tourism policies and strategies are often inadequate, as
they fail to respond to the current needs of businesses and the changing market
environment (Halkier 2014/Najda-Janoszka and Kopera 2014).
Internal or organisational barriers often arise from the very nature of the industry, which
includes large numbers of small businesses that often lack the financial and human
resources and the skills necessary to innovate. Lack of knowledge and skills in the areas of
technology, innovation, and change and knowledge management (Najda-Janoszka and
Kopera 2014), as well as high personnel turnover, shows that human resources pose a
major challenge to innovation. In addition, stakeholders absorptive capacityi.e., their
capacity to acquire and retain the knowledge necessary to innovate (Cohen and Levinthal,
1990) appears to be low. This situation is thought to stem from the prevalence of small
and medium-sized enterprises (SMEs) in the tourism industry (Baggio and Cooper 2010).
When it comes to innovation processes, lack of formal, clear and permanent processes
hinders innovation in the tourism industry. Also worthy of note is the fact that businesses
show little interest in the process due to their low innovation capability and to the
difficulty of protecting tourism innovations (Najda-Janoszka and Kopera 2014). Adding to
this situation is the low level of trust among tourism organisations (Rnningen 2010).
Although the tourism industry claims to place consumers/tourists at the forefront, they are
virtually never involved in innovation processes (Najda-Janoszka and Kopera
2013/Guimont and Lapointe 2015). Tourist involvement could be achieved by opening up
the innovation process (Chesbrough 2005/Lapointe and Guimont 2014), notably through
co-creation of the tourist experience (Neuhofer et al. 2012).
With the rise of social media, it is becoming ever harder to keep viewing tourists merely as
passive consumers (Gretzel et al. 2014). Todays tourists expect the industry to cater to their
complex and individual needs (Gretzel et al. 2014). Tourists must be considered fullfledged players involved in the market (Neuhofer et al. 2012/Gretzel et al. 2014), and
potential co-creators of the products and services intended for them.
Co-creation is primarily an approach that places the user, or tourist, at the heart of the
destinations processes and strategies. It offers a new perspective on the market, resources,
tourists and technology, incorporating tourists ideas and creativity into the design of
tourist products and services (Tussyadiah and Zach 2013). This integrated approach is
thought to boost innovation. Co-creation means looking at the destination in a new light
and developing tools, techniques and capacity for involving not only tourists (Tussyadiah
and Zach 2013), but also other stakeholders in the tourism systembasically, working with
tourists, not for tourists. Accordingly, co-creation involves opening up the innovation
process to players outside the organisation.
Open innovation can be understood as the opposite of the vertical, internal approach to
innovation. It is a new paradigm based on the assumption that businesses must use both
internal and external ideas in the innovation process (Chesbrough 2006). It views research
and development as an open system: since the knowledge needed for innovation may be
found outside the organisation, external sources of knowledge should be identified and
incorporated into the innovation process (Chesbrough 2006).
Open innovation, which relies in large part on the co-operation of various stakeholders,
does entail a number of constraints. Several barriers to adopting that new approach to
innovation stem from human resistance to change (Lapointe and Guimont 2014). Chief
31

barriers include fear of giving an edge to competitors (Veer et al. 2012), lack of human and
financial resources, uncommitted managers, lack of time, and inappropriate innovation
strategies (Chesbrough and Crowther 2006/Meer 2007). Moreover, public institutions fail
to take on a leadership role in fostering collaboration and innovation (Najda-Janoszka and
Kopera 2013).
The low propensity of tourism businesses to collaborate on innovation (Najda-Janoszka
and Kopera 2014/Hjalager 2002) highlights the need for an effective and efficient
intermediary that can improve collaboration and stimulate innovation. Since DMOs play a
key role in ensuring competitiveness and co-operation at the destination level, one might
think that they would take on a leadership role in fostering collaboration on innovation.
Tourism SMEs that struggle to implement the management systems required to spur
innovation (Rnningen 2010) could turn to DMOs or other organisations for knowledge
and information that would allow them to innovate. But the truth is that DMOs seem to
have trouble adapting to the current technology revolution (Gretzel et al. 2006/Neuhofer et
al. 2012) and therefore positioning themselves as innovation leaders. In fact, public
institutions overseeing tourism are often said to hinder innovation (Weiermair
2004/Najda-Janoszka 2013). Their governance structure appears to prevent them from
moving away from their traditional promotion-based approach and building their new
strategies around innovation (Halkier 2014). They are also struggling to create an
environment that promotes collaboration and innovation (Najda-Janoszka 2013). The
difficulty in getting tourism stakeholders to collaborate is now a major hurdle to
innovation in tourism (Halkier 2014/ Najda-Janoszka 2013) and forces tourism
organisations to seek other innovation models.
2 Living labs as a mean to spur innovation
The rise of the Internet and social media prompted a shift in approaches to the market,
resources and individuals (Vargo and Lusch 2004/Prahalad and Ramaswamy 2004). As
information technology develops and social media emerge, it is becoming harder to ignore
individuals power and desire to get involved (Prahalad and Ramaswamy 2004). Value
creation used to be considered the sole prerogative of businesses, but nowadays value is
co-created by both individuals and businesses (Prahalad and Ramaswamy 2004). Cocreationa new, individual-oriented way to look at how the market workshas driven
organisations to design new ways of involving and collaborating with individuals (Kotler et
al. 2010). Innovation, which was traditionally restricted to academics or research and
development departments, now increasingly calls on users as co-creators. Methods are
being modelled on innovation networks or ecosystems from the IT industry, including LLs.
LLs fall within the open innovation paradigm and involve a user-centric approach. They
provide physical regions or virtual realities in which stakeholders form public-privatepeople partnerships (PPPP) of firms, public agencies, universities, institutes, and users all
collaborating for creation, prototyping, validating, and testing of new technologies,
services, products, and systems in real-life contexts (Westerlund and Leminen 2012: 7).
Stakeholders collaborate on common value-creation objectives by co-developing products
and services focused on user needs (ifrer et al. 2012).
With the LL approach, users must be at the centre of research or innovation efforts.
Instead of attempting to understand users or consumers through studies, some
organisations now prefer to directly involve users in their actual innovation process
(Westerlund and Leminen 2011). This co-creation approach stimulates innovation and
delivers a number of benefits: better grasp on consumers latent needs, lower risk of failure
in product and service design, shorter lead times for new products and services, and higher
profits (Westerlund and Leminen 2011). The LL addresses simultaneous processes of
32

exploration and exploitation as suggested by Almirall and Wareham (2011), although it

appears that living labs are particularly good for exploration purposes. However, living
labs also hold a lot of potential in terms of retention of generated knowledge, especially
when successive cases run on the same living lab infrastructures (Schuurman et al. 2013).
We believe that the process holds promise for stimulating tourism innovation. Through
LLs, tourist service providers will obtain insight to what tourists actually want and will
have an opportunity to improve and develop new services targeted to different customer
segments (Lenart et al. 2014). Such insight could not only enable identification of new
markets, but also spur innovation, development and product improvement (Buhalis and
Amaranggana 2014) through more frequent interactions among stakeholders in a
partnership. Interactions between users (tourists) and providers of technology and tourist
services being a key catalyst for innovation (Hjalager 2002), LLs could create increased
collaboration opportunities through a common platform where stakeholders would share,
discuss, assess and design various solutions (Lenart et al. 2014). In addition, LLs have the
potential to become innovation facilitators (Schuurman et al. 2014/Lapointe and Guimont
2014) and thus create what DMOs are struggling to build: an environment that promotes
co-operation among tourism industry stakeholders to enable innovation (Najda-Janoszka
2013).
The tourism industrys innovation challenges led us to implement a LL process to promote
exploration and retention of information and observe how LL participants collectively and
individually address the barriers to innovation that they face.
3 The living lab process implemented in Rivire-du-Loup
Over the years, many regions around the globe have looked at tourism and made it the
cornerstone of their development strategies. An explosion in the number of destinations
ensued. While many of them are small, they still need to deal with the technology
revolution that is transforming the tourism industry. Investigations into how a smaller
destination manages to integrate new innovation practices are therefore extremely
relevant. The essentially rural territory of the Rivire-du-Loup area makes it an interesting
object of study, with unique dynamics compared to large urban centres. In the Quebec
territorial context, the Rivire-du-Loup area is relatively far from large outbound markets
(Montreal and Quebec City) and is not considered a hub for international tourism. It is a
rural area with a favourable geographical setting, along the vast St. Lawrence
estuary. Featuring interesting landscapes (forest, estuary and islands), the area is home to
Rivire-du-Loup, a town of 19,400 people, and a dozen smaller municipalities that bring
the total population to 27,700.
The areas DMO is a private association with a membership of some 230 tourism
organisation and businesses. Members also include the municipalities, which provide
nearly 50% of the operating budget through public funds. The association is very active,
focusing on the traditional DMO roles: information and promotion. It fosters dialogue and
supports development initiatives as far as its resources allow.
Based on the above list of barriers (adapted from Najda-Janoszka and Kopera 2014), the
area (its tourism businesses and its DMO) struggles with the following:
Limited IT skills and resources
Small businesses, often with high personnel turnover because of their seasonal
nature, giving organisations limited absorptive capacity for technology and
innovations
Businesses often lacking a strong knowledge and change management culture;
some level of inertia among organisations; poor understanding of the value of
innovation; risk aversion
33

Very basic levels of trust and collaboration, especially compared to open

innovation standards
On the flip side, businesses and stakeholders are willing to innovate in spite of
existing barriers
These observations show the value and challenge of boosting innovation and stakeholder
innovation capability, and provide the backdrop for our action research: Co-creation of a
tourist experience enhanced by technology, in the context of a living lab.

4 The projects LL process and the stakeholders involved

The DMOs initial goal was to update its sightseeing routes (available on paper maps) and
to record podcasts providing tourists with complementary information about the sights,
and about rural attractions in particular. LLio, the regions LL, approached the DMO and
offered to co-lead a co-creation process involving stakeholders and tourists working
together to develop a technology proposal for the sightseeing route update. The facilitated
open innovation process would be funded through a Quebec government research
program, and the DMO had to be comfortable with not knowing the outcome in advance.
A private-sector partner was also approached to be involved not as a mere supplier, but as
a full-fledged co-creator of a technology-enhanced tourist experience.
Our action research in a LL context involves three stakeholder groups: the LLio, a LL
associated with the local teaching institution, the DMO, and a local web developer. A
steering committee made up of one representative from each group is in charge of project
coordination. Here is the breakdown of stakeholder groups in a typical PPP partnership in
a LL context (Almirall and Wareham 2008):
Population (beneficiary): The DMO and its members
Private sector : The web developer
Public (institutional/research) sector: LL with his researchers and students
User groups involved in co-creation
Tourism stakeholders: 19 volunteers (DMO members) who commit for two years.
Recruited from the pool of DMO members following a tourism industry discussion
event, using personal invitations, and through a general call for participants.

Tourists: 21 Volunteers. French-speaking tourists who use information and

communications technology (ICT) and own a tablet or smartphone. Recruited from
the DMOs visitor centre, the DMOs Facebook page and the living labs panel.

Tourism providers take part in co-creation workshops both in situ and on the online
platform. Tourists participate online only. We use in situ to characterized the
participation in local co-creation workshops (not to be confused with in situ as in the
context that refers to information from customers/users that originate from a real-life,
value co-creation situation (Edvardsson, 2012). We believe that The in-situ open
innovation workshop could serve as an entry point in OI for private stakeholders (Lapointe
and Guimont, 2014).
Our action research in a living lab context relies on phases, cycles and activities adapted
from the FormIT approach (Bergvall-Kreborn et al. 2009). The first iteration of the fivephase process took place during Year 1 of the project. Here is a more detailed chronology
(table 2), over 24 months:

34

1. Planning
(6
months)
>

2.
Concept 3.
Prototype 4. Innovation
design
design
design
(2
(2
(2
months) >
months)
months)
>
>

5.
Implementati
on
/ beta
testing
(3
>
months)

Stakeholder
integration
Steering
committee
meetings
Recruitment of
tourism
stakeholders
Recruitment of
tourists
Initial meeting
of stakeholders

Co-creation
workshops with
the stakeholders
Three rounds of
interactions with
the tourists to
capture
their
needs,
expectations,
ideas,
suggestions,
reactions
and
feedback

Co-creation
workshops with
the
stakeholders.
New round of
interactions
with the tourists
focusing on the
journey map, in
order to test the
prototype

Development
of the beta
application
Tests with the
stakeholders
and tourists

Public launch
of the beta
version
Real-life
testing during
the summer to
assess
the
applications
userfriendliness
and
the
overall tourist
experience

1. Planning and 2. Concept review 3.

Prototype
adjustments
and
design
(1 month)
experimentation (2 months)
with new options
(2 months)

4. Innovation
design

second beta
version and/or
complementar
y modules
(2 months)

5.
Implementati
on
(2 months)

Second iteration (15) (9 months)

Table 2 - Adapted FormIT process

During Year 2, the co-creation process will be repeated. This second iteration will allow for
adjustments to the collaborative working mechanisms and the application itself. The same
steps and stakeholders will be used (although additional participants could be onboarded).
In addition to recruiting and involving various stakeholders, and designing the co-creation
phases, the project involved implementing an information gathering and sharing
infrastructurea collaborative infostructure, a digital environment for networking
(Najda-Janoszka and Kopera 2014). The group of stakeholders uses the collaborative
innovation lab at the teaching institution, the living labs collaborative web platform
(WordPress website with posts and a forum) and a closed Facebook group for co-creation
and for viewing information obtained from tourists. Tourists are surveyed primarily
through online surveys, and they can access profiles and summaries on the collaborative
platform. Shared online forms and documents are also used for specific interactions (e.g.,
co-creation of common representations based on commented texts). Examples of in situ
and online asynchronous interactions between tourism providers and tourists :
Stakeholders (tourism providers) make hypothesis about tourists needs and
preference, during co-creation workshops.
Then, by the mean of an online survey, the tourists express their preferences, needs
and habits (about technology, travel and technology in travel). Open and narrative
forms are preferred to let tourists express themselves. We use the same survey with
the tourism providers to have a comparison point (and because they are tourists
too).
35

During a new workshop, tourism providers use the synthesis of the responses to
better conceptualize the new technology enhanced experience.
The tourists are then questioned again, building on the synthesis of their previous
answers and the concept they are being presented.

This kind of asynchronous exchanges goes on at every phases. At each step, everyone gets
a better view of the whole concept and everybody can see the impact of the suggestions of
its group/community. The communities cumulate informations, knowledges and valuable
insights about the tourism experience 2.0 : the way to co-create it and what are the real
needs and aspirations of tourists.
The role of the lead researcher in this process is to oversee the living lab process, prepare
co-creation workshops, and facilitate co-creation both in situ and online. He leads the
experimentation component. He also documents the process (planned versus completed
tasks) as well as user participation and engagement to describe how innovation capability
is growing, and the drivers and barriers at play. The collaborating researchers support the
co-creation process, deliver specific workshops on technology and the tourist experience,
and help to document and characterise the growth in innovation capability.
5 Stimulating innovation
Now that Phase 1 of the project has been completed, we can observe changes in
stakeholder innovation capability. The project is acting on both the barriers and the
innovation process. Changes were observed in all three of the abovementioned barrier
categories.
Gaps in IT competency in the stakeholder population had been identified at the outset of
the LL project. After Phase 1, we can see that the gaps have shrunk and that stakeholders
IT skills have grown. First, upgrading efforts were made: stakeholders received training on
co-creation and on technologys role in tourism. The fact that participants actively
contribute to the collaborative platform and share best tourism and IT practices on the
projects Facebook page provides further evidence of the shrinking gaps. Moreover,
participants are embracing IT and tourism discourse and concepts during in situ meetings.
We can also observe changes in the innovation management culture. Project stakeholders
told the researchers that they have incorporated open innovation tools into their
management practices. New behaviours include using innovation project management
template and involving stakeholders in innovation efforts. The matrix used by the project
researchers to characterise and describe innovation initiatives was adopted by other
stakeholders, who now manage their own projects in a more participative and user-centric
fashion.
For the DMO, the project was a catalyst for change and made it aware of its role in driving
members towards innovation. A rethinking of the DMOs planning strategy ensued. The
five-yearly tourism forum was organised using a participative and collective intelligence
approach focused on members as users and on community dynamics. Furthermore, the
DMO applied knowledge management strategies to identify best practices, but also in an
effort to place tourists and their knowledge at the heart of planning activities. We can
conclude that the project has shifted the focus to userswhether DMO members or
touristsas a potential source of innovation.
The new focus on tourists as users was also observed among other project stakeholders. As
the project advanced and the mobile application which is its core deliverable started to
take form, stakeholders increasingly suggested recourse to the tourist panel, whereas such
suggestions used to be made only by researchers. Stakeholders now spontaneously suggest
that questions be put to tourists instead of looking for answers themselves, and researchers
36

have to adapt the surveys sent to the tourist panel accordingly. As a result, the innovation
process is increasingly centred on the needs of tourists/users.
The project has also lifted or reduced process-related barriers, formalising the innovation
and solution-seeking process among stakeholders. The result has been integration of in
situ open innovation practices (Lapointe and Guimont 2014) by project stakeholders in
their own development activities. Two local development agents have incorporated such
processes into their operations. Another example would be the tourism forum based on
open innovation practices organised by the DMO in order to identify high-priority projects
for strategic planning. Lastly, the technology provider involved in the project is a
committed participant in the LL process: it finds the process inspiring and does not
attempt to derail it and impose its regular technology development process instead. The
provider has also expressed interest in using the LL process for future development efforts.
As for external barriers, the project has fostered a climate of trust within the industry.
Collaboration opportunities among stakeholders have flourished. In addition, stakeholder
engagement has been maintained: since the very beginning, not one participant has left; in
fact, new people were onboarded as the project advanced. In addition, information sharing
among stakeholders has grown, whether in situ, on the collaborative platform or on the
projects Facebook page. Lastly, the related projects jointly initiated by project
stakeholders testify to the climate of greater trust that now exists.
Stakeholders have not only raised their innovation capability in the specific context of the
project, but also developed new innovative projects of their own beyond the scope of the
LL project that is the focus of our research. We regard such spin-offs as the main indicator
that participation in a LL project has raised stakeholders innovation capability. Four spinoff projects are already under way, spurred by the acquisition of knowledge about the 2.0
tourist experience and the integration of co-creation skills. The projects are as follows:
New inspiration/search module better suited to visitors needs on the DMOs
website
Creation of a research and development unit, as part of the web developer training
program, that works on developing a technology-enhanced experience (Estimote
and iBeacon) for a local attraction
Launch of a joint geocaching/treasure hunting project by the town and county
departments of cultural development
New action research project aimed at turning an island in the St. Lawrence into a
tourist destination using LL-inspired collective intelligence processes
The collaborative process yielded a concept more fitting than the DMOs original goal
(before getting involved in the project, the DMO wanted to develop podcasts). The mobile
application concept will help visitors to plan, navigate and share their discovery
experience. Users can find inspiration, select attractions, and edit, share or comment on
preset routes.
However, limitations and challenges were encountered. The objective was to recruit
more than 25 tourists (up to 100). We currently have 21. Recruitment is challenging, notably
because tourists do not form communities of technology users (unlike users of a given
phone brand or fans of a specific video game, for instance). Recruiting and involving
participants from a community that does not exist formally is a difficult task. Lack of
community spirit also makes interaction more challenging. Our action research will help
to document methods and tools for promoting more effective interaction and reinforcing
the culture of co-creation in tourist communities. Lastly, as we document the co-creation
process followed by tourism practitioners and stakeholders, we notice that individuals
motivations and willingness to subscribe to a given goal depend on their respective fields
of expertise. The living lab facilitator is very helpful in reconciling divergent views,
37

specifically reconciling the DMOs natural tendency to focus mostly on promotion and
staging, but less on technology and co-creation (level 1 of the Typology of technology-
enhanced tourism experiences [Neuhofer et al., 2014; 346]) with the web developers
natural tendency to propose a mobile tool offering maximum reliability but not
necessarily suited to co-creation, which requires some degree of randomness and
compromise in the programming. While the process expressly provides for ways to
address such situations, constant adjustments are needed. It will be interesting in the
future to document this challenge using the framework of the technology enhanced tourist
experience (Neuhofer et al., 2014).
6 Conclusion
In spite of the abovementioned limitations, our results lead us to conclude that tourism
stakeholders involved in the project have raised their innovation capability. Initially,
Rivire-du-Loups tourism community faced many of the typical barriers to innovation in
the tourism industry. However, the first year of action research has shown us that the
selected LL approach has indeed reduced several barriers to innovation at the destination
level. The capacity of LLs to stimulate innovation, which has already been observed in
more technology-intensive industries (Schuurman et al. 2014) but also in the tourism
industry (Lenart et al. 2014/ifrer et al. 2014), is confirmed in our case. Change can be seen
in the way that stakeholders embraced the technological tools and the LL process, and in
the emergence of a climate of greater trust that generated spin-offs. Accordingly, this
research also confirms results from our past work on open innovation (Lapointe and
Guimont 2014) and the potential of LLs as intermediaries to open up the innovation
process, but also as a means to raise stakeholder innovation capability. The power of the
LL to boost innovation capability stems from the co-creation process which, as part of this
research, helps to establish an innovation management culture and a climate of trust
among stakeholders.
Researchers role in the process is threefold: they are sparks that move the LL forward,
facilitators, and critics who reassess the process and its outcomes in collaboration with
stakeholders. Based on our experience, we can say that while researchers can step away
from the spark role, the other two roles tend to stick. This is evidenced by spin-offs from
our project: although such spin-offs were not initiated by researchers, the stakeholders
involved still turn to researchers for help in facilitating and critically assessing the process.
In addition, the LLs role as an intermediary and innovation facilitator enables innovation
leadership in an industry whose very structure (prevalence of SMEs and fragmentation
across sectors) makes it hard for a champion of innovation to arise.
While the process implemented with the Rivire-du-Loup DMO has enabled tourism
stakeholders to take part in a structured innovation process, the ultimate question remains
whether it will stand the test of time and be widely adopted by the community. The LL
originally set up for co-creating a technology-enhanced tourist experience would need to
become a more permanent entity providing innovation support, as requested by tourism
stakeholders. Spin-offs provide evidence that a new culture of innovationmore
specifically open innovationis emerging in the industry. Nevertheless, it is too early to
say that innovation is now institutionalised.
7 References
Almirall, E., & Wareham, J. (2008). Living Labs and open innovation: roles and
applicability. The Electronic Journal for for Virtual Organizations and Networks, 10(3), 2146.

38

Baggio, R., Sigala, M., Inversini, A., & Pesonen, J. (2014). Information and Communication
Technologies in Tourism 2014. In Z. Xiang & I. Tussyadiah (Eds.), . Cham: Springer
International Publishing.
Baggio, R., & Cooper, C. (2010). Knowledge transfer in a tourism destination: the effects of
a network structure, (May 2008), 2628.
Bergvall-Kareborn, B., Holst, M., & Sthlbrst. (2009). Concept design with a living lab
approach. In Hawai International Conference on Systems Science (HICSS 42), Big Island,
Hawaii. (pp. 110). Big Island, Hawaii.
Buhalis, D., & Law, R. (2008). Progress in information technology and tourism
management: 20 years on and 10 years after the InternetThe state of eTourism research.
Tourism Management, 29(4), 609623.
Chesbrough, H. (2006). Open innovation: a new paradigm for understanding industrial
innovation. Open Innovation: Researching a New Paradigm.
Chesbrough, H., & Crowther, A. (2006). Beyond high tech: early adopters of open
innovation in other industries. R&d Management, 36(3), 229236.
Cohen, W. M., & Levinthal, D. A. (1990). Absorptive capacity: a new perspective on
learning and innovation. Administrative science quarterly, 128-152.
Gretzel, U., Fesenmaier, D. R., Formica, S., & OLeary, J. T. (2006). Searching for the
Future: Challenges Faced by Destination Marketing Organizations. Journal of Travel
Research, 45(2), 116126.
Gretzel, U., Fesenmaier, D. R., & OLeary, J. T. (2014). The transformation of consumer
behaviour. Tourism Business Frontiers, 918.
Guimont, D., & Lapointe, D. (2015). Co-creation of a tourist experience enhanced by
technology, in the context of a Living Lab. ENTER2015 : Volume 6 Short Paper - E-Review
of Tourism Research, 01/2015(Enter 2015(6).).
Halkier, H., Kozak, M., & Svensson, B. (2013). Innovation and Tourism Destination
Development. European Planning Studies, 22(8), 15471550.
Hjalager, A.-M. (2002). Repairing innovation defectiveness in tourism, Tourism
Management, 23, pp. 465-474.
Hjalager, A.-M., & Nordin, S. (2011). User-driven Innovation in TourismA Review of
Methodologies. Journal of Quality Assurance in Hospitality & Tourism, 12(4), 289315.
Kotler, P., Kartajaya, H., & Setiawan, I. (2011). Welcome to Marketing 3.0. Hoboken, NJ,
USA: John Wiley & Sons, Inc.
Lapointe, D., & Guimont, D. (2014). Private stakeholders and open innovation in Qubec:
What are the perspectives for living labs? In ENoLL OpenLivingLab Days proceedings 2014.
Leminen, S., Westerlund, M., & Nystrm, A. (2012). Living Labs as Open-Innovation
Networks, (September), 611.
Lenart, G., Pucihar, A., & Maleic, A. (2014). User-Centered Design of a Web-Based
Platform for the Sustainable Development of Tourism Services in a Living Lab Context, 7,
251266.
Meer, H. Van der. (2007). Open innovationthe Dutch treat: challenges in thinking in
business models. Creativity and Innovation Management, 16(2).
Neuhofer, B., Buhalis, D., & Ladkin, A. (2014). A Typology of technology-enhanced tourism
experiences. International Journal of Tourism Research, 16(4), 340-350.
Neuhofer, B., Buhalis, D., & Ladkin, A. (2013). Experiences, co-creation and technology: A
conceptual approach to enhance tourism experiences. CAUTHE 2013: Tourism and Global
Change: On the Edge of Something Big, 562.
Najda-Janoszka, M., & Kopera, S. (2014). Exploring Barriers to Innovation in Tourism
Industry The Case of Southern Region of Poland. Procedia - Social and Behavioral Sciences,
110, 190201.

39

Najda-Janoszka, M. (2013). Innovative Activity of Small Tourist EnterprisesCooperation

with Local Institutional Partners. Journal of Entrepreneurship, Management and Innovation,
vol.9, no.1, 17-32.
Neuhofer, B., Buhalis, D., & Ladkin, A. (2013). A Typology of Technology-Enhanced
Tourism Experiences. International Journal of Tourism Research, 16(4), 340350.
Porter,
I.
(2013).
Une
cible,
1831
messagers.
Retrieved
from
http://www.ledevoir.com/societe/actualites-en-societe/381429/une-cible-1831-messagers
Prahalad, C. K., & Ramaswamy, V. (2004). Co-creation experiences: The next practice in
value creation. Journal of Interactive Marketing, 18(3), 514.
Pallot, M., Trousse, B., Senach, B., & Scapin, D. (2010). Living Lab Research Landscape:
From User Centred Design and User Experience towards User Cocreation. In In First
European Summer SchoolLiving Labs' (Vol. 0).
Rnningen, M. (2010). Innovation in the Norwegian Rural Tourism Industry: Results from
a Norwegian Survey. The Open Social Science Journal, 3(1), 1529.
Schumpeter J. (1934). The Theory of Economic Development. Boston, Harvard University
Press.
Schuurman, D. et al. (2013) Open Innovation Processes in Living Lab Innovation Systems :
Insights from the LeYLab in Technology Innovation Management Review. November 2013 :
28-36.
Sifrer, A., Pucihar, A., Borstnar, M. K., & Lenart, G. (2012, May). Development of the
prototype solution for user involvement in the Living Lab approach. In MIPRO, 2012
Proceedings of the 35th International Convention (pp. 1705-1708). IEEE.
Tussyadiah, I., & Zach, F. (2013). Social Media Strategy and Capacity for Consumer CoCreation among Destination Marketing Organizations. Information and Communication
Technologies in Tourism 2013, 242.
Vargo, S., & Lusch, R. (2004). Evolving to a new dominant logic for marketing. Journal of
Marketing,117.
Veer, T., Lorenz, A., & Blind, K. (2012). How Open is Too Open? The Dark Sideof
Openness Along the Innovation Value Chain. In 35th DRUID Celebration Conference 2013.
Barcelona.
Weiermair, K. (2004). Product improvement or innovation: what is the key to success in
tourism. Innovations in Tourism UNWTO Conference.
Westerlund, M. & Leminen, S. (2011). Managing the Challenges of Becoming an Open
Innovation Company: Experiences from Living Labs, (October), 1925.

40

Girls Making History Summary Report

Communities and culture
network funded research pilot
Penny Evans a, Roz Hall a
a

Knowle West Media Centre (KWMC), University of Bristol

pennyevans@kwmc.org.uk

Abstract
Girls Making History (GMH) is a Bristol-based project designed and led in a co-productive
partnership, by Knowle West Media Centre (KWMC) and a group of local young women
aged 13-24. The project asks how, by harnessing the expertise of young womens direct
experience of violence and coercive control in their relationships, digital tools might raise
awareness of the cultural normalisation of partner violence in teenage relationships and
social networks. A six-month research phase ran from January to June 2014, which was an
opportunity to critically reflect upon how involvement in this co-produced project might
support these young women to transform their understanding of their own experience,
develop new imagined futures and, on a wider scale, transmit new ways of understanding
externally into real and virtual communities.
During this pilot study the young women collaborated with engineers, digital experts and
artists in the design of two digital platforms. In line with the principles of the project this
research was co-produced with the young women, KWMC staff and researchers
collaboratively developing and conducting all elements of the research programme.
1 Introduction
Girls Making History (GMH) is a Bristol-based project designed and led in a co-productive
partnership, by Knowle West Media Centre (KWMC) and a group of local young women
aged 13-24. The project asks how, by harnessing the expertise of young womens direct
experience of violence and coercive control in their relationships, digital tools might raise
awareness of the cultural normalisation of partner violence in teenage relationships and
social networks. A six-month research phase ran from January to June 2014, which was an
opportunity to critically reflect upon how involvement in this co-produced project might
support these young women to transform their understanding of their own experience,
develop new imagined futures and, on a wider scale, transmit new ways of understanding
externally into real and virtual communities.
During this pilot study the young women collaborated with engineers, digital experts and
artists in the design of two digital platforms. In line with the principles of the project this
research was co-produced with the young women, KWMC staff and researchers
collaboratively developing and conducting all elements of the research programme.
41

2 Rationale
Violence against women and girls remains a national crisis. In 2009 the NSPCC published
a ground-breaking report, which, for the first time, documented teenage partner violence
in the UK context4. The study reported significant numbers of both young men and
women perpetrating violent behaviours in their intimate relationships. Young women
were shown to be experiencing violence more frequently and conveyed a significantly
higher degree of negative impacts on their wellbeing than their male counterparts, with
three-quarters of young women reporting experiencing emotional violence, one third
reporting sexual violence and one quarter reporting physical violence.
3 Pilot Project Aims

To develop our understanding of community as envisioned by teenage girls.

To enable teenage girls to overcome barriers to discussing the normalisation of
violence and to understand the value of dissent, through a programme of !upskilling, peer mentoring and co-creation in digital media.
To build confidence in this group of young women to challenge such normalisation
and enable them to support, through a wider campaign, other teenage girls at!risk of
becoming involved in abusive / violent relationships.
To develop our understanding of the potential use of digital tools as!supportive
spaces for teenage girls experiencing intimate partner violence.
To explore how these spaces can be employed as transformative arenas subverting
the !normative nature of teenage intimate partner violence.

4 The process of co production

The research design, responsive to the projects aims, was deliberately outlined in a loose
and flexible manner, anticipating an iterative process that engaged with the emerging
findings of the project. Four work strands were identified and this served as a guide map to
aid the collaborative design of the six-month workshop series.
Girls Making History was borne out of a commitment to collaborative research and
creating partnerships that are meaningful in terms of research contribution, community
development and enhancing self-knowing5 amongst all involved. By using co-production
approaches that have been developed and refined over years of practice at KWMC, there
was a desire to support academic partners to leap outside of their comfort zones into an
unknown and unpredictable territory, to draw on different sites and types of knowledge.
Through the embracing of uncertainty we hoped to open up new possibilities and
contingencies6 for academic research, in particular, to situate young women in a place and
space that recognised the expertise of their experience and could speak to their
marginality as a site of resistance7. By widening and diversifying participation in the

4

40% of teenagers are accepting aggressive behaviour from dating partners, 1 in 4 girls have experienced an
abusive relationship, 1 in 3 girls have experienced sexual violence. (NSPCC (2009) Partner exploitation and
violence in teenage intimate relationships, Barter, C et al)
5
Butler, J. (2005). Giving an account of oneself. Fordham Univ Press.
6
Pearce, J. (2008) We make progress because we are lost: Critical Reflections on Co Producing Knowledge
as a
Methodology for Researching Non-Governmental Public Action, NGPA Working Paper Series. LSE.
7
Hooks, Bell. 1990, Choosing Margin as a Space of Radical Openness, The Feminist Standpoint Theory
Reader:
Intellectual and Political Controversies, Ed. Sandra Harding, (2004), 153-160. Routledge, London.
42

production of knowledge, ideas and capacities we hoped to create a pathway in which the
possibly disruptive truths of young womens experience could acquire greater authority8
and enable the micro-local to reach out into wider power structures. Ultimately GMH
sought to foster transformational outcomes 9 targeting individual and collective
empowerment for progressive social change10, developing reflexively a collaborative
experience of learning.
5 Relevance of the KWMC Approach
GMH expressly situated the experiences and opinions of all those participating as equal,
bringing together digital experts, campaigners, domestic violence specialists, academics
and young women affected by teenage partner violence to experiment with what might be
developed if multiple experts could co-produce together. This approach has close affinity
to models of action research11 and responded to the principle that people are able to
theorize about their lives and experiences and act in self-directed and consciously political
ways to change their own communities12.
The Products
Through this process two digital tools were developed:
Digital Tool 1: The Emoti-Meter
The diamond fits in the hand and the two halves rotate, allowing you to align the colours
and play with colour combinations. The bottom half of the keyring features six coloured
buttons. Each button has a small counter connected to it.

Attached to the chain is a how to guide, which introduces the idea that colours relate to
emotions and that you can privately assign an emotion to each colour. Every time you
press for example, the green button, representing the nervous emotion, the counter clocks
it up. So at the end of the day, or a week, you can reveal the counters and compare how
many times you have felt nervous, scared, or excited. This monitoring of your emotions

8

Brigstocke, Julian. (2013) Democracy and the Reinvention of Authority. Problems of Participation:
Reflections on Authority, Democracy, and the Struggle for Common Life. Ed. Noorani Tehseen, Claire
Blencowe, and Julian
Brigstocke. Authority Research Network; 7-12. Pg. 11.
9
Heron, J. & Reason, P. (2008). Extending epistemology within a cooperative inquiry. In P. Reason & H.
Bradbury
(Eds.) Handbook of action research: Participative inquiry and practice (2nd ed.) (pp. 366380). London: Sage.
10
Banks, S. & Armstrong, A, et al. (2014). Using co-inquiry to study co-inquiry: community-university
perspectives on research collaboration. Journal of Community Engagement and Scholarship 7(1).
11
Bradbury, H., & Reason, P. (2003). Action Research An Opportunity for Revitalizing Research Purpose and
Practices. Qualitative Social Work, 2(2), 155-175.
12
Gatenby, B. & Humphries, M. (2000), February. Feminist participatory action research: Methodological and
ethical issues. In Women's Studies International Forum (Vol. 23, No. 1, pp. 89-105). Pergamon.

43

over a self-organised duration of time gives the user chance to gather an overview of their
own emotional wealth.
Digital Tool 2 - The Game
The Knowing the Signs game was developed with the Games Hub in Bristol. Three
games designers came to a workshop to speak to the young women and get a sense of what
the project was about. Unlike many of the projects that the game designers have worked
on it was a challenge to create a game with such serious content.

The girls told the game designers their stories and about the research that they had been
doing. The most prevalent thing that came from the discussion was a need for young
women to recognise what an unhealthy relationship actually is, hence the name, Knowing
the Signs. The game puts the user in the third person, acting as if they are looking at an
abusive relationship from the outside. This is a well-known strategy when discussing
abusive relationships with young people, as they find it easier to reflect on other peoples
situations than their own.
The game gives the user a series of scenarios from which they have to choose an answer.
Some of the decisions are not necessarily the correct answer, but the game is about
building confidence and peace levels. For example; He tells you not to wear a dress the
options would be: wear it anyway/ tell him that hes upset you but change anyway/
immediately change. The first option would build your confidence levels but lessen your
peace levels. The more confidence you build during the game, the easier it becomes to
leave the relationship.
6 Challenges of working with this issue
Two key issues became apparent throughout the developmental stages of the digital tools:
- What if the abusive partner found it and used it as a means to create additional power,
for example if it was a diary app to write about your experiences?
- Often, when youre in an abusive relationship you dont realise it.
Both the emoti-meter and the game address these issues. If the abusive partner was to find
the emoti-meter it would be easily explainable, the user is just monitoring their emotions
rather than it being directly about abuse. Therefore, it is about the emotional wellbeing of
the girl and educating her to start thinking about the patterns of her emotions. For
example, if she was counting up high numbers of anger then the website would suggest
reasoning and help. The game addresses the issue about the realisation of being in an
abusive relationship and understanding the complexities of what it is like to be in one.

44

7 Benefits and impact of the process

Sharing stories was a central mechanism through which GMH engaged with the expertise
of the young women involved in the project. Though GMH was expressly not attempting
to create a therapeutic space, there was significant hope that the sharing of stories might
operate as a form of praxis through which the collective could begin to think
transformatively about relationship violence in the context of their own lives as well as
considering the wider societal issue. GMH recognised the potential of group work as a
tertiary prevention through which those who have lived with partner violence could
reduce harm by overcoming and make sense of their experiences 13 using skilled
facilitation and mutual learning through sharing with other women of similar
experience14.
The space for individual stories to emerge was within the context of practical
skills/learning orientated workshops, envisaged as a shared and dialogical experience. In
terms of listening and being listened to, some of the young women reported a process of
transformation through which being trusted as a listener imbued them with the
confidence that they would be listened to in a non-judgemental manner, for example:
I kind of spoke a lot with confidence [at GMH] and like normally I dont. And I dont really
speak out a lot like I have no confidence so I kind of hide away in the corner and stuff but
in being there it kind of gave me a sort of voice that I wasnt kind of scared to say what I felt
and I was quite happy to be open and honest and like it kind of felt like when youre
listening it felt like other people had like you had other people had confidence in you to
hear what theyve gone through.
What is indicated in this quote is how young women involved in GMH have experienced
personal development; as their confidence has increased so has their sense of their own
potential to make positive change within their own lives and the lives of others in their
community. They have also developed greater understanding of relationships,
criminology, the law and everyday sexism. Furthermore, they have developed an
understanding of how these things relate to the media and so they have become
increasingly aware of how they might utilise the very skills they have developed during
GMH, in, for example, photography, in ways that are counter to the pervasive
normalisation of domestic violence through everyday sexism.
8 Conclusions
GMH was a unique and ambitious project that employed an experimental and challenging
methodology to collaboratively develop understanding of the normalisation of teenage
partner violence and the ways in which digital tools might intersect in such phenomena.
KWMC continues to work collaboratively with the young women involved in the project
and is currently exploring additional funding opportunities to further develop the digital
tools designed during the project.
The young women involved are keen that the project and their involvement is expanded
upon, so that the two products can be produced, but also so that young men can also
explore some of the societal pressures there are around masculinity, in order to address
and counter those pressures that young men may feel encourage them to be perpetrators
of domestic violence. Young women involved are also keen to co-productively work with

13

Mullender, A. and Hague, G. (2000) Reducing domestic violence ... What works? Women Survivors Views.
Home Office Briefing Note, London: Home Office.
14
Hester, M. & Westmarland, N. (2005). Tackling domestic violence: effective interventions and approaches.
Home Office Research, Development and Statistics Directorate.

45

other services and initiatives across the city, to make the most of and share the learning
they have experienced to date.
Copyright: Knowle West Media Centre/Law School, University of Bristol. Copyright images:
Knowle West Media Centre!
Visit: http://kwmc.org.uk/projects/girlsmakinghistory/

46

Smart City Living Lab

City as a place
Bergstrm, Maija a
a

Forum Virium Helsinki

maija.bergstrom@forumvirium.fi

Abstract
Smart cities are often discussed in the context of technology development or virtual
services. However, smart cities are still real places where people live, work and play.
Livability a target often discussed in smart city strategies is also connected to the
particular places where urban dwellers spend their every day lives. The aim of this paper is
to show how both qualitative research and art projects can be used to offer information
about the identity of the place. This information may be used as a starting point towards
placemaking in a citizen-driven Smart City Living Lab. This is demonstrated by three
cases. They showed the need for distinctiveness, re-occurring events and happenings to
produce a feeling of a place, and a narrative that stems from the history of the place. Ideas
on Smart City would develop better if there would be something concrete, something that
could be experienced, touched, seen, heard; something that would represent the Smart
City for the citizens.
Keywords
Smart city, Kalasatama, Placemaking
1 Introduction
1.1 Placemaking in a Smart City
A smart city is a different kind of city than the one we consider normal: adaptive and
sustainable, open and democratic, creative and well educated and directed by real-time
information. Three factors can be found in definitions of a smart city: technology factors,
institutional factors and human factors (Nam & Pardo, 2011). Nam & Pardo suggest that a
rather technology-centered definition still prevails, but more emphasis on the human
factors would be needed: a socio-technical view of a smart city.
The active development of an area the whole city or just a part of it to a certain
direction can be further elaborated in the terms of placemaking. The starting point for this
47

should be the understanding of the current state of the place, its sense of place which
means how the place can be identified, how it stands out from its surroundings (Auburn &
Barnes, 2006). The disctinctiveness means more than just the physical surroundings,
including also the narratives and mythologies linked to a place (Wortham-Galvin, 2008).
Smart cities are often discussed in the context of technology development or virtual
services. Smartness may reside in the cloud. However, smart cities are still real places
where people live, work and play. Livability a target often discussed in smart city
strategies is also connected to the particular places where urban dwellers spend their
every day lives. Further, smart city development offers new possibilities to placemaking:
e.g. new layers to the experience via augmented reality solutions or possibilities to add
changing digital tags, sensors and data to selected locations. Thus, smart cities are
interesting also in the context of placemaking and offer new possibilities to create
interesting, multifunctional, shared spaces.
When talking about the sense of place, or the identity of a place, we are referring to
peoples shared ideas of this place. From a social psychological perspective these shared
ideas can be understood as its social representations, a concept used for researching
various phenomena, developed by Serge Moscovici (1961/2008). The place itself has no
power on defining itself, but these definitions come from the outside, the people and the
media. This idea of a place stems from its past (deAlba, 2011) and stretches towards the
future. With the use of the word identity, the personality and uniqueness of the place is
emphasised. As each person has their own, distinctive understanding of a place due to
their personal history, values and attitudes, the challenge is to capture this diversity, but
try to classify and arrange it in a meaningful and informative manner.
Here, three cases on the place identity of Kalasatama are described and then discussed
further in the context of placemaking in a smart city. The first case is an empirical research
project (Bergstrm, 2015), the next ones are art projects of Eskus, the center for performing
arts running a community arts project Kalasataman taidetalkoot15.
The aim of this paper is to show how both qualitative research and art projects can be used
to gain information about the identity of the place, and how this information may be used
as a starting point towards placemaking in a citizen-driven Smart City Living Lab.
2 Smart Kalasatama project
2.1 Kalasatama as a place
Kalasatama is a new, developing inner city district in Helsinki, Finland. The construction
work in Kalasatama started after the old port moved away from the area in 2008, and the
first inhabitants moved to the area in 2012. The work will continue until the 2030s. A total
of 20,000 inhabitants will reside in the area and 8,000 jobs are expected.
Kalasatama is a place undergoing a massive regeneration. It has an industrial and
maritime history as a port, surrounded by wholesale markets and marketplace to sell the
imported goods and a gas plant to produce energy a protected, historical building of
Suvilahti that was recently turned into a cultural center. After the plans emerged to build a
new residential area in Kalasatama, and the old port moved away in 2008, the area came to
be known from its opportunities for temporary uses. The city officials accepted festivals,
graffiti fences and urban gardening to take place in Kalasatama and Suvilahti as a strategy
for developing the place through culture (Kriv, 2013). In Kalasatama, the highest buildings
in Helsinki insofar will be built. The topic awoke some controversy and the process was

15

In both of the art projects presented here, the author of this paper has been in collaboration with
Kalasataman taidetalkoot -community art project of Center for Performing Arts (Esitystaiteen keskus).
48

delayed because of complaints to the court. Now the construction work has started and
shopping center REDI with its first two towers should be completed in 2018.
2.2 Smart Kalasatama
The aim of Smart Kalasatama project is to co-develop Kalasatama as a smart city with the
residents and other actors in the area. To drive this process forward, Kalasatama is used as
a Living Lab where new smart services can be co-created and tested in the real
environment. For this work to have a firm base, the residents are invited to participate in
the process. A way to start the co-creation is to work together and define the local identity.
Smart Kalasatama project grows from the collaboration between the City of Helsinki,
companies and residents. The aim is to inspire residential participation and create new
business and innovations. The Smart Kalasatama is a development environment that
brings together the research, development and innovation activities of businesses, and the
development of city services. Companies and the city bring their projects to the mix; the
ideas then are being developed, tested and composed together with the inhabitants and
those working in the area.
Smart Kalasatama is a 6Aika spearhead project of the City of Helsinki, coordinated by
Forum Virium Helsinki. It is part of the Six City Strategy Open and smart services
carried out by the largest cities in Finland (Helsinki, Espoo, Vantaa, Tampere, Oulu and
Turku). In 20152017 Smart Kalasatama is funded by Six City Strategy (ERDF/HelsinkiUusimaa Regional Council), the City of Helsinki and the Finnish Ministry for Employment
and the Economy. The budget for three years is approximately 900 000 euros.
2.3 Kalasatama as a living lab
Kalasatama forms a geography based Living Lab for the Smart Kalasatama project. The
area houses approximately 2000 residents in 2015. A collaborative working space will be
opened in autumn 2015 to support different development activities such as workshops,
hackathons and seminars. Kalasatama Developers Club will be launched to support the
smart development of the area and further support connections between different actors
on the area. Minor financial supports will be available for entrepreneurs to support product
development and agile testing.
Westerlund and Leminen (2011), following the view of Ballon et al. (2005) define living labs
as physical regions or virtual realities where stakeholders form public-private-people
partnerships (4Ps). 4P-model is an attempt to move from dual-partnership of public and
private, towards a more democratic multi-stakeholder companionship where the resident
and the customer are at the heart of development activities (Anttiroiko, 2010). This also
supports well the beforementioned need to shape the definition of smart cities into a more
human-centered one. Weeckman et al. (2013) have emphasized the need to establish ways
for community interaction and stakeholder engagement in order to succed in
implementing living lab projects and create user-centerd innovation.
The role of the Smart Kalasatama project in the living lab could be defined as what
Westerlund & Leminen (2011) call enabler, or orchestrator, organizing the network
meetings and enhancing collaboration as well as offering collaborative working space and
methodological knowledge to support the living lab activities. The project also has to
establish ways for effectively communicate and engage its networks: the users and other
stakeholders.
3 Case studies

49

Case 1. Researching the shared ideas on Kalasatama

Three different kind of data sets were collected and analysed in the study (Bergstrm,
2015):
a) public documents on communication between the city and residents and other
stakeholder
b) hand drawn mental maps of Kalasatama by the citizens of Helsinki
c) word association lists related to trigger words Kalasatama, Suvilahti and High
buildings in Helsinki from the same citizens.
The public documents were analysed according to methods suggested by discursive
psychologists (e.g. Potter & Wetherell, 1987; Billig, 1996/1987; Juhila 1993). Especially, the
controversies and the logic of argumentation were looked at closely. The mental maps
were analysed for their spatial dimensions as well as for the symbols and contents in a
similar manner as in Milgram and Jodelet (1976) study on psychological maps of Paris.
Word associations were classified and analysed for their positive, negative or neutral
quality, as indicated by the respondents.
The results show that the way in which Kalasatama was represented in the ideas of citizens
was not very diverse or specific, according to descriptions given by the respondents for the
word associations (Bergstrm, 2015). The views of Kalasatama were very consistent, and
were defined by proximity to the sea, the subway, and novelty. On the other hand,
Suvilahti, the old industrial area near Kalasatama, now a cultural center, which is
considered as a part of Kalasatama, triggered a more vivid set of descriptions: summer,
parties, music, a certain style of living. This could be seen as a result of recurring events
and happenings that have shaped the representation and formed an identity for the area.
This was further indicated by the numerous amounts of names of places associated with
Suvilahti. Even places (cafe, event arenas) that were situated further away and so could
have been associated with broader Kalasatama area, were associated with Suvilahti. This
seems to stem from the way the representation of Suvilahti is strongly anchored and
likened to the category of culture.
When the core of the representation has formed, it will guide further notions about the
phenomena (Abric, 2001). The visual dimension of Suvilahti, with its gasholder, graffiti,
and the skatepark, support the idea of modern urbanism, which draws on nostalgia for the
industrial era and the freedom of action in urban space.
Case 2. KA:LAB installation and living room
KA:LAB was a two-week-long temporary living room and art installation. Construction
and development company NCC offered an empty office space for the project. During
these two weeks, a collaboratively made installation grew in KA:LAB, shaped by visiting
groups and individuals.
In KA:LAB video material from interviews with the residents were shown, and in art
workshops different groups created art to be shown at the installation. Local people
brought baked goods to be served in the living rooms cafe, and the local communities held
their meetings in KA:LAB during the two weeks. The installation was a collaborative
process, and the use of the space was part of the artwork. By using a space that would
otherwise be quiet and empty, the people of Kalasatama expressed their wish for a lively,
active city.

50

Part of the installation, a map of Kalasatama built by visitors, is now in possession of

Helsinki City museum and will be shown again on 2016. The map was one attempt to
archive the fleeting and changing ideas of a place that is constantly and so dramatically
changing. The installation differed from many previous temporary art projects in that it
was held in the middle of the new residential area, not in Suvilahti cultural center or its
surroundings.
Case 3. Kalasatama map service -community art project
During KA:LAB, the art project held its first workshop, where the working methods were
tested, aiming at producing thematic routes. By walking these routes a newcomer could
explore Kalasatama like a tourist, seeing local specialties and learning about the history of
people who live or used to live there.
The workshops were partially based on previous study (Bergstrm, 2015), but the data
collection methods were modified so that it could be collected in different workshops. The
following events were held during spring 2015: a workshop table at a flea market in
Suvilahti, three workshops with teenagers at the Centre for Performing arts and two
workshops with elderly people in a senior house. The selection of people for the
workshops, were guided by the need to easily reach groups that would be interested in
working with the project. At the flea market a very wide spectrum of people attended, the
teenagers were attending as a part of school program and the seniors were attending as a
part of recreational program in their senior house. Approximately 150-200 people
altogether attended these events.
The drive-in workshop at the flea market was aimed at producing a large amount of
material that could be further developed and worked with at the following events. The
people were asked what words they associated with Kalasatama (see blue notes on a map
in Picture 1.), and which places they think would be worth seeing in Kalasatama (yellow
notes in Picture 1.). We also asked people to pick up one of the photos of Kalasatama that
we have selected, and write the reason why they found it interesting on the other side of
the photo. Most of the people that attended the workshop didnt reside in Kalasatama, so
this way we produced an outsiders perspective to the area.

51

Figure 1. Map of Kalasatama produced in workshops. Photo: Anastasia Artemeva

Figure 2. The old gas plant of Suvilahti in Kalasatama. Photo: Wikimedia Commons, Leena Tarjamo

The gasholder and old industrial building of Suvilahti (in Picture 2.) awoke vivid images in
the participants:
In my imagination, this has been the abandoned amusement park. When Im walking past, I hear
the wagons of the rollercoaster rattling.
52

Unusual building and its history.

Ive never known what this building is. Dont know it yet It makes me think of the Wall of
death, the motordrome in a circus.
The map was used in a workshop for the teenagers as a means of understanding other
peoples ideas on Kalasatama. After that they produced their own performances to the
urban space.
With the seniors of Kalasatama we worked the material further: asking them to find
something that they agreed or disagreed with. We collected truths about Kalasatama and
then found meaningful places from the area.
At the workshops we produced mind maps of the history, present and the future of
Kalasatama with the seniors. They connected Kalasatamas history to its maritime past,
the boats, the trade and sailors. They also made a connection to a nearby residential areas
of Srninen and Kallio, recently gentrified inner city areas, that are known from their
notorious past: bars, violence and prostitution. The present state of Kalasatama is
connected to urban culture, represented especially by Suvilahti area. The residential area
is not very well known by the wider citizen community. The residents often face questions,
if the area is only for the rich people, despite of the fact that most of the participants didnt
count themselves as being especially wealthy, but very ordinary citizens. The seniors saw a
very optimistic vision of a metropolitan, urban future ahead, but some uncertainty
prevailed of how to get there.
In these workshops we found interesting notions on how Kalasatama is seen among the
residents and the outsiders. There are two different perceptions to the area: a wider one,
that includes also parts of the surrounding areas that are seen linked to Kalasatama. This
view comes close to the results of Case 1. The narrower one, that includes mostly the new
residential area, was more popular among the residents. This narrower conception might
be further emphasised by the limited transportation opportunities and few walking routes
that the residents must use every time they need to leave the new residential area. The
residents also are aware of the subareas that altogether form the broader Kalasatama
area. The names of subareas were quite seldom mentioned among the non-residents that
attended the workshops.
4 Discussion
4.1 Place identity of Kalasatama
The three cases demonstrate means to produce descriptions of the area. This could serve
as a base for the placemaking. The first case, a research project (Bergstrm, 2015), consisted
of an empirical analysis of different data sets: public documents, hand-drawn maps and
word associations. The level of analysis was a collective one: in the discourses and the
visual material, certain descriptions of the place were widely shared by the participants.
Together the analysis of the data offered a rich description of Kalasatama, and supported
the idea of a place shared by human geographers such as Tuan (1977/2007, 171-172) who
emphasizes the meaning of distinctiveness, history and significant events, and Friedmann
(2010) who suggests that small scale places that serve as arenas for recurring event and
ritual produce the feeling and experience of a place.
The two latter cases were art projects, although the Kalasatama map service used
partially similar methods for collecting material, as was used in the research of Bergstrm
(2015). The art projects are not aimed to produce knowledge per se in a similar manner as
empirical research does, but rather reflect the feelings and ideas of people together with
53

them. KA:LAB installation and living room aimed at producing a shared space for
citizens to use: making a place in an empty space. The installation was a joint production
of the residents mediated by the artists.
4.2 Engaging citizens in placemaking
The act of producing a vision of a place is a powerful one. A collective vision was produced
in the latter art project too, the Kalasatama map service. It worked the individual notions
and experiences forward, by grouping the material thematically, pulling together typical
or similar ideas together with the citizens. This is very similar to the analysis of qualitative
data, where the researcher moves back and forth between the data and the theory (see e.g.
Braun & Clarke, 2006). Here, the participants produced their own material and at the same
time built on the previous interpretations of other participants. It could be said, that the
interpretation here was done by the participants, where in the research project it was done
by the researcher. Giving voice for the people, is one of the aims for research in social
sciences. In the arts project, the aim was to help people give the voice for themselves.
The understanding of a geographical place and the shared meanings that are given to it by
the people living in the area and other citizens are part of the operational environment of
the living lab. Veeckman et al. (2013) very relevantly emphasized the need to establish ways
for community interaction and stakeholder engagement. These ways of gaining new indepth understanding of a place that also serves as a living lab, help to set a firm base to
open, user-centered innovation. It helps to direct and focus the user-participation to areas
of development that are most meaningful and most relevant to them. It also helps defining
which notions should be given more attention when communicating the living lab
activities for the users and residents in an interesting manner. In this paper I have
suggested some possible methods for gaining this understanding. As Veeckman et al. (2013)
have suggested, the interplay between the characteristics of the living lab environment,
the living lab approach, and the innovation outcome could and should be analyzed in a
more detailed way after these first steps towards a truely open, user-centered innovation
platform.
When developing projects like Smart Kalasatama as a truly open innovation platform,
understanding the shared ideas and representations of the people should be followed by
well-planned possibilities to take part in living lab activities. Living labs should be a step
towards a more democratic multi-stakeholder 4P-model (on 4P-model see e.g. Anttiroiko,
2010). Hence, their true purpose will not be fulfilled if the participation stays only on the
level of consulting and gaining understanding peoples ideas.
5 Conclusions
Placemaking is an active, collaborative process. Linda Rutherford wrote on a blog post in
Project for Public Spaces (May 13, 2014) about placemaking: I like to think of it as
crowdsourcing meets urban and community planning. This comparison is a very good
one. Placemaking could be understood as a goal-directed journey from the starting point
towards a new vision, done together by the community; public authorities, charities, or an
organised group of citizens, in collaboration with businesses and other organisations.
Making a place means that there must be distinctiveness, re-occurring events and
happenings and a narrative that stems from the history of the place. Ideas on Smart City
would develop better if there would be something concrete, something that could be
experienced, touched, seen, heard; something that would represent the Smart City for the
citizens. As our next step in the Smart Kalasatama project we are opening a collaborative
working space for the citizens. The understanding of the place that these cases offer, help
54

us to understand what might be the problems that should be addressed, and what are the
strengths of the area. It also helps to produce a narrative, a story for Kalasatama as a smart,
new and developing place.
6 References
Abric, J.-C. (2001). A structural approach to social representations. Teoksessa K. Deaux &
G. Philogne (Eds.), Representations of the social. Bridging theoretical traditions (s. 42-47).
Oxford, UK: Blackwell.
Auburn, T. and Barnes, R. (2006), Producing place: A neo-Schutzian perspective on the
'psychology of place' , Geographical Review, 26(1), 38-50.
Bergstrm, Maija (2015). Kalasataman rakentaminen sanoista ja sementist. Tutkimus uuden
alueen konstruoimisesta ja sosiaalista representaatioista. Masters thesis, University of Helsinki.
Billig, M. (1996). Arguing and thinking. Cambridge: Press Syndicate of the University of
Cambridge (originally published in 1987).
Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77-101.
De Alba, M. (2011) Social representations of urban spaces: a comment on mental maps of
Paris. Papers on Social Representations, 20, 29.1-29.14.
Friedmann, J. (2010). Place and Place-Making in Cities: A Global Perspective. Planning
Theory & Practice, 11(2), 149-165.
Juhila, K. (1993). Miten tarinasta tulee tosi? Faktuaalistamisstrategiat viranomaispuheessa.
In Jokinen, A., Juhila, K. & Suoninen, E. (ed.) Diskurssianalyysin aakkoset. Tampere:
Vastapaino.
Kriv, M. (2013). Don't Plan! The Use of the Notion of 'Culture' in Transforming Obsolete
Industrial Space. International Journal Of Urban & Regional Research, 37(5), 1724-1746.
Milgram, Stanley & Jodelet, Denise (1976). Psychological maps of Paris. In Harold
Proshansky, William Ittelson & Leanne Rivlin (ed.), Environmental psychology: People and
their physical settings, 104-124. New York: Holt Rinehart and Winston.
Nam, T. & Pardo, T. A. (2011) Conceptualizing Smart City with Dimensions of Technology,
People, and Institutions. The Proceedings of the 12th Annual International Conference on Digital
Government
Research,
282-291.
Available
at:
demo.ctg.albany.edu/publications/journals/dgo_2011_smartcity/dgo_2011_smartcity.pdf
Potter, J. & Wetherell, M. (1987). Discourse and Social Psychology. Beyond Attitudes and
Behaviour. London: Sage Publications.
Rutherford, Linda (2014). Blog post, May 13, 2014: Project for Public Spaces. Why public
places are the key to transforming our communities. Available at:
http://www.pps.org/blog/why-public-places-are-the-key-to-transforming-ourcommunities/
Tuan, Y. (2007). Space and Place: The perspective of experience. Minneapolis, MN: University
of Minnesota Press (originally published in 1977).
Wortham-Galvin, B. D. (2008). Mythologies of Placemaking. Places, 20(1). 32-39.

55

Living Lab theory and tools

Session II

56

The wearable Living Lab:

how wearables could support
Living Lab projects
Tanguy Coenen a, Lynn Coorevits b & Bram Lievens a
a

iLab.o, iMinds-SMIT, Vrije Universiteit Brussel

b
iMinds-MICT, Universiteit Gent
tanguy.coenen@vub.ac.be

Abstract
Part of the added value of living lab or other in-situ research resides in its capability to
capture contextual and personal data of end-users in real life environments. In this paper,
we present an architecture for the wearable living lab (WELLS), allowing the collection of
user experience feedback in a more permanent and unobtrusive way. It enables mainly the
user researcher to capture contextual and experience data during a Proxy technology
assessment, a formative or a summative software evaluation.
Keywords
Wearables, living lab, evaluation
1 Introduction
Wearables provide intrinsically different opportunities for the research process of several
application domains ranging from health to consumer research. We perceive wearables as
any computing device that can be worn on the body (from smart-textiles to smartbracelets, -jewelry and watches). They enable the monitoring, tracking and controlling of
several human activity types. A distinguishing feature is that, unlike other devices such as
smartphones, wearables are often worn almost permanently and therefore allow the
continued and longitudinal capturing of data. The market for wearables is looking
promising and wearable computing devices are increasingly hitting the market, like smart
and sport watches (e.g. Pebble, Apple Watch,...), fitness trackers (e.g. Fitbit, Jawbone
UP3,...) or smart glasses (e.g. Google Glass). Consumer-grade wearables allow the
measurement of different types of user data in a less obtrusive and often more objective
way than current, mainstream living lab data collection methods , i.e. observation
techniques or surveys. Also, software development is more and more taking place in the
context of mobile devices, where the interactions that the user has with the system are
mediated by the context in which the use takes place (at work, at home, in the car, in
bed,...). The assessment of such systems requires an in-situ or in the wild evaluation.
Current application developers and researchers try to evaluate the software product by
logging the actions of the users. However, logging data on itself has its limitations as it
cannot track all contextual elements nor the individuals personal experiences related to
the usage of the application. Therefore, (additional) experience sampling is used to capture
57

in the moment experiences of users. Both techniques allow to track users in the wild, but
requires specific research methods that afford the continuous evaluation of the users
response to the system.
Katz (2001) concluded that there is an invisible side of emotions that cannot be induced
from observations of user behavior alone. One way to measure emotional fluctuations is
by researching their physiological changes via skin conductance in order to understand
arousal. Due to the specific set-up and technology required, such measurements were only
possible in a dedicated physical location. Because of the ongoing commercialization of
wearables allowing measurement of behavior as well as psychological changes, wearables
provide research opportunities in living lab environments. To allow this, the challenge
which was already addressed in e.g. the work of Kocielnik et al (2013) or Sano & Picard
(2013), to make such measurement work outside of controlled lab conditions, must be
further taken up for application in living lab conditions. In other words, we must ask how
these measurements can be made in the in-situ evaluation conditions that distinguish
living lab projects from other design approaches.
In this position paper, we discuss how wearables can be used in living lab projects (and by
extension other types research with an in-situ component) to perform novel types of
systems evaluation. To do this, we first discuss the process structure of living lab projects,
which informs us on what living lab stages can benefit from unobtrusive systems
evaluation. Then, we proceed to discuss how this can be done with the means that are
currently or in the near future on the consumer market. We focus on consumer-grade
devices, as it is important that the user of the device become as accustomed to it as
possible, in order not to bias the data (Henandez et al 2014). Consumer-grade devices are
designed to accomplish exactly such accustomedness. Another reason for focusing on
consumer-grade wearables is that evaluations in living labs are often large-scale. This can
only be achieved if the wearables to be used are already owned by the user, or because
their cost of acquisition is low. In sum, this paper discusses how the need for in-situ
evaluations that exist in living labs can be met by the opportunity of new wearables
becoming available as consumer devices.
2 Prototype evaluation in Living Lab projects
Living lab-research is a state-of-the-art methodology aiming at the involvement of endusers in the innovation process. Although there are several ways of integrating consumers
in the innovation process, living labs are interesting because of the context for
understanding customer co-creation. Living labs are experimental platforms where endusers can be studied in their everyday context (Eriksson et al. 2005). They confront
(potential) users with (prototypes or demonstrators of) products and/or services in the
innovation process (Schuurman & De Marez 2012). As a distinguishing factor of living labs
is that the systems they produce are tested in non-laboratory conditions, but in the wild
(Westerlund & Leminen 2011, Ballon 2015). By introducing this in a real life environment,
experience based learning and discovery (facilitating serendipity) is made possible.
Through real-time iterations the design and development actions are constantly being
validated. This will help developers to make more informed decisions, thus increasing the
likelihood of success (Trimi & Berbegal-Mirabent, 2012).
Various types of process models for living labs have been proposed in literature (e.g.
Pierson & Lievens 2005, Schaffers et al 2008, Tang & Hmlinen 2012, Bergvall-Kreborn
et. al 2009). Recently, Coenen & Robijt (2015) proposed the Framework for Agile Living
Labs (FALL) process model as a way to guide both researchers and practitioners in how to
perform a living lab project. This process model was shown to be compatible with most of
the existing process model literature.
58

In the FALL approach to a living lab project, there are 2 stages in which user feedback can
be of use, i.e. the problem formulation and the so-called BIEL phase. In the problem
formulation phase, users can be brought in contact with existing technologies that are
configured to mimic the behaviour of the prototype that the project team has in mind. This
is done through a Proxy Technology Assessment (PTA) (Pierson et al 2006). In the iterative
build, intervene, evaluate and learn (BIEL) phase, two different types of evaluations can be
performed to get user feedback on a prototype: formative or summative evaluation. In
formative evaluation, the prototype is of a very provisional nature and the aim is to get
feedback that allows the living lab team to create better prototypes after it. In summative
evaluation, the objective is to evaluate a more stable prototype and find out how effective
and efficient it is to be able to report this to some project stakeholder (client who
commissioned the project, customers, societal actors,). These three types of prototype
evaluation are the domains where we see potential in living lab projects for the application
of research methods that are supported by wearables.
3 WELLS or how wearables can be used to support prototype evaluation in living lab
projects
Based on our experiences in conducting real life evaluations of prototypes in living labs, as
well as the literature on wearables, usability research and affective computing, we propose
the architecture for a Wearable Living Lab Software Evaluation System (WELLS) depicted
in Figure 2. Various roles are described as part of FALL, of which the user researcher is the
role that is involved in carrying out the evaluation of prototypes. It is therefore the user
researcher that will be the main consumer of the data generated by WELLS. Taking into
account the opportunities of the wearable devices and the research needs for in-situ,
contextual user-centered living lab research we focus on two main types of data-sets
collected through wearables: bio-data and context data.

Figure 1: Architecture for a Wearable Living Lab Software Evaluation System (WELLS)

59

4 Bio-data
In our choice of sensors for measuring bio-data to be included at the core of the WELLS
system, we concentrate on sensors that can provide a measure of the arousal experienced
by the user. Arousal is a general term describing an emotional response by the user. The
valence of the response can be either positive or negative representing emotions such as
excitement and frustration. We interpret arousal according to the two factor theory of
Schachter Singer (1962), assuming that arousal is the product of physiological arousal and
the cognitive processes that respond to a particular situation(=context and stimuli)
provoking the emotion. The cognition of the user will determine whether the physiological
arousal will be perceived as anger or happiness. As sensors can detect arousal but not the
valence of the emotion, the WELLS model also adds contextual data in order to enrich and
by so interpret the captured bio data better. Measuring arousal will detect positive or
negative experiences and therefore identifying moments and actions where a prototype
can be improved on.
Various physiological measures can be used to indicate arousal, among which blood
pressure, heart rate, heart rate variability (HRV), electrodermal activity (EDA), pupil
diameter (Sano & Picard 2013) as well as facial expression (Albert & Tullis 2013). However,
not all of these measures can be collected using unobtrusive devices. For example the
measurement of pupil dilation requires the constant training of a camera on the subjects
pupil. By investigating current hardware offerings, we found that what can currently be
measured in the least obtrusive way are electrodermal activity and heart rate.
4.1 Electrodermal activity
Electrodermal activity is a measure of sweat excreted by the eccrine glands, which are
innervated by the sympathetic nervous system. (Hedman et al 2012). Electrodermal
activity has been used widely to measure the response of the sympathetic nervous system
to events in our environment. In the past, devices that were used to do this were often
cumbersome to the wearer, resulting in feelings of stress resulting from wearing the device
itself. Recently, devices like Philips Discrete Tension Indications (DTI-2) or the Empatica
E4 Wristband have become available that allow EDA measurement by wearing a relatively
unobtrusive, watch-like device. The latest in wearable fitness trackers, like the Jawbone
UP3 wristband allows to measure electrodermal activity through the use of a device type
that is becoming more and more mainstream on the consumer market. These mainstream
devices will reflect a natural setting for participants and will allow Living Lab researchers
to maintain a real life research setting.
4.2 Heart rate
Heart rate and heart-rate variability, or the fluctuations in beat-to-beat intervals, can be
used to detect stress and arousal (Albert & Tullis 2013, Choi et al 2012). There are more
devices that can measure heart rate than EDA, making it a pragmatically useful addition to
the WELLS architecture. Indeed, heart rate can be measured by mainstream fitness
wearables by wearing a sensor band on the torso. However, wearing such a band is
obtrusive and few are the people who would want to wear it permanently, beyond their
fitness workout. But as for EDA, new devices are becoming available that can be worn on
the wrist and that also measure heart rate.

60

5 Context data
5.1 Video and audio
Video can be used to capture the interactions of the user in a rich way and has been proven
to be highly valuable to detect possible improvements of the user experience. Within
usability research, video has been used in various way to capture and detect user
interactions and experience. However, this has either been in a controlled lab environment
or with very obtrusive equipment when performing this in-situ. For a pervasive game
called Playground, we collected user data with smart glasses, as they are able to capture
the users context when using the system in a rather easy and less obtrusive way. Players
were asked to play the game on smartphone, while wearing these smart glasses and
performing the think aloud protocol. The latter is a standard method in which participants
thinking aloud as they are performing a specified amount of tasks. Because the data is
recorded, developers can look back at the experiences and reactions of the users and
interpret the data. As pervasive games are not only about the use of the game on a mobile
device, but also about the experience of the player with the physical environment in which
the game is set. By using smart glasses, both dimensions (use of device and interaction with
physical context) could be captured in rich detail.
Our experience within the Playground project indicated that the use of smart glasses is
promising but still a number of issues need to be tackled. First, they are less unobtrusive
than expected. For example, Google Glass constantly shows a display in ones field of
vision. It is therefore hard for the user to forget the presence of the recording device. In
addition, battery life can be a hurdle in smart glasses. The battery life time of Google Glass
is limited (one hour of video recording), making long experience sessions difficult. In
addition, there were issues with the video framing. As the camera in the Google glass is
located in the top-right corner of the glass frame, the recorded video did not always catch
the interface on the smartphone. Finally, often the video was not focused on the interface,
but on other objects in the field of view. This resulted in blurry recordings of the interface,
making it hard to identify certain interface elements.
An alternative to using smart glasses is devices that are used for LifeLogging or the longterm recording of video in daily life. One example is the Narrative clip 2, which allows up
to 30 hours of video recording through a device that can be clipped on clothing and is
around 10 by 10 cm large. However, many of these devices do not record audio. Our
experience within Playground learned that audio is often essential as extra information
layer to allow users to express their frustration and provide additional context on what it is
that they are experiencing.
5.2 Experience sampling
Experience sampling is a research methodology in which users are asked to self-report on
various elements (such as experience, context,) on specific moments (this could be timebased, event-based,). Huang et al (2014) and Sano & Picard (2013) e.g. used experience
sampling, via short questionnaires distributed through a smartphone app, to query the
user on their emotional state of mind. In their research they demonstrated that gathering
survey data can be of great use in identifying emotions such as stress related to a specific
context. The captured data also linked with a whole batch of collected metrics like EDA,
heart rate and others. However, in systems evaluations in living labs, taking place in reallife, naturalistic environments, the identification of stress needs to take place within a brief
time interval from the event that caused the affective response. This needs to be done to be
able to identify instants that caused an affective response, making aggregate experience
61

sampling per day ineffective as retrospective self-reporting will not reveal these specific
moments. Therefore, more fine-grained approaches need to be included to measure affect.
Two options are possible: (1) based upon real-time data of the wearables in which the
reporting is triggered by certain data-points. But this would require a permanent online
connection and could also results in too many interactions and request. (2) based upon
predefined intervals in which the user is asked to evaluate her aggregate affect level in a
previous time interval. Such an approach would not produce labels on the exact moments
when stressful events took place, but being able to search down the search space would be
useful in analysing the large amounts of data that will be produced by a large-scale
evaluation using the WELLS infrastructure.
5.3 Location, event logs and time
Keeping track of location is important to be able to gather context data. Such location data
can come from geolocation based on GPS, wifi or bluetooth data as is often done in mobile,
geolocative apps. A well-known drawback of such apps is the rate at which they drain a
smartphones battery. Situations where location is less accurate but more energy efficient
can be built based on iBeacon solutions. In such cases, the captured data only shows in
what general area the user resides. An infrastructure cost is associated with such a
solution, as the area in which the user interaction takes place needs to be fitted with
iBeacons. Next to location, movement is also an important contextual element, which can
easily be determined due to the accelerometers within smartphones and wearables. This
data stream is also important, as it can determine if a user is moving or not. Keeping event
log data is another important aspect of evaluating data during in the wild trials. Such log
data can indicate bugs and can provide useful context information to be able to further
make sense of the users behavior. Finally, keeping accurate time is essential
for synchronizing data in the research interface when data is coming from multiple
sensors (Banaee et al 2013). Therefore, having timestamps that are in sync for all
measurements is critical.
6 Smartphone
The smartphone plays an important central role. Firstly, most wearables need a
smartphone to make connection to various (cloud-based) services. Therefore it can be used
to bundle the incoming data and send it over to the data repository in an efficient way
(stability, robustness). In the future, wearables may become directly connected to
internet, bypassing the need for smartphone. Secondly, the smartphone can help in
keeping all sensor data sets synchronized.
7 Data repository
The data repository should store the data coming from as many sources as required. It
should be able to receive real-time or batch data from the smartphone. In addition, this
data should be stored in such a way that privacy requirements are met.
It is important to keep in mind that other metrics can be of use when measuring user
affect. As more wearable sensors will become available, it will be important to make the
architecture as extensible as possible, to be able to quickly add new sensors to the
system. The data repository, and the way in which it is accessed, provides the main point
in the architecture where such future compatibility needs to be realised. This will require
flexible APIs that can accommodate data coming from a wide array of sensors as well as
an efficient identification mechanism.
62

8 Research interface
The purpose of the research interface is to allow user researchers to make sense of the data
that is produced by the sensors and the smartphone. Especially relevant are moments in
the users experience that have generated arousal while using the application being
evaluated. In order to support sense making by user researchers, the WELLS data should
be visualised, mined and made queryable.

Figure 3: Wearable Living Lab research interface mockup, adapted from Kocielnic et al (2013). Galvanic Skin
Respons (GSR) is synonym to eletrodermal activity (EDA). Foto is a still from a Google Glass recording of a
user session in Playground Kortrijk

8.1 Data mining

The amount of data produced by sensors over a longer period of time can be very large. In
order to make sense of such sensor data, which comes in the form of a time-series,
analytical methods are necessary. Banaee et al (2013) provide a survey of the methods used
in data mining and machine learning in the area of medical applications, that is relevant to
the aims of this paper. In their literature review, they found support vector machines,
decision trees, neural networks, hidden markov models, gaussian mixture models and
rule-based models to be in use.
It is important to distinguish EDA and HRV fluctuations resulting from movement from
electrodermal activity resulting from emotional response. This can be achieved by
applying rule-based filtering, one of the main data mining approaches discussed by
Banaee et al (2013). The data from the 3-axis accelerometer is crucial in distinguishing both
types of EDA and HRV response. Rules can be built that disregard EDA data from periods
in which the user was moving along one of the 3 axis in a value that exceeds a certain
threshold.
The main objective in the analysis of data coming from user experience evaluation in
living labs should be on finding instants that coincide with increased arousal from users.
These instants can then be labeled as possible candidates for further analysis. By
juxtaposing data that has been labeled in such a way from EDA or HRV with video, audio
and location data, a rich picture will emerge of what features and contextual
circumstances can be stressful or enjoyable to the user.

63

8.2 Visualisation
In order to be useful, the system needs to visualize the data in a way that facilitates sense
making. The video and audio data will be the main way in which the user researcher will
come to understand what exactly was going on at a particular point in time. However, the
video and audio data of a certain evaluation can be extensive and if the trials are largescale, user researchers can not be expected to go over all the produced video and audio for
each participant. Therefore, combining the video/audio feed with data mining techniques
on data coming from other sources can be powerful to discover moments in the data where
the user got frustrated with using the software.
Kocielnic et al (2013, 2014) experimented with collecting work stress-related data using
EDA over a prolonged period of time. A visualisation was created that combined sensor
data from EDA and accelerometer with labels that were extracted by interfacing with
online agendas. Not only did this approach deliver an instrument for the analysis of the
data, it was also able to present the findings back to the user and in this way obtain better
self-reporting. By combining a visualisation of EDA data with calendar data and
questionnaire data, users were able to make sense of their experiences in terms of the
relative stressfulness of different events in their day-to-day professional life.
8.3 Privacy issues
Continuous tracking poses many privacy challenges that neither users nor stakeholders
are ready to deal with. Wearables constantly collect, transmit and store data that is
considered personal, private, sensitive and confidential by users. This brings many
benefits to the researchers, as it can provide them with access to end-users latent
experiences and fill the gap between the saying versus doing paradigms. Yet for users this
constant surveillance and sousveillance can lead to privacy threats and risks. Sousveillance
is the recording of an activity by a participant in the activity typically via small wearables
or other small portable technologies (Mann, Nolan & Welmann, 2003). Users are currently
less aware of these privacy concerns because wearables are a relatively new thing. In
addition, wearables are often shaped as accessories that never posed any privacy threats to
users such as glasses, watches, etc. Motti & Caine (2014) identified several privacy concerns
from users about wearables related to the device and the application itself, the data being
captured and the sensors. Especially their last two concerns are of importance to living lab
research. Data specific privacy concerns are related to the issues of users that certain data,
when combined, could have critical implications. Next to having the explicit consent of the
users, it will also be necessary to inform the users on which type of data will be collected
during the scope of the research, what will be done with that data as well as to foresee the
ability to delete all data afterwards (in line with the EU policy on the right to forget).
9 Discussion
We have described the general architecture of a wearable living lab and touched upon
certain opportunities and challenges. However we are still in the phase of putting the
integrated WELLS architecture into practice. The benefit of WELLS is that there are no
aspects of the architecture that can not be implemented using todays consumer-grade
wearables technologies, web-based programming languages, database systems, machine
learning and visualisation techniques. What remains to be investigated, however, is the
accuracy of the different devices that are and will become available and the quality of the
data. In addition, some other concerns exist in terms of battery power, data heterogeneity,
form-factor, ruggedness and ease of use.
64

Battery power has been a constant issue in many recent technology developments. Certain
applications, like geolocalisation, can drain a smartphone battery at a much increased rate
compared to a normal smartphone application. The WELLS system will only remain
operational for as long as all of its components remain active. How long we can expect this
to be still remains unclear, but is an important unknown to figure out in order to allow
long-term measurements of user experience. Data heterogeneity will result from different
devices creating different data in a variety of formats. Handling this so that for example all
data feeds can be combined and accurately synchronized is a hurdle which we still need to
tackle. Form-factor is an important aspect, as it influences the obtrusiveness that is
experienced by the user. For example the form factor of the Google Glass proved to be
sub-optimal, as users would not feel comfortable with wearing the device. For devices in
the wearable living lab to function well, they should be as invisible as possible, both for the
user as its environment. Video-capturing devices are available that allow more discreet
video-recording, but many of them also have their limitations, like lack of audio recording
capability. Still, previous research on wearables shows that even for more common devices
such as wristbands it takes a while for users to get accustomed to the device and start
behaving naturally (Hedman, 2011). Therefore we will need to investigate how the use of a
WELLS architecture impacts the natural use of the systems under investigation.
Rugedness is important, as the wearables living lab needs to remain operational in various
environmental condition. Especially continued active when it is raining seems essential.
Finally, the WELLS architecture should be coordinated through an app that runs on a
smartphone and that is able to handle all the interactions that are needed with the user. As
with all applications, it will be necessary to make sure that the user experience design of
this app, and the way in which it integrates with the wearables, is created in such a way
that it is as easy to use as possible.
The WELLS architecture can be of use in each of the three living lab phases discussed
above: a PTA, a formative prototype evaluation or a summative prototype evaluation.
WELLS will be most useful in situation in which mobile technologies are evaluated.
Indeed, these are the technologies that are hard to evaluate in usability lab conditions, as
many of the users reactions will result from the combination of the software and the
context in which the user resides.
Ballon (2015) found that, although there is heterogeneity in the existing living lab
approaches, they share four characteristics: (1) the discovery of unexpected usage and new
service opportunities, (2) the evaluation or validation of new digital technology solutions
by users, (3) a familiar usage context, and (4) a medium or long term research angle. We
believe it is clear that WELLS can add novel aspects to each of these characteristics but can
not go into detail on how this is the case due to space restrictions.
We plan to use this architecture in many of our ongoing research and development
projects. One example is a pervasive city game in which players can use their smartphone
to play games in the urban environment. Another is a project in which the aim is to
provide resistive schyzophrenic patients with cognitive behavioral therapy over wearable
devices. Both projects are examples that will need users to experience the system in a
prologued way, underlining the need for a research approach that can collect, analyze and
visualize in the wild and continuous data, originating from an extensible set of sensors.
10 Conclusion
We have presented an architecture for the wearable living lab, a system to collect user
experience feedback in living lab projects. The user researcher is the main consumer of
the WELLS data and can use the system in a living lab project during a PTA, a formative or
a summative software evaluation. Its relevance for Living Lab or other in-situ research
65

resides in its capability to capture contextual and personal data of end-users in real life
environments. The next steps are to build, gradually, a concrete instance of WELLS and
to explore and evaluate the different elements addressed in this paper. This will entail
creating a system that can support a living lab research approach (and in-situ research in
general) that can collect, analyze and visualize in the wild and continuous data, originating
from an extensible set of sensors.
11 References
Albert, W., & Tullis, T. (2013). Measuring the user experience: collecting, analyzing, and
presenting usability metrics. Newnes.
Ballon, P. (2015). Living Labs. The International Encyclopedia of Digital Communication
and Society. John Wiley & Sons
Banaee, H., Ahmed, M. U., & Loutfi, A. (2013). Data mining for wearable sensors in health
monitoring systems: a review of recent trends and challenges. Sensors, 13(12), 1747217500.
Bergvall-Kareborn, B., Hoist, M., & Stahlbrost, A. (2009). Concept design with a living lab
approach. In System Sciences, 2009. HICSS09. 42nd Hawaii International Conference on
(pp. 110). IEEE.
Choi, J., Ahmed, B., & Gutierrez-Osuna, R. (2012). Development and evaluation of an
ambulatory stress monitor based on wearable sensors. Information Technology in
Biomedicine, IEEE Transactions On, 16(2), 279286.
Eriksson, M., Niitamo, V. & Kulkki, S., 2005. State-of-the-art in utilizing Living Labs
approach to user- centric ICT innovation - a European approach., Sweden.
Hedman, E. "The Frustration of Learning Monopoly: The Emotional Tension of Entering a
New Game Encounter," The Ethnographic Praxis in Industry Conference, Boulder.
September 18-21
Hedman, E., Miller, L., Schoen, S., Nielsen, D., Goodwin, M., & Picard, R. W. (2012).
Measuring autonomic arousal during therapy. In Proc. of Design and Emotion (pp. 1114).
Hernandez, J., Riobo, I., Rozga, A., Abowd, G. D., & Picard, R. W. (2014). Using
electrodermal activity to recognize ease of engagement in children during social
interactions. In Proceedings of the 2014 ACM International Joint Conference on Pervasive
and Ubiquitous Computing (pp. 307317).
Huang, S. T.-Y., Kwan, C. M. Y., & Sano, A. (2014). The moment: a mobile tool for people
with depression or bipolar disorder. In Proceedings of the 2014 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (pp. 235238).
ACM.
Katz, J. (2001). How Emotions Work. London: University of Chicago Press.
Kocielnik, R., Sidorova, N., Maggi, F. M., Ouwerkerk, M., & Westerink, J. H. (2013). Smart
technologies for long-term stress monitoring at work. In Computer-Based Medical
Systems (CBMS), 2013 IEEE 26th International Symposium on (pp. 5358).
Kocielnik, R. D. (n.d.). LifelogExplorer: A Tool for Visual Exploration of Ambulatory Skin
Conductance Measurements in Context. In Proceedings of Measuring Behavior 2014.
Mann, S., Nolan, J. & Wellmann, B. (2003)., Surveillance & Society 1(3),331-355
Pierson, J., & Lievens, B. (2005). Configuring Living Labs For A Thick Understanding Of
Innovation. Ethnographic Praxis in Industry Conference Proceedings, 2005(1), 114127.
Pierson, J., Jacobs, A., Dreessen, K., Lievens, B., Van den Broeck, I., & Van den Broeck, W.
(2006). Walking the interface: uncovering practices through proxy technology assessment.
EPIC 2006 Proceedings, 2426.
Sano, A., & Picard, R. W. (2013). Stress recognition using wearable sensors and mobile
phones. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine
Association Conference on (pp. 671676). IEEE.
66

Schaffers, H., Garcia Guzman, J., & Merz, C. (2008). An Action Research Approach to Rural
Living Labs Innovation. Proceedings of the Cunningham and M. Cunningham (Eds),
Collaboration and the Knowledge Economy: Issues, Applications, Case Studies. IOS Press.
Schachter, S., Singer, J. (1962). Cognitive, Social and Physiological Determinants of
Emotional State, Psychological Review, 69, 379-399
Schuurman, D. & Marez, L. De, 2012. Structuring User Involvement in Panel-Based Living
Labs. Technology Innovation Management Review, (September), pp.3138.
Tang, T., & Hmlinen, M. (2012). Living lab methods and tools for fostering everyday life
innovation. In Engineering, Technology and Innovation (ICE), 2012 18th International ICE
Conference on (pp. 18). IEEE.
Trimi, S. & Berbegal-Mirabent, J. (2012). Business model innovation in entrepreneurship.
International Entrepreneurship and Management Journal, 8(4), 449465.
Westerlund, M., & Leminen, S. (2011). Managing the challenges of becoming an open
innovation company: experiences from Living Labs. Technology Innovation Management
Review, October 2011.

67

Change Laboratory as a method

of innovation management
in an Urban Living Lab
Virpi Lund & Soile Juujrvi a
a

Laurea University of Applied Sciences

virpi.lund@laurea.fi

Abstract
Urban living labs (ULL) challenge researchers and developers with bottom-up approach
and diverse problems which require appropriate tools for innovation management. The
Change Laboratory (CL) is such a promising method which has widely been used in
organizational development but not yet applied in living lab contexts. The present paper
describes the application of the CL in the suburban area in Southern Finland. Residents,
city planners, civil servants and non-governmental organizations (NGOs) were invited to
join the workshop process aiming at enhancing collaboration and establishing shared
goals for urban development. Problems were analyzed and reinterpreted with the CL tools
revealing contradictions in current practices. The results suggest that mirror data and
conceptual tools as intervention methods are powerful stimulus for problem-analysis and
goal-setting in urban and community development.
Keywords
Change Laboratory, living labs, residents, stakeholder collaboration, urban development
1 Introduction
Cities can be said to be self-organizing organisms, involving complex problems, such as
excessive bureaucracy, social segregation, poor image, or economic differences (Baynes,
2009). Living laboratories provide a promising approach to handle those problems and
innovate new solutions in urban areas, requiring bottom-up innovation process and
collaboration among multiple stakeholders. One of the success factors for urban living
labs is a composition of all actor roles: enablers, utilizers, providers and users (Leminen,
Westerlund & Nystrm, 2012). Enablers represent public-sector actors providing strategy,
visions, and networks, and allocating financial resources; utilizers are enterprises, public
sector actors, and NGOs willing to experiment services and products; providers represent
research and development institutions offering innovative methods and taking care of
systematic augmentation of knowledge. Finally, users are residents and other people
habiting the area who give opinions, participate in co-creation and empower other
residents in developmental initiatives (Juujrvi & Pesso, 2013).
Urban Living Lab (ULL) has been defined as a forum for innovation that integrates
residents and other stakeholders to develop and test new ideas, systems and solutions in
complex and real contexts (JPI, 2013). ULLs capture many essential features identified in
68

the living lab literature (Westerlund & Leminen, 2014). First, they represent an ecosystem
or networks involving multiple stakeholders which are motivated by different objectives
but would benefit from collaboration. Prerequisites for successful collaboration are
learning for interaction and handling power issues (Hakkarainen & Hyysalo, 2013). Second,
ULLs provide tools for enhancing and implementing public and user involvement.
Resident involvement has been regarded crucial for speeding up innovation process and
enhancing participatory democratic practices; despite that recruiting residents is
challenging and they need targeted support (Friedlich, Karsson & Federley, 2013;
Veeckman & Graaf, 2015). Third, ULLs can be seen as an innovation management tool for
building networks and user involvement in urban development (Edwardsson et al., 2012;
Westerlund & Leminen, 2014).
Notwithstanding all above-mentioned aspects are important, the present paper is focused
on innovation management that has received little attention in past studies. The Change
Laboratory is a formative intervention method based on the theory of expansive learning
(Engestrm, 1987, 2007; Virkkunen & Newnham, 2013) which has been widely applied for
promoting innovation and learning within organizations but not explicitly used in regional
or urban development to date. The aim of this paper is to present the application of the CL
with the emphasis on one of its distinguishing features: the analysis of contradictions. We
argue that the CL provides a comprehensive conceptual framework for innovation
management, and effective tools for analyzing and solving multidimensional urban
problems through collaborative learning. Our challenge is to modify it to respond to the
needs of ULL that gathers actors across variety of organizations and sectors lacking explicit
shared goals, rather than from single organizations with officially defined objectives.
The remainders of the paper are organized as follows. The target area of the ULL initiative,
Espoo Centre, is next described, followed by the description of the CL and its recent
application, Community Workshops. Then the application of the CL is exemplified by the
case Community Space emerged in the workshops, focusing on the phase of problemanalysis (analysis of contradictions). In particular, we wish to illuminate one of the
distinguished CL tools, using double stimulation, as means of enhancing problem-solving
and collaboration. Last, the paper is finalized by conclusions. Our aim is to describe the
implementation of the Change Laboratory method and its possibility to reveal the
contradictions of everyday practice.
2 Espoo Centre as an ULL platform
This study is a part of a three-year participatory action research (Kemmis &McTaggart,
2000) aiming at examining and enhancing residents participation and developing efficient
means for residents and stakeholders collaboration in urban development. The focus area
is a part of the municipal district in the city of Espoo, in southern Finland, consisting of the
administrative centre of the city with a railway station and two shopping malls,
surrounded by two neighbourhoods of 17 000 people altogether. The area is characterized
by different historical layers in terms of construction and waves of migration, mainly
refugees, from 1970s onwards. Cultural and ethnic diversity in daily life is reflected in a
high proportion of immigrants and over 70 of spoken languages. In the light of social and
economic indicators, the area represents the least advantaged area in the City of Espoo,
mainly due to the concentration of social housing (Residents welfare in Espoo, 2013). The
area has also multiple strengths, such as good transportation and availability of services,
surrounding nature with indoor activities, and historical buildings. The area has been a
focus for dozens of research and development projects in last decades, including current
renovation projects by the city.

69

The present research project is motivated by two main observations: low user involvement
in previous development projects and a lack of systematic collaboration between various
local stakeholders and developers. Recent interviews with stakeholders however revealed
their strong will to develop collaboration and engage residents in development activities
(Juujrvi, 2014, Juujrvi & Lund, forthcoming). It was concluded that the living lab
approach could provide an appropriate innovation platform for a systematic collaboration
initiative that would bring together local residents and other stakeholders with
overlapping aims to share their interests and build collaboration. The CL was primarily
selected as a suitable method, because its aims at expanding participants' understanding
about the objects of development work that would furthermore lead to shared goals
enabling purposeful collaboration. Second, the CL starts with the analysis of
contradictions in current work practices that are historically molded and outdated to
respond to urgent challenges in work. It seemed obvious that current practices in urban
and community development were heterogeneous originating from various historical
layers and developed to respond individually to emerging urban problems at different
times. We therefore expected that the analysis of contradictions could be essential for
creating sustainable new ways of collaboration.
3 Change Laboratory as an intervention method
CL is a formative intervention method widely used in organizational development which
has roots in cultural-historical activity theory (Vygotsky, 1978; Leontjev, 1978), and its
Finnish application of developmental work research (Engestrm, 2007; Engestrm,
Virkkunen, Helle, Pihlaja & Poikela, 1996; Virkkunen & Newnham, 2013). In the culturalhistorical theory, work can be conceptualized through the activity system with six
components: the subject, the object, mediating artifacts, the rules of participation,
community, and the division of labor. The activity system is usually depicted as a trianglemodel (see Figure 1) (Engestrm, 1987, 2001). The activity system is a unit of analysis in the
CL; it is used to describe the entity of activity with the changing object. A crucial part of
the analysis is contradictions within and between the components which are seen as a
driving force for a change (Ilenkov, 1977, 1982). Contradictions are systemic, structural
obstacles that need to be broken away from new forms of activity to emerge. Each activity
system has its own history against which contradictions and challenges can be understood.
The analysis of the internal contradictions of the activity system helps CL participants to
shape and renegotiate the shared object and work on other components as well.

70

Tools, Instruments

Object Outcome

Subject

Rules

Community

Division of Labour

Figure 1. The structure of human activity (Adapted from Engestrm, 1987, p.78)

Collaboration among participants in the CL is based on expansive learning which means

new understanding in concrete situations with the elements (components) of the activity
system. Expansive learning forms a cycle where questioning and analyzing contradictions
necessitate changes in current practices, leading to innovating new models and ways of
working. The cycle of expansive learning is interwoven in the phases of the CL process
(see Figure 2). Usually the CL process has from five to ten successive sessions. In our case,
we made a shortened version of five workshops for ULL purposes covering five phases
with the experimentation period of two months. The main reason for shortening was that
people participated on the voluntary basis and, therefore, it could be difficult to maintain
their participation over long time.
6.
Spreading
and
consolidati
ng

5.Implementi
ng the new
model

1. Charting

the
situation

2.
Analyzing
the
situation

4.
Concretizin
g and
3. Creating
testing
a new model
the new
model
Figure 2. The phases of Change Laboratory process (Virkkunen & Newnham, 2013, p. 17)

4 Community workshops
The research project Vlittvt Valittavat Verkostot (Caring and Sharing Networks in
English) arranged the Community Workshops in winter-spring 2015. The procedure
71

involved five successive workshops with two-week intervals and the experimentation
period of two months between the fourth and five workshops. The workshops were
scheduled to start at 04.30 p.m. in a local city hall and took approximately two and half
hours. Forty-seven invited people attended to the workshops, varying from 30 to 38 across
the workshops. The participants involved residents and members of resident associations,
managers of regeneration projects, city planners, public servants and experts in the city
administration, representatives of non-governmental organizations and local parishes, and
managers of shopping malls. The workshops were managed by a consultant qualified for
practicing the CL method in collaboration with four researcher-interventionists who acted
as group facilitators in four groups. For research purposes, all workshop activities were
recorded and documents photographed.
The analysis of the present paper is confined to the second workshop in which participants
with similar interests grouped together and were instructed to analyze present troubles
and contradictions in current practices. The authors listened to the recordings several
times, reviewed prior observations by the group facilitators and analyzed the group
documents. Finally, one of the groups was selected a case to illustrate the workshop
process.
5 Double stimulation as an intervention method
Collaboration and learning in the CL is based on the method of double stimulation
(Vygotsky, 1978; Sannino & Engestrm, 2015). Double stimulation enables human being to
use volition and agency for transforming the contradictory circumstances with the help of
the external artifacts providing meaningful tools (Sannino, 2015). There are two types of
stimulus material: the mirror data as the first stimulus and conceptual tools as the socalled mediating second stimulus. The mirror data is ethnographic data collected in
various ways such as observing events, interviewing actors, shadowing practices, or
studying documents representing critical viewpoints, trouble and problems in current
practices. The mirror data can be collected by participants or research interventionists.
The purpose of the mirror data is to make distracting and deviating things in current
practices visible; it has a crucial role in identifying contradictions in the current system of
activity. It is presented to participants in order to challenge various interpretations and get
them engaged in seeking new solutions. Tasks, models and questions given by researcherinterventionists serve as the second mediated stimulus that helps participants to analyze
and interpret the mirror data. These two types of stimulus serve different purposes; the
mirror data makes participants confront unpleasant facts of the current activity, whereas
theoretical tools help them to distance themselves from the emotionally difficult situation.
Movement between concrete observations and abstractions is important, because it fuels
participants learning and real change (Virkkunen & Newnham, 2013).
6 Case Community Space
The purpose of the second workshop was to analyze present troubles and contradictions in
current practices and ways of working. The session was started with a speech of one
researcher-interventionists that summarized the outcomes of the previous workshop and
introduced the role of the mirror data in problem-definition and problem-solving. In
addition, another researcher gave a short presentation about the history of the residential
area and its distinctive characteristics. Participants then were divided in four groups of
interest each focusing on a certain developmental challenge identified in the previous
workshop. It was expected that every challenge would have an activity system of its own,
reflecting a definite aspect of general development.
72

This paper is focused on a group dedicated to further process the theme Community
Space that had emerged in needs-charting in the previous workshop. There had been a
chronic lack of public non-commercial premises where residents and associations could
meet and organize social activities. The group members represented various stakeholders
who had a sort of premises on their agendas but who did not know interests of each other
that well. There have been several individual attempts to find and establish such premises
for citizens, but they had constantly failed due to the bureaucratic model of municipal
governance and a lack of political will. The specific mirror data was built to illustrate this
complicated situation based on earlier observations and interviews with stakeholders:
people talked a lot about the residents house, however referring to various existing
premises and different kind of concepts. This contradiction had been presented as a
PowerPoint slide to participants in the beginning of the workshop and competing
expressions by various stakeholders were further elaborated by one of the researcherinterventionists (Figure 3).

Occupying
empty City
Hall
Low
treshold
social
services

Old chapel

Dimensions of
Residents'
House
Old
railway
station
building

Community
Centre

Culture
Centre
in a hotel
Figure 3. Dimensions of Residents House (Salin, 2014)

The purpose of the mirror data is to stimulate discourse that would reveal one-sided
perspectives and repeating myths in current practices and point out the need of change
(Engestrm, 1987, 2007). The participants were asked to pinpoint problems, interruptions
and conflicts in present practices and identify tensions and contradictions against the
mirror data. The following excerpt from the group conversation exemplifies how
participants start to build the big picture of the problem by sharing their own
presumptions.
Amanda (a worker of the city planning department): I have heard so often that
people want a community space, so I am curious, why this issue does not go any
further.
73

Josh (a second -generation immigrant): I am interested in having an office for my

association, we have tried to buy a place, but it fails constantly. But we keep on
doing this again, and we do have an intention to purchase a place.
Macy (a worker of an international encountering place): Taking into account how
much we have space available here, it is a pity, that they are vacant and not
utilized.
Jill (a worker of the cultural department): We do have space and we can offer it to
others, too. There are places available, but how can we utilize them? We have a
happy problem; we can provide spaces.
Macy: The problem is who is managing this? Someone said that the NGOs should
be the ones who coordinate but they do not have money to finance activities. The
money has to come from elsewhere; the associations do not have money! The
problem is that there should be the municipality involved who would lead this,
and then in collaboration, we can do things.
Tim (a leading staff member of social services): Coordination is lacking here. And
the perspective; should social services be involved or should it be cultural
activities, this has not been crystallized yet.
Jane (a resident, a social worker): There are a plenty of organizations with money,
if we only could have so-called heart which could be a prime mover for more
networking, and when it bubbles over coordination starts to happen just by
coincidence The groups could find each other, if they had a place.
Mike (an architect of city planning): The aims are not yet defined. After that it
would be easier to find a place.
Keith (a director of a NGO): Do we have a vision? Mushrooms have its mycelium
beneath the ground, but we need caps for the mushroom. Thats why we need the
community space, to enable the mycelium functioning, we are the mycelium with
the activities.
Jane: We cannot go and use premises owned by the municipality, because then we
will be involved with the housing department, and we will have rental payments,
limitations and troubles. The aim is to enable residents activities in evenings and
weekends too, it should function as free as possible to enable its flourishing
Jill: It pops up more and more all the time, I cant outline the big picture, there is
so much happening all the time, isnt there already enough of places and
activities? Do we need more premises? Could it be so that residents premises are
located in different places, do we need only one space?
Amanda: Premises cost, who takes the responsibility? We do have places
available, but no one dares to ask how they are utilized. Should someone just ask?
Keith: The problem is connected to the place. As citizens, we can allow time, the
organizations can allow other resources, but we need the place. Associations are
financed according to their activities, not according to premises.
Jane: Coordinating needs resources, running activities needs walls! I have been
involved in many meetings and groups dealing with the future residents house,
but they are all abolished. The discussions have been going on in many arenas.
You are certainly confused about which place you are gonna to plan and draw for
us?
Mike: Well, there are a plenty of places, but I dont know what kind of premises
you want.
Tim: These issues have been taken care of separately, in different boxes.
Keith: The municipality should be the core layer: (providing) the place and the
coordinator. After that the civil society will arrive and produce other stuff: time,
resources, activities, and then everything starts blooming.
74

Keith: The thing is that everyone starts to protect own resources, instead we
should give resources to our shared target.
The summary of the discourse was documented in a three-piece clipboard and its findings
were explained to other groups (see Table 1). The phases of the clipboard was to show the
present practice with the disturbances and tentative suggestions for the future. It was not
necessary to invent yet complete ideas for the solutions of the problem; the thorough
processing was more crucial.
Disturbance/problem
- A heart that collects
stakeholders
together,
layers of the mushroom?
-Who
takes
care
of
responsibilities,
management
and
coordination?
-Overall picture is missing,
aims are unclear.
-How to combine the
resident house and its
ideology to the architectural
planning process?

Present Practice
- everyone is
working
inboxes
of their own

New Ideas
-something common
-coping an already existing
concept of the residents
house
-occupying new space
-assisting
the
ongoing
planning process of the
residents house

Table 1. The analysis of the disturbances of the Community Space present practice

As explained above, analytical tools provide another component of the double stimulation
method. In the CL sessions, the triangle model of the current activity system is usually
used for this purpose. The triangle model illustrates the structure of the present practice
and reveals tensions and contradictions embedded. It also helps participants to shape and
advance a joint object in a visible way. This phase is strongly guided by researcherinterventionists questions: why there is a need to a change, who are involved, what are the
tools needed, who belong to the community, according to what principles the work is
divided, what are the rules, instructions and limitations? The participants are also
instructed to mark the recognized contradictions within and between the components of
the activity system by flashes. The triangle model constructed by the participants is shown
in Figure 4.

75

Figure 4. The analysis of the structure and contradictions of the present practice

To summarize the outcomes of the group work, the analysis of the current practices
revealed that there have been a number of actions and initiatives by single residents, small
groups and larger communities to promote community spaces in the area. The main
obstacles have been a lack of funding and suitable premises, as well as confusion about
which stakeholders should take responsibility to advance and organize community spaces.
All interested stakeholders had not even met each other before. Stakeholders have not
been able to act alone which has caused frustration. The present discourse revealed that
group members have a plenty of resources for creating a joint community space: time,
activities, experienced practitioners, premises, and even a public initiative with a budget
for the future residents house. Many of them were ready to take more responsibility for
planning, organizing and implementing a joint community space or the residents house,
but the shared vision with specific aims should be created at the first place. There were
still competing views on the ideology and purpose of the potential joint community place.
The group members recognized urgent needs for further coordination of collaboration.
Political decision-makers should get involved in the planning process, and various options
for financing should be sorted out.
As a result of the workshop session, the participants succeeded to form a big picture and
shape a joint goal for future activity. The intervention tools helped them to elaborate
scattered and unfinished ideas and build new knowledge. The atmosphere was
constructive and enthusiastic, enabling creating relations and mutual learning. The
experiences from other groups not reported here were quite similar, showing that the
presentation of various types of data (excerpts for residents interviews, a letter to a local
newspaper) elicited emotional responses and got the participants engaged. The chosen
analytical tools proved challenging but manageable within the time limits. Gained
knowledge provided a basis for planning experiments in the following workshops.
7 Conclusions
The findings highlighted the importance of the analysis of contradictions and conflicts in
current practices in order to create shared understanding and appropriate goals for future
development. The mirror data served as a powerful stimulus for shared reflection and
learning, whereas the triangle-model of activity system as a conceptual tool made current
contradictions visible to participants, enhancing understanding of the emerging shared
object. Double stimulation boosted stakeholder collaboration that is critical in solving
multidimensional urban problems. The participants started to take charge of the
development process by exercising agency provoked by the mirror data and double
stimulation.
It is worth noting that the present ULL activity was driven by the researchers who had an
opportunity for deriving the appropriate mirror data from the projects multiple databases.
Another option would have been to have participants gather the mirror data themselves.
This however proved too challenging within the scheduled workshop programme.
Successful mirror data is emotionally touching, and may provoke powerful and
contradictory feelings that must be handled carefully by facilitators in order to maintain
constructive atmosphere.
As the final conclusion, the Community Workshops based on the CL provided an
innovation platform and helped to build new practices in urban development through
collaboration between various stakeholders. The sessions succeeded to build a creative
encounter of multiple perspectives with complementary expertise and resources needed in
problem-solving and creating new models of practice (Miettinen, 2014). The CL provides a
structured procedure for managing innovation process, still allowing room for creativity of
76

individuals. The major challenge might be a relatively big number of participants that is
typical for urban living lab networks. Therefore, the CL method needs to be modified to
meet needs of heterogeneous groups and actors involved in urban development.
8 References
Baynes, T. (2009). Complexity in urban development and management. Historical
overview and opportunities. Journal of Industrial Ecology, 13(2), 214227.
Edwarsson, B., Kristensson, P., Magnusson, P. & Sundstrm, E. (2012). Customer
integration within service development A review of methods and an analysis of in-situ
and ex-situ contributions. Technovation, 32, 419429.
Engestrm, Y. (1987). Learning by Expanding: an Activity-theoretical Approach to Developmental
Research. Helsinki; Orienta-Konsultit.
Engestrm, Y. (2007). Putting Vygotsky to work: The change laboratory as an application of
double stimulation. In H. Daniels, M.Cole & J.V.Wertsch (Eds.), The Cambridge companion
to Vygotsky (pp. 363-382). Cambridge, UK: Cambridge University Press.
Engestrm, Y., Virkkunen, J., Helle, M., Pihlaja, J. & Poikela, R. (1996). Change laboratory as
a tool in transforming work. Lifeling learning in Europa, 1(2), 1017.
Friedrich, P., Karlsson, A. & Federley, M. (2013). Report 2.1. Boundary Conditions for successful
Urban
Living
Labs,
SubUrbanLab,
http://jpi-urbaneurope.eu/wpcontent/uploads/2014/01/Report-Boundary-conditions-for-successful-Urban-LivingLabs.pdf, accessed April 2015
Hakkarainen, L. & Hyysalo, S.(2013). How do we keep the living laboratory alive? Learning
and conflicts in living collaboration. Technology Innovation Management Review, December
2013, pp. 1622. http://timreview.ca/article/749, accessed April 2015.
Ilenkov, E.V. (1977). Dialectical logic. Essays in its history and theory. Moscow, Russia:
Progress.
Ilenkov, E.V. (1982). The dialectics of the abstract and the concrete in Marxs capital. Moscow,
Russia: Progress.
JPI
(2013).
Urban
Europe.
Joint
call
for
proposals
2013.
http://www.ffg.at/sites/default/files/downloads/call/jpi_ue_pilot_call_2__call_text_final_2013-06-18.pdf, accessed April 2015
Juujrvi, S. (2014). Osallisuudesta toimijuuteen Espoon keskuksessa.[From participation
towards agency in the Espoo centre] Eetvartti 3/2014. Espoo: The city of Espoo.
Juujrvi, S. & Lund V. (forthcoming). Engaging residents: Insights from Participatory
Action Research into Urban Living Labbing. In E. Amirall, S. Leminen & M.Westerlund
(Eds.) Living Labs Innovating by Co-Creating with Users in Real-Life Environments.
Juujrvi, S.& Pesso, K. (2013). Actor roles in an urban living lab: What can we learn from
Suurpelto, Finland? Technology Innovation Management Review, November 2013,
http://timreview.ca/article/742, accessed April 2015.
Kemmis, S. & McTaggart, R. (2000). Participatory action research. In N.Denzin & Y.Lincoln
(Eds.) Handbook of Qualitative Research (pp. 567605). Thousand Oaks: Sage.
Leminen, S., Westerlund, M. & Nystrm, A-G (2012). Living Labs as open-innovation
networks. Technology Innovation Management Review, October 2012,
611,
http://timreview.ca/article/602, accessed April 2015.
Leontev, A.N. (1978). Activity, consciousness, and personality. Englewood Cliffs, NL: PrenticeHall.
Miettinen, R.. (2014). Creative encounters, collaborative agency, and the extraordinary act
of the meeting of a need and an object. In A. Sanninio & V. Ellis (Eds.), Learning and
Collective Creativity: Activity-theoretical and Sociocultural Studies, (pp.158-173). New York:
Rotledge.
77

Residents welfare in Espoo (2013). [Espoolaisten hyvinvoinnin tila 2013]. Espoo, Finland:
City
of
Espoo,
http://www.espoo.fi/fiFI/Espoon_kaupunki/Tietoa_Espoosta/Tilastot_ja_tutkimukset/Hyvi
nvointi/Espoolaisten_hyvinvoinnin_tila_2013(39045), Accessed June 2014.
Salin, O. (2014). Dimensions of Residents House. Unpublished material.
Sannino, A. (in press.). The principle of double stimulation: A path to volitional action.
Learning, Culture, and Social Interaction.
Sannino, A. & Engestrm, Y. (2015). Co-generation of societally impactful knowledge in
change laboratories (forthcoming).
Veeckman, C., and Graaf van deer, S. (2015) The city as Living Laboratory: Empowering
citizens with the Citadel Toolkit. Technology Innovation Management Review, March 2015,
pp.617, http://timreview.ca/article/877, accessed April 2015.
Virkkunen, J. & Newnham, D. (2013). The Change Laboratory. Rotterdam: Sense Publishers.
Vygotsky, L. (1978). Mind in society. The Development of Higher Psychological Processes.
Cambridge: Harvard University Press.
Westerlund, M. & Leminen, S. (2014). The multiplicity of research on innovation through
living labs. Paper presented at The XXV ISPIM Conference in Dublin, Ireland on 8-11 June
2014. The publication is available to ISPIM members at www.ispim.org.

78

Involvement of end-users in innovation process: towards a

user-driven approach of innovation A qualitative analysis of
20 Livings Labs
Perrine VANMEERBEEK a, Lara VIGNERON b,
Pierre DELVENNE , Benedikt ROSSKAMP b, Mlanie ANTOINE c
b

CRIDS, UNamur
University of Lige
c
SPIRAL, ULg
perrine.vanmeerbeek@unamur.be
b

Abstract
Initially developed to bridge the gap between research and market, a Living Lab can be
described as an open, innovative and collaborative process based on three core
characteristics: user involvement in the innovation process, experimentation in real-life
context and the gathering of stakeholders in public-private-people partnerships. The
paper focuses on user involvement and provides an insight on how this principle is put in
practice, through the qualitative analysis of interviews conducted with twenty Living Labs
in Europe and Canada. Our results interestingly point out that practice of user-driven
approach, unlike what is promoted in Living Lab theory, is rather limited. Indeed, many
Living Labs do not involve users for the ideation phase at all, and those ones usually use a
user-centered approach for the following steps. Besides, when observed, user-driven
approach is often restricted to the ideation phase. We can though ask ourselves the
question of what is, or should be, living in a Living Lab? Furthermore, our results suggest
that a user-driven approach seems more suitable when a Living Lab aims to create social
value than when its objective is to create economic value.
Keywords
Living Labs, open innovation, user involvement, qualitative study
1 Introduction
Initially developed to bridge the gap between research and market, a Living Lab can be
described as an open, innovation and collaborative process based on three core
characteristics: user involvement in the innovation process, experimentation in real-life
contexts and the gathering of stakeholders in public-private-people partnerships (Dub et
al., 2014).
Because it is based on the diversity of actors to be engaged and on a strong focus on
innovations users, the Living Lab methodology can be considered as a social innovation.
Besides, by involving users in the long run, Living Labs aim to speed up the innovation
process and to reduce the risks related to market failure or public backlash with regard to
new products. Unlike traditional approaches, where innovation lies in the experience and
the creativity of professionals, the Living Lab values users tacit knowledge to reach an
79

alignment between supply and demand. In this perspective, professionals are considered
as experts in technology and users are considered as experts in usage.
Living Labs are supposed to be valuable for every stakeholder of an innovation project. In
theory, private and public sector as well as citizens are concerned. Dub et al. (2014) refer
to Santoro and Bifulcos model of value creation the KBS Framework (PRO-VE, 2005)
to talk about the value created in a Living Lab. Three dimensions are highlighted:
knowledge, social and business (p. 18). This value can be modelled by the
compromise between the knowledge created by experimenting, the economic value
produced, and the social benefits generated by a specific experimentation project. Each
project will have a particular identity within this framework. (p. 18, our translation)
What does user involvement mean in practice? In order to assess user involvement in
Living Lab projects, inspiration might be found in the New Product Development
literature. Hoyer et al. (2010) outline and discuss a conceptual framework that focuses on
the degree of consumer co-creation in new product development (NPD). According to
their framework, the degree of co-creation activities in NPD is a function of both the scope
of co-creation activities as well as the intensity of these activities. The scope of co-creation
refers to the propensity of firms to collaborate with consumers across all the stages of the
NPD process which include ideation, product development, commercialization, and
postlaunch activities. Thus, firms that are highest in their scope of co-creation look out for
collaboration with consumers in all of these stages. Intensity of co-creation refers to the
extent to which firms rely on co-creation to develop products within a particular stage of
NPD. Thus, firms that are highest in their intensity of co-creation in a particular stage of
product development rely exclusively on consumers for their development activities in
that stage. (Hoyer et al., p. 288)
Considering the scope dimension, it may be argued that users in Living Labs take part in
every step of the innovation process. Involving users in the early development of a
product or service enables to ensure that the product or service corresponds not only to
users natural behaviours but also to users life context (Dub et al., 2014, p. 31, our
translation). Similarly, Ballon, Pierson and Delaere (2005) studied different types of test
and experimentation platforms for broadband innovation. They introduce the Living Lab
concept as a type of test and experimentation platform and explain that the novel aspect is
that it involves users in all stages of R&D and all stages of the product development
lifecycle, not just at the end phases as, for example, in more classical field trials or user
testing of products (Ballon, Pierson and Delaere, 2005, p. 9).
Considering the intensity dimension, Living Labs may be presented as user-driven
methodologies more than user-centered methodologies. To better grasp what is meant by
user-driven methodologies, we can rely on the work of Almirall, Lee and Wareham
(2012), who carried out four case studies of Living Labs and divided their methodologies in
four different categories. (1) User centered: users are mostly passive subjects of study; (2)
Design driven: designers take the lead; (3) Participatory: users are considered on equal
ground with the rest of the partners in a co-creative process; (4) User-driven: where the
user is the one who drives the innovation process. On one end of the spectrum, users are
regarded as subjects of observation, such as in human factors, ergonomics, or applied
ethnography. On the other extreme, users are co-creators, such as in the case of lead users
or open source communities. In the middle, we find the majority of methodologies, such as
co-design, design thinking, and design-driven innovation. (Almirall, Lee and Wareham,
2012, p. 17)
The notion of empowerment might also be helpful to understand the extent to which users
are involved. Bergvall-Kareborn and Stahlbrst (2009) analyse how two Living Labs
involve citizens in the design of an e-service. They insist on the empowerment of users.
Inherent in being a partner, from an end-user perspective, is the power of choice. People
80

always can choose if, when and to what extent they want to participate (BergvallKareborn and Stahlbrst, 2009, p. 367). According to the authors, the fact that user needs
and ideas are traceable in the concepts, prototype and finished product is highly
important.
To sum up, the question we focus on is the following: to what extent users have the power
to influence the process and the nature of innovations? Dub et al. (2014) argue that Living
Labs consider users on the same level as other stakeholders and enable the development of
new products and services that are conform to their needs. In our point of view, this rather
corresponds to the participatory level defined by Almirall, Lee and Wareham (2012).
With user-driven methodologies, users are the ones who make decisions. Literature draws
our attention on the fact that reality might be nuanced. Users tacit knowledge might be
captured without giving them any power on what will be made of the knowledge. We
might also observe Living Labs where users are given full power on the whole innovation
process.
2 Methodology
The paper is based on the intermediary results of a research project financed by the
Walloon Region (Belgium) to assess the interest in and the conditions for the
establishment of a Living Lab approach in the health sector in Wallonia. The first six
months of the project were dedicated to in-depth interviews with a selection of twenty
Living Labs. Based on a comparative analysis, a variety of strategies and practices were
identified in regard to various needs. The paper highlights the user involvement
dimension and questions the link it may have with a Living Labs values and missions,
either economic-, social- or knowledge-oriented or a combination of them.
Extensive mapping of Living Labs was based on four information sources: ENoLL
members, Living Lab pointed out by the CETIC (Research Center in Information and
Communication technologies, Gosselies, Belgium), word of mouth and scientific
publications. In total, we selected 28 Living Labs according to two criteria: the
diversification of sources (not to only have Living Labs members of ENoLL) and the
existence of a connexion to the health sector. Each Living Lab was contacted by phone or
mail. Out of these, twenty Living Labs agreed to take part in our research: two in Belgium,
thirteen in France, three in the Netherlands, one in Sweden and one in Canada. Semistructured interviews, mostly with managers, were made on site. Two interviews were
made by Skype: the one with the Swedish Living Lab and the one with the Canadian
Living Lab. All interviews were recorded and extensively transcribed. Qualitative thematic
analysis was made in multidisciplinary team.
3 Analysis
Involvement of users, as we just argued, is one of the most important features in Living
Labs. End-users can be involved in the innovation process in many different ways, at
different moments, and with a variable intensity. We distinguish three ways of involving
end-users (the color refers to the table below):
- User-centered innovation (yellow): that is when end-users are taken into account in
the innovation process, but not as stakeholders, by answering questionnaires or
testing prototypes for example. Users are asked for their specific opinion about a
product or service.
- Co-creation of innovation with users (orange): this is a higher level of end-user
involvement, which includes them as stakeholders in innovation projects.

81

Users-driven innovation (red): this is the highest level of end-user involvement,

which corresponds to innovation embodied by end-users. For example, users can
participate to the Living Lab governance or they can also be stakeholders and even
one of the project leaders.

We also distinguish three projects phases: for each phase, end-users could be involved in a
specific way. The first step is the ideation: where does the innovations idea come from?
Who decides to innovate about a specific thematic, to develop a service or a product? The
second step is the development of a prototype or a solution: whos developing the solution
or the prototype? And the last step is the experimentation: how is the product or the service
tested? In what kind of environment: real or realistic?
We should point out that this sequence of phases is used for analysis and comparison
purposes: not every Living Lab talked about any of these phases, just like not all of them go
through these three phases. Furthermore, some Living Labs have a very specific
innovation process that does not correspond to those three phases.
The following table sums up our results for each Living Lab, about how they involve endusers in their innovation process. Each color represents one way of involving end-users in
Living Labs. Note that the grey boxes mean that the Living Lab does not include end-users
at all. We also draw the attention of the reader on the fact that one Living Lab was
withdrawn from the analysis. This Living Lab is dedicated to the rehabilitation of a
brownfield. It does not develop innovation projects strictly speaking, but it is in charge of
developing land aspects, supporting business creation and connecting researchers with
companies. Therefore, assessing end-users involvement in innovation projects was not
relevant in that case.

A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P

End-users
Researcher
Seniors
Inhabitants
Undefined
Inhabitants
Undefined
Teachers & students
Seniors
Museums
Government
Inhabitants
Health structure
Patients
Citizens
Citizens
Hospital services
Patients
Citizens
Seniors / patients
Citizens
Citizens
Citizens
Health care Professional
Patients

Ideation

Development

Experimentation

82

Q
R
S

Seniors
Patients
Health care Professional
Undefined

The table clearly shows that the most used way of involving end-users is the co-creation
(there is a majority of orange boxes). We also observe that most of the Living Labs
endorsing a user-driven approach restrict their user-driven approach to the ideation
phase. More specifically, we observe a user-driven innovation during the ideation phase in
seven Living Labs. In other words, it seems uncommon for Living Labs to be user-driven in
the development and experimentation phase. Therefore, it seems even more unlikely to
happen if users were not involved at the very beginning of a project, which is the ideation
phase. It might be very important for a project to use that approach from the starting point,
though. Indeed, in this way, chances are increased that the innovation meets users
expectations. Users will probably choose innovations responding to most of their
preoccupations.
We stress that only two Living Labs have user-driven approach in the development phase.
Among these, one Living Lab has a user-driven approach in the experimentation phase
too. From our sample, this is the only Living Lab with a user-driven approach through all
the innovation process. It is important to note that those two specific Living Labs do not
have predefined innovation process phases. They describe a process which is undefined a
priori and which is largely constructed by users (and maybe other stakeholders) along the
project. This is very different from the process described by some other Living Labs where
phases and the way phases are interrelated to each other are identified beforehand like
in a stage-gate process. These results might suggest that most Living Labs prefer to resume
control over the project once the idea is found, at least in term of process. In those cases,
we might wonder what criteria are driving decision-making and who exactly is in charge.
For example, how is valued the innovative potential of specific ideas stemming out of
users minds?
To continue with, we found that most of the Living Labs that do not involve users for the
ideation at all (grey boxes) are the ones using user-centered approach for the following
steps (yellow boxes). This observation is very interesting considering the literature (e.g.
Dub et al. 2014, Ballon, Pierson and Delaere, 2005). If Living Labs are supposed to be more
user-driven, why do nearly half analyzed Living Labs have user-centered approach (at
least for some of their projects) and do not involve users from the beginning of innovation?
Can those Labs really be called living? If not, then what are they, exactly, and why would
they call themselves Living Labs? More generally, what shall we understand from the
fuzzy understanding of what is, or should be, living in a Living Lab?
To conclude with, we observe two tendencies of end-user involvement in the Living Labs
we have analyzed. On the one hand, Living Labs that involve end-users during the entire
innovation process mostly use the co-creation innovation (orange boxes), and sometimes
use the user-driven approach (red boxes). On the other hand, Living Labs that only involve
end-users during the experimentation phase (to test prototypes for example) have an
approach that is more user-centered (yellow boxes).
Another question emerges from complementary results gathered in the framework of the
project: is there a link between user involvement and the values and missions pursued by
Living Labs? As suggested by Dub et al. (2014), we relied on Santoro and Bifulcos KBS
Framework (PRO-VE, 2005) which highlights three dimensions to assess values:
knowledge, social and business. Results along these dimensions are based on
crossing elements related to:

83

values that are explicitly defended by Living Labs ;

missions they pursue ;
indicators of success ;
their intellectual propriety strategy.

Despite the seeming simplicity of the KBS-Framework, it was really hard to characterize
the twenty Living Labs in those terms. Our difficulties may be explained by the fuzzy
boundaries between the three dimensions and the way they overlap each other both in
speech and practices. More specifically, the complexity to define if Living Labs are creating
economical or social value is linked to the hybridization of Living Labs concerns
(among which financial autonomy, relationship with public sector and social usefulness).
The hybridization is also linked to the strong connection between knowledge and
business that drives the economic policy in Europe since the Lisbon agenda. While the
industrial production is moving south, knowledge has become the Norths engine for
productivity and economic growth. The term knowledge-based economy stems from this
fuller recognition of the place of knowledge and technology in modern OECD economies.
(OCDE, 1996, p. 3)
So, is there a link between user involvement and the values and missions pursued by
Living Labs? Considering the results presented above, we decided to focus on the ideation
phase, which appears to be decisive in understanding the place given to users. Remind
that, according to our results: (1) Living Labs that involve end-users during the entire
innovation process mostly use the co-creation innovation and sometimes use the userdriven approach; (2) Living Labs that only involve end-users during the experimentation
phase assume an approach that is more user-centered.
When results along these two dimensions (user involvement and values/missions) are
crossed, with a focus on the ideation phase, observations can be represented as follow:

Out of eight Living Labs with a user-driven approach (see the red crosses), six are situated
in the social area. Two Living Labs with exclusive economical missions have user-driven
approach. Most of the Living Labs taking up a user-driven approach defend social values
and social purposes, at least partly. We should also point out that, out of nine Living Labs
that do not involve users in the ideation phase (see the grey crosses), seven are situated in
the economic area. As a conclusion, a user-driven approach seems more suitable when the
objective of the Living Lab is to create social value then when its objective is to create
economic value. Does it mean that a strong economic approach of the Living Lab
methodology automatically exclude user-driven projects? At least, our results invite to
question the rhetoric of Living Labs that are largely considered as deeply engaging endusers.
84

4 Conclusion
The aim of our paper was to provide an insight on how users are actually involved in the
innovation processes that are central to Living Labs missions. Two trends emerge from
our results. On the one hand, Living Labs that involve end-users during the entire
innovation process mostly use the co-creation innovation, and sometimes use the userdriven approach. On the other hand, Living Labs that only involve end-users during the
experimentation phase have an approach that is more user-centered.
Results also interestingly point out that practice of user-driven approach, unlike what is
promoted in Living Lab theory, is rather limited. Besides, when observed, user-driven
approach is often restricted to the ideation phase. This might suggest that most Living
Labs prefer to resume control over the project once the idea is found, at least in terms of
process. In those cases, we might wonder what criteria are driving decision-making and
who exactly is in charge.
Our results also indicate that most of the Living Labs that do not involve users for the
ideation at all are the ones using user-centered approach for the following steps. Nine
Living Labs are concerned, at least for some of their projects. Can those Labs really be
called living? If not, then what are they, exactly, and why would they call themselves Living
Labs? More generally, what shall we understand from the fuzzy understanding of what is,
or should be, living in a Living Lab?
Furthermore, results suggest that a user-driven approach seems more suitable when the
objective of the Living Lab is to create social value than when its objective is to create
economic value. Of course, future research should further investigate this hypothesis
based on preliminary results. Nevertheless, this points at a wide gap between the official
rationales of Living Labs, claiming that more user engagement will improve marketization
and quality of innovations, and what they do in practice, which is a rather limited users'
engagement. Why is that? Is it just a matter of control, as if managers would fear to
develop what they say they strive for? Are economic approaches and user involvement
mutually exclusive, contrarily to what the literature says? If user-driven methodologies are
easier to implement in social and knowledge creation contexts, to what extent less
pressure on economic results is part of the explanation? Is it more difficult for businessoriented Living Labs to take distance from the traditional innovation stage-gate process
widely used in companies? What material and procedural conditions inhibit users of
taking a more active role in the Living Lab processes? More fundamentally, are these
preliminary results just an unintended consequence of uncompleted attempts at
developing new approaches to innovation with a Living Lab vocabulary or a gloss that
investors project on a process that is mostly left unchanged with regard to real engagement
of end-users? What role do framework conditions (e.g. public funding schemes) play and
how do stakeholders take up the messages and use the resources?
We should stress that our paper is based on the discourse of Living Lab managers. Up to
now, users have not been interviewed. Managers might either take an instrumental and
reductionist stance towards user-involvement processes or they might tend to overemphasize user-involvement. The education and role of the manager in the Living Lab
structure may determine the framing he has over the Living Labs activities. This is one
reason to continue to study Living Labs with participant observation and interactions with
users.
5 References
Almirall, E., Lee, M. and Wareham, J. (2012). Mapping Living Labs in the landscape of
innovation methodologies. Technology Innovation Management Review. September 2012: 12-18.

85

Ballon, P., Pierson, J. and Delaere, S. (2005). Test and Experimentation Platforms for
Broadband Innovation: Examining European Practice. http://dx.doi.org/10.2139/ssrn.1331557
Bergvall-Kareborn, B. and Stahlbrst, A. (2009). Living Lab: an open and citizen-cenric
approach for innovation. Int. J. Innovation and Regional Development, 1(4), 356-370.
Dub, P. (dir.), Sarrailh, J., Billebaud, C., Grillet, C., Zingraff, V. and Kostecki, I. (2014). Le
Livre Blanc des Living Labs (1st ed.). Montral : Umvelt.
Hoyer, W. D., Chandy, R., Dorotic, M., Krafft, M. and Singh, S. S. (2010). Consumer
coreation in new product development. Journal of Service Research, 13(3), 283-296.
OCDE (1996). The Knowledge-Based Economy. Science, Technology and Industry Outlook,
Paris: OCDE.

86

A hypothesis driven tool to structurally embed user and

business model research within Living Lab innovation
tracks
Ruben DHauwers a, Olivier Rits a, Dimitri Schuurman b, Pieter Ballon a
a

iMinds-SMIT-VUB
iMinds-MICT-Ghent University
ruben.dhauwers@iminds.be

Abstract
Living Labs are a structured approach to open innovation and have a potential to focus on
the triple combination of technology, user and business model research. Hypothesis
driven living labs ensure that the living lab project remains process - and goal oriented.
Thus far, the iMinds Living Lab has been using the validation board of Ries (2010) to track
the validated learning of hypotheses around the user research aspects. However, the
validation board only takes into account user research hypotheses and learning and does
not incorporate business model research aspects. In this paper iMinds Living Lab
researchers introduce a hypothesis driven living lab framework (LLAVA matrix Living
Lab Assumption and Validation matrix) incorporating both user and business model
learning.
Keywords
Living Lab; Open Innovation; Lean Management, Business Model Innovation; SME
1 Introduction
Living Labs are often referred to as public-private-people partnerships (4Ps) (Westerlund
and Leminen, 2011). Based on a meta-review of the Living Labs literature, Schuurman
(2015) defines Living Labs as an organized approach (as opposed to an ad hoc approach) to
innovation consisting of real-life experimentation and active user involvement by means
of different methods involving multiple stakeholders, as is implied in the Public-PrivatePeople character of Living Labs. Moreover, he also concludes that Living Labs are
emanations of both Open Innovation and User Innovation practices, as external inputs,
including end-user contributions, are used to iteratively design and co-create the
innovation in development. This opening of the innovation process and the involvement
of external actors in a structural process have the potential to increase the value and
sustainability of the business model of the innovation (Baccarne et al., 2013).
Not many Living Lab initiatives combine the technological, user and business aspect of
innovation through open innovation approaches. Rits et al. (2015) point out that the use of
the Living Lab to explicate and validate an actual business model seldom occurs. The
approach suggested by Rits et al. (2015) implies the structural embedding of business
87

models in Living Labs leading to collaboration between different types of researchers and
viewpoints into the living lab platform.
Learning to grow a new venture or implementing business model innovation is not a
matter of ideation but of iterative experimentation (Thomke 2002). The selection of the
right tools and methods to structure and optimize such iterative innovation processes is
key to the success rate of a business in the ever-changing economy (Brem and Viardot,
2013). Living labs are complex partnerships, as they facilitate university-industry
relationships but also relationships between large companies, SMEs and start-ups. The
need to include not only several stakeholders (universities, large companies, SMEs and
start-ups) but also several types of researchers (technical, user and business researchers)
into the living lab process as discussed above, does however substantially increase the
complexity of the Living Lab projects.
As a result of the missing link between business modelling and living labs (Rits et al. 2015),
the different innovation tools available today either focus on strategy and business
modelling, or focus on the process of iterative innovation tracks and Living Labs
specifically. None of these tools take the combined effort of strategy/business modelling
and iteration/living lab into account.
This paper aims to introduce a hypothesis driven living lab framework, which incorporates
innovation track design and management and business model innovation allowing living
lab researchers to efficiently embed and link user and business modelling research. The
focus of the paper is on innovation of Living Labs with start-ups and SMEs as instigators.
2 Tools for hypothesis driven Living Labs
The best know business model innovation tool today is probably the business model
canvas proposed by Osterwalder (2010). The business model canvas is a strategic
framework to develop an innovative business model. This framework does not link to
living lab research explicitly nor does it provide clear guidelines on the process of
designing and especially iterating the business model, which often leads to a mere fillingin exercise as stipulated by Verrue (2014). So the business model canvas is more focused on
the strategy than it is on processes.
The Value Proposition Canvas (Osterwalder, 2015) is a derivative or rather subcategory of
the business model canvas, making the different sub-categories of the 2 above mentioned
components (user and value proposition) much more explicit. Coorevits et al. (2014)
compare the usage of the validation board (Ries, 2011) with the Value Proposition Canvas
in a Living Lab Framework. The validation board is proposed by Ries (2011) in the Lean
Startup where it is used to support start-ups and innovators in being focused on the
process of solving the problem or need of a customer through validated learning.
The Value Proposition Canvas (Osterwalder, 2015) focuses on the needs of the customers
(customer jobs, gains and pains) and the corresponding value proposition (gain creators,
pain relievers and products and services). Coorevits et al. (2014) conclude that the Value
Proposition Canvas addresses strategy, not processes, while the validation board addresses
processes with only limited incorporation of the strategy. Through tests within 4 SME
Living Lab cases, they conclude that the Value Proposition Canvas did not provide value
throughout the Living Lab project, as it focused more on the strategy and not on the
process. The validation board is a tool that identifies not only customer specific strategy
assumptions, but also forces researchers to link these to research steps for validation. As
such the validation board provided more value by guiding researchers and innovators
through the research decision-making processes within Living Labs. In short, the
Validation Board was suggested as a tool to develop hypotheses and to set (research) goals
within Living Labs.
88

Additionally, Coorevits et al. (2014) conclude that the validation board does not take into
consideration all aspects which influence the business model, such as customer touch
points, partners, revenue model and cost structure:
Using the Validation board, only the value proposition is researched .A framework to enable
validation based business models in Living Labs: integration of business model aspects such as
customer touch points, partners, revenue model and cost structure.
Breuer & Ketabdar (2012) propose the Business Modeling toolkit, which focuses on eight
basic components for validated learning. The eight components are quite similar to the
nine components of the Business Model Canvas. The Business Model Toolkit components
can be mapped one-on-one to a component of the Business Model Canvas, except for the
Capabilities component, which aggregates the Key Activities and Key Resources
components of the Business Model Canvas. So at least on the strategic side, they take the
same approach as the business model canvas. Breuer & Ketabdar (2012) have improved this
strategy-oriented approach by adding a process-oriented framework for tracking and
improving the maturity of each business model component.
Although the business model kit comprises both a strategy and a process focus, some
issues arise for use in a living lab innovation track. On the strategy side, some important
business model components are missing or at least not explicitly mentioned. The customer
need, addressed by Osterwalder (2015) and by Ries (2011) are both key components when
designing and performing user research. The aspect of incorporating competition and the
differentiation compared to the competition of an SME are core, as well, to understand the
business model. On the process side, the tool is focused rather on creating common
ground for teams and allowing for self-assessment based on maturity of assumptions. The
sustainability of business models is however not only determined by maturity of
validation.
3 Methodology
The LLAVA (Living Lab Assumption and Validation) matrix was developed based upon
the iMinds Living Lab experience with over 40 SME cases. The methodology utilized in
section 4 and 5 to set up the LLAVA matrix was based on the theory of action research.
Action research is a flexible spiral process, which allows actions (change, improvement)
and research (understanding, knowledge) to be achieved at the same time (Dick, 2002). In
this research, the researchers applied action research on the development of a hypothesis
driven tool to structurally embed user and business model research within Living Lab
innovation tracks. Initially, the researchers started from existing tools utilized for
managing Living Lab innovation tracks, but the continuous cycle of action and reflection
(Dick, 2002) helped the researchers to further develop the LLAVA matrix in section 4 and
5.
In order to receive initial insights in the usefulness of the LLAVA matrix in Living Lab
tracks, the researchers generated a discussion and conclusion in section 6 and 7. At the
moment, no complete SME/Start-up innovation tracks have followed the current design of
the LLAVA matrix. Therefore, in order to generate and analyse data, the participant
observation methodology (Yin, 2009) including an active observation and interference of
researchers was utilized. In practice, this consisted of 10 business model workshops guided
by iMinds Living Lab researchers utilizing the LLAVA matrix. Conclusions were drawn
out of these ten cases utilizing pattern matching between different workshops (Yin, 2009).
The sample of companies diverts from companies who are in a very early stage (less then
one year) to companies which have a more stable business model and which are in a less
exploratory phase. After the workshops, feedback was gathered on the impact the LLAVA
89

matrix had on the conversation. A limitation to the research is that the data is merely
focused on the strategic implications of the LLAVA matrix as the case studies were focused
on one single workshop and not on managing the progress of a Living Lab innovation
tracks due to the fact no entire Living Lab track was managed using the LLAVA matrix. For
further research case studies need to be done in order to investigate the effectiveness of the
strategic, process and assessment purpose of the LLAVA matrix as it will be mentioned in
section 5.
4 Results: LLAVA matrix: a Hypothesis Driven Living Lab tool with integration of
Business Models
The LLAVA tool was created on top of an aggregation of principles from Ries (2011), the
Osterwalder Value Proposition Design (2015), the business model matrix of Ballon (2007),
the business model canvas of Osterwalder (2010) and Porters five forces model (1985) and
translated into a set of strategic component. The action research methodology offered
spiral approach of action and reflection action research (Dick ,2002). The link of business
model references to the proposed hypothesis driven Living Lab framework can be found in
figure 1.
Pain&Relief&

Customer&
Hypothesis&

Pain&

Product&
&&
Service&
Gain&

Gain&Creator&

Problem&
Hypothesis&

Custo
mer&
Jobs&

Solu6on&&
Hypothesis&

Valida6on&Board,&Ries&
Value&proposi6on&Design&
(Osterwalder)&

Five&Forces,&Porter&

Exis6ng&
Compe6
tors&

Bargaining&
customer&

Bargaining&
supplier&

New&Entrant&

Subs6tute&

Business&Model&Matrix,&
Ballon&
Value&
Network&

Business&Model&&Canvas,&
Osterwalder&

Ac6vity&

Cost&

Cust.&
Rela6n&
Channe
l&

Customer&

Resour
ce&

Value&Proposi6on&

Value&
Proposi6on&

Partner&

Financial&
model&

Func6onal&
Architectur
e&

Revenue&

Figure 1: Link of business model references to the proposed hypothesis driven Living lab framework

The validation board (Ries, 2011) puts the customers at the core of its framework and
therefore focuses on customer hypothesis, (customer) problem hypothesis and solution
hypothesis. The approach is process oriented, more specifically for highly iterative (and
lean) innovation processes allowing for structured learning and pivoting (Coorevits, 2014).
Osterwalder (2015) introduced an alternative with the value proposition design. As
mentioned above, Coorevits (2014) found that it is less applicable to the process oriented
Living Lab trajectories, but the modules of pain and gain are included in the framework.
As discussed above, this process-oriented approach of the validation board is missing some
important strategic components that need to be taken into the innovation track at the
strategic level. Osterwalders (2010) business model canvas has the customer segment and
value proposition components in common with the Validation Board, but analysing the
other components, it was found that for most of the SME projects, one or more of these
components were not always relevant or important to the case at hand. Therefore, the
90

researchers decided to build a matrix based spread sheet format but add different business
model modules.
The researchers added different modules to the validation board based on utilizing a spiral
approach of reflection and action by initially testing the validation board, identifying
additional needs for other modules and subsequently trying those additional modules in
workshops with SMEs and instigators. The value proposition layer is similar to Ries (2011)
and with Osterwalder (2010) and (2015). In order to capture the marketing and positioning
of an SME, the iMinds Living Lab researchers replaced the value proposition by the
value promise. Entrepreneurs have an idea in mind, often based on a certain technology,
which is clear to the start-up/SME but not to the prospective client. Most customers will
not purchase a solution SMEs/start-ups offer because of the technical components, but for
the value the solution offers to them. Many start-ups tend to give a technical description of
their business idea. The goal of utilizing a value promise is to pitch the idea in a more
effective and simple way to enable entrepreneurs to sell and market the idea effectively.
Based on the spiral approach of action research, researchers identified there is a need to
make a clear division between the value promise and the solution. The solution segment in
LLAVA is determined by technical components as well as non-technical components, part
of the product and/or service. The solution refers to the functional architecture of Ballon
in the business model matrix (2007). The functional architecture comprises of the technical
systems composed of at least one building block (or module), governed by certain rules (or
intelligence) that interwork (or not) with other technical systems through predetermined
interfaces. The composition of the solution in the key modules and technical systems
enables the researcher and the entrepreneur to identify the differentiation of the
innovation compared to the competition. This division is less explicitly included in
Osterwalder (2010), even though the difference can be important in certain innovations.
The value network definition is an alternative to the broad market based approach of the
business model matrix of Ballon (2007), though the applicability is more adapted to living
labs and to start-ups and SMEs. Ballon (2007) is focused on creating a market overview, in
order to describe IT market from a researchers' perspective. The LLAVA matrix is applied
in Living Labs and on specific cases with SMEs and start-up, so therefore the purpose of
the value networking is different. The researchers goal is to understand the market
dynamics, more specifically: who creates value, who delivers value, who consumes value
and how is the value captured. Based on these elements, researchers get an overview of the
market dynamics in order to define the potential positioning of the innovation in the
market. Additionally, through the different value networks, the entrepreneur and the
business model researchers can identify potential sales channels..
Ballon (2007) included the financial model in the Business model matrix, which described
the revenue model and the revenue sharing model. Osterwalder also (2010) takes into
account the revenues model, where the pricing level and the pricing model are being
discussed. The researchers opted to utilize the definition of Willingness to pay, which
comprises of the pricing model and the pricing level, and in cases where revenue sharing is
applicable this section can be utilized. The action research has shown many start-ups and
SMEs face difficulties identifying their pricing model and pricing level, and thus this needs
to be included in the LLAVA matrix. It is important to note that the willingness to pay has
an important link with how pressing the customer need is and to the associated value the
SME/start-up promises to deliver. The value network is linekd to the willingness to pay,
especially the ability of the start-up/SME to capture the value.
One last, important, missing pillar in Ries (2011), Osterwalder (2010) and in Ballon (2007), is
the competition and the differentiation of a SME/start-up. Competition refers to the five
market forces of Porter (1985), which draws from five forces model. The five forces make up
the attractiveness of a market. The five forces are rivalry within the industry, threat of new
91

entrants, and threat of substitutes, bargaining power of suppliers and bargaining power of
buyers. The rivalry of the industry can help to identify the difficulties to enter the market
in case of many existing strong players and the need to differentiate compared to them in
order to be able to enter the market. In case there are several new entrants entering the
market it shows the market is attractive, but that differentiation can depend on how the
start-up differentiates compared to other start-ups who challenge the status quo in the
market. For some products and/or services one can find possible substitutes, which can
serve as an alternative to the specific service and/or product. In the LLAVA matrix the aim
is to identify what the differentiation can be compared to the existing and upcoming
companies in the industry and which alternatives exists as a substitute.
Taking these seven parameters into account, the researchers created the LLAVA matrix
(Figure 2). The LLAVA matrix has a similar spread sheet based lay out to the validation
board as Ries (2010), enabling to represent different eco-systems. Per column, a customer
segment can be filled out with its respective common need, value promise, solution, value
network, competition and willingness to pay.

Figure 2: LLAVA: Living Lab Assumption and Validation Matrix

5 The Purpose of LLAVA: Living Lab Assumption and Validation matrix

LLAVA has been created to serve 3 main purposes:
A strategic purpose by clarifying and designing the business model. Additionally,
the research questions around assumptions and facts, which need to be validated in
the Living Lab research, are generated around a workshop with the LLAVA matrix.
A process purpose in order to enable researchers to track and visualize the different
research results based on the validated assumptions.
An assessment purpose in order to analyse and challenge the business model of the
SMEs based on the 3 guidelines of focus, differentiation and coherence.
92

5.1 The strategy oriented purpose of LLAVA

Similar to the business model canvas (Osterwalder, 2010), the LLAVA is used as a tool to
create a common understanding and vision on the different key elements of the business
model and the strategic roadmap of the start-up/SME. The LLAVA matrix is adapted to
the specific components needed for the management of a Living Lab.
In order to kick-off the Living Lab track, the researchers facilitate a workshop based on the
LLAVA matrix. By putting their business model on paper, the instigator clarifies the
business model. Additionally, the LLAVA matrix ensures a common understanding
between the instigator team and the research team within the Living Lab.
Similarly to Ries (2011) a division is made between unknown aspects, assumptions and
facts. In contrast to the business modelling toolkit (Breuer, 2012), the iMinds Living Lab
researchers do not incorporate 5 maturity levels in order to simplify the process for the
entrepreneur and for the researchers.
When filling out the LLAVA matrix, researchers utilize a colour code indicating whether a
cell is an assumption (green), a fact (red) or whether a cell is completely unknown (blue) as
the example in figure 3 shows. Concerning the research questions, an assumption needs to
be validated and unknowns need to be explored.

Figure 3: LLAVA assumptions, facts and unknowns

The task of the researcher is to identify the assumptions and facts in the business model,
which need to be validated in the Living Lab. The output of the workshop is mainly a
clarification of the research questions around assumptions and facts.
5.2 The process oriented purpose of LLAVA

93

The second purpose of the LLAVA matrix is to manage and define the innovation track
through the Living Lab by the researcher. The LLAVA matrix is incorporated in the overall
management and process of the Living Lab:

First, it is utilized as a framework to track the learning of different research steps

concerning the business model. After each research step, LLAVA matrix the
content is updated together with the validation of the content (assumption vs. fact
vs. unknown) during STEERCOs.
Second, the LLAVA matrix is utilized to create an overview of the business model
in order to aid the research steps. The components and the content of LLAVA are
utilized to create topic guides for interviews, co-creation sessions, surveys and tests.

5.3 The assessment purpose of LLAVA

LLAVA serves as a tool to analyse business models throughout the different research steps.
During STEERCOs after each research step, the business model is analysed by business
model researchers. The analysis is utilized to challenge the instigator on the user inputs
and the alignment of the business model.
Three principles are utilized to analyse the business models of the instigator:
Focus: Does the SME/start-up have enough focus within the business model to
implement the business idea completely taking into account the principle of the
opportunity cost. Since resources are scarce relative to needs, the use of resources
for one activity prevents their use in others. Translated to the case of the
entrepreneur, this means that if he or she decides to target several customer
segments, will he or she be able to execute it completely in order to deliver the
intended value proposition? The role of the business model researcher is to identify
whether the entrepreneur is sufficiently focused.
Differentiation: Does the SME/start-up offer a sufficiently differentiated business
value for the client compared to competitors? What are the core
components/capabilities of the SME/Start-up that competitors do not have? Does
the customer perceive the value of the SME/start-up higher as the value of
competitors or substitutes? In short, what is the reason why this specific team of
innovators need to implement this solution, and not another team?
Coherence: In order for a business model to generate the desired business value,
the business model building blocks needs to be coherently designed. More
concretely, this means that the customer segments their needs need to be
coherently communicated in the value promise, which also needs to be solved as
promised through he solution in line with the communicated pricing model. The
positioning of the company in the value network needs to be in line with the other
components in order for the SME/start-up to deliver value and to capture the value.
The task of the business model researcher is to identify contradictions in the model.
6 Discussion
Based on the ten workshops targeted on the strategic purpose of the LLAVA matrix,
iMinds researchers were able to gather different inputs on the utilisation of the LLAVA
matrix based on the participant observation methodology (Yin, 2009) based on different
cases. The four main outcomes and points of improvement of the LLAVA matrix are
discussed below based on a concrete case of a workshop. The process oriented purpose
94

and the assessment purpose of the LLAVA matrix were not tackled, as it concerned only
one workshop, and not an entire Living Lab track.
i.

ii.

iii.

iv.

Making a distinction between assumptions and facts has proven to be challenging

but useful. The latter are easily mixed and entrepreneurs are often unaware of the
importance of the difference, which exposes them to the risk of making the wrong
assumptions, which might potentially lead to failure. In order to capture this value,
the researcher leading the workshop needs to be able to validate the facts and
assumptions an entrepreneur indicates, as the entrepreneur might be overly
positive about certain facts. Making a distinction between assumptions and facts
helps to identify the important research questions which were previously invisible
for the instigator. An example is the case of an entrepreneur setting up a loyalty
system with different local shops (Oliva Card). The entrepreneur did not
distinguish between facts and assumptions, and therefore went into an
implementation mode, while some assumptions were not validated. Through the
LLAVA matrix, the distinction between assumptions and facts became clear to
Oliva Card, as well as the need to validate the unclear assumptions.
The LLAVA matrix helps entrepreneurs to focus on a certain customer segment
and its business model during a workshop. Entrepreneurs face difficulties to
identify the focus of their business model and consequently of their start-up. Some
entrepreneurs try to target several customer segments with different business
models, which does not enable the to focus on the execution of a specific case. The
researcher has the important role to challenge the entrepreneur to focus. As an
example, Motosmarty is a start-up focusing on creating a Digital Driving Pass
enabling drivers to improve their driving behaviour and to receive reductions from
insurance companies based on their driving behaviour. Motosmarty initially had
many different customer segments and business models in mind, indicating a clear
lack of focus. The LLAVA matrix helped to get an overview of the complexity of
serving different customers and the extra effort required to do so. Combined with
efforts to challenge Motosmarty, they found a focus on one single business model,
which enabled them to focus.
During workshops with several team members, the LLAVA matrix sparked
different discussions on different opinions about directions between team
members. By asking for more concrete and specific use-cases, the discussion can be
sparked by the researcher, which evolves into more concrete use cases leading to
more focus. This helps to make the research questions of the Living Lab research
more concrete and is key to create a good outcome of the Living Lab. As an
example case we can mention Yagram, a start-up that designs solutions around
digital health facilitated by mobile technologies. During the workshop based on the
LLAVA matrix, there clearly were different, sometimes contradictory views on their
business model by different team members. The LLAVA matrix provided a clear
overview and discussion tool to facilitate the discussion in order to create a
common ground by asking for other specific use cases by the researcher.
As a point of improvement to the LLAVA matrix, different researchers indicated
that the value network could become rather complicated in case of many
stakeholders involved, which was the case in three workshops. The spread sheet
approach did not enable the researchers and the start-ups to create an overview of
the market dynamics. A specific example of such a case is a health care project:
Zebra Telemedicine. Zebra telemedicine aims to offer telemedicine solutions in
ambulances for improved stroke care. Zebra Telemedicine involves several
stakeholders in a complicated ecosystem (hospitals, ambulances, insurance
95

companies,). The business model researchers task is to create a structure and

model adapted to the case of the entrepreneur, which proves to be challenging. In
the future, the researcher will investigate further in what way value network
analysis can be combined with the LLAVA matrix in the case of complicated ecosystems.
7 Conclusions
The LLAVA (Living Lab Assumption and Validation) matrix is a proposed framework for
hypothesis led living lab tracks which includes business model learning. LLAVA is focused
on the strategy, process and assessment of user and business model learning in a living
lab trajectory. The LLAVA matrix was created out of the experience of over 40 SME Living
Lab projects combined with models and frameworks taken from literature based on action
research.
The LLAVA matrix has gone through a preliminary test with ten start-ups which we used
as cases in our research. The main outcomes were focused on the strategic purpose of the
LLAVA matrix. More specifically, the division between facts and assumptions, the ability
to focus and to be specific and the power to spark discussions have added to the quality of
the workshop. Researchers need to be aware they have an important task to properly
support the instigator on these three levels (strategic, process en assessment). As such the
LLAVA matrix aims to strike the right balance between being specific enough as to
efficiently enable the research team in offering that support to the instigator and being
high-level enough as to be applicable to the many different projects with a different
context.
This balance seems to be especially difficult for the value network aspect of the business
model. In the case of complicated ecosystems for example, the LLAVA matrix is
potentially more effective when combined with a value network analysis. Further work is
required to improve this aspect of the LLAVA matrix.
The iMinds researchers will continue to update the analysis and design of the LLAVA
matrix while applying the LLAVA matrix on entire Living Lab tracks and additionally on
how the value network analysis of complicated ecosystems can be supported through the
LLAVA matrix.
8 References
yvri 1,(2014), Towards a customer-centric tool for building value propositions, ENOLL,
2014
Ballon P. (2007), Business Modelling revisited: the configuration of control and value, Info,
Volume 5, Issue 9, Special Issues: The Redesign of Mobile Business
Breuer H., Mahdjour S. (2012) Lean Venturing: Entrepreneurial Learning to Model and
Grow New Business, ISPIM South-Korea 2012
Bastiaan Baccarne UGent, Dimitri Schuurman UGent and Constantijn Seys UGent(2013)
The XXIV ISPIM Conference : Innovating in Global Markets : Challenges for Sustainable
Growth, Proceedings.
Coorevits, L, and D. Schuurman. (2014) "Hypothesis Driven Innovation: Lean, Live and
Validate." XXV ISPIM Innovation Conference. 2014.
Dick, B. (2002) Action research: action and research [On line]. Available at
http://www.aral.com.au/resources/aandr.html
Eisenhardt, K. (1989) Building theories from case study research, Academy of
management review, Vol. 14, No. 4, pp. 532-550.
96

Niitamo, V. et al., 2006. State-of-the-Art and Good Practice in the Field of Living Labs.:
Proceedings of the 12th International Conference on Concurrent Enterprising: Innovative
Products and Services through Collaborative Networks. Milan, pp. 341348.
Osterwalder A., Pigneur Y., Clark T. (2010) Business Model Generation . Hoboken,
NJ:Wiley
Osterwalder, 2015, Value Proposition Design: How to Create Products and Service
Customers Want, Wiley
Porter, M. (1985) Competitive Advantage: Creating and Sustaining Superior Performance.
New York: Free Press.
Reason, P. & Bradbury, H., (2001) (Ed.) The SAGE Handbook of Action Research.
Participative Inquiry and Practice. 1st Edition. London: Sage
Ries, E., 2011. The Lean Startup: How Todays Entrepreneurs Use Continuous Innovation
to Create Radically Successful Businesses 1st. ed., New York: Crown Business.
Schuurman, D. (2015). Bridging the gap between Open and User Innovation?: exploring the
value of Living Labs as a means to structure user contribution and manage distributed
innovation (Doctoral dissertation, Ghent University).
Teece D. J. (2010) Business Models, Business Strategy and Innovation , Long
RangePlanning Vol. 43, 172-194
Thomke, S. (2002). Experimentation Matters. Unlocking the Potential of New
Technologies for Innovation, Harvard Business School Press: Cambridge, MA.
Verrue J., (2014), A critical investigation of the Osterwalder Business Model Canvas: an indepth case study
Westerlund, M. & Leminen, S., 2011. Managing the Challenges of Becoming an Open
Innovation Company: Experiences from Living Labs. Technology Innovation
Management Review, (October), pp.1925.
Yin, R. K. (2009) Case Study Research: Design and Methods. Thousand Oaks, California:
SAGE Publications, Inc.

97

Living Labs As Innovation Platforms:

The Key Constructs
Habib, C. a, Westerlund a, M. & Leminen, S. b
a

Carleton Universitys Sprott School of Business, Ottawa, Canada

b
Laurea University of Applied Sciences, Finland
Mika.Westerlund@carleton.ca

Abstract
Despite the growing popularity of using living lab as innovation platforms, little is known
about their characteristics. We use a qualitative research approach to identify constructs
that help us to understand the concept of living lab. We used theoretical constructs from
the literatures on user innovation, co-creation and living labs to analyze European
Network of Living Labs (ENoLL) membership applications. The results from the content
analysis of 40 applications revealed nine constructs that are characteristic to living labs: 1)
objective, 2) governance, 3) philosophy, 4) stakeholders, 5) funding, 6) advantages, 7)
communications, 8) infrastructure, and 9) methods. These findings provide new insight
that helps us to provide a research-based definition of living lab platforms.
Keywords: Living Labs; Platform; Definition; Constructs; ENoLL
1 Introduction
In todays rapidly changing global economy, economic success requires group creativity
that is facilitated through interactive processes (cf. Holst, 2007). The use of living labs (LLS)
has become increasingly popular over the last decade, because they offer a multiplestakeholder platform for innovation in real-life contexts. Although Thomas Knight coined
the term living laboratory in 1749 (Westerlund & Leminen, 2014), the roots of modern LLS
are often associated with MIT professor William Mitchells real home environment for
investigating the application of smart home systems in the day-to-day activities of humans
(Eriksson et al., 2005), for sensing, prototyping, refining, and validating technologies within
the context of real-life scenarios (ENoLL, 2010). However, many studies seem to ignore
American scholars who experimented with LLS before they were associated to MIT (cf.
Moffat, 1990; Tarricone, 1990; Bajgier et al., 1991).
Since its conception, LLS has evolved into many fields of research and applications. The
literature has reviewed living lab concepts, methodologies, and research streams (cf.
Flstad, 2008; Dutilleul et al., 2010; Fulgencio et al., 2012; Westerlund & Leminen,
2014). Nonetheless, scholars in the field not only disagree on the definition of LLS
(Nystrm et al., 2014) but also on the components that make LLS unique from other
innovation platforms (DellEra & Landoni, 2014). Leminen (2013) argues that the lack of a
proper definition is the cause of disconnected research. Multiplicity of definitions has
limited the understanding of innovation practices in LLS. Thus, there is a need for
research on how LLS view their operations.

98

This study aims to identify the key constructs of LLS by using a qualitative research
approach. We review previous literature on LLS and compare labs with user innovation
and co-creation for the purpose of identifying constructs by which LLS can be examined in
terms of their defining characteristics. We use these constructs to analyze 40 ENoLL
membership applications in order to provide a research-based definition of living lab
platforms. The derived constructs help with understanding LLS as innovation platforms.
The study concludes with implications derived from our analysis.
2 Literature Review
2.1 User Innovation
More and more companies are shifting the task of revealing and understanding user needs
to users themselves. By providing users with innovation toolkits and various resources,
companies can outsource the innovation activity to customers and other stakeholders, and
bundle these actors into the companys own product development process (von Hippel &
Katz, 2002; Bogers & West, 2012). Toolkits can be introduced into user communities, i.e.
groups of users who share and disseminate information about a particular good
(Parmentier & Gandia, 2013), and therefore put the users to work to harness new and
reliable innovation (Sawhney & Prandelli, 2000).
To encourage participation and contribution, companies must support users intrinsic and
extrinsic motivations. The former is the internal gratification a member receives from
participating/achieving a goal within the community, and the latter refers to the external
forces that encourage participation regardless of intrinsic presence. Extrinsic motivation
include, e.g., recognition by the firm (Jeppesen & Frederiksen, 2006), peer reputation
(Hertel et al., 2003), monetary incentives (Jeppesen & Lakhani, 2010) and reciprocity of
solutions. In addition to motivational factors, proper leadership can steer the evolution of
projects and choose the best fitting solutions. Despite hierarchical coordination possibly
dispiriting the intellectual creativity, such governance structure needs to be in place to
allocate roles and tasks to the members (Bonaccorsi & Rossi, 2003).
The most common problem companies are facing when utilizing user innovation is how to
create a business model to profit from user innovation (Franke & Shah, 2003). To this end,
proprietary business models can attempt to solicit license agreements from the innovators
(West & Gallagher, 2006). Management of intellectual property (IP) is central to
controlling knowledge and determining ownership of the innovation (Bogers & West,
2012), especially since strong IP regimes by the firm can retard the innovation spirit of the
user community (von Hippel & Katz, 2002).
2.2 Co-Creation
Co-creation extends the user innovation process by appropriating ideas from customers
and stakeholders to enhance the product and create new experiences (von Hippel &
Oliveira, 2011). It engages participants in collaboration to develop a we competency
rather than a differentiated you and I interaction (Lee et al., 2012; DeFillippi & Roser,
2014). This means working together and consolidating resources over a network
(Gassmann et al., 2010). Customers participating in co-creation may not receive direct
social or economic value (Chen et al., 2012). Rather, intrinsic factors such as enjoyment
(Fuller et al., 2007), a sense of belonging (Zhang, 2010), or potential career advancement
(Wasko & Faraj, 2005) contribute to their participation in co-creation.
Co-creation consists of five areas: co-ideation, co-evaluation, co-design, co-test, and colaunch (Russo-Spena & Mele, 2012). Co-ideation means that members propose innovative
99

ideas to the community, which are then discussed and refined. Co-evaluation focuses on
the appraisal of the ideas; high-ranking ideas are reviewed by top management for
business potential and passed onto others to determine the costs and benefits of
implementation. Co-design is the implementation of approved ideas and requires
resources such as tool-kits and knowledge. Co-testing helps refine the new product and
gain feedback before launching to market; the pre-commercialized product is tested,
refined, and presented iteratively until it reaches satisfactory levels. Finally, co-launch
means that the product is released to market and will have early adopters who promote it
via word-of-mouth.
Lee et al. (2012) argue that co-creation improves the architecture of products (resulting in
better quality) and lowers the costs of production. Due to the parallel nature of
collaborative development (cf. Russo-Spena & Mele, 2012), the product lifecycle is
shortened allowing for faster launch and increased speed to market (DeFillippi & Roser,
2014). In addition, the diversified collaborative network enables organizations to become
more efficient and agile for rapid scaling (Adler et al., 2011). Furthermore, co-created
innovations have lower risk of market failure because they are associated with higher
customer satisfaction, positive word-of-mouth, and a lower likelihood of customers
seeking out competitive solutions.
2.3 Living Labs
LLS are an innovation approach that benefits the creation of products and services
(Liedtke et al., 2012). Building on co-creation, LLS provide physical and organizational
infrastructures (Ponce de Leon et al., 2006) and a methodology to coordinate the
experimentation process within the test environment (Almirall & Wareham, 2011). LLS are
based on user-driven approach and the open involvement of many stakeholders (Nystrm
et al., 2014). LLS engage diverse members to collaboratively undertake projects and
develop and validate innovations (De Ryuter et al., 2007; Schuurman et al., 2011;
Westerlund & Leminen, 2011). Trust is necessary to facilitate the equal and fair exchange of
knowledge, resources, and efforts (Leminen & Westerlund, 2012).
LLS give insight into consumers daily interactivity with firms offerings in the real-life
contexts (Mulder, 2012). Prior research on LLS (e.g., Pierson & Lievens, 2005) analyzes the
activities of consumers; especially their uses of goods in relation to other market offerings
that may be disjoint (Liedtke et al., 2012). Information about usage patterns are often
collected through the digital network (Intille, 2002), and analyzed to identify patterns and
opportunities (Edwards-Schachter et al., 2013). Members are encouraged to socialize,
suggest ideas, and engage in innovation development (cf. Russo-Spena & Mele, 2012). The
approach mitigates the risks associated with market commercialization (Liedtke et al., 2012)
and result in sustainable advantages for the company (Mattson, 2010).
Users that participate in LLS are selected from consumer groups, lead user communities,
research organizations, or firms employees (Niitamo et al., 2012). They are not seen as
passive respondents (Schuurman & De Marez, 2012) and are more than an object for
testing and feedback (Schaffers et al., 2007). The living lab provides them with resources to
convert their ideas into products and services (Sanders & Stappers, 2008). The industry
partners take on the role of designers and join LLS to access external ideas provided by the
others. They use the labs resources, networks, and techniques to find opportunities and
develop solutions that meet the needs of users (Levn & Holmstrm, 2008). Researchers
are stakeholders who focus on the generation of knowledge (DellEra & Landoni, 2014).
LLS offer benefits to participants, including networking opportunities and access to
funding and resources (Niitamo et al., 2012). Research conducted with LLS often yields
unique knowledge (Dutilleul et al., 2010). The lab carries out research, development, and
100

experimentation with products and services (Schaffers & Turkama, 2012). Consumers
know their needs and preferences while firms have the resources and know-how to
produce goods (OHern & Rindfleisch, 2009). Thus, the lab collects data from users
(Flstad, 2008) to study behavior and co-create outcomes for the society (Kanstrup et al.,
2010). Such knowledge can validate the innovation and ensure initial demand for the
product prior to commercialization (Almirall & Wareham, 2011). According to Kreborn et
al. (2010), the intangible activities and values (e.g., employee support, supplier value,
managerial tasks, and societal value) that help business develop and monitor their wellbeing are part of the living lab mandate.
2.4 Common Constructs
A comparison of the three reviewed innovation concepts reveals that they share six
defining constructs:
(1) Stakeholders: Parties who are involved in the innovation process.
(2) Objectives: The advantageous benefits of the output from the innovation process.
(3) Governance: The manner in which the decisions in the innovation process are
made.
(4) Tools: The resources required to carry out innovation activities.
(5) Motivation: The reasons why stakeholders participate and the techniques used to
promote participation.
(6) Business Appropriation: The direct or indirect means to capture monetary value
from the innovation outputs.
3 Methods
According to Baxter and Jack (2008), case study research can facilitate the exploration of
LLS allowing for multiple facets to be revealed especially when little is known of the
phenomenon or its boundaries are unclear. A case study approach can yield theory that is
unified and grounded in practice (Eisenhardt, 1989). Thus, we use case studies and content
analysis (CA) on the text generated from the cases. CA is a systematic technique used to
evaluate qualitative content by converting textual data into a numerical form that can be
subjected to quantitative analysis (Wolfe et al., 1993).
This research was limited to the qualitative data extracted from 2011/2012 ENoLL
applications which consisted of: 1 Australian, 4 Belgium, 2 Colombian, 1 German, 1
Denmark, 10 Spanish, 5 French, 2 Greek, 1 Irish, 4 Italian, 2 Mexican, 1 Polish, 1 Saudi, 1
Slovenian, 2 Turkish and 2 British data sets. We narrowed the data set from 332 cases down
to 40 by focusing on labs that had both application and profile completed. First, we
prepared detailed write-ups of cases (within-case analysis) to summarize relevant
information. Then, we used CA on the write-ups to find themes. For CA, we conducted
manual pre-editing of the data to simplify sentence structures into singular context
phrases and convert words into clearly defined nouns. We developed the coding rules
used to observe the units within the text by constructing an Excel macro formula:
[=OR(IF(ISNUMBER(SEARCH("KEYWORD", A2)),1,0))]
The macro was used to group phrases based on the specified key word. A group termed
OTHER was added to each search for two purposes: (i) highlight phrases that were not
categorized, and (ii) highlight phrases that were categorized multiple times. Using the
phrases that were categorized into their respective themes, we were able to further explain
each constructs composition and count the occurrences. Enfolding literature step was
used to connect the literature to the findings from the research. This step involved
101

determining what is similar and conflicting, and why such variances exist. By making the
connections, we could assure that the results are correct and descriptive.
4 Results
After an analysis of the data, it was apparent that the literature-provided constructs
required modification. Whereas some of the constructs found in the data were similar with
the literature (stakeholders, objective, governance), new constructs (philosophy, funding,
advantages, communication, infrastructure, methods) turned out to be useful in
understanding LLS (Table 2).
Construct
Objective

Definition
Scope
The positive impact that the Collaboration, social impact,
innovation output is expected to business
development,
produce
economic development, user
impact, test bed, framework
Governance
A structural or procedural model by Managerial process,
which decisions for the innovation managerial structure
projects, process or organization are
made
Philosophy
Mindset of the organization that is Innovation
culture,
reflected in their level of openness and intellectual property rights
collaboration
Stakeholders
Entities that add value to the living lab Participants and their role
Funding
The means by which the living lab Public
funding,
private
financially supports its innovation funding, revenue stream of
activities
labs business model
Advantages
The benefits the stakeholders gain Product outcome, framework,
from
their
membership
and social
value,
business
participation within the living lab
development,
validation,
resources,
networking,
knowledge, investment and
marketing
Communication The channels, technology and Online
presence,
media
techniques
used
to
network presence,
person-to-person
stakeholders for information exchange interaction
Infrastructure
The
necessary
resources
and Software tools
specialized equipment required to hardware, sensors, facilities
carry out the innovation activities
Methods
The procedural steps used for the Attracting participants, ethics,
inception,
development
and motivational rewards, user
deployment of innovation
support, data collection, idea
generation, design, testing, and
commercialization
Table 1: Emergent constructs from LLS cases

Appendix 1 illustrates the relative occurrences of scope within each emergent construct.
Objectives
The studied LLS develop innovations by the communal effort of various actors
(Collaboration). They prioritize teamwork and establish joint operations to mutually
102

manage incubation space, state-of-the-art technology, and knowledge databases for

optimal creativity, cost-reduction, and ecosystem. They pursue social impact on regions by
improving citizen involvement in the community, developing technologies that better
meet the needs, and building up urban infrastructures. Moreover, labs offer business
development to companies by creating resources and services (e.g., product research,
incubation space, market trend analysis, and education). They foster employment and
entrepreneurship (Economic Development), creation of customized and holistic solutions,
and the development of digital infrastructure. Lastly, they provide test beds and a
framework for experimenting and testing products in real settings with users (Figure 1).
Governance
It was difficult to identify a governance structure for LLS, but the responsibilities of
governance group include: setting the labs vision, making investment decisions, managing
IP, organizing activities, appointing roles, maintaining lab infrastructures, and planning
research. The governance group ensures that the activities meet the goals by monitoring
the performance of the lab. They take on the administrative and managerial work. The
governing group is also responsible for the project level decisions. They select the projects
to pursue and assign the appropriate members to oversee and run the activities and create
user-centric research methodologies. The legal forms of LLS in their respective order of
highest occurrence are: Private, Public-private partnership, Public-private-people
partnership, Public, and undefined (Figure 2).
Philosophy
The methods of managing IP in LLS are: Consortium Agreement, OEM, Licenses, Opensource, Case-by-case, Law, and Other. LLS set forth rules and regulations regarding the
use, sharing, and licensing of IP prior to the initiation of a project within the Consortium
Agreement. The agreements can outline the distribution of cost and gains for each
member depending on their role and investment in the developments. These set of rules
must be signed by all members. Labs can also give the originators (OEM) full rights to
determine the extent of the IPs usage, or manage IP for each project between the
participating members (Figure 3). LLS innovation culture encourages collaborative work to
achieve innovation and other goals. Labs ensure that the members respect one another
and share knowledge. They reduce barriers through free access to knowledge, and use
open standards to enable integration and access to free tools (Figure 4).
Stakeholders
Stakeholders could not be efficiently analyzed because the data were not properly
formatted for CA. However, subjective review of the cases suggested the diverse
involvement of companies, universities, unions, governments and public bodies, financers,
civic organizations and associations.
Funding
The majority of funding comes from government grants and private investments. In
addition, consulting provides revenues when the living lab receives payment for services
rendered to third parties. The services offered in living lab vary, because each lab has a
different focus. LLS offer consulting including digital marketing, data collection and
analytics, training, and product evaluations. Moreover, LLS produce income from their
outputs in the form of royalties and/or sales. However, the members only make profit
when they succeed in the commercialization of the products. Labs also lease their
resources such as facilities and equipment to third parties for event purposes, lab research,
or development (Figure 5).
Advantages
LLS offer various benefits to their members: business development, knowledge, resources,
networking, validation, marketing, social value, and investments. Supportive activities aid
lab members to achieve business goals (Business Development). Members benefit from
103

management support, advisory teams, project development, other members

experiences/expertise, and education. Members can take advantage of labs research
facilities, incubation space, technologies, and knowledge content. Members can make new
connections to access new industries or markets. Through the structured process that
enables collaborative work (Framework), LLS help accelerate the development of products
at low cost, higher quality, and establish an initial market presence. Their associated
activities and ecosystem create visibility for members brands, and add legitimacy to
members businesses (Figure 6).
Communication
Two-way communication aims at achieving open dialogue for collaborative work in LLS,
and helps for brainstorming ideas and gaining feedback from members. LLS also need to
consistently update the members of their progress and ongoing activities. Labs use
communication for self-promotion to brand, legitimize, and gain public recognition.
Furthermore, the technology used for communication serves as management tools, e.g., a
database for hosting shared content, tracking project tasks, and collecting and appraising
ideas. Figure 7 shows the online, media, and in-person modes of communication. Figure 8
details online channels and Figure 9 details media channels. Figure 10 details in-person
channels. Figure 11 illustrates the reasons for communication.
Infrastructure
LLS have five types of infrastructure necessities: facilities, networks, hardware, software,
and sensors. All LLS appear to have facilities, dedicated or shared, to host in-person
activities such as events, workshops, and testing in a test-bed. Facilities are either owned
by the lab or a stakeholder who permits their use. Information technology infrastructure
(networks) includes servers used to host the web technologies and data that facilitate
collaboration. Hardware, software, and sensors vary from lab to lab depending on their
intended use. In particular, sensors are used within the test environment for observing
user behavior and collecting usage data.
Methods
LLS gain users from associations, events, and random sources such as hot-spots or housing
authorities. Before their involvement, the lab informs the users of their role and project
objectives, and gain written, voluntary consent. Using lead users as influencers, and
extrinsic rewards, the lab motivates the users to contribute to the project. The lab also
provides training and tools. During the idea generation, the users and other members
discuss problems, brainstorm solutions, and set initial requirements. Universities and
small companies often convert the requirements into designs and prototypes. Under the
guidance of research experts the solutions are tested with users in real-life environments
where data is collected through monitoring technology, digital activity logs, and
surveys/interviews. Academics then analyze the data to understand the impact of the
solutions. Labs often leave the commercialization efforts to companies, but can use its
ecosystem and experts to promote and adopt the solutions (Figure 12).
5 Conclusion
The nine constructs (objective, governance, stakeholders, philosophy, funding,
advantages, communication, infrastructure, methods) provide a multi-faceted perspective
to understanding LLS. Although such constructs could be considered common to
innovation platforms in general, they provide a thematic perspective to examining and
describing LLS that could be later compared to other innovation platforms. Using the
constructs, we can now define LLS in a new manner: A living lab is a sociotechnical platform
with shared resources, collaboration framework and real-life context, which organizes its
stakeholders into an innovation ecosystem that relies on representative governance, open104

standards, and diverse activities and methods to gather, create, communicate, and deliver new
knowledge, validated solutions, professional development, and social impact. New knowledge
refers to identified problems, ideas for solutions, novel information, content generation,
and (scientific) discoveries. Validated solutions include the co-creation, testing, and
validation of solutions. Researchers can use this definition to guide their research, refine
the constructs, and contribute to even a more precise definition of LLS.
Although the new definition of living lab platforms is based on an analysis of how LLS
describe themselves in public documentation, it is a significant contribution to the current
literature. The study provided further knowledge of the constructs that give rise to the
definition. That is, common constructs drawn from innovation literatures (user
innovation, co-creation, living labs) that are associated with living labs were only partially
supported by the empirical study. For instance, the empirical study revealed nine key
constructs as opposed to six derived from the literature review, and only three of them
matched perfectly. Moreover, the study revealed communication as an important
construct that previous research has not emphasized (cf. Mulder et al., 2008). Surprisingly,
the study did not highlight stakeholder roles, user engagement, and real-life contexts as
the key constructs of LLS (cf. Nystrm et al., 2014). This may be related to the fact that
applications reflected an early stage of living lab activity, and that the study searched for
common aspects within the three literature streams, whereas real-life context is a unique
aspect of living labs. The research helps researchers, entrepreneurs and managers
understand the advantages of LLS (business development support, access to resources and
partnership networks, as well as product ideas, validation, and commercialization), and
join a LLS that provides a particular benefit. Finally, stakeholders can look into the
implications of each construct and theme to form LLS that best suits their goals and is
aligned with their society/stakeholders to optimize their innovation process.
Limitations of the research included restricted number of analyzed cases due to resource
constraints. A larger data set could refine the discovered constructs as descriptors of LLS
and lead to a more detailed explanation of the results. Likewise, more recent data
consisting of 2013-2014 may be more accurate and reliable compared to the 2011-2012 set
used. Interpretation of the data is dependent on the researchers understanding of the
subject. Thus, content analysis was used to limit the bias of human interpretation.
However, codifying a semantic category counter based on the frequencies of occurrences is
difficult due to diversity of the cases. Data that are nouns, such as names, require
additional work to determine their equivalent pronoun (e.g., user, designer). This problem
occurred for the infrastructure and stakeholder constructs. This issue may also be related
to the fact that we were unable to identify living labs where users in a dominant role (cf.
Leminen, 2013). The data requires extensive formatting prior to analysis, which means
heavy investment of time and effort.
We propose the following future work to be done: (i) the discovered constructs could be
confirmed using a larger set of data, (ii) future researches could focus on the individual
constructs to deepen the knowledge of LLS, and (iii) the constructs may be applied to other
innovation concepts to examine unique patterns in those concepts, and (iv) other type of
data should be incorporated to avoid cause-effect problems associated with analyzing
characteristics of members based on their membership applications.

105

6 List of references
Adler, P., Heckscher, C. & Prusak, L. 2011. Building a collaborative enterprise. Harvard
Business Review, 89(8): 94-101.
Almirall, E. & Wareham, J. 2011. LLS: Arbiters of mid- and ground-level innovation.
Technology Analysis & Strategic Management, 23 (1): 87-102.
Bajgier, S.M., Maragah, H.D., Saccucci, M.S. & Verzilli, A. 1991. Introducing Students to
Community Operations Research by Using a City Neighborhood as Living Laboratory.
Operations Research, 39(5): 701-709.
Barcenilla, J. & Tijus, C. 2012. Ethical issues raised by the new orientations in ergonomics
and LLS. Work: A Journal of Prevention, Assessment and Rehabilitation, 41: 5259-5265.
Baxter, P. & Jack, S. 2008. Qualitative Case Study Methodology: Study Design and
Implementation for Novice Researchers. The Qualitative Report, 13(4): 544-559.
Bogers, M. & West, J. 2012. Managing Distributed Innovation: Strategic Utilization of Open
and User Innovation. Creativity & Innovation Management, 21(1): 61-75.
Bonaccorsi, A. & Rossi, C. 2003. Why Open Source software can succeed. Research Policy,
32(7): 1243-1258.
Chen, L., Marsden, J, & Zhang, Z. 2012. Theory and Analysis of company sponsored value
co-creation. Journal of Management Information systems, 29(2): 141-172.
DeFillippi, R., & Roser, T. 2014. Aligning the co-creation project portfolio with company
strategy. Strategy & Leadership, 42(1): 30-36.
DellEra, C. & Landoni, P. 2014. Living lab: A methodology between user-centred design
and participatory design. Creativity and Innovation Management, 23(2): 137-154.
De Ryuter, B., van Loenen, E. & Teeven, V. 2007. User Centered Research in
ExperienceLab. European Conference on Ambient Intelligence 2007, Darmstadt,
Germany: 305313.
Dutilleul, B., Birrer, F.A.J. & Mensink, W. 2010. Unpacking European LLS: Analysing
Innovations Social Dimensions. Central European Journal of Public Policy, 4(1): 6085.
Edwards-Schachter, M., Matti, C. & Alcantara, E. 2013. Fostering Quality of life through
Social Innovation: A Living Lab Methodology Study Case. Review of Policy Research,
29(6): 672-692.
Eisenhardt, K. 1989. Building Theories from Case Study Research. Academy of
Management Review, 14(4): 532-550.
ENoLL. 2010. William Mitchell, father of the Living Lab concept, passed away this
weekend. Available at http://www.openlivinglabs.eu/news/bill-mitchell-father-living-labconcept-passed-away-weekend
Eriksson, M., Niitamo, V. & Kulkki, S. 2005. State-of-the-Art in Utilizing LLS Approach to
User-centric ICT innovation A European approach. Lule University of Technology.
Flstad, A. 2008. LLS for innovation and development of communication technology: a
literature review. Electronic Journal for Virtual Organizations and Networks, 10: 99131.
Franke, N. & Shah, S. 2003. How Communities Support Innovative Activities: An
Exploration of Assistance and Sharing among End-Users. Research Policy, 32(1): 157178.
Fuller, J., Jawecki, G. & Mhlbacher, H. 2007. Innovation creation by online basketball
communities. Journal of Business Research, 60(1): 6071.
Fulgencio, H., Le Fever, H. & Katzy, B. 2012. Living Lab: Innovation through Pastiche (a
research linking disparate and discorded ontology), eChallenges 2012 Conference: 1-8.
Gassmann, O., Enkel, E. & Chesbrough, H. 2010. The future of open innovation. R&D
Management, 40(3): 213-221.
Hertel, G., Niedner, S. & Herrmann, S. 2003. Motivation of Software Developers in Open
Source Projects: An Internet-Based Survey of Contributors to the Linux Kernel. Research
Policy, 32(7): 11591177.
106

Holst, M. 2007. Enabling boundary-crossing collaboration for innovation Issues for

collaborative working environments. Lule University of Technology.
Intille, S.S. 2002. Designing a Home of the Future. IEEE Pervasive Computing. April-June:
7682.
Jeppesen, L. & Frederiksen, L. 2006. Why do Users Contribute to Firm-Hosted User
Communities? The Case of Computer-Controlled Music Instruments. Organization
Science, 17: 4563.
Jeppesen, L. & Lakhani, K. 2010. Marginality and Problem Solving Effectiveness in Broadcast Search. Organization Science, 21: 10161033.
Kanstrup, A., Bjerge, K. & Kristensen, J. 2010. A Living Laboratory Exploring Mobile
Support for Everyday Life with Diabetes. Wireless Personal Communications, 53: 395-408.
Kreborn, B., Eriksson, C., Sthlbrst, A. & Scensson, J. 2010. A Milieu for Innovation
Defining LLS. Lulea University of Technology, Sweden.
Kreborn, B., Anders, B., Lassinantti, J., Davoli, L., Kuenen, S., Palmquist, L., Parns, P.,
Sthlbrst, A., Synnes, K., Wennberg, P. 2012. User Toolkits for Citizen-centric Mobile
Service Innovation. Proceedings of the e-Challenges Conference.
Lee, S, Olson, D. & Trim, S. 2012. Co-innovation convergences, collaboration and cocreation for organizational value. Management Decision, 50(5): 817-831.
Leminen, S. 2013. Coordination and Participation in Living Lab Networks. Technology
Innovation Management Review, 3(11): 514.
Leminen, S., Westerlund, M. & Nystrm A.-G. 2012. LLS as Open Innovation Networks.
Technology Innovation Management Review, 2(9): 6-11.
Levn, P. & Holmstrm, J. 2008. Consumer co-creation and the ecology of innovation: a
living lab approach. The 31st Information Systems Research Seminar in Scandinavia.
Liedtke, C., Welfens, M., Rohn, H. & Nordmann, J. 2012. LLS: user-driven innovation for
sustainability. International Journal of Sustainability in Higher Education, 13(2): 106-118.
Mattson, J. 2010. Developing a strategic abstraction tool for service innovation. Journal of
Strategic Marketing, 18: 133144.
Moffat, A.S. 1990. China: A Living Lab for Epidemiology. Science, 248: 553-555.
Mulder, I., Velthuas, D. & Kriesn, M. 2008. The Living Labs Harmonization Cube:
Communicating Living Lab Essentials. eJOV Exceutive The Electronic Journal for
Virtual Organizations and Networks. 10: 1-14.
Mulder, I. 2012. Living Labbing the Rotterdam Way: Co-Creation as an Enabler for Urban
Innovation. Technology Innovation Management Review. September: 39-43.
Niitamo, V., Westerlund, M. & Leminen, S. 2012. A small-firm perspective on the benefits
of LLS. Technology Innovation Management Review, September: 44-49.
Nystrm, A.-G-, Leminen, S., Westerlund, M. & Kortelainen, M. 2014. Actor roles and role
patterns influencing innovation in LLS. Industrial Marketing Management, 43: 483-495.
OHern, M. & Rindfleisch, A. 2009. Customer co-creation: A typology and research agenda.
Review of Marketing Research, 6: 84106.
Parmentier, G., & Gandia, R. 2013. Managing Sustainable Innovation with a User
Community Toolkit: the case of the video Game Trackamaion. Creativity & Innovation
Management, 22(2): 195-208.
Pierson, J. & Lievens, B. 2005. Configuring LLS for a 'thick' understanding of innovation,"
Ethnographic Praxis in Industry Conference.
Ponce de Leon, M., Eriksson, M., Balasubramariam, S. & Donnelly, W. 2006. Creating a
distributed mobile networking testbed environment through the LLS approach.
Testbeds and Research Infrastructures for the Development of Networks and
Communities, 2nd International Conference on TRIDENTCOM.
Russo-Spena, T. & Mele, C. 2012. Five-Co-s in innovating: a practice-based view. Journal
of Service Management, 23(4): 527-553.
107

Sanders, E., & Stappers P. 2008. Co-creation and the new landscapes of design. CoDesign,
4(1): 518.
Sawhney, M. & Prandelli, E. 2000. Communities of Creation: Managing Distributed
Innovation in Turbulent Markets. California Management Review, 42: 2454.
Schaffers, H., Cordoba, M.C., Hongisto, P., Kallai, T., Merz C. & van Rensburg, J. 2007.
Exploring business models for open innovation in rural LLS. 13th International
Conference on Concurrent Enterprising (ICE), Sophia-Antipolis, France.
Schaffers, H. & Turkama, P. 2012. LLS for Cross-Border Systemic Innovation. Technology
Innovation Management Review. September: 25-30.
Schuurman, D. & De Marez, L. 2012. Structuring User Involvement in Panel-Based LLS.
Technology Innovation Management Review. September: 31-38.
Schuurman, D., De Moor, K., De Marez, L. & Evens. T. 2011. Living Lab research approach
for mobile TV. Telematics and Informatics, 28: 271282.
Tarricone, P. 1990. A tale of two laboratories. Civil Engineering. 6(7): 50-53.
Wasko, M., & Faraj, S. 2005. Why should I share? Examining social capital and knowledge
contribution in electronic networks of practice. MIS Quarterly, 29(1): 3557.
West, J. & Gallagher, S. 2006. Challenges of Open Innovation: The Paradox of Firm
Investment in Open-Source Software. R&D Management, 36: 319331.
Westerlund, M. & Leminen, S. 2011. Managing the Challenges of Becoming an Open
Innovation Company: Experiences from LLS. Technology Innovation Management
Review, 1(1): 19-25.
Westerlund, M. & Leminen, S. 2014. The multiplicity of research on innovation through
LLS. XXV ISPIM Conference, Dublin, Ireland.
Wolfe, R., Gephart, R., & Johnson, T. 1993. Computer-facilitated qualitative data analysis:
potential contributions to management research. Journal of Management, 19(3): 637660.
Von Hippel, E. & Katz, R. 2002. Shifting Innovation to Users via Toolkits. Management
Science, 48(7): 821-837.
Von Hippel, E., & Oliveira, P. 2011. Users as service innovators: The case of banking
services. Research Policy, 40(6): 806-818.
Zhang, Z. 2010. Feeling the sense of community in social networking usage. IEEE
Transactions on Engineering Management, 57(2): 225239.
Appendix 1 Illustrations of the results
Figure 1 Objectives
25
20
15
10
5
0

108

Figure 2 Governance/legal structures

35
30
25
20
15
10
5
0

Figure 3 IPR management techniques

35
30
25
20
15
10
5
0
Agreement

OEM

License

Open

Case-by-case

Law

Other

109

Figure 4 Innovation culture

60
50
40
30
20
10
0
Collective
Effort

Accepting Open Source Agreement

Members

Open
thinking

other

Figure 5 Revenue streams

60
50
40
30
20
10
0
Consulting

Royalties

Rent

Membership
Fees

education

Other

110

Figure 6 Advantages to members

20
18
16
14
12
10
8
6
4
2
0

Figure 7 Modes of communication

45
40
35
30
25
20
15
10
5
0
Online

Media Presence

Physical Presence

Misc

111

Figure 8 Online communication channels

60
50
40
30
20
10
0
Website Social Media

Virtual

Blogs

Forums

Mobile

Other

Publications

Ads

radio

magazines

other

Figure 9 Media channels

25

20

15

10

0
tv

press

112

Figure 10 In-person communication channels

30
25
20
15
10
5
0

Figure 11 Reasons for communication

40
35
30
25
20
15
10
5
0

113

Figure 12 Methods categories used for innovation

25
20
15
10
5
0

114

Getting personal
Exploring the usage of persona
in order to optimize the involvement
of a living lab panel
Sara Logghe a
a

iMinds-MICT-Ugent
sara.logghe@iminds.be

Abstract
iMinds Living Labs started with living lab research in 2009. Living lab research involves
gathering user feedback on innovations implemented in a real-life context (Eriksson et al.,
2005). This can be facilitated by means of a panel-based approach (Schuurman et al., 2012).
In order to keep a panel motivated for participating in living lab research it can be
beneficial to generate a sense of belonging to a community. Logghe et al. (2014) examined
the motivations and behavior of the panel members and concluded that there are four
groups of panel member types, each with their own motivations and behavior patterns.
But how can a living lab get to know its panel members better? How can every panel
member be approached in their preferred way? How can every panel member be
stimulated to keep on participating in living lab research? How can a community feeling
be created? In order to gather more information about each panel member type, we
developed a four way segmentation of the panel which we translated into four distinct
persona. These persona will be used as a basis for community building, a future panel kit,
experimenting with research approaches, supplemented with other methodologies.
Keywords
Persona; User research; Living Lab research; Community building; Quantitative research;
Qualitative research;
1 Introduction
One possibility for users to contribute to an innovation and for companies to involve users
during their development phase is living lab research. Living Labs are an organized
approach (as opposed to an ad hoc approach) to innovation consisting of real-life
experimentation and active user involvement by means of different methods involving
multiple stakeholders (Schuurman, 2015). The living lab approach is a framework that
introduces new ways of managing innovation processes (Sthlbrst, 2008). The underlying
idea is that peoples ideas, experiences, and knowledge, as well as their daily needs of
support from products, services, or applications, should be the starting point in innovation
115

(Bergvall-Kareborn & Sthlbrst, 2009). The living lab approach is also a form of open
innovation, because technology can be developed and tested in a physical or virtual reallife context, and users are important informants and co-creators in the tests (Kusiak, 2007).
Therefore, living labs are user-centric with user involvement as an essential characteristic
of living lab research. Not only are users empowered by living labs (Veeckman et al., 2013),
living labs depend on the involvement and motivation of these users in order to generate
useful user contribution (Schuurman, 2015). It is important to get to know the motivations
and behavior of users who participate in living lab research in order to keep them
motivated to participate in living lab research. This paper aims to shed light on using
persona to gather insights on the motivations and behavior of living lab panel members.
2 Methodology
2.1 What preceded: scenarios as a user-centered design method
Participatory or cooperative design focuses on the eventual users of a system or
application (Pruitt & Grudin, 2003). Early participatory design efforts were explicitly
focused on improving the quality of working life for those workers most at risk of
unrewarding consequences of information technology (Ehn, 1988) and included a strong
focus on sociopolitical and quality of life issues.
Realistic scenarios appeared to be a perfect tool for design: they depict the work practices
one hopes to support (Pruitt & Grudin, 2003). Scenarios are a natural element of personabased design and development. In Carrolls words (2000), a scenario is a story with a
setting, agents, or actors who have goals or objectives, and a plot or sequence of actions
and events. Given that scenarios have actors and personas come with scenarios, the
distinction is in which comes first, which takes precedence. Actors or agents in scenariobased design are typically not defined fully enough to promote generative engagement.
The lifelessness of characters in such scenarios has been critiqued from a writers
perspective (Moore, 1999) and by scenario-based design researchers who suggest using
caricatures, perhaps shocking or extreme ones (Bdker, 2000; Djajadiningrat et al., 2000).
They argue for extreme characters were personas try to expose those emotions and
character traits which remain hidden in scenarios for supposedly real-life characters
because they are incorrect or embarrassing (Djajadiningrat et al., 2000). That is also why
Bdker (2000) argued for caricatures, unrealistic extremes that are more engaging, more
memorable.
According to Grudin & Pruitt (2002) personas can help restore elements which get lost
within participatory design approach: (1) long-term engagement with particular
participants, and the empathy, commitment and deep understanding that such
engagement can bring and (2) attention to the sociopolitical and quality of life issues that
marked much of the early work, including values, fears, aspirations, and so forth.
Contextual Design (Beyer & Holtzblatt, 1998), a powerful approach for obtaining and
analyzing behavioral data, is a strong candidate for informing personas. Ethnographic data
may help the most in developing realistic personas, when available in sufficient depth.
Quantitative data may be necessary in selecting appropriate personas, but does not replace
observation (Pruitt & Grudin, 2003).
2.2 What are personas?
Personas are a method for enhancing engagement and reality. It is a medium for
communicating data that are collected using other research methods (Grudin & Pruitt,
2002). They are an interaction design technique with considerable potential for software
product development. Personas are an established method for bringing different types of
116

users to life (Cooper, 1999). They are fictional people and have names, likenesses, clothes,
occupations, families, friends, pets, possessions, and so forth. They have age, gender,
ethnicity, educational achievement, and socioeconomic status. They have life stories, goals
and tasks. They are not agents or actors in a script, they are people (Grudin & Pruitt,
2002).
Second, a persona is described in narrative form. This narrative has two goals: (1) to make
the persona seem like a real person, and (2) to provide a vivid story concerning the needs of
the persona in the context of the product being designed (Miaskiewicz & Kozar, 2011). The
use of abstract representations of users originated in marketing, but Coopers (1999) use of
personas, their goals, and activity scenarios are focused on design (Mikkelson & Lee, 2000).
Well-crafted personas are generative: once fully engaged with them, you can almost
effortlessly project them into new situations. In contrast, a scenario covers just what it
covers. Persona use brings sociopolitical issues to the surface. Each persona has a gender,
age, race, ethnic, family or cohabitation arrangement, socio-economic background, work,
or home environment. This provides an effective avenue for recognizing and perhaps
changing assumptions about users (Pruitt & Grudin, 2003).
Grudin & Pruitt (2002) feel that persona use needs to be complemented with a strong,
ongoing effort to obtain as much quantitative and qualitative information about users as
possible, to improve the selection, enrichment, and evolution of sets of personas. The
highest priority segments get fleshed out with user research including field studies, focus
groups, interviews and further market research. Links between persona characteristic and
the supporting data should be explicit and salient. Communicating about your personas
should be multifaceted, multimodal, on-going, and progressively unfolding. Generally,
they think of the persona effort as an on-going campaign (Grudin & Pruitt, 2002).
In the same vein, Nielsen (2002) argues that personas, as described by Cooper (1999), are
too flat to engage designers. With examples from movie manuscripts she suggests the
development of characters with richer personalities and better descriptions (Johansson &
Messeter, 2005). McGinn & Kotamraju (2008) on the other hand, take the broader
perspective of Mulder & Yaar (2007) and Pruitt & Adlin (2006), that there are many types of
personas, which are differentiated by the methods used to create them.
2.3 Why use personas?
Personas can create a strong focus on users and work contexts through the fictionalized
settings. The act of creating personas can help to make assumptions about the target
audience more explicit. Once created, personas help to make assumptions and decisionmaking criteria equally explicit. Personas are a medium for communication, a conduit for
information about users and work settings derived from ethnographies, market research,
usability studies, interviews, observations, and so on. Personas utilize the power of
narrative and storytelling to enhance attention, memory, and organization of detailed user
data (Grudin & Pruitt, 2002).
Similarly, two other important benefits, prevention of self-referential design and challenge
assumptions, point to the personas ability to establish a truly consumer-centered design
attitude (Miaskiewicz & Kozar, 2011). As Alan Cooper (1999) and others have observed,
personas can engage team members very effectively. They also provide a conduit for
conveying a broad range of qualitative and quantitative data, and focus attention on
aspects of design and use that other methods do not (Pruitt & Grudin, 2003).
Personas create a strong focus on users and work contexts through the fictionalized
setting. They utilize our minds ability to extrapolate from partial knowledge of people to
create coherent wholes and project them into new settings and situations. The act of
creating personas makes explicit our assumptions about the target audience (Grudin &
Pruitt, 2002).
117

Determining an appropriate range of personas is a balancing act between having too few,
which means that sufficient capability variation cannot be adequately represented, and
having too many, which makes them more difficult to apply and makes it harder to
maintain focus. Drafting a prototype set of personas and then iteratively updating is a good
way of getting the balance right (Hosking et al., 2010). That is what we are planning to do
with the iMinds Living Labs panel persona.
2.4 iMinds Living Labs personas
Based on Pruitt & Grudin (2003) we also used a central foundation document (attachment
1) for each persona as a storehouse for information about every persona (data, key
attributes, photos, reference materials, and so on). Note that the foundation document is
not the primary means of communicating information about the persona to general team
members. Likewise, the foundation documents do not contain all or even most of the
feature scenarios. Instead, the foundation document contains goals, fears, and typical
activities. Links between persona characteristics and the supporting data are made explicit
and salient in the foundation documents (Pruitt & Grudin, 2003).
In the original notion of personas, as presented by Cooper (1999), they are rich but static
descriptions of fictive users. Once they are established, the contents of their description is
frozen. In contrast we tend to follow Johansson & Messeter (2005) who worked with a
dynamic take on representing the user where we allow new information and different
perspectives to enrich the user description as deeper knowledge of users. We chose to
continuously construct personas in order to allow the user to become an entity of its own
right in design worlds. Personas are typically based on interpretations of interviews,
surveys or studies. We tend to follow this methodology, in contrast with Johansson &
Messeter (2005) who suggest that the interpretation process should become part of the
design process. To create the iMinds Living Labs persona we used previous research on
the panel members as a foundation for our persona. A first already completed research
step was a survey on motivations of the iMinds Living Labs panel members to participate
in living lab research in 2013 (figure 1).

FIGURE 1: MOTIVATIONS FOR THE IMINDS LIVING LABS PANEL MEMBERS TO PARTICIPATE IN LIVING LAB RESEARCH

Goodwin (2001) tells us to use patterns of behavior to drive the persona development. So
that is why next to this survey (n=725), we analyzed the behavior of our panel members and
noticed three types of behavior. The behavior of 19.403 respondents between 2010 and 2013
was analyzed. In total, 22 Living Lab projects took place during this period with a total of 59
research activities in which panel members could participate. According to the frequency
of their Living Lab participation, the panel members were divided into three groups: active
(3.4%), sleeping (33.3%) and passive (62.4%) panel members. We also discovered a small
118

group of panel members being extremely active. We called them the alpha users (0.8%). In
order to find out whether they have other motivations to participate in living lab research
than regular panel members, we interviewed 15 of these alpha users and found out that
they do have other motivations (table 1).

TABLE 1: COMPARED MOTIVATIONS FOR ALPHA USERS AND REGULAR PANEL MEMBERS

During a later research project, we compared the participation of the different panel
member types in different research phases. In a first step we recorded which user types
participated in every co-creation session (qualitative research) held in 2014. We expected
alpha users to represent the largest amount of participants, but we noticed that sleeping
users also were frequently represented. We defined an extra group of users as other,
these are friends or family of our panel members joining the panel members at co-creation
sessions.

TABLE 2: COMPARING PANEL MEMBER TYPE PARTICIPATION IN CO-CREATION SESSIONS OF 2014

As a next step, we compared the participation of all our panel member types in the
different research phases we organized in 2014. We noticed that there are significant
differences between the four panel member types.

119

TABLE 3: PARTICIPATION OF THE PANEL MEMBER TYPES IN DIFFERENT RESEARCH PHASES HELD IN 2014

Finally, we gathered all this data compose central foundation documents as advised by
Pruitt & Grudin (2003) (attachment 1). Based on this documents, we created four persona of
the iMinds Living Labs panel members (figure 1). Because iMinds Living Labs researchers
work with the panel members in a rather intense way, we know our panel members in a
sometimes quite personal way. We had some panel members in mind during the creation
of our personas, but we never used private information to complete the personas. We tend
to follow Pruitt & Grudin (2003) who state that personas should be seen as complementing
other approaches, or used where another approach is impractical. Therefore we organized
a co-creation session and survey in march 2015 to complement our personas in order to
brainstorm about a future community platform.
We thus based our research on the findings of Tu et al. (2010) who propose an approach
that combines the quantitative and qualitative methods in the creation of the personas. In
this way, the final personas are more representative and less ambiguous upon which all
the team members can agree. As a result we created four iMinds Living Labs persona (1)
Passive Peter, (2) Sleeping Sophie, (3) Active Alain and (4) Alpha Anna. For each persona
(figure 1) we defined the preferred research type, the main motivation and the research
question we should try to answer. We also dedicated an adoption phase (Rogers, 1976) to
every persona, based on the experiences with the iMinds Living Labs panel members.

FIGURE 2: IMINDS LIVING LABS PERSONAS

The four persona (figure 1) are already in use for communicative issues such as different
types of communication forms (texting versus email), different survey elements (e.g. socio
120

demo we already know), focus on the reward or not, In the nearby future, iMinds Living
Labs wants to build a community platform inspired by and together with representative
panel members for these four persona. With this platform we can connect interests, topics,
research steps, of panel members with their persona to get even more detailed
information on our panel members to organize our research in the way our panel
members prefer. We can adapt invitations, locations, research types, research approaches,
rewarding, based on these persona. It became clear to us that personas focus attention on
a specific target audience. The method helps establish who is and consequently who is not
being innovated for. Personas explicitly do not cover every conceivable user. They also
help focus sequentially on different kinds of users.
3 Conclusion
In order to get a better understanding of the different types of panel members we created
persona. Following Tu et al. (2010), who propose an approach that combines the
quantitative and qualitative methods for the creation of personas, we designed four
persona of iMinds Living Labs panel members (Passive Peter, Sleeping Sophie, Active
Alain and Alpha Anna). We gathered data from earlier research on the iMinds Living Labs
panel and supplemented the persona with information on the panel members from a cocreation session with our alpha users the panel members who participate the most and in
both qualitative and quantitative research- and we surveyed the alpha users who were
absent on this session. Contrary to McGinn & Kotamraju (2008) we created personas from
existing data because we did not start a living lab in order to create persona. Also, this
method takes less time. Although the data has not been gathered specifically for the
purpose of creating personas, we can state that the data is real because every aspect of the
persona was gathered from different living lab projects and research phases in an
academic way. The time to complete the data gathering and analysis does take longer than
the data-driven persona development of McGinn & Kotamraju (2008). Still, the persona
give more insights in the behavior, motivations and interests of the different panel
member types. In this way the living lab can try to offer services which are convenient for
every panel member. It is important to pay attention to every panel member persona,
because in order to cumulate representative feedback a living lab needs to gather
information from different types of people.
In making choices to create your personas, it becomes clear that choices have
consequences. The iMinds Living Labs personas will be used to guide participant selection
for future studies and could be used to filter out data from sources not matching one of the
persona profiles. Related to this is the temptation towards persona reuse. There can also be
a temptation to overuse personas. At worst, they could be used to replace other usercentred methods, ongoing data collection, and product evaluation. They should augment
existing design processes and enhance user focus. In other words, personas are not
without problems and can be used inappropriately, but based on experience and analysis
it has extraordinary potential (Grudin & Pruitt, 2002). Persona use does require decisionmaking. It is not a science. If not used appropriately, any powerful tool can take one down
the wrong path, as in lying with statistics or using non-representative video examples
(Pruitt & Grudin, 2003).
For future research, we tend to follow Johansson & Messeter (2005) and aim at continuing
to broaden and enrich our understanding of the user through design moves where early
concepts, ideas and mock-ups function as probes. In our future work we will continue to
explore and develop the persona of the iMinds Living Labs panel members in order to
communicate and involve our panel members in the way every panel member prefers.

121

4 Bibliography
Beyer, H., & Holtzblatt, K. (1997). Contextual design: defining customer-centered systems.
Elsevier.
Bergvall-Kareborn, B., & Sthlbrst, A. (2009). Living Lab: an open and citizen-centric
approach for innovation. International Journal of Innovation and Regional Development, 1(4),
356-370.
Bdker, S. (2000). Scenarios in user-centred designsetting the stage for reflection and
action. Interacting with computers, 13(1), 61-75.
Carroll, J. M. (2000). Making use: scenario-based design of human-computer interactions. MIT
press.
Cooper, A. (1999). The inmates are running the asylum:[Why high-tech products drive us crazy
and how to restore the sanity] (Vol. 261). Indianapolis: Sams.
Djajadiningrat, J. P., Gaver, W. W., & Fres, J. W. (2000, August). Interaction relabelling and
extreme characters: methods for exploring aesthetic interactions. In Proceedings of the 3rd
conference on Designing interactive systems: processes, practices, methods, and techniques (pp.
66-71). ACM.
Ehn, P. (1988). Work-oriented design of computer artifacts (Vol. 78). Stockholm:
Arbetslivscentrum.
Eriksson, M., Niitamo, V. P., & Kulkki, S. (2005). State-of-the-art in utilizing Living Labs
approach to user-centric ICT innovation-a European approach. Lulea: Center for Distancespanning Technology. Lulea University of Technology Sweden: Lulea.
Goodwin, K. (2001). Perfecting your personas. Cooper Interaction Design Newsletter.
Grudin, J., & Pruitt, J. (2002, January). Personas, participatory design and product
development: An infrastructure for engagement. In PDC (pp. 144-152).
Hosking, I., Waller, S., & Clarkson, P. J. (2010). It is normal to be different: Applying
inclusive design in industry. Interacting with Computers, 22(6), 496-501.
Johansson, M., & Messeter, J. (2005). Present-ing the user: constructing the persona. Digital
Creativity, 16(04), 231-243.
Kusiak, A. (2007). Innovation: the living laboratory perspective. Computer-Aided Design and
Applications, 4(6), 863-876.
Logghe, S., Baccarne, B., & Schuurman, D. (2014). An exploration of user motivations for
participation in Living Labs. In International Society for Professional Innovation Management
Conference.
McGinn, J. J., & Kotamraju, N. (2008, April). Data-driven persona development.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 15211524). ACM.
Miaskiewicz, T., & Kozar, K. A. (2011). Personas and user-centered design: how can
personas benefit product design processes?. Design Studies, 32(5), 417-430.
Mikkelson, N., & Lee, W. O. (2000). Incorporating user archetypes into scenario-based
design. In Proc. UPA.
Moore, G. A. (1999). Crossing the Chasm (2nd edn, 1st edn in 1991 by HarperCollins
Publishers).
Mulder, S., & Yaar, Z. (2006). The user is always right: A practical guide to creating and using
personas for the web. New Riders.
Pruitt, J., & Grudin, J. (2003, June). Personas: practice and theory. InProceedings of the 2003
conference on Designing for user experiences (pp. 1-15). ACM.
Pruitt, J., & Adlin, T. (2010). The persona lifecycle: keeping people in mind throughout product
design. Morgan Kaufmann.
Rogers, E. M. (1976). New product adoption and diffusion. Journal of consumer Research,
290-301.

122

Rosson, M. B., & Carroll, J. M. (2009). Scenario based design. Human-computer interaction.
Boca Raton, FL, 145-162.
Schuurman, D. (2015). Bridging the gap between Open and User Innovation?: exploring the value
of Living Labs as a means to structure user contribution and manage distributed
innovation (Doctoral dissertation, Ghent University).
Schuurman, D., Lievens, B., De Marez, L., & Ballon, P. (2012, July). Towards optimal user
involvement in innovation processes: a panel-centered living lab-approach. In Technology
Management for Emerging Technologies (PICMET), 2012 Proceedings of PICMET'12: (pp.
2046-2054). IEEE.
Sthlbrst, A. (2008). Forming future IT the living lab way of user involvement.
Veeckman, C., Schuurman, D., Leminen, S., Lievens, B., & Westerlund, M. (2013).
Characteristics and Their Outcomes in Living Labs: A Flemish-Finnish Case Study.
In XXIV ISPIM Conference: Innovating in Global Markets: Challenges for Sustainable Growth.
Tu, N., Dong, X., Rau, P., & Zhang, T. (2010, October). Using cluster analysis in persona
development. In International Conference on Supply Chain Management and Information
Systems.

123

Living Lab cases

Session III

124

Care Living Labs Flanders:

Social and Open Innovation
Mark Leysa, Lukas Versteelea, b
and Lien Potsa, c
a

Vrije Universiteit Brussel, Brussels, Belgium

Vrije Universiteit Brussel, Brussels, Belgium
c
KIO, interuniversity consortium on innovation in elderly care, Belgium
mleys@vub.ac.be
b

Abstract
In 2012 the Flemish Minister of Innovation launched an initiative to enhance innovations
in elderly care using a living lab methodology: the Care Living Labs Flanders (CLLF). The
main purpose of CLLF is to accelerate innovations tackling social and economic challenges
in elderly care. The main difference between many other types of living labs (also in
other countries) and the CLLF is this differentiation between projects and platforms.
Today, six platforms host 23 projects.
This paper tries to contribute to the question on how living lab (LL) methodologies could
integrate insights of social and open innovation perspectives as a ground to optimise the
LL methodology. Based on a content analysis of the ambitions and scope of the Flemish
government programme, this paper tried to disentangle a number of (scientific) debates to
which the CLLF can be connected to better identify the role of a platform (LL) and the role
of projects in the strategy to seek innovative approaches in elderly care.
In the start-phase of the implementation most attention was spent on implementing the
projects, rather than giving form to the LL as an infrastructure. None of the initiators had
clear and well developed ideas on the role and the form of what could be a LL. Many
efforts have been paid by the support organisms in CLLF to introduce and explain the
notion of a LL, especially with the ambition to create sustainable LL infrastructures.
The diversity of partners in a platform can trigger innovations, but also make meaningful
collaboration difficult. Collaboration between LL partners is a crucial issue. As they are
influenced by their profession, discipline, organisational membership, socio-cultural
habits, their own vocabulary, etc. this collaboration should be better structured and
governed. For this purpose lessons can also be learned from the (inter organisational)
network literature.
Key words
Living lab, Social innovation, open innovation, elderly care
1 Introduction
This paper tries to contribute to the question on how living lab methodologies could
integrate insights of social and open innovation perspectives as a ground to optimise the
LL methodology. The paper addresses the question how a LL method supports
innovations in elderly care and how this is taking form in Flanders.
125

In 2012 the Flemish Minister of Innovation launched an initiative to enhance innovations

in elderly care using a living lab methodology: the Care Living Labs Flanders (CLLF). The
main purpose of CLLF is to accelerate innovations tackling social and economic challenges
in elderly care. Flanders is confronted with a so-called double greying of society: one out
of four Flemish people is older than sixty. This ratio will increase over time to one out of
three in 2040. People who are older than eighty will triple in the next decades. This
demographic evolution is expected to vastly increase the demand of care (Department of
Economics, Science and Innovation, Flemish Government, 2013). Moreover, a shortage of
nurses and other professional care givers is expected in Europe (Ellenbecker, 2010).
Connected with these developments a movement is initiated where elderly persons strive
to stay and to be cared for at home as long as possible. Older adults prefer to live in their
own houses as long as possible and as close as possible to their (social) environment and
family (Panigrari, 2009). Professional care systems should only support on a needs-based
basis.
The CLLF program aims to stimulate innovations on the one hand tackling problems in
elderly care and on the other hand enhancing innovative economic activities. From a
social and policy perspective one is seeking innovative approach of organising support and
care for an ageing population, to maintain elderly persons as long as possible at home, and
to create opportunities to increase their active participation in society. The new models of
elderly care focusses a lot more on self-care and informal caregivers, complemented and
integrated (when needed) with professional care. It endorses the central position of the
elderly person in the care process. Moreover, Flanders is also seeking connection with the
internationally emerging trend of active ageing, seeking ways to activate elderly persons
and develop strategies to make or let them participate in society. If these innovations can
also contribute to increase the economic assets of Flanders in a care economy, this would
be very welcomed.
This paper is based on a literature review, document analysis and interviews with the
partners involved in CLLF. Part of the reflection is experimental based relying on
participatory observations in implementation activities in the programme.
Section one discusses the CLLF programme design. The following section discusses the
specificities of the CLLF from the perspective of social and open innovation. The last
section draws some conclusions and issues for further reflection.
2 Care Living Labs Flanders
The CLLF was set up to tackle challenges in elderly care. The objective of the initially
three year program is to create and test new care concepts, services, processes and
products in a participatory process with the end users. The program expects the Care
Living Labs (platforms) (CLL) to become sustainable and that the CLL methodology will
enhance economic as well as social innovations in elderly care.
The Agency for Innovation by Science and Technology (IWT, 2012bis) defines the Care
Living Labs as: a structured test environment where organisations can test their innovative
technologies, products, services and concepts using a representative group of individuals
who can be used as testers for innovations in their own habitat and workplace (IWT,
2012b). The CLL offers a platform or infrastructure where products and services can be
tested in a real life environment. It should be conceived as an experience room for users,
and testing will be used as input for further development of the innovation. The end user
(client, patient, informal caregiver and professional care giver) should be highly involved
in the design and adaptations of the innovations (Zorg Proeftuinen, 2015).

126

The Flemish government has developed an initially three year programme framework
rather than a closed set of guidelines for platforms or projects, opening the door for a wide
range of initiatives (Zorg Proeftuinen, 2015).
2.1 Design of the program
The program has platforms and projects at its core (figure 1). These are supported by three
organisms (Sounding Board Committee, Programme Office and Scientific Consortium).

Figure 1: Structure of the 'Care Living Labs Flanders'

A Program Office (PO) supports and facilitates both projects and platforms for developing
their living lab methodology as an instrument for innovation in care (Zorg Proeftuinen,
2015).
The Scientific Consortium KIO (Knowledge Innovation Centre for Elderly Care) collects
scientific knowledge about the platforms and projects, realises knowledge transfer and
develops a model of indicators to monitor the impact of projects and platforms (Zorg
Proeftuinen, 2015).
The Sounding Board Committee consists of representatives from Flanders' Care (a
Flemish organisation stimulating medical and care innovations) and various actors from
the field. This committee can bring in ideas and reflections to support the implementation
of the programme (Zorg Proeftuinen, 2015).
2.2 Platforms and projects
Today, six platforms host 23 projects (Figure 2). The main difference between many other
types of living labs (also in other countries) and the CLLF is this differentiation between
projects and platforms.

127

Figure 2: Active platforms and projects in the 'Care Living Labs Flanders'

A platform consists of an infrastructure and hosts a database and a network of a test

population (a panel of end-users). The panel consists of elderly, informal caregivers,
volunteers, professional caregivers and partner organizations.
End-users are supposed to be actively involved in conceptualisation, development,
research and evaluation of the innovation process (IWT, 2012bis). Every platform needs to
have a users committee: there is no predefined form or format to involve this userscommittee: the from and activities are left to the responsibility of the platforms.
On these platforms, projects can test their product, process or service. So each platform
can host a range of projects. The timeframe of developing the projects is limited. Every
platform is governed by a mix (public and private organisations, individual professionals)
of partners (multidisciplinary perspective).
The scope or identity of the platforms differs: some platforms define their activities more
on technological and person oriented activities to support elderly, while other platforms
focus more on social community based approaches. The domains of the projects vary from
adaptations to individual housing, to technological support in home care, over food
delivery innovations to community based support systems. The range of projects and
platforms is clearly a result of the bottom-up call and illustrate how a wide range of
meanings is given to seek innovation in the organisation of elderly care.
The number of sectors involved in the platforms varies between four (AZoB, LiCalab,
CareVille and Online Buurten) and six (AIPA and InnovAGE). The number of partners in
a platform ranges between five and seventeen16.

16

A more detailed description of the platforms and programs can be found in the Flemish written report
planevaluatie and some English written background information on platforms and programs can be found
on http://www.zorgproeftuinen.be/en
128

3 Thinking grounds of CLLF

In this section we briefly sketch approaches to which the CLLF is connected. We try to
trace back how the CLLF government programme aims to support the development of
adequate innovation environment. A number of issues can be found explicitly in the
programme documents, other aspects are more implicit to the CLLF.
3.1 Social innovation
Although not explicitly stated in the program documents, some direct links can be made
between CLLF and social innovation. The concept of social innovation is initially used to
differentiate it from technological innovation, although openings are created towards the
use of technology to find solutions. Technology is merely one component in the process of
(social) innovation because technology has to become embedded in social practices.
Secondly, social innovations sees that the drivers for innovation are not only
entrepreneurs but that other parties can surely contribute to the development of ideas
and new social practices. Thirdly, social innovation mainly focuses on social issues or
problems, very often those where governments of markets did not fill in activities:
sometimes referred to as institutional voids.
Some key characteristics of social innovation can be deduced from a sample of definitions
(see box).
Social innovation is the development and implementation of new ideas (products,
services and models) to meet social needs and create new social relationships or
collaborations. It represents new responses to pressing social demands, which affect the
process of social interactions. It is aimed at improving human well-being. Social
innovations are innovations that are social in both their ends and their means. They are
innovations that are not only good for society but also enhance individuals capacity to act.
European Commission (2013)
Social innovation seeks new answers to social problems by: identifying and delivering new
services that improve the quality of life of individuals and communities; identifying and
implementing new labour market integration processes, new competencies, new jobs, and
new forms of participation, as diverse elements that each contribute to improving the
position of individuals in the workforce. Social innovations can therefore be seen as dealing
with the welfare of individuals and communities, both as consumers and producers. The
elements of this welfare are linked with their quality of life and activity. Organisation for
Economic Cooperation and Development (OECD) (2010).
Social innovations are new ideas (products, services and models) that simultaneously
meet social needs and create new social relationships or collaborations. In other words, they
are innovations that are both good for society and enhance societys capacity to act.
(Murray, Caulier-Grice & Mulgan (2010))
Common elements in these definitions are the focus on social problems, needs and
demands, on participative approaches, on capacity development to develop alternative
(innovative) ways to react to social problems. The empowerment of society and social
groups is a key dimension of social innovation. Initiators of innovations are not necessarily
the more traditional actors such as industry or science but can also be end users or
societal partners. Social innovations can be generated from within any sector public, private or
social or from citizens and social movements. They may generate financial value, but dont have
to. (Gabriel, 2014; Bulut, Eren & Halac, 2013) Ideas from within society become the starting
point for the innovation.

129

Social innovation was introduced as a counterweight to the policies and discourses that
originally focussed on hard innovations, in particular technological innovations. The
social innovation paradigm assumes that a top-down imposing of change strategies or
solutions can harm the acceptability and effectiveness of an innovation. In social
innovation the search for value is less concerned with profit and more with issues such as
quality of life, solidarity and well-being. The social innovation process itself is an outcome
as it produces social capital (European Commission, 2011) which can foster social
participation. However it is also clearly stressed that social and other forms of innovation
are not mutually exclusive: in many cases technological innovations are triggered by social
innovations (Bulut et al., 2013).
Due to the complexities of (health and social) care problems, different levels of
policymakers are seeking for alternatives to organise care. Social problems, also in the field
of health care, are very often wicked problems. A wicked problem is difficult or
impossible to solve because of incomplete, contradictory, and changing requirements that
are often difficult to recognize (Rittel & Webber, 1973). Many factors are involved and
causal relationships are numerous, interrelated and difficult to identify (van Bueren, Klijn
& Koppenjan, 2003). Wicked problems require collaboration among multiple actors, as
interventions by single agencies have limited or even perverse effects. For many of these
wicked problems the traditional market fails to come up with adequate innovation
strategies as well as the state or civil society. It is assumed that a participative process,
involving a number of actors and stakeholders who have a vested interest in solving a
social problem also directly empowers the beneficiaries (capacity building). It is believed
that complex and wicked social problems require collective action from a network
perspective (van Bueren et al., 2003). Government departments are often poorly equipped
to tackle the complexity of the problems which cut across sectors and civil society often
lacks the capital, skills and resources (Murray et al., 2010). In the care of elderly people
collaboration is required between health care, social care and voluntary sector agencies as
well as with informal caregivers. Single agency based initiatives may have negative effects
because of a lack of integration (see also paragraph on open innovation infra). Because of
complexity of the problems and the objective to develop a patient centred approach of
elderly persons, numerous viewpoints and perspectives can contribute to a better
understanding of a problem.
The underlying assumption of social innovation is that investing in human needs is
potentially key for growth and tackling social challenges. Although certainly not grounded
in social innovation, Porter and Kramer (2011) come up with comparable ideas rooted in
their shared value concept: social and economic progress will be realised when
innovative ideas create both social and economic value.
3.2 Open Innovation
The CLLF approach is embedded in the open innovation paradigm. Chesbrough (2003)
argued that closed innovation as a paradigm should be replaced by open innovation,
which integrates external ideas and knowledge mainly focussed on technological and
economic issues. The open innovation paradigm stresses the relevance of interactions
between different stakeholders within the innovation processes, as a reaction to older
more linear approaches to innovation. The open innovation logic suggests that valuable
ideas can come from inside or outside the company (Chesbrough, 2006). Open innovation
recognises that views, knowledge and capacity to be innovative is not to be found in a
single corporation (Chesbrough & Crowther, 2006; Herskovits, Grijalbo & Tafur, 2013).
Collaborative open practices will lead to solutions that go far beyond the scope of what
any individual actor can. Ideas and different points of view from a wide range of
130

stakeholders become shared to develop new understandings of a problem and new

perspectives on solutions (Chesbrough, 2003; OECD 2008).
3.2.1 Helix and ecosystems thinking
The open innovation approach was initially grounded in the so called triple helix
thinking (Etzkowitz and Leydesdorf, 2000). The Triple Helix refers to universityindustry-government interactions (OECD, 2008). It is proposed as a spiral model of
innovation representing reciprocal relations in different moments in the knowledge
process. It holds three dimensions: (1) internal transformation of each of the helices
(UniversityIndustryGovernment), (2) mutual influence among the three helices and (3)
creation of trilateral networks resulting from interaction between the three helices. The
spiral Triple Helix model positions the university as a strategic actor in the innovation
process and assumes that research bodies, the government and industry can contribute to
a country's economic growth through the development of generative relationships. The
transfer of scientific and technological knowledge from universities to industry and vice
versa is considered as a conditioning factor in a country's economic development. It
focusses on processes where the university is proactive in putting knowledge to use for
firms. Firms are expected to raise their technological level as they collaborate with
academia to enhance their capacity through knowledge sharing. Government is
considered as one of the partners in this process as it will create the conditions, as well as it
is a partners in the sharing of knowledge.
The core Triple Helix model is broadened as the Quadruple Helix, (even quintuple
helix adding other groups) where government, industry, academia and civil society work
together to co-create the future and drive structural changes. The Triple Helix places the
emphasis on knowledge production and innovation in the economy and addresses the
knowledge economy, while the Quadruple Helix introduces the perspective of the
knowledge society, and knowledge democracy for knowledge production and innovation.
(Carayannis, Barth, & Campbell, 2012). The underlying reasoning is that innovation is not
only a matter of bringing in technical knowledge, grounded in interaction with different
groups or partners, but also requires a different culture, learning how to collect and grasp
different sources of relevant knowledge.
Open innovation is often directly connected to (business) ecosystems thinking. The
ecosystems theory relies on the assumptions of open innovation and ultimately addresses
the question of value creation (co-created shared value). It is a theory focussing on the
question on how synergies can be created between partners coming from different
backgrounds. Borrowing insights from ecology, an ecosystems framework addresses issues
of interdependency and collaboration between many different types of players and the
environmental conditions of these actors and relationships. It focuses on the organisms
constituting an ecology and on the conditions that impact on the type of organisms that
can exist as well as on the relationships between these organisms. The underlying
theoretical assumption is that environmental conditions also determine what types of
organisms will live or die.
Ecosystems approaches stress the importance from the start of an innovation initiative to
clearly map the players (the organisms) in the ecology, and focus on a clear
understanding on what the characteristics and expectations are of players engaging in an
innovation approach. It also stresses the importance to better understand the
environmental conditions in which both the individual players and well as the
collaboration itself functions in order to understand how individual players, the
collaboration and the users could get value from the efforts.

131

The ecosystems approach is clearly associated with Porter & Kramers (2006) notion of
shared value creation. Shared value creation focuses on identifying and expanding the
connections between societal and economic progress. It focuses on creating economic
value in a way that also creates value for society by addressing its needs and challenges.
Moreover, the concept of shared value blurs the line between for-profit and non-profit
organizations. In their logic companies can create economic value by creating societal
value. Through three distinct ways: (a) reconceiving products and markets, (b) redefining
productivity in the value chain, and (c) building supportive industry clusters at the
companys locations. Each of these are part of the mechanisms to create shared value as
improving value in one area will give rise to opportunities in the others.
3.3 Collaborative networks
In previous sections we discussed arguments that collaboration and open sharing of
knowledge is considered a key for solving innovative solutions for social wicked
problems. In situations where one has to rely on complex knowledge, and where the
sources of expertise are dispersed, the locus of innovation will be found in networks of
learning, rather than in individual firms (Powell, Koput, & Smith-Doerr, 1996). The traces
of collaboration can also be clearly found in open innovation, ecosystem or helix thinking.
Collaboration is grounded in the assumption that by joining forces, parties can accomplish
more as a collective than they can achieve by acting as independent agents.
Interorganizational networks in which actors with different backgrounds are involved are
seen as one of the critical success factors for innovation (Brass, Galaskiewicz, Greve, &
Tsai, 2004; Hoberecht, Joseph, Spencer, & Southern, 2011; Provan & Lemaire, 2012; Popp,
MacKean, Casebeer, Milward, & Lindstrom, 2013). People construct (and have to construct)
their meaning on problems and potential solutions based personal preferences,
backgrounds, professional training, education, organizational affiliations. Based on this
insights it is not difficult to understand that people differ in their process of giving
meaning and why these characteristics predispose them to see the world in different ways.
The creation of inter-organizational networks can be a strategy for developing an
organizational structure that is able to create change and/or be more responsive to change,
as it will on a structural way create a model where these different viewpoints are
coordinated.
From a different viewpoint, a certain branch of innovation literature also shows that
innovation is strongly influenced by clusters, or geographic concentrations of firms,
suppliers, service providers, academic or research types of institutions, logistical
infrastructure. These clusters can be field specific.
However there are also known challenges to working in inter-organizational networks
(e.g., achieving consensus on the network purpose and goals, culture clashes, loss of
autonomy, coordination fatigue, the development of trusting relationships, power
imbalances) that should be considered. Network literature differentiates between more
informal social networks and more formalised forms sometimes studied as whole
networks and emerging networks are to differentiated from mandated networks.
Trust, power and contract are mechanisms that govern activities inter-organizational value
networks and these mechanisms seem to be different according to the stage of service
innovation. A mandated network can provide a powerful incentive for organizations to
work together. An emergent network, on the other hand, may start with higher levels of
trust due to its voluntary nature. Allowing sufficient time for trust building is critical to the
longer-term effectiveness of all networks (Popp et al., 2013). Overall network effectiveness
can be significantly enhanced when network goals and interests are understood and
accepted through meaningful involvement by multiple members of organizations in the
132

network, especially for those organizations that are critical to overall network success
(Provan & Lemaire, 2012). Three interrelated themes for effective network development
and growth, are network governance, management and leadership of and in networks, and
network structures.
Many questions remain on how to increase the effectiveness of these collaborative
networks especially to better understand what works in what circumstances in a particular
domain (see Provan & Fish, 2007). It raises the question on how collaboration between
different types of partners can be managed and maintained. Argued that in the field of
whole networks, structure and governance (a way of coordinating interactions between
different types of partners), should be the subject of future research.
3.4 Sustainability and scaling up of innovations
To maximise the impact innovative interventions found to be effective in tackling needs,
should be become part of wider social practices. However the transfer of new knowledge
from a lot of local initiatives and test projects into practice, is sub-optimal and many
individual initiatives become institutionalized.
Porter and Kramers core argument that creating shared value will be more effective and
more sustainable than other types of initiatives is highly reproduced today in policy
making discourses (also on the European Level). Indeed, the question on how investments
in innovation research and pilot applications can lead to more sustainable solutions is
considered as a key question Sustainability refers to the stadium where innovations have
become the norm in the system, when they have become integrated or become the
mainstream way of working. Many innovations (especially in a care setting) rarely become
embedded in the wider system (Greenhalgh, Robert, Bate, Macfarlane, & Kyriakidou,
2005).
Sustainability is connected to the notion of scaling-up. Scaling-up is the process by which
interventions (shown to be effective and efficient on a small scale) are expanded under real
world conditions into broader policy or practice. The concept of scaling up differs from
routine adoption as an intervention tries to reach to new settings or target groups. In
public health a recent stream of literature discusses the barriers of scaling up public health
interventions. Research learns that scaling up can be supported by producing information
on effectiveness, reach, and adoption, human, technical, and organizational resources,
costs, intervention delivery arrangements, contextual factors, and applying appropriate
evaluation approaches. If these scalability considerations are addressed in the funding,
design, and reporting of intervention research, it could advance the quality and usability of
research for decision makers, and improve uptake and expansion of programs into
practice (Milat, King, Bauman, & Redman, 2013).
3.5 Living labs supporting innovation
Paying more attention to the role of users as contributors the innovation process is part of
the new open and social innovation paradigm. However, the collaboration process
between different players including the users very often lacks structure and governance.
Therefore many efforts are paid to seek a (sustainable) environment for a wide range of
users to propose and carry out projects in order to prepare the rolling out of practices in
real-life. LL have been introduced as a user-driven open innovation business citizens
government (academia) partnership (an ecosystem) which enables users to take an active
part in the research, development and innovation process also allowing for early
assessment of the implications of new (technological) solutions and testing the
effectiveness and efficiency of innovative services and business models. Living labs (LL)
133

are being promoted as a method to implement the principles of open innovation:

platforms where all actors, including end users can interact and new ideas can be captured
(European Union, 2014; Edwards- Schachter, Matti, & Alcantara, 2012). LL are conceived as
open-innovation ecosystem in a real- life setting to create value, based on co-creative
collaboration between all actors in society (Leminen, Westerlund, & Nystrm, 2012).
There is a large number of Living Labs in Europe with a variety of characteristics, as this
and previous ENoLL conferences demonstrate. So it is impossible to identify precise
working methodologies, because working practices to realise these ambitions depend on
the context, the sector or field characteristics, the players involved and the resources
available.
4 Discussion and Conclusion
In the context of Care Living Lab Flanders, a multitude of stakeholders from a wide range
of sectors are collaborating in pursuit of innovative solutions that respond to the needs of
the elderly persons. The ambition of the CLLF program is to foster intersectorial
collaboration, including business sectors, local governments, organised social actors and
local communities to engage in the development and testing of potential interventions that
stand at the verge of real-life implementation. Based on a content analysis of the ambitions
and scope of the Flemish government programme, this paper tried to disentangle a
number of (scientific) debates to which the CLLF can be connected. Our ambition with this
exercise is to better identify the role of a platform (LL) and the role of projects in the
strategy to seek innovative approaches in elderly care.
CLLF responds to the principles of open and social innovation connect with helix and
ecosystem thinking and addresses the question of sustainable innovative interventions
that can be scaled up. The program focusses on social value and not only for financial and
economic value. CLLF combines social with economic ambitions. Value creation should go
beyond business value or financial value but include social value creation regarding
like effectiveness of care, efficiency of care, quality of care and quality of life, solidarity, etc.
The government call to obtain funding allowed for a lot of freedom (bottom-up approach)
within a general framework in the way a LL infrastructure could be given form. However
what we have seen in the beginning period of the implementation that most of the
attention was spent on implementing the projects, rather than the LL as a support
infrastructure. A lot of time and energy was spent in trying to mobilise a user panel. The
implementation experience also learned that many projects struggled with a lot of nonanticipated activities that had to be performed within the CLLF. None of the initiators had
well developed implantation strategies on the role and the form of what could be a LL, and
according to what collaborative approach one would work. In the first years of
implementation many efforts have been paid by the support organisms in order to
introduce and explain the notion of a LL, especially with the ambition to create sustainable
LL infrastructures. A learning process is ongoing on how different partners can become
committed to work and identify the (common) LL goal(s) and visions. If a LL wants to
become sustainable, the partners need to have a shared vision on the role of the platform
and on the position of the platform in relation to their projects.
The diversity of partners in a platform can trigger innovations, but also make meaningful
collaboration difficult. Collaboration between LL partners is a crucial issue. As they are
influenced by their profession, discipline, organisational membership, socio-cultural
characteristics, their own vocabulary (Hovenier, de Joode & Bijsterveldt, 2014), this
collaboration should be better structured and governed. The diversity of organizations can
be a source of conflicts and failing projects (Du Chatenier, Verstegen, Biemands, Mulder &

134

Omta, 2009). One factor of effective collaboration depends on the way these interactions
are stimulated, structured and governed.
5. References
Agency for Innovation by Science and Technology (IWT) (2012bis). Handleiding
platformen en projecten voor de proeftuin Zorginnovatieruimte Vlaanderen: Brussel.
van Bueren, E.M., Klijn, E. & Koppenjan, J.F.M. (2003). Dealing with Wicked Problems in
Networks: Analysing an Environmental Debate from a Network Perspective. Journal of
Public Administration Research en Theory. 13(2), 193-212.
Brass, D., Galaskiewicz, J., Greve, H. & Tsai, W. (2004) Taking Stock of Networks and
Organizations: A Multilevel Perspective. Academy of Management Journal. 47(6). 795817.
Bulut, C., Eren, H. & Halac, D. S. (2013). Which one triggers the other? Technological or
Social innovation? Creativity Research Journal. 25(4), 436-445.
Carayannis, E.G., Barth, T.D. & Campbell, D.F.J. (2012). The Quintuple Helix Innovation
model: global warming as a challenge and driver for innovation. Journal of Innovation and
Entrepreneurship. 1(2).
Chesbrough, H. (2003) Open Innovation. The New imperative for Creating and Profiting From
Technologie. Boston: Harvard Business School Press.
Chesbrough, H. & Crowther, A. K. (2006). Beyond high tech: early adopters of open
innovation in other industries. R&D Management. 36(3), 229-236.
Chesbrough, H. (2005). Open Innovation: A New Paradigm for Understanding Industrial
Innovation.
Department of Economics, Science and Innovation (EWI). Flemish Government (2013).
Geraadpleegd op 28/10/2014 op http://www.ewi-vlaanderen.be/ewi/nieuws/ouderenzorgvan-de-toekomst-zorg-proeftuinen-vlaanderen-gestart.
Du Chatenier, A., Verstegen, J., Biemands, H., Mulder, M. & Omta, O. (2009). The
Challenges of Collaborative Knowledge Creation in Open Innovation Teams. Human
Resource Development Review. 8(3), 350-381.
Edwards- Schachter, M.E., Matti, C.E., Alcantara, E. (2012). Fostering Quality of Life
through Social Innovation: A Living Lab Methodology Study Case. Review of Policy
Research. 29(6), 672-692.
Ellenbecker, C. (2010). Preparing the nursing workforce of the future. Policy, Politics, &
Nursing Practice, 11(2), 115-25.
Etzkowitz, H. & Leydesdorff, L. (2000). The Dynamics of Innovation: From National
Systems and 'Mode 2' to a Triple Helix of University-Industry-Government Relations.
Research Policy, 29(2), 109-123.
European Commission (2013). Guide to social innovation. European Commission
European Commission (2011). Empowering people, driving change: social innovation in the
European Union, Luxemburg: Publications of the European Union.
European Union (2014). Social Innovation. A Decade of Changes. Geraadpleegd op
13/04/2015 op http://espas.eu/orbis/document/social-innovation-decade-changes.
Gabriel, M. (2014). Making it big. Strategies for scaling up social innovations. Londen: Nesta.
Greenhalgh, T., Robert, G., Bate, P., Macfarlane, F. & Kyriakidou, O. (2005). Diffusion of
Innovations in Health Service Organisations. A Systematic literature review. Massachusetts:
Blackwell Publishing.
Grieco, C., Michelini, L. & Iasevoli, G. (2014). Measuring Value Creation in Social
Enterprises: A Cluster Analysis of Social Impact Assessment Models. Nonprofit and
Voluntary Sector Quarterly. 1-21.

135

Harrisson, D., Chaari, N. & Comeau-Valle, M. (2012). Intersectoral alliance and social
innovation : When corporations meet civil society. Annals of public and cooperative
economics. 83(1). 1-24.
Herskovits, R., Grijalbo, M. & Tafur, J. (2013). Understanding the main drivers of value
creation in an open innovation program. International Entrepreneurship and Management
Journal. 9. 631-640.
Hoberecht, S ., Joseph, B ., Spencer, J ., & Southern, N . (2011) . Inter-organizational
networks: An emerging paradigm of whole systems change. OD Practitioner. 43(4), 2327.
Hovenier, C., de Joode, E. & van Bijsterveldt, M. (2014). Innoveren in de zorg is samenwerken:
Samenwerkingsgids.
Leminen, S., Westerlund, M. & Nystrm, A. (2012). Living Labs as open- innovation
networks. Technology Innovation Management Review. 6-11.
Marcy, R. T. & Mumford, M. D. (2007). Social innovation: Enhancing creative performance
through causal analysis. Creativity Research Journal. 19, 123-140.
Milat, A.J., King, L., Bauman, A., & Redman, S. (2013) The concept of scalability: increasing
the scale and potential adoption of health promotion interventions into policy and
practice. Health Promotion International. 28(3), 285298.
Murray, R., Caulier-GFrice, J. & Mulgan, G. (2010). The open book of social. London: Nesta
OECD
(2008).
Open
innovation
in
global
networks.
http://www.oecd.org/science/inno/openinnovationinglobalnetworks.htm. Accessed on
February 2 2015.
OECD (2010). Entrepeneurship and Innovation. OECD.
Popp, J., MacKean, G., Casebeer, A., Milward, H. B., & Lindstrom, R. (2013). Interorganizational networks: A critical review of the literature to Inform practice.
Porter, M.E. & Kramer, M.R. (2006). Strategy and Society: The link between competitive
advantage and corporate social responsibility. Harvard Business Review. 1-15.
Powell, W.W., Koput, K. & Smith-Doerr, L. (1996). Interorganizational Collaboration and
the Locus of Innovation: Networks of Learning in Biotechnology. Administrative Science
Quarterly 41(1), 11645.
Provan, K. & Fish, A. (2007). Interorganizational Networks at the Network Level: A Review
of the Empirical Literature on Whole Networks. Journal of Management. 33(3), 479-516.
Provan, K.G. & Lemaire, R.H. (2012). Core Concepts and Key Ideas for Understanding
Public Sector Organizational Networks: Using Research to Inform Scholarship and
Practice. Public Administration Review. 72(5), 638- 648.
Rittel, H.W.J. & Webber, M.M. (1973). Dilemmas in a General Theory of Planning. Policy
Sciences. 4, 155-169.
Wainwright, S. (2002). Measuring impact: A guide to resources. London: National Council for
Voluntary Organization (NCVO)
Zorg Proeftuinen (2015). http://www.zorgproeftuinen.be/en. Accessed on April 14 2015.

136

Experimenting with location-based

service applications: exploring
a new methodology for Living Labs
Uschi Buchinger a, Heritiana Ranaivoson a, Pieter Ballon a
Karel Verbrugge b
a

iMinds-SMIT-Vrije Universiteit Brussel, Belgium

b
iMinds-MICT-Universiteit Gent, Belgium
hranaivo@vub.ac.be

Topic
Living Lab domain specific cases (privacy and value of data); Living Labs involved service
design and user experience
Previously published
No
Abstract
This paper argues and illustrates how experiments could support and complement
traditional Living Lab research methods. It takes as a starting point insights and results of a
pre-study on the value of location privacy, which was conducted outside of a Living Lab.
The experimental approach aims at assessing how smartphone users value their location
privacy vs. benefits brought by location-based special offers and deals. Results from the
pre-study show that participants demand more recompense in the revision of their bids
regardless of how we reverse the conditions (i.e. enabling or disabling special offers and
deals). Explanatory qualitative data is needed to understand the reasons and intrinsic
factors of such decisions. As an eventual resolving, we propose the replication and
continuation of the study in a Living Lab, which can result even in a multitude of
complementary research methods.
Keywords
Privacy, location-based services, experimental economics, living labs

137

1 Conception/Idea
In the traditional understanding Living Labs are used as a research and development
environment where a multitude of stakeholders create and validate innovations like
services, products or application enhancements in collaborative, multi-contextual
empirical real-world settings. They therefore are one way to bring technology into the
market (Ballon, Pierson & Delaere, 2005). This is especially true and relevant for the ICTenabled application sector that is mainly concerned with the enhancement of innovation,
inclusion, usefulness and usability of ICT and applications in the society (Eriksson,
Niitamo, & Kulkki, 2005). Living Labs have thus proven to constitute a useful setup to hold
co-creation sessions and focus groups to let users partake in the design and shaping of
(technological) products and services (Mulder & Stappers, 2009; Veeckman, Schuurman,
Leminen, Lievens, & Westerlund, 2013). However, using other research methods is yet
rarely exploited in Living Labs despite their vast potential to facilitate it.
The aim of the paper is to present how experimental methods could benefit Living Lab
research. Experimental research is a now common social research method that remains
largely out of the scope of Living Labs, in particular experimental economic research. To
reach this aim, the paper focuses on an experimental research case that has been
conducted around users perception of location-based service (LBS) applications, in
particular the way LBS use ones personal data. The paper concludes on how this peculiar
case could be expanded based on a Living Lab setting. Thus the authors also aim at
providing a framework that could be replicated in other Living Lab research fields. More
generally, introducing new methodologies for Living Labs is important especially to allow
Living Labs to succeed in the long run.
2 Experimental research: advantages and drawbacks
This paper illustrates and argues that economic experimental research can be an
interesting contribution to Living Labs.
Experimental research has been adapted for, and applied to a multitude of areas and
research fields, such as technology design (Staelens, 2012), psychology or social science
(Ariely, 2008). In principle an experiment requires the arrangement and management of
two groups: an experiment group and a control group, which are exposed to experimental
manipulation of variables. Usually an independent variable is manipulated to test the
effects of the dependent variable between the groups. Real experiments command
strictness on several conditions, e.g. the recruitment and division of participants (Bryman,
2012). Given the limited possibilities that such strictness entails, deviations from this
method are common. Quasi-experiments are normally less constrained. They fulfil many
but not all criteria of a real experiment. Yet, they are often a realistic alternative and
provide valuable results (Bryman, 2012). In a Living Lab setup, an experiment could
introduce a more subtle form of analysis, where, different to usual Living Lab research,
research objects and questions are not directly exposed to the participants but aid or
spearhead the development phase.
The greatest benefit of experiments are their artificiality (Webster & Sell, 2014). In
experiments only the theoretically presumed causes of certain phenomena are selected
while other factors are eliminated or minimized. This allows to obtain settings that would
not happen in real life. One can be more certain than with any other design about
attributing cause to the independent variables (Vogt & Johnson, 2011). Most importantly,
such settings can then be replicated, notably to allow other researchers to check the results
(Bryman, 2012; Webster & Sell, 2014). This is unlike findings in natural or real-life settings.

138

On the other hand, experiments receive a lot of criticism due to their artificiality
(Webster & Sell, 2014). Actually all their advantages turn out to be disadvantages with a
different view on them: subjects are isolated from the context that might influence their
behaviour (Vogt & Johnson, 2011); therefore experiments cannot mirror any real setting
(Webster & Sell, 2014); and it may be inappropriate to generalize results beyond the
experimental context (Vogt & Johnson, 2011).
Beyond the apparent paradox, experiments are more fit for some research purposes and
less for others. They cannot attempt to simulate all the complexities of particular settings.
But they are well-suited to test the validity of theories (Webster & Sell, 2014).
3 Research field: value of location privacy
The experimental structure presented in this paper is situated in the field of location
privacy and the value of personal data collected by LBS applications. It especially
considers the economic aspect of these applications by relating it to location-based
commercial benefits. Benefits often constitute products and services promoted by
commercial entities (here: third parties) via the LBS applications (Cramer, Rost, &
Holmquist, 2011). In return, app providers share user data with these third parties (Chang
& MacMillan, 2012).
For the users, this concept is ambivalent: On the one hand, commercial benefits are
appealing as they enable monetary gains in their shopping and leisure activities. On the
other hand, users might take up a sceptic attitude toward such applications as research and
media alike reveal detrimental data sharing mechanism. They principally condemn the
collection and usage of personal data for commercial interest (Chang & MacMillan, 2012;
Couts, 2013; Fuchs, 2012).
In this area, research methods that resolve around direct questionnaires have to be careful,
as they have to recognize this ambivalence, also presented in the privacy paradox. The
paradox describes the discrepancy or relation between individuals intention to disclose
personal information and their actual personal information disclosure behaviors
(Norberg, Horne, & Horne, 2007 p.100). While the former is supposedly very limited, the
actual disclosure contradicts this intention. This is especially true for mobile applications
and can be based in great parts on the fact that there are only few or no alternatives to the
privacy-intrusive app versions. In an exaggerated form, users often seem to face a trade-off:
the more personal data they disclose, the more benefits they gain.
Consequently, users start to weigh and valuate the benefits and implicitly the value of their
own data. Our research is mainly based on the disclosure of location data and juxtapose it
opposite the value of location-based commercial offers, deals and promotions.
4 Encompassing research approach - the SoLoMIDEM17 project
The experimental conception was part of a research and development project on the
matter of location-based service applications. The aim of the project was to build a secure
and privacy-friendly mobile app development environment that allows exploitation of big
data for multiple stakeholders whilst respecting the users rights and preferences of
disclosure or concealment of personal data. The platform and LBS app prototypes
developed in the project are foreseen to be tested and adapted with the therein-situated
Living Lab.

17

Project outline can be found at: https://www.iminds.be/en/projects/2014/03/18/solomidem The project

entails to make available an iterative Living Lab to monitor users and gather feedback in the context of
converging Social, Local and Mobile consumption patterns. An entire workpackage is working on the
management and organization of the Living Lab.
139

In this paper, the project is used as a case study to demonstrate the expected possible and
positive spillovers between different research methods, in particular in using experimental
research in Living Labs. We argue that Living Labs cannot only be used for the creation
and design of products and services, but moreover can herald the start of complementary
research work.
Due to data collection and management tools, Living Labs typically hold enormous
potential as a recruitment and experimentation pool where profiling and selection of
participants is easily possible and deployable.
5 Concept and background
The framework of the experimental setup is based on two previous studies in the area of
location-based services: Cvrcek, Kumpost, Matyas, & Danezis (2006) and Danezis, Lewis, &
Anderson's (2005),,which share a similar experimental structure. They pretend to recruit
volunteers for a study about localization of mobile phones. They then assess the
recompense that people request in return for participating in the study and revealing their
location information for one month.
We roughly follow this method to herald multiple further studies and experiments and the
actual development of prototype applications as foreseen in the project. However, there is
a need to adapt the previous research to the current technological state-of-the-art: For
smartphone users nowadays, the localization of their devices, especially within
applications, is no longer unfamiliar. This feature can provide a multitude of benefits like
viewing ones own, or friends position, get route descriptions or see nearby localities or
points of interest on a virtual map. Consequently users have developed experiences,
expectations and demands towards them. The applications developed in the project have
to understand current market versions and entail, alter and improve their conception.
In this research, the commercial component of LBS apps is central to derive the value and
perceived benefit of this feature for LBS apps. As mentioned, present-day LBS applications
often include commercial entities (third parties) to promote their products and services to
users based on the collection of their data and bring monetary benefits (Cramer et al., 2011;
Hui & Png, 2006). Coming from this viewpoint we try to assess if people value the
concealment of location information more or less than access to special offers and
discounts from third parties and outline how application providers can react.
We formulated the following hypotheses:
H1: People value their location information more than access to special offers and deals.
H1a Enabling special offers and deals leads to people increase the amount they
demand to participate in the study.
H1b: Disabling specials offers and deals leads to people decrease the amount they
demand to participate in the study.
H1c: The increase (in H1a) is higher than the decrease in H1b.
6 Methodology Auction structure
To identify the discrepancy between the value of location data versus the value of locationbased commercial deals and offers we use a first-price sealed-bid auction (Milgrom &
Weber, 1982) introduced and used by Danezis et al., (2005) and Cvrcek, Kumpost, Matyas,

140

& Danezis (2006). We will adapt this concept to look for the minimum bid that participants
claim for their participation in the research about location information.
Constituted as a quasi-experiment, the study will work with a division of the sample into
two groups (Group 1 and Group 2) that will be exposed to different configurations and
manipulation of variables. The initial cover note will be the same for both. We will start
with an introduction to a research study on location-based applications, which focuses on
the collection and processing of individuals location information. It will inform the
addressees that we recruit participants for a study about LBS applications. For the study,
participants will need to download a smartphone application that will record the
whereabouts for a stated period of time at a certain level of accuracy. The application will
have certain features and will allow interaction with participants, which are:
Seeing ones location on an interactive map
Sharing location through social media such as Facebook
Collecting badges based on the usage of location
In addition, and only students of Group 1 will receive the information that via the
application it will be possible to
Get special offers and discounts from nearby merchants
The addressees will have to specify how much monetary compensation they demand to
participate in the study. We will inform them that we only include the 60 people with the
lowest bids in return for the requested amount. By this, we strive to evaluate the smallest
amount that people are willing to accept for disclosing their location information.
Following the cover note, several questions about smartphone usage, location-based
services and the attitude towards commercial location-based benefits will be asked asked.
These questions enable and support subsequent profiling.
In a timely follow-up e-mail, we will notify the respondents that the features of the
application are changing. The feature Get special offers and discounts from nearby
merchants will be abolished for Group 1 while added for Group 2. We will record if and to
what amount this change influences the changing of bids; or in other words, we will
record if participants bid higher, lower or the same amount for participating in the study
under the new conditions.
For the Living Lab testing, two prototype applications will be developed and the 80
selected members in the groups will receive the application they were informed of lastly
(i.e. Group 1 will get the application without special offers and deals; Group 2 will get an
application with special deals and offers). Special offers and deals are possible due to the
collaboration with a project partner with a network of local merchants and commerce. The
business network and testing is foreseen be located in the city of Leuven in Belgium. After
the actual conduction of the tracking, follow-up questionnaires and focus groups are
planned in multiple research fields. We plan to evaluate the value of the application and
the general attitude towards the technology, the design and functionality and to develop
the apps further as in regular co-creation approaches. Moreover, the aim is to include
questions about the utility and appeal of third party offers to cover the economic
viewpoint. Thirdly, from a user research perspective we enquire the self-estimation about
the fairness of the recompense and the motivation to participate. Again, due to the division
into two experimental groups and the exposure to two different app configurations, the
differences shall be juxtaposed opposite each other and the differences shall draw
inferences about the commercial component of the application.

141

6.1 Initial or pre-study to the current research

Before the Living Lab research will be conducted, the first part of the research has already
been tested, though outside of a Living Lab setting. In a limited pre-study we
conceptualized and tested the basic configuration of the setup and user addressing and
collection of bids but the actual tracking was not conducted. We however pretended that
we would and revealed the deception to participants only after we collected the change of
bids.
The pre-study served multiple purposes:
outlining how to preserve the basic research framework introduced by Danezis et
al., (2005) while altering the conditions and features of the research object to reflect
current, real-life, state-of-the-art technology
discovering the average and the modal value that people ask for participating in the
study, i.e. what needs to be considered for compensation
evaluating the initial reaction of participants towards the option to receive or notreceive commercial offers and discounts in a location-based service application
assessing which data needs to be asked or ascertained to enable profiling and
segmentation of users in succeeding analyses
estimating how bid the groups have to be to deliver significant results in
comparison

In other words, in the pre-study, the tracking with the application was not conducted for
real. Only the collection of the bids and the change of conditions were enquired.
Participants were informed about the fictional nature of the study and were compensated
with a voucher.
7 Preliminary results
As stated previously, the experimental setup is conceived for comparing two test groups.
For the pre-study, addressees were divided according to the two participating universities,
the Vrije Universiteit Brussel (Group 1) and the Universiteit Gent (Group 2). Group 1 was
initially informed about the possibility to Get special offers and deals from nearby
merchants via the application. This parameter was disabled in the follow-up e-mail.
Group 2 was not informed about this feature originally; it was enabled in the follow-up email.
The final pre-study sample included 94 participants, 55 in Group 1 and 39 in Group 2.
Table 5 compares the results of Group 1: initial bids and revision of the bids.

Mean
Median
Mode
Minimum
Maximum

Bid
84.20 EUR
40.00 EUR
30 EUR
.0
750 EUR

Bid revision
91.51 EUR
Disabling
50.00 EUR
special
offers and 50 EUR
.0
deals
900 EUR

Table 5: Statistics for bids and revision of bids - Group 1 (n=55)

In average, people claimed 84,20 to partake in the study. This value increased by 7,31
(6,16%) after the change of the apps condition. While it was previously 30 that was
mentioned most often as a compensation, it changed in the alteration of the question to
142

50 (followed by 100 and 30). We assume that for the participants, the change of
condition constitutes a loss of benefits when the service of having benefits from merchants
is abolished. The maximum value rose to 900 from 750.
Looking at the delta between the original bids and the revision, for Group 1, 55% did not
change their bids, 31% increased the amount, 14% decreased the amount.
The 39 people from Group 2 first got no information about special offers and deals in the
outlined application. For them, this condition was enabled in the second mail whereupon
they could change their original bids. Table 6 shows the result for this group.

Mean
Median
Mode
Minimum
Maximum

Bid
48.00 EUR
35.00 EUR
50
1
250

Bid revision
52.10
Enabling
40.00
special
offers and 50
deals
1
270

Table 6: Statistics for bids and revision of bids - Group 2 (n=39)

On average, the students in this Group claimed 48, only 60% of the amount of their
opponents. After the revision, this value increased by approximately 8,5% or 4,10 . Similar
to the first group, 54% did not revise at all, 36% increased and 10% decreased their bids.
The direct comparison shows that whatever way the conditions change i.e. enabling or
disabling special offers and deals in average students ask for more compensation in the
revision of their bids. However, the majority of participants in both cases did not change
their bids. They might thus be indifferent for this feature of the application.
Within the limited sample size we answer the hypotheses preliminary:
H1: People value their location information more than access to special offers and deals.
H1a: Enabling special offers and deals leads to people increase the amount they
demand to participate in the study.
This hypothesis refers only to Group 2. Within the current sample this statement
can be verified if we start out from the average bid. Participants in this group
increased their bids in average by 8,5%. Certainly multiple factors that we have not
enquired can be the reason for this increase. Privacy of location information is only
one explanation. Others can be that the participants suspect extra effort for this
feature (coming with learning costs, unwanted advertisement, etc.) that they are not
prepared to accept. Therefore we cannot draw inferences from this analysis directly
to privacy concerns.
Moreover, the analysis showed that the majority of participants (54%) did not
change their bids at all.
H1b: Disabling special offers and deals leads to people decrease the amount they
demand to participate in the study.
This hypothesis is based on the conditions given to Group 1. Contrarily to the
assumption, disabling special offers and deals equally lead to an increase in the
amount the participants ask. Thus, the hypothesis must be falsified for the sample.
Paradoxically, in absolute values the increase in the amount is even higher than in
Group 2 (7,31 (6,16%) vs. 4,10 (8,5%)). We assume this result emerges because
participants feel that a benefit, which was initially promised, is withdrawn. Yet
again, no explicit link to (location-) privacy issues can be made. We assume that in
143

general the participants of this group were not concerned about privacy or data
sharing with third parties via the application.
As in the previous group, we can see that the majority of participants did not
change their bid between the first query and the revision.
H1c: The increase (in H1a) is higher than the decrease in H1b.
Seen that we had to falsify the previous hypothesis, this applies also to H1c. Both
groups raised their bids after the change of the conditions for the application.
It is important to point out that the sizes of the test groups have been small; comparison
between them is not significant. To achieve this, bigger sample groups are necessary.
Additional, explanatory qualitative data is needed to understand the reasons and intrinsic
factors of the decisions. This can also influence the development of the later technology.
At the moment we can only assume multiple factors causing the increase or decrease of
the bids. The assumptions demand adequate follow-up research. As an eventual resolving
we propose the replication and continuation of the study in a Living Lab.
7.1 Envisaged deployment in Living Lab setting
As indicated, the pre-study fulfilled multiple purposes though the current intention is to
deploy this research in a Living Lab where it is possible to further involve the participants.
Living Labs and quasi-experiments are in essence very different but we argue that they are
complementary rather than incompatible. There is a huge contrast between both in terms
of artificiality with Living Labs aiming at providing a naturalistic context while
experiments aim at isolating the impact of tested independent variables. Such contrast
may explain why experimental methods are not typical in Living Labs. However,
expanding quasi-experiments into Living Labs allows, after a preliminary phase of
verification of isolated specific phenomena, to expand the understanding of dynamics
within the Living Labs. Being able to control variables is the essence of an experiment, and
it can help explain economic or psychological processes that are otherwise hard to explain
or infer from surveys, interviews or cocreation methods.
The Living Lab approach allows to support the constant, transparent and direct
communication with the participants. Also the (random) sampling of the participants or
the pre-division into LBS-users and non-LBS-users is facilitated in such an environment
compared to universities or other sample pools. The research entails the management of
two similar app prototypes and two test groups that enable a broad range of further
research possibilities and direct comparison between groups in all further research. In the
Living Lab it is foreseen to execute the proposed study for the requested compensation
followed by research throughout the tracking phase and post-evaluation.
A multitude of questionnaires and surveys are foreseen in different directions. Regarding
the privacy aspect, they entail participants perception of privacy intrusion or (undesired)
personal data sharing especially regarding the location data. The economic angle will
focus on the value of the commercial benefits. We prepare to evaluate the self-estimation
and assessment of the users versus the data from the actual usage. Regarding the design
and technology, the aim is to get insights for the value of the application, the design and
look and feel and suggestions for improvements. Naturally, these follow-up and/or
comparative research can leverage a multitude of methods, amongst them focus groups,
in-depth interviews, surveys and other forms of questionnaires.
Finally, while we have argued that Living Labs and quasi-experiments are complementary,
there are rmethodological challenges. Living Labs by definition have always been about
involving all stakeholders in innovation, and covering all relevant aspects of the
144

innovation, including functionality, design, usability and business model aspects. In such a
view, user-driven innovation and open innovation are supposed to go hand in hand.
However, in practice, Living Labs have focused on user innovation as well as societal
impacts. Therefore, making the shift (e.g. to introduce quasi-experimental approach) is a
challenge because most Living Labs have good experience working with end-users,
setting-up user panels, etc. The case we have developed gives oneway to remedy this based
on the strengths of a Living Lab, i.e. its user panel and user involvement.
8 Conclusion
Though Living Labs proved their usefulness and deployability for some time, only a
limited potential of Living Labs is used mainly the shaping and development of
technology in co-creation sessions with users. In this paper, we have argued that Living
Labs can and should be used for other (social) research methods, such as experimental
research, in particular with an economic approach. The paper also explains why and how,
thanks to the management of its participants, Living Labs can build a perfect pool to
conduct experiments (or - less restrictive - quasi-experiments). Furthermore, they can
facilitate recruiting, communicating, sampling and profiling of participants more easily
than any other means of engaging participants. Living Labs therefore enable a longer
engagement of participants in the experimentation and eventual combinations with other
research methods.
A field that we see profiting from the deployment in a Living Lab is research on locationbased services and its controversial position between personal data sharing and benefit
dispersion. Usually, studies in this area are risking distortion and inaccuracy, caused
amongst others by the privacy paradox or discrepancy between the responses of people
and their actual behavior. By (initial) deception, the experimental approach aims to bypass
this problematic or make the people aware of its existence to derive more accurate results.
The experimental approach suggested in this paper has been previously tested with two
research groups according to two universities in Belgium but was limited to only to the
first part of the study. Like in previous research, participants were deceived to reveal their
request for participating in a study that was not conducted for real. The deception was
used as a trick to collect data about the value of personal location data. Two LBS
applications were outlined and provided to the two test groups: one where special offers
and deals were enabled, one where it was disabled. Paradoxically, both groups raised their
bids after the change of the conditions for the application. Further research is needed to
find an explanation of this result and recommendations for application providers.
We argue that in a Living Lab setting, this (so far fictional) study could be conducted for
real and bring more relevant value of a multitude of research fields such as privacy, the
value of data, the economics of personal data sharing and the design and technology of
modern LBS applications. The (quasi-) experimental approach proposed here is
complementary to the typical Living Lab approach. In conclusion, this would contribute to
Living Labs mission of bridging experimental setting and real-life use or adoption of
technology.
9 Bibliography
Ariely, D. (2008). Predictably irrational: the hidden forces that shape our decisions (1st ed). New
York, NY: Harper.
Ballon, P., Pierson, J., Delaere, S. (2005). "Test and experimentation platforms for
broadband innovation: examining european practice", in Conference Proceedings of 16th

145

European Regional Conference by the International Telecommunications Society (ITS), Porto,

Portugal, 4-6 September.
Bryman, A. (2012). Social research methods (4th ed.). Oxford; New York: Oxford University
Press.
Chang, E., & MacMillan, D. (2012, August 2). Foursquare Says Merchant Services Will
Provide Bulk Of Revenue [Online Version of a multinational mass media house]. Retrieved
July 20, 2012, from http://www.bloomberg.com/news/2011-08-02/foursquare-ceo-saysmerchant-services-will-provide-bulk-of-startup-s-sales.html
Couts, A. (2013, January 6). Terms & Conditions: If you care about privacy, dont use
Foursquare. Retrieved January 8, 2013, from http://www.digitaltrends.com/web/termsconditions-foursquare/
Cramer, H., Rost, M., & Holmquist, L. E. (2011). Performing a check-in: emerging practices,
norms and conflicts in location-sharing using foursquare. In Proceedings of the 13th
International Conference on Human Computer Interaction with Mobile Devices and Services (pp.
5766).
Stockholm,
Sweden:
ACM.
Retrieved
from
http://dl.acm.org/citation.cfm?id=2037384
Cvrcek, D., Kumpost, M., Matyas, V., & Danezis, G. (2006). A study on the value of location
privacy (pp. 109118). Presented at the 5th ACM Workshop on Privacy in Electronic Society
(WPES), Alexandria, Virginia. Retrieved from http://dl.acm.org/citation.cfm?id=1179621
Danezis, G., Lewis, S., & Anderson, R. J. (2005). How much is location privacy worth?
Presented at the Fourth Workshop on the Economics of Information Security, University
of
Cambridge,
United
Kingdom.
Retrieved
from
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.236&rep=rep1&type=pdf
Eriksson, M., Niitamo, V.-P., & Kulkki, S. (2005). State-of-the-art in utilizing Living Labs
approach to user-centric ICT innovation-a European approach. Lulea: Center for DistanceSpanning Technology. Lulea University of Technology Sweden: Lulea. Retrieved from
http://www.vinnova.se/upload/dokument/verksamhet/tita/stateoftheart_livinglabs_eriksso
n2005.pdf
Fuchs, C. (2012). The Political Economy of Privacy on Facebook. Television & New Media,
13(2), 139159. http://doi.org/10.1177/1527476411415699
Hui, K.-L., & Png, I. (2006). Economics of Privacy. In T. Hendershott (Ed.), Handbooks in
Information Systems. Economics and Information Systems (Vol. 1, pp. 471499). UK: Elsevier
B.V. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=786846
Milgrom, P. R., & Weber, R. J. (1982). A Theory of Auctions and Competitive Bidding.
Econometrica, 50(5), 10891122. http://doi.org/10.2307/1911865
Mulder, I., & Stappers, P. J. (2009). Co-creating in practice: results and challenges. In
Collaborative Innovation: Emerging Technologies, Environments and Communities (Proceedings of
the 15th International Conference on Concurrent Enterprising: ICE 2009, Leiden, The Netherlands,
2224 June 2009). Centre for Concurrent Enterprise: Nottingham, UK. Retrieved from
http://www.ictusagelab.fr/ecoleLL/sites/default/files/202_Ingrid_Mulder_and_Jan_Stappers
.pdf
Norberg, P. A., Horne, D. R., & Horne, D. A. (2007). The privacy paradox: Personal
information disclosure intentions versus behaviors. Journal of Consumer Affairs, 41(1), 100
126.
Staelens, N. (2012). Assessing the importance of audio/video synchronization for
simultaneous translation of video sequences. Multimedia Systems, 18(6), 445457.
http://doi.org/10.1007/s00530-012-0262-4
Veeckman, C., Schuurman, D., Leminen, S., Lievens, B., & Westerlund, M. (2013).
Characteristics and Their Outcomes in Living Labs: A Flemish-Finnish Case Study. In
XXIV ISPIM Conference: Innovating in Global Markets: Challenges for Sustainable Growth.
Retrieved
from
146

http://www.researchgate.net/profile/Carina_Veeckman/publication/259174995_Characterist
ics_and_Their_Outcomes_in_Living_Labs_A_FlemishFinnish_Case_Study/links/542003d80cf203f155c298af.pdf
Vogt, W.P., Johnson, R.B. (2011), Dictionary of Statistics & Methodology: A Nontechnical Guide
for the Social Sciences, SAGE.
Webster, M., Sell, J. (2014), Why Do Experiments? in Webster, M., Sell, J. (eds),
Laboratory Experiments in the Social Sciences, Academic Press p.5-22.

147

Organisation of labour, quality of work,

and relational coordination
in Care Living Labs
Leen De Kort a, Ezra Dessers a, Geert Van Hootegem a
a

Centre for Sociological Research (CeSO) of the KU Leuven (Belgium)

Leen.DeKort@soc.kuleuven.be

Abstract
A growing interest can be noticed in living labs as a method to test and develop
innovations in health and social care. This paper argues that, apart from the care clients,
also care professionals deserve attention when care living labs are designed. The Care
Living Labs program in Flanders (Belgium) is used as a case to show that innovations in a
living lab context can profoundly influence the work of the care professionals. After a brief
theoretical description of the concepts organisation of labour, quality of work and
relational coordination, the paper explains how these concepts are applied in the Flemish
Care Living Labs. The paper concludes that care professionals can best be involved from
the start as a separate target group in future care living labs.
Keywords
Organisation of labour, quality of work, relational coordination, care, living labs
1 Introduction
With the aging of the population, the number of older adults in Europe increases, leading
to a growing number of people in need of some form of chronic care (European
Commission & Economic Policy Committee, 2014). This demographic trend can also be
seen in Flanders (Belgium) (Paulus, Van Den Heede, & Mertens, 2012). In order to tackle
this disequilibrium, innovations in the older adult care are becoming increasingly
important. Recently new methodologies that help stimulate innovation have made their
appearance. One of those methodologies is the living labs approach (Mulvenna et al.,
2010). The living labs grew from the idea that innovations should be tested in real life
situations by the actual end users. In the field of older adult care, the older adults are the
end users. Because of the growing need for chronic care for older people, in 2012, an
extensive, three-year, living labs based program has been initiated in Flanders to
stimulate innovation in the domain of older adult care. The program goes under the name
Care Living Labs (in Dutch: De Vlaamse Zorgproeftuinen). A scientific consortium is
entrusted with the scientific support of the program. The program is funded by the
Flemish Agency for Innovation by Science and Technology (IWT), and kicked off in
September 2013. It comprises 23 projects, grouped in six regional platforms. The projects
are very diverse, with topics including technology, housing, informal care networks,
148

mobility, care and cure integration, and care management. Enhancing the quality of life of
older adults being the purpose of this initiative, older adults were defined as the target
group. As can be seen from the list of topics above, innovations in care involve more than
testing technologies or user interfaces. Even when technology finds a place in care Living
Labs, it is always as part of a broader social innovation. Not only older adults, but also
other stakeholders are involved in such a social innovation. The innovations may directly
or indirectly influence daily practices of caregivers and care professionals. This paper
argues that, apart from the care clients and the informal caregivers, also care professionals
should be involved as a target group in a care living lab. A care professional is defined as
someone who acquired a care-related certificate and is legally authorized to offer care or
advise about care, in exchange for a remuneration (FOD Sociale Zekerheid, 2014).
The paper has four parts. First, the relevance of care professional-related issues in the Care
Living Labs in Flanders is described. Second, an overview is given of the three concepts
that are used to analyse these issues: organisation of labour, quality of work, and
relational coordination. Third, the application of these concepts in living lab-related
activities are presented. The paper ends with some concluding remark.
2 The Care Living Labs in Flanders
As explained in the previous Section, in the Flemish Care Living Labs, six platforms were
established who conduct all together 23 projects. Although the care professionals were not
explicitly declared as a target group, it was implicitly mentioned that they are part of the
end users, by stating that testers in the own working environment should be involved.
Based on the original platform and project plans a plan evaluation study was made (Leys et
al., 2015). The study describes the goals, target groups, collaborations, organisation of
labour, technology and the scope of the platforms and the projects. It was striking that
none of the platforms expressed the aim to work around themes as organisation of labour
and quality of work. The concept of organisation of labour indicates the way in which a
set of tasks, needed to create a product or service, is divided into different work packages
(Van Hootegem, 2000). Quality of work refers here to the stress and health risks resulting
from ones job and is seen as the result of the combination between the amount of
workload and the ability to handle challenges the job imposes on the worker (Van
Hootegem, 2000). Only one project plan explicitly mentioned these aspects. It needs to be
said that when the Flemish government in 2012 launched the call to establish care living
labs in Flanders, it was not stated that platform and project proposals needed to pay
attention to the organisation of labour or to the quality of work of the care professionals.
The scientific consortium however introduced this dimension, because the plan evaluation
study revealed that aspects of organisation of labour and the related quality of work could
be crucially important for the projects within the six platforms. An example is that various
project plans strive to apply technological tools to increase cross-organisational
coordination, encompassing a possible risk for increased bureaucratization. Another
example is that the introduction of care managers could draw attention away from the
underlying problem, being the fragmentation of the organisation of labour. A third
example is that various projects are focused on care integration at the micro-level of care
professionals and informal caregivers, which could affect tasks and roles of both groups.

149

3 Organisation of labour, quality of work, and relational coordination

To capture the concepts organisation of labour and quality of work, a sociotechnical approach (De Sitter, 1994) is used. The notion of relational coordination
is coming from the work of Jody Hoffer Gittell (2002).
3.1 Organisation of labour
The concept of organisation of labour refers to the way in which a set of tasks,
needed to create a product or service, is divided into different work packages (Van
Hootegem, 2000; Van Hootegem et al., 2008). Essentially there exist two ways of
dividing labour. A functional organisation of labour stands for an organisation that
is focused on the different operations needed to complete a product or service.
These different operations are identified and isolated in specialized units. This
extensive division asks for a lot of additional coordinating tasks, which are usually
assigned to a separate unit, located higher in the organisational hierarchy. On the
other hand, when organising labour in streams, the focus is on types of products
or clients, and work is organised in parallel streams. Relatively independent teams
are formed around the different types of orders. The way in which these tasks are
divided, influences the amount of job control. When labour is organised in
streams, job control can be higher than when work is organised in a functional
way. With a functional organisation of labour, each unit is dependent on the work
of the units in which the specialized tasks earlier in the production process happen.
Therefore a high level of alignment is needed. To be able to overlook all the
different alignment tasks that are needed, coordination will be centralized on a
higher level, which results in less job control in individual jobs and units.
Organising labour in streams allows for a more decentralized approach because
there is less need for alignment. This reduced need for alignment makes it possible
to assign more job control to individual jobs, teams and units. (Van Hootegem et al.,
2008).
Within the context of the Flemish Care Living Labs, the organisation of labour
actually concerns the organisation of care processes. These are the processes that
need to coordinate between interoperating care professionals and organisational
units (Lenz & Reichert, 2007). A care client rarely has to deal with only one care
unit. Because of the functional way in which traditional care organisations in
Flanders often are organised, they usually run through a chain of activities located
in different units (Kuipers, 1992). Activities of these different unites need to be
planned, prepared, executed and supported. The more alignment is needed
between these different tasks, the higher the chances that malfunctions emerge.
This can result in errors and long waiting times for the care clients (Lenz &
Reichert, 2006). Rethinking the organisation of the care process in terms of care
client flows may radically diminish malfunctions that arise from alignment
problems (Kuipers, 1992).
3.2 Quality of work
The interpretation of quality of work comes from the Job Demand / Control
(JD/C) model of Karasek (1979). This model has been validated by over 30 years of

150

scientific research. The model describes how functional characteristics of a job can
influence stress and health risks. The model seeks to predict stress and health risks
based on the features of the job within the organisation (and not based on other
factors, such as personal characteristics of the person who performs the job). The
tasks which are connected to a specific job have regulatory requirements: when
malfunctions and deviations appear and when unexpected events happen,
decisions have to be made to deal with these. The decision-making power of the
worker determines whether the worker is able to face these demands.
In the JD/C model, two dimensions regarding the characteristics of jobs are defined
(figure 1): on one side there are the job demands or the workload, on the other side
there is the level of job control, or, in other words, the decision-making power one
has over his/her job. The JD/C model indicates that psychological strain is not so
much the result of the workload, but the result of the joined effect of workload and
the level of decision-making power a worker has to handle the demands that result
from this workload. Four different types of jobs arise from the combination of these
two dimensions. The jobs in which both job demands and job control is low are
called passive jobs. In low strain jobs, the job demands are low, but the job
control is high. Probably the worst case is the high strain job, in which the job
demands are high, but job control is low, which makes coping with challenges
difficult. In active jobs a good equilibrium is found between high job demands and
high job control. Employees with an active job have the ability to regulate their
own work. They can autonomously develop working strategies and improve and
change these when the tasks demand so. Karasek (1979) concludes that a
demanding job as such is not necessarily stressful, as long as the worker has
enough decision-making power to cope with challenges related to the job.

Figure 1: The JD/C model (Karasek, 1979)

151

The JD/C model has been validated across the years in many different studies. The
negative impact of high strain jobs on health and wellbeing has been described in
many papers. Next to a poor general health (Karasek, 1990; Sderfelt et al. 1997) it
can lead to mental strain (Karasek, 1979), emotional exhaustion, physical
complaints and job dissatisfaction (De Jonge et al., 2000), burn-out and rebellious
behaviour (Bakker et al. 2001), increased blood pressure and cortisol levels (Fox et
al. 1993; Everson et al. 1997; Schaubroeck & Merrit, 1997), and an increased risk at
cardio-vascular diseases (Johnson & Hall, 1988).
3.3 Relational coordination
Care processes usually require a lot of cooperation and coordination between
different units and between people working in different disciplines. Organisational
rules define how this alignment officially happens. Research by Gittel (2002)
demonstrates that the performance of a care organisation is not only mediated by
the way the care process is formally structured, but also by more spontaneous
forms of coordination. For these spontaneous forms of coordination the term
relational coordination was coined. In relational coordination much importance is
attached to the role of relationships. Shared goals, shared knowledge and mutual
respect are considered to be the key factors for effectively coordinating a care
process. Gittell states that when workers have shared goals, they might be
motivated to move beyond the sub goals attached to their own tasks, which would
make them act in order to contribute to the overall care process. Shared knowledge
could enable the care professionals to see how their own tasks fit with the tasks of
colleagues and could therefore give them an understanding of the overall work
process. Having respect for the work of others would have as a consequence that
workers will value the contribution of other actors in the care process and it might
make them consider the impact of their own actions on others. This could lead
them to act with respect to the overall care process. She suggests that when these
characteristics of a relationship are present frequent, it could stimulate an in time,
accurate and problem solving way of communicating. This type of communication
in turn, would decrease obstacles to coordination and thus enable employees to
more effectively manage their task interdependencies. As a result more effective
and qualitative performances could be expected (Gittell, 2002).
4 Care Living Lab activities
Although the initial platform and project plans do not mention organisation of
labour, quality of work, and relational coordination, they do describe the need for
cross-organisational coordination and integration of care between care
professionals and informal caregivers. Also the installation of new functions such
as case managers was indicated. These actions can affect the tasks and roles for the
care professionals. Thus, the scientific consortium took steps to engage the
platforms and projects to work on the concepts of organisation of labour, quality
of work, and relational coordination.
In a first step, the platforms were asked to add the care professionals involved in
their projects to their user panels. Surveys were developed to assess the quality of
work and relational coordination in different groups of care professionals involved

152

in the projects. Quality of work was conceptualized through the dimensions job
demands and job control. Relational coordination concerns both the characteristics
of the relationships (shared goals, shared knowledge and mutual respect) and
characteristics of communication (frequency, accuracy, constructiveness and in
time). These surveys are send out to the care professionals in the user panels. The
results of the surveys should give a general indication of the situation regarding
quality of work and relational coordination in the projects, and provide concrete
starting points to tackle issues concerning these themes.
In a second step workshops will be organised per platform in which the projects
can dive deeper into these matters. During the workshop the concepts organisation
of labour, quality of work, and relational coordination are first briefly
introduced, after which people from the different projects will apply the concepts
to the care processes in which their innovations take place. With support from
researchers from the scientific consortium, they will investigate whether their
innovations influences the organisation of labour and the quality of work of the
care professionals that are involved in their projects. The results from the survey
that was mentioned above will serve as a starting point. After identifying specific
issues, the projects can look into possible structural interventions, with support
from the scientific consortium.
With the information retrieved from the surveys and the workshops, the scientific
consortium can draw lessons concerning the use of living labs methodology in the
care domain. The collected information should allow to answer the question to
what extent it would be useful to involve care professionals and in which way this
can best be done. Finally the information can help to understand in which way an
optimized organisation of labour could be a breeding ground for care innovations.
5 Conclusion
Innovations in older adult care involve more than testing technologies or user
interfaces. These innovations always take place in a broader social context.
Therefore, when using a living lab approach to test and develop innovations in
older adult care, also the care professionals are an important end-user group. The
case of the Flemish Care Living Lab program shows that this end-user group is not
always given a lot of attention in care living lab activities. This paper gives an
overview of three concepts that can be used to analyse care professional related
issues. These concepts are organisation of labour, quality of work and relational
coordination. Also does the paper present how these concepts can be applied in
living lab related activities.
The current research in Flanders suggests that organisation of labour, quality of
work and relational coordination can play an important role in care living labs. For
that reason, care professionals should best be involved as a separate target group in
future care living labs.
6 Acknowledgements
This research is part of KIO, an interuniversity consortium studying innovations in
elderly care in Flanders. The consortium consists of Mark Leys & Lien Pots (OPIHVUB), Ellen Gorus & Charlotte Brys (GERO-VUB), Ezra Dessers, Geert Van

153

Hootegem & Leen De Kort (CESO-KUL), Marc Jegers & Lukas Versteele (iCher),
Patricia De Vriendt (GERO-VUB, Arteveldehogeschool) & Juul Lemey
(Arteveldehogeschool), Bart Jansen (ETRO-VUB), Bart Mistiaen & Bart Grimonprez
(HOWEST). The consortium is financed by the Flemish Agency for Innovation by
Science and Technology (IWT).
7 Reference list
Bakker, A.B., Demerouti, E. & Euwema, M.C. (2005). Job resources buffer the
impact of job demands on burnout. Journal of Occupational Health Psychology, 10(2),
170 180.
Dhondt, S., Pot, F. & Oeij, P. (2012). Social innovation of work and employment. In:
H.W. Franz, J. Hochgerner & J. Howaldt (Eds.), Challenge social innovation. potentials
for business, social entrepreneurship, welfare and civil society (pp. 261-274). Berlin,
Germany: Springer Verlag.
de Jonge, J., Bosma, H., Peter, R. & Siegrist, J. (2000). Job strain, effort-reward
imbalance and employee well-being: a large-scale cross-sectional study. Social
science & medicine, 50(9), 1317-1327.
De Sitter, L.U. (1994). Synergetisch produceren. Human resource mobilisation in de
productie: een inleiding in de structuurbouw. Assen, The Netherlands: Van Gorcum. [in
Dutch]
European Commission (DG ECFIN) & Economic Policy Committee (AWG). (2014).
The 2015 Ageing Report: Underlying Assumptions and Projection Methodologies.
European Economy, 8.
Everson, S.A., Lynch, J.W., Chesney, M.A., Kaplan, G.A., Goldberg, D.E., Shade,
S.B., Cohen, R.D., Salonen, R., Salonen, J.T. (1997). Interaction of workplace
demands and cardiovascular reactivity in progression of carotid atherosclerosis:
population based study. British Medical Journal, 314, 553-558.
Federale overheidsdienst (FOD) sociale zekerheid. (2014). Wet betreffende de
erkenning van de mantelzorger die een persoon met een grote zorgbehoefte
bijstaat. Publicatie: 2014-06-06. [in Dutch]
Fox, M. L., Dwyer, D. J. & Ganster, D. C. (1993). Effects of stressful job demands and
control on physiological and attitudinal outcomes in a hospital setting. Academy of
Management Journal, 36(2), 289-318.
Gittell, J.H. (2002). Coordinating mechanisms in care provider groups: relational
coordination as a mediator and input uncertainty as a moderator of performance
effects. Management Science, 48(11), 14081426.
Johnson, J.V. & Hall, E.M. (1988). Job strain, work place social support, and
cardiovascular disease: a cross-sectional study of a random sample of the Swedish
working population. American Journal for Public Health, 78, 1336-1342.
Karasek, R. A. (1979). Job demands, job decision latitude, and mental strain:
implications for job redesign. Administrative Science Quarterly, 24 (2), 285-308.
Karasek, R.A. (1990). Lower health risk with increased job control among white
collar workers. Journal of Organisational Behavior, 11(3), 171-185.
Kuipers, H. (1992). Sociotechniek in de gezondheidszorg. Paper gebaseerd op een lezing
gehouden tijdens het symposium Sturing van zorgprocessen. [in Dutch]

154

Lam, A. (2004). Organisational innovation. In: Fagerberg, J., Mowery, D. & Nelson,
R.R. (Eds.), Handbook of innovation. Oxford, United Kingdom: Oxford University
Press.
Lenz, R., Reichert, M. (2007). IT support for healthcare processes premises,
challenges, perspectives. Data & Knowledge Engineering, 61, 39-58.
Leys et al. (2015). Zorg Proeftuinen Vlaanderen. Een inhoudelijke vergelijking van
de platformen en projecten bij de start. Brussel, Belgium: Kennisplatform Innovatie
Ouderenzorg. [in Dutch]
Mulvenna, D.M., Bergvall-Kareborn, B. Galbraith, J.W. Martin, S. (2010). Living labs
are innovation catalysts. In Howlett, R.J. (Ed.), Innovation through knowledge transfer
(pp. 253-264). Berlin, Germany: Springer-Verlag.
Paulus, D., Heede, K. & Mertens, R. (2012). Position paper: Organisatie van zorg voor
chronische zieken in Belgi. KCE Reports. Tilburg, The Netherlands: Federaal
Kenniscentrum voor de Gezondheidszorg. [in Dutch]
Schaubroeck, J. & Merrit, D.E. (1997). Divergent effects of job control on coping
with work stressors: the key role of self-efficacy. Academy of Management Journal, 40,
738-754.
Sderfeldt, B., Sderfeldt, M., Jones, K., Ocampo, P., Mountaner, C., Ohlson, C.G., &
Warg, L.E. (1997). Does organisation matter? A multilevel analysis of the demand
control model applied to human services. Social Science and Medicine, 44, 527534.
Van Hootegem, G. (2000). De draaglijke traagheid van het Management. Tendensen in
productie- en personeelsbeleid. Leuven, Belgium: Acco. [in Dutch]
Van Hootegem, G., Van Amelsfoort, P., Van Beek, G. & Huys, R. (2008). Anders
organiseren en beter werken: Handboek sociale innovatie en verandermanagement.
Leuven/Voorburg, Belgium: Acco. [in Dutch]

155

Creating a Smart City Vision in a Living Lab

Case Study of Smart Kalasatama
Vision-building Process
Veera Mustonen a
a

Forum Virium Helsinki/ Aalto University, Helsinki

Veera.mustonen@forumvirium.fi

Abstract
This paper examines collaborative vision-building process as a key ingredient of a
Smart City Living Lab. As a case study, the Smart City district Living Lab, Smart
Kalasatama, vision-building process is analyzed. Part of the process was
benchmarking various Smart City visions and objectives. Most Smart city visions
were analyzed as rather system-centric. Contrary to that, the outcome of this Living
Lab vision process was integrating systemic and human goals. One more hour of
own time a day Smart City vision statement was co-produced in public-privatepeople partnership (PPPP). As a conclusion, two hypothesis are formed proposing
that Living Lab methods and PPPP-process (1) yield more balanced yet humancentric Smart City visions which (2) appear engaging to a variety of stakeholders.
Keywords
Smart City, Urban Living Lab, Vision, PPPP-process, Quadruple helix, Smart
Kalasatama
1 Introduction
Living Labs have become places to drive open innovation in real life context.
Recently many Living Labs have started to operate in the Smart City context
addressing real urban development (Cosgravea;Arbuthnot;& Tryfonas, 2013). How
the Living Labs impact what the Smart City innovations aim at? This paper
examines how a Living Lab can contribute to a shared Smart City district vision
building process. To date, there is very little research of the collaborative vision
building in Living Lab context. In order to understand the ingredients of visionbuilding the Smart City visions are approached through four different analytical
focus.
First, the dimensions of the Smart City visions are scrutinized using two different
frameworks. The first framework is used in many Smart City concept analysis: topdown vs. bottom-up approaches to Smart City development (Townsend, 2013)
(Breuer;Walravens;& Ballon, 2014). The second framework contrasts the ends the

156

cities are seeking. Here we hypothesized that most cities take the ultimate vision of
Smart City development to be linked either to improve sustainability or quality of
life. A balanced middle is proposed for both frameworks. Secondly, this paper
analyzes how the two frameworks appear empirically in Smart City visions.
Thirdly, we present and analyze quadruple helix model vision-building process
case study of Smart Kalasatama vision building-process. The fourth analytical focus
is to evaluate the outputs and early outcomes of the vision-building process. We
conclude by providing two hypothesis of how a Living Lab process of Smart City
vision-building impacts the content of the vision.
This paper is structured so, that first we discuss smart city and benchmarking of
smart city visions and propose a framework for those.. Then we present the case
study of the Smart Kalasatama district programme Living Lab vision-building
process. Finally we analyze and discuss the learning.
2 Building a Smart District Kalasatama in Helsinki
Building smart city districts and creating smart city initiatives has become a major
trend in municipal development during the past decades (Neirotti;Marco;&
Cagliano&al, 2014) (Andrea & Peter, 2011). Respectively, the City council of Helsinki
decided 2013 that Kalasatama will be the model district of smart urban
development, thus the primary lieu of the Smart City innovation in Helsinki.
Kalasatama can be viewed as part of the global Smart City trend, where the
municipality is seeking new ways to address public issues with diverse partnerships
and utilizing ICT and data as enablers. The Smart City concept has been defined in
many ways (Hollands, 2008) (Neirotti;Marco;& Cagliano&al, 2014) (European
Parliament, 2014)) and the concept remains controversial. Still, a Common
frameworks can be identified behind the definitions, consisting of the drivers
(urbanization, climate change, digitalization), the enablers (ICT technology, data),
the collaborative public-private-people partnership (PPPP-) model and the targets
(resource efficiency, better quality of urban life and economic growth)
(Mustonen;Koponen;& Spilling, 2014), (Neirotti;Marco;& Cagliano&al, 2014)
(Chourabi & al., 2012).
The new Kalasatama district of Helsinki, under construction on former harbor is
located by the sea close to the old city center. The district will see some 175 hectares
re-developed with a total of 1.3 million square metres of homes, offices and service
areas. Construction of Kalasatama began in 2011 and will continue until the
2030s, by when it will be home to over 20,000 residents and a working district for
8,000 people. The City hall initiated a smart city innovation platform program
called Smart Kalasatama. The program has wide scope and concept, including a
number of different industries and different solutions from smart grids to eHealth
and smart retail. Further, the wish of the public governors is that innovations
developed and tested in Kalasatama will scale up elsewhere thus creating jobs and
wealth in the region.
Globally, a lot of critique of the smart city concept has been presented, especially
that the approach is too technocentric, top-down, driven by big global industry
players (such as Cisco or IBM) and leading municipalities to vendor/technology
lock-in (Kitchin, 2014). However, that critique seems already outdated, as the
second wave of the smart city development emphasizes more the role of

157

municipalities, entrepreneurs and citizens. There is a considerable bottom-up

interest in smart city scene to utilize new digital enablers to create more
participatory urban solutions and increase livability (Hill, 2013). The Smart City
concept has evolved from rather techno-driven approach to more smart citizen
centric. (Baccarne;Schurman;Mechant;& Marez, 2014). The urban Living Labs have
a role to play to facilitate quadruple helix collaboration, to balance the interests
and enable engagement of the citizens in the smart city development. (Paskaleva,
2011).
3 Smart Kalasatama Living Lab
To make Smart Kalasatama grow from the genuine collaboration between the city,
companies, academia and the residents a Living Lab approach was chosen. Smart
Kalasatama Living Lab can be seen as an engagement platform (Ramaswamy &
Gouillart, 2010) following open innovation 2.0 quadruple helix model, where the
municipality,
industry, academia and citizens collaborate to co-create new smart city ideas,
services and prototypes (Baccarne;Schurman;Mechant;& Marez, 2014).
The City of Helsinki assigned the co-ordination of Kalasatama Living Lab activities
to its development and innovation agency Forum Virium. The City finances the
Lab with help of other public funds. The Living Lab started virtually in the fall of
2013 and is looking for physical premises to open in the fall of 2015. Currently the
Living Lab defines itself as an orchestrator, or innovation intermediary, of Smart
City development. The focus in Kalasatama is in agile smart city service piloting.
There are around 20 projects in the program portfolio and the Living Lab works to
define new solutions to be co-created and tested in Kalasatama. A key role of the
Living Lab is to find system integrators to drive co-designing and testing of new
Smart City solutions such as Mobility as Service trials or Internet of Things LANs
serving sharing economy in residential buildings. The Living Lab facilitates new
project creation, ecosystem building, helps with collaborative methods and
organizes workshops and other events. The projects themselves are owned and
carried out by city departments, large and small companies, NGOs and resident
organizations. The funding of the projects varies accordingly, some of them being
large scale industry investments, others procurements by the City itself. The Living
Lab is involved in finding funding for small partners (SMEs, NGOs) and is running
itself some small-scale agile piloting.
All in all, Smart Kalasatama Living Lab orchestrates a complex network with
hundreds of different stakeholders coming from diverse different public and
private organizations, having expertise in varied domains. Thus, the Living lab
faces the challenge of leading a complex system where the stakeholders have very
different targets for the new district being built. From the beginning, the Living Lab
has chosen a strategy to embrace the complexity: All stakeholder groups from
quadruple helix model are encouraged to participate. The smart district
development allows multitude of different and even competing approaches
simultaneously. Still, to keep the loose network together, a common purpose or
shared vision was needed. The necessity to share a loose goal to direct the different
smart city initiatives towards the same direction, initiated a vision process.

158

4 Role of Living Lab in Forming a Shared Vision of a Smart City District

So, upon starting the Kalasatama Living Lab work 2013, the need for common vision
to form a basis for multistakeholder collaboration. The importance of a shared
vision as success factor of Smart Cities is already recognized by institutions, such as
EU Open innovation strategy and policy group stating that: Shared value is the
value created at the intersection of corporate performance and society when big problems
are solved. Shared value is best achieved in the context of a shared vision. (Curley &
Salmelin, 2013).
In a complex network, a good vision engages a variety of stakeholders. It defines a
common dream state and leads the network loosely to the same direction.
Moreover, a common vision is needed to operationalize smart city implementation
in the city organization. As a study (ARUP, 2013) of European Smart Cities suggests,
a vision of smart is needed to supports strategic alignment and investment in
technology across departments. Furthermore, it enables coupling SMEs and
bottom-up development with larger scale city-driven initiatives.
The sought after vision was defined as the dream state of which Smart Kalasatama
will be known in 20 years by the project steering group (which consisted of the City
and industry directors) in January 2014. Further, the criteria for the vision consisted
of the following attributes: The vision should be compelling and inspiring for most
of the diverse stakeholders, showing common direction for the smart city
developers and still being
broad enough to encompass variety of mindsets. The vision should be born in
Helsinki, underlining the local needs and history of the district and thus also
differentiating it both nationally and internationally.
The main motivation of the vision building process in the Living Lab was to fortify
the commitment of the stakeholders representing all the sides of the quadruple
helix and having different interests. Thus a common vision addressing
simultaneously at least potentially controversial goals such as social inclusion,
economic growth, sustainability and improved quality of living, could become an
essential tool to negotiate possible conflict in the Smart City development.
5 Benchmarking Smart City Visions
Staring point of our vision building process was to benchmark other smart city and
district visions. As Hollands (Hollands, 2008) already remarks, cities are branding
themselves as smart. The self-celebrating branding of smart cities can be seen as
municipal storytelling, similar to the smart city corporate storytelling
(Sderstrm;Paasche;& Klauser, 2014). Regardless how the visions relate to reality,
we benchmarked the smart city visions as sought after futures. Our special interest
was in smart district development, so we paid attention to such cases as Masdar
(Abu Dhabi), Royal Seaport (Stockholm) and Nordhavn (Copenhagen). The
benchmarking material was used as props in the subsequent participatory
workshops. Our benchmarking activities consisted of study visits and desktop
studies. Copenhagen, Dubai and Masdar in Abu Dhabi were visited. Desktop
studies concentrated on European leading smart cities (as according to EU study
(European Parliament, 2014)) and neighbouring capitals: Wien, Amsterdam,
Barcelona, Stockholm, Copenhagen, London, Lyon, Paris, Oslo, Tallinn. Material

159

for the benchmark was the web sites of the cities/districts, studies and reports made
of the smart cities (e.g. (ARUP, 2013) (Bakiti;Almirall;& Wareham, 2013) and
conference materials.
5.1 Smart City Visions and Objectives- Learnings from the benchmark material
Smart City development is often seen to embrace all aspects of digitally enabled
urban development, so forming a clear vision is not trivial. Most cities bluntly state
that their aims are sustainability (resource efficiency/carbon neutrality), livability
(quality of life) and economic growth through their smart city initiatives. Thus, they
repeat the basic targets of any Smart City initiative, without adding local flavor or
more direction in their visions. Unlike the corporations, the smart city visions do
not emphasize the technological advancement, rather they tend to focus on the
municipal benefits. In our benchmark material we found few clear visions. Vision
statements may be foreign to public administration, which rather tries to express its
wide responsibility and accountability. Many of the cities we studied mixed vision,
mission and strategic objectives. So, we decided to scrutinize a greater variety of
objective statements than just straight vision statements.
Sustainability seems to the most underlined objective of Smart City initiatives,
following earlier popular concepts such as eco city, sustainable city or green
city. Many cities and districts have bland sustainability related visions like
Stockhoms Royal Seaport: A World class environmental urban district (City of
Stockholm, 2015). Copenhagens vision is to be the first carbon neutral capital by 2025
(City of Copenhagen, 2015). That is both concrete and ambitious sustainabilitydriven vision. Copenhagen has also sub-objectives sand strategies serving the big
target or carbon neutrality. For example the Danish capital has very clear bicycling
vision supported by concrete targets and policies.
One Smart City district having a concrete vision is Masdar City in Abu Dhabi.
Their vision is to drive sustainable energy industry (Wikipedia, 2015). Many urbanists
have overlooked Masdar as not a city at all rather a smart technology driven
campus built in the middle of desert. However, Masdar has never claimed to
become an European style city and has not sought citizen driven approaches.
Instead it has tried to follow its own vision, working on new energy related
innovations with strong industry partners. Masdar had indeed combined the two of
Smart City benefits, innovation (economic growth) and sustainability, in its vision
statement.
The vision of smart city bringing happiness to all (Emirates 24/7, 2014) was taken by
Dubai during spring 2014. That vision is truly human-centric putting the subjective
experience to be the ultimate measure of the advancement of the Smart City. Dubai
has followed the vision by launching a happiness index to measure its Smart City
success (Dubai Smart Gov, 2015). As we predicted, later during the year 2014 more
and more cities started to mix sustainability and wellness in rhetorical level in their
smart city visioning. Good example of that is Wien, which has a very
comprehensive Smart City strategy. Still, they keep their vision in the generic
Smart City definition level: quality of life, resource preservation and innovation (City of
Wien, 2015). In Barcelona Smart City Expo 2014 a big number of cities were proudly
presenting that Smart City is not just about smart solutions, but about smart
citizens. Instead of yielding interesting visions, the mixture of different targets

160

seems to have turned the Smart City discussion even more abstract and all
embracing. Apart from Dubai, clear vision statements emphasizing the quality of
life are hard to find. In Dubais case the connection between the Smart City
development and happiness is yet not clearly pronounced, although happiness is
measured (Dubai Smart Gov, 2015).
The cities benchmarked do not combine the sustainability and quality of living
targets in their visions. Closest comes City of Barcelona (City of Barcelona, 2015),
which had a human-centric vision statement Barcelona as a City of People to win the
award of European innovation capital.
Amsterdam is often appointed as one of the leading Smart Cities in Europe in (e.g.
(European Parliament, 2014)). However, their strategy has been not to have a vision,
just an innovation platform to bring quadruple helix model partners together to
innovate (City of Amsterdam, 2014). Their philosophy is to enable bottom-up urban
development and not to restrict it by centralized top-down visions.
Most other European cities benchmarked had similar visions/objective statements
as Wien or Stockholms Royal Seaport. All in all, the level of the Smart City/district
visions/ objective statements was not very crisp, even though the actual
development of those cities/districts may be of high quality.
Table below summarizes the main benchmarks.
City/District

Vision/Objective statement

Copenhagen

First carbon neutral capital by 2025.

Stockholm: Royal Seaport

A world class environmental urban district

Wien

Quality of Life, sustainability, Innovation

Amsterdam

-(bottom-up)

Barcelona

Barclona as a city of people.

Masdar City

Driving sustainable energy industry.

Dubai

Happiness of all citizens.

To sum up our benchmarking, we placed the visions in two axis. The X-axis places
sustainability and quality of life targets in different ends, leaving the balanced
sweet spot in the middle. The other axis we used to examine the cities vision
statements was top-down vs. bottom up. The current academic Smart City
discussion favors a balanced middle for successful smart city development
(Breuer;Walravens;& Ballon, 2014) (ARUP, 2013). According to research, change and
innovation have better chance to arise when top-down and bottom-up systems and
processes are integrated (Shephard & Simeti, 2013).
Interestingly, most of the examples above, are rather top-down. Although the
outcome of Smart Cities in Masdar and in the Scandinavian capitals is very
different, their approach and vision is focusing on the systemic top-down

161

efficiency. Likewise in Dubai, the system of happiness is centralized, the state

optimizing the happiness of the people top-down.
The smart city visions benchmarked were mainly stories written by the cities,
which explains their rather centralized top-down approach. The cities suggest large
scale (infra) projects pushing new solutions to cities to achieve common good. This
approach is at least partially contradictory to the Living Lab as a Smart City
platform (Breuer;Walravens;& Ballon, 2014) paradigm to involve citizens and other
bottom-up developers like SMEs in the process. Although, we do not know the
processes behind in forming the benchmarked smart city vison/objective
statements, it seems that citizens and other bottom-up focused stakeholders were
not involved. Seems that as the first wave of Smart City vision were rather technocentric and driven by the industry, the current wave is municipally led.
The learning from the benchmarking taken to the Kalasatama vision process, was
to focus on the middle and try to encorporate views of all the quadruple helix
groups.
The table below summarizes the benchmarks in two axis:

Smart&City&Visions&
Masdar&

Top0down&

Dubai&

Stockholm:*Royal*Seaport*
**
Copenhagen&

Wien&
Barcelona& Quality&of&Life&

Sustainability/&
Resource&eciency/&
Energy&eciency&
&

Amsterdam&
Bo4om0up&

6 The Kalasatama Vision building process

The Vision-building process consisted of a series of workshops and benchmarking
activities facilitated by a team of five from Forum Virium. All in all almost 200
people participated in the vision forming process presenting all stakeholders of the
Quadruple Helix Model. They participated in seven separate workshops during
winter-spring 2014. The final decision was made in August 2014 in a steering group
meeting chaired by the deputy Mayor of the City of Helsinki.The seven workshops
organized had different participant profiles, including civil servants and
government representatives, residents, academia, designers, large industry and
SME managers, start-ups, ICT & building technology industries, local
kindergarden kids, local senior home residents and city leaders.

162

December'2013,March2014:'Material'collec6on'and'crea6on.'
Civil&servant&workshop&12.12.&
Smart&city&benchmark&reports&&
Stakeholder&interviews&

Material&collecFon&

March'2014':'Workshop'4.3.''
Mixed&audience&(PPPP)&
April:'Workshop'11.4.''
ICTC&and&building&industries&workshop&

Analysis&

CoCthinking&

Benchmark&to&Masdar&City&
May'2014:''
Two&residenFal&workshops&

CoCthinking&

Benchmark&trip&to&Copenhagen&(Nordhavn)'9.5.'

TesFng&and&reframing&
June'2014:'Summing'up,'dierent'proposals'
Mixed&(PPPP)&workshop&and&City&directors&workshop&

Final&decision&August&
2014&
Picture 1: Process model

In total, the vision process lasted nine months. So between the workshops the
people involved interacted with each other, discussed the process in different
formal and informal settings and learnt considerable amounts about smart city
development on the go.
All the workshops were tightly structured and lasted 2-3 hours each using a variety
of co-working methods. In each workshop a common target was presented: To be
able to tell in one sentence what Smart Kalasatama is about. One method used to
issue this question was to ask participants to fill in a sentence what Kalasatama will
be known in 20 years of time.
7 The Final Vision Statement
Forming a compelling vision statement proved to be a challenging puzzle. On the
one side there were the different viewpoints of the PPP partners, on the other side
the all-embracing general Smart City targets. The differentiation requirement of
the vision statement both nationally and internationally had to be considered. The
learning from the benchmarks suggested us to seek a deliberate middle point in
both of the axis taken as a framework to analyze the visions.
Finally, the vision-building process ended up to a statement, that each citizen living
in Kalasatama should gain one more hour of own time a day. In a smart district the
time savings may result from good urban planning, where live, work and play take
place in proximity. Smart mobility, logistics and services coming to the citizen may

163

contribute to the time savings. One more hour of own time a day vision shifts the
focus into smart lifestyle supported by services.
One more hour statement has taken a human-centric approach: all the smartness of
a Smart City should eventually turn to give more capabilities or choice to the
citizens by saving his/her time. One more hour tries to incorporate the
sustainability aspect in the personal lifestyle. Thus it is deliberatively in the
balanced sweet spot of Smart City visions, taking the citizen to be central part of
the system of systems aiming simultaneously at resource efficiency and quality of
life. Or one could argue, that it may integrate the approaches: you may create timesaving solutions from top-down (e.g. traffic infra including subway and EV infra)
and pair them with bottom-up solutions (e.g. sharable EV services). One more hour
incorporates the citizen as a key actor to make a Smart City successful.
8 Learnings of the workshop process
The Kalasatama vision building process reinforced the presumptionthat the
viewpoints of the different quadruple helix participants differed. As expected, the
City emphasized more systemic goals such as reduction of CO2 emissions, other
sustainability measures and better overall public management. The City also
pushed systemic solutions to the district: smart grid, waste collection tubes and
open public governance led by the City hall. The City wishes to have control over
the quality of construction. Citizen viewpoints were emphasizing fluent everyday
life, services, social aspects and surprises. The developers (representing large and
small companies developing smart city solutions) had quite different standpoint.
They would like to have Kalasatama as a test bed with minimal regulation, where it
is easy to test new solutions. Developers were excited of the possibility of
innovation and open horizon of future technologies and business. The developers
paid more attention to means than the ends.
Still, despite their different strategies, the different quadruple helix stakeholders
seemed to share a lot of common values like sustainability, resource efficiency and
urban livability. The workshops with mixed PPPP participants worked best. They
resulted the richest in detail and more through-out outcomes, where more
processing was done at the workshop to synthetize different view points. So, it
seems that heterogenous participant groups foster learning and may accelerate
innovation as argued in Living Lab literature (Leminen;DeFillippi;& Westerlund,
2015). The diverse interests might also yield conflicts or tensions. In this case, the
tensions did not explode. Still, a key role of a Living Lab as a process facilitator
could be seen to negotiate the different views and possible conflicts when they
emerge.
Also the participants seemed to value more the workshops where they met
different kind of actors. They learned more. Especially, the residents seem to rather
participate in events and workshops where they can meet public or private
developers. They feel that their input is taken into a process and they also learn
more of the expert processes.
9 Conclusion

164

The Smart City Living Lab vision-building process described in this paper, yielded
clearly differentiating smart district vision: Kalasatama will be so smart and
resource efficient that it will promise one more hour for a resident. Most other
Smart City vision and objective statements analyzed were rather system-centric
(top-down), emphasizing either sustainability goals or aiming at livability and
economic growth from the top-down perspective. Based on our learning, living lab
PPPP co-creation methods are powerful in mixing different viewpoints and finding
balanced outcomes. Based on this process, we could suggest two hypotheses, which
could form a basis for a further research:
Hypothesis1: Living Lab methods and PPPP-process yield more balanced yet humancentric Smart City visions (and strategies).
Hypothesis2: Living Lab methods and PPPP-process yield more engaging Smart City
(or other domain) visions, than visions created e.g. by City (or another process/
domain owner) alone.
Based on only this particular case study, we can not generalize the Hypothesis1 to be
true in other cases. More studies and examples are needed to validate it.
The Hypothesis2 was supported by our experience: an unexpected Smart City vision
raises discussion, comments and activates brainstorming and curiosity to
participate among all quadruple helix stakeholders. However, as our Smart City
Living Lab is just less than two years old is too early to evaluate weather a
compelling vision helps to achieve longer term commitment and targets. Obviously
more cases and research are needed to validate this kind of hypothesis.
Kalasatama Smart City Living Lab managed to take the role of a participatory
platform in the vision-building process. The outcome of the vision process merged
top-down and bottom-up approaches as well as integrated seemingly contrasting
ends of sustainability and quality of life. One could conclude, that Living Labs as
truly open innovation platforms, may take a powerful role in balancing visions of
smart cities to engage all different stakeholders.
10 Bibliography
Andrea, C., & Peter, D. B. (2011). Smart Cities in Europe. Journal of Urban Technology
, 18 (2), 65-82.
ARUP. (2013). International Case Studies on Smart Cities. The Department for
Business Innovation and Skills. London: Arup.
Baccarne, Schurman, Mechant, & Marez, D. (2014). The role of Urban Living Labs
in a Smart City. The Proceedings of ISPIM Conference for Sustainable Economy and
Society. XXV, pp. 1-14. ISPIM.
Bakiti, Almirall, & Wareham. (2013). A Smart City Initiative: the Case of Barcelona.
Journal of Knowledge Economy , 4 (2), 135-148.
Breuer, J., Walravens, N., & Ballon, P. (2014, June). Beyond Defining the Smart City.
Meeting Top-Down and Bottom-Up Approaches in the Middle. TeMa. Journal of
Land Use, Mobility and Environment , 153-164.

165

Chourabi, & al. (2012). Understanding Smart Cities: An Integrative Framework.

Hawaii International Conference on System Sciences proceedings. 45th, pp. 2289-2297.
IEEE computer society.
City of Amsterdam. (2014, 4 15). Amsterdam Smart City. Retrieved 4 26, 2015, from
Amsterdam Smart City: http://amsterdamsmartcity.com/
City of Barcelona. (2015, 4 26). Retrieved 4 26, 2015, from iBarcelona:
http://ibarcelona.bcn.cat/en
City of Copenhagen. (2015, 4 26). https://stateofgreen.com/. Retrieved 4 26, 2015, from
StateofGreen: https://stateofgreen.com/en/profiles/city-of-copenhagen
City of Stockholm. (2015, 4 26). Retrieved 4 26, 2015, from Stockholmroyalseaport:
http://www.stockholmroyalseaport.com/en/
City of Wien. (2015, 4 26). Smart City Wien Framework Strategy_o
ne
page.
Retrieved
4
26,
2015,
from
Smart
City
Wien:
https://smartcity.wien.gv.at/site/files/2014/09/SmartCityWien_FrameworkStrategy_
english_onepage.pdf
Cosgravea, E., Arbuthnot, K., & Tryfonas, T. (2013). Living Labs, Innovation Districts
and Information Marketplaces: A Systems Approach for Smart Cities. (2. P.6.
Volume 16, Ed.) Procedia Computer Science , 16, 668-677.
Curley, M., & Salmelin, B. (2013). Open Innovation 2.0: A New Paradigm.
OISPGOpenInnovation20ANewParadirm-WhitePaper, (pp. 1-12).
Dubai Smart Gov. (2015, 4 26). Dubai Smart Government has officially activated
Happiness Meter in 14 government entities so far. Retrieved 4 26, 2015, from
Government
of
Dubai:
http://dsg.gov.ae/en/mediacentre/DegNews/Lists/NewsCentre/DispForm.aspx?ID=1
46
Emirates 24/7. (2014, 3 5). Mohammed launches Dubai Smart City. Retrieved 4 26, 2015,
from Emirates 24/7: http://www.emirates247.com/news/government/mohammedlaunches-dubai-smart-city-smartphone-to-be-key-pivot-2014-03-05-1.540576
European Commission. (2014). Open Innovation 2.0 -Yearbook 2014. Luxembourg:
European Commisssion.
European Parliament. (2014). Mapping Smart Cities in the EU. European Union.
Hill, D. (2013, 2 1). On the smart city: Or, a 'manifesto' for smart citizen instead. Retrieved
6 15, 2015, from City of Sound: http://www.cityofsound.com/blog/2013/02/on-thesmart-city-a-call-for-smart-citizens-instead.html#more
Hollands, R. (2008). Will the real smart city please stand up? City , 12 (3), 303-320.
Kitchin, R. (2014). The real-time city? Big data and smart urbanism. (SpringerLink,
Ed.) geoJournal , 79 (1), 1-14.
Leminen, DeFillippi, & Westerlund. (2015). Paradoxical tensions in living labs. The
Proceedings of The XXVI ISPIM Conference. Budapest: ISPIM.
Mustonen, V., Koponen, J., & Spilling, K. (2014). lyks kaupunki -Smart City. Katsaus
fiksuihin palveluihin ja mahdollisuuksiin. Helsinki: Ministry of Transport and
Communications.
Neirotti, Marco, D., & Cagliano&al. (2014). Current trends in Smart City Initiatives:
Some stylised facts. Cities , 38, 25-36.
Paskaleva, K. (2011). The smart city: A nexus for open innovation? Intelligent
Buildings International , 3 (3), 153-171.
Ramaswamy, V., & Gouillart, F. (2010). The Power of Co-creation. New York: Free
Press.

166

Sderstrm, O., Paasche, T., & Klauser, F. (2014). Smart cities as corporate
storytelling. City , 18 (3), 307-320.
Shephard, M., & Simeti, A. (2013). Whats so smart about the Smart Citizen? In H. D.
A., Smart Citizens (pp. 13-18). Manchester, UK: FutureEverything Publications.
Townsend, A. (2013). Smart Cities: Big Data, Civic Hackers, and the Quest for a New
Utopia. New York: W.W. Norton& Company.
Wikipedia. (2015, 4 26). Masdar City. Retrieved 4 26, 2015, from Wikipedia:
http://en.wikipedia.org/wiki/Masdar_City
Wikipedia. (2015, 4 26). World's most liveable cities. Retrieved 4 26, 2015, from
Wikipedia.org: http://en.wikipedia.org/wiki/World%27s_most_liveable_cities

167

Use of Living Lab

in Innovative Public Procurement:
Case Keyless Home Care
Haukipuro Lotta a, Torvinen Hannu a, Vinm Satu a, Mattila Pasi a
a

University of Oulu, Finland

Abstract
Recent shifts in public procurement from traditional towards innovative has
opened up new opportunities for operators providing tools and services for userdriven development. Living Labs have strong expertise in the field thus being
appropriate actors in innovative public procurement activities. This paper presents
empirical findings of an innovative public procurement case in health care field
where the service provided by a Living Lab was in significant role. Living Lab was
involved in the process from the beginning until the end of the procurement. The
results indicate that use of Living Lab tools and services may foster innovative
public procurement in the field of health care.
Keywords
Innovative Public Procurement, Living Lab, User Engagement, Home Care
1 Introduction
In present research, innovation is defined as novelty or reform with significant
productivity-, economic efficiency- or other value-adding effects on the
organizations performance (Yliherva 2006). Lee, Olson and Trimi (2012) see
innovations necessary for public sectors better productivity and new more costeffective operations. The desire to create new innovations is one of the key drivers
behind the utilization of new differentiated public procurement approaches (Edler
& Georghiou 2007). An extremely prominent source for innovations are customer
interfaces favourable to the exchange of know-how, information, viewpoints,
experiences, cultures and resources (Yliherva 2006). The possibility to enhance the
involvement of the end-user in the procurement process is partly the result of
advanced technology, and partly citizens increased enthusiasm to participate in
the co-production of the services (Bovaird & Loeffler, 2012).
According to Uyarra & Flanagan (2009), public procurement is seen as having a
potentially crucial role in enhancing the innovations in Europe, thus creating
wellbeing. As current trend in public procurement is to add innovativeness through

168

end user involvement, a new opportunity has opened for Living Labs to provide
their expertise of user-driven methods and tools to be utilized in public
procurement field. So far, the involvement of end users into procurement process
has raised some interest and attempts in practice as well (Ng et al., 2013).
A long-standing and several successful co-development projects with public sector
conducted Living Lab OULLabs,18 located at the University of Oulu, Finland, was
providing Living Lab services in an innovative public procurement of the City of
Oulu. The aim of this Keyless home care process was to implement public
procurement of a keyless mobile door opening service for home care in a new,
innovative way through product testing and user involvement. The aim of the city
was to purchase service that allows home care personnel to open customers door
with a mobile device. The Keyless home care projects product testing phase was
implemented within an EU funded, Living Lab environment developing project19.
The project team implemented Keyless home care purchase product testing, led by
a usability specialist. PATIO forum (www.patiolla.fi) was used for involving home
care workers who were testing the service in a real environment. As a result,
innovative public procurement of a mobile door opening service where product
testing was for first time in significant role, was successfully implemented.
Furthermore, as a result of the process, the winning mobile door opening service
was put in use in certain districts of the City of Oulu home care, and the aim is to
further spread it to the whole city.
2 Innovative public procurement
The public sector acquired infrastructure and services can be regarded as essential
for maintaining society's economic and social structures (Lhdesmki & Kilkki,
2008). Traditional competitive bidding procedures carried out in public
procurement have experienced in recent years an increasing pressure for change,
including increased demand for services due to aging population, challenging
economic situation and as a result of new technology solutions (Pekkarinen et al.,
2011; Jamali, 2007). According to Aho et al. (2006), along changing procurement
environment, innovative public procurement has emerged as a central theme in the
2000s demand-driven innovation policy at the national level as well as in the
European Union. Specifically, new market-oriented public procurement models
aimed at generating innovations but also accelerating the spread of innovations by
strengthening the demand for new solutions, have attracted attention. Increased
demand-orientation may have blurred the line drawn between the public and the
private sector as in the private service provider carry increasing responsibility for
public procurement holistic implementation. Furthermore, public procurement
has multitude of social goals to serve, and its use as innovation tool entails various
challenges (Uyarra & Flanagan, 2009).
General benefits of innovative procurement in addition to increased innovation
have been considered to consist of increased higher overall efficiency of
procurement, quality, efficiency, risk management and transparency (Majamaa,
2008a; Yescombe, 2007). Innovative public procurement, in a wider perspective,

18

19

Oulu Urban Living Labs www.oullabs.fi/en

MAINIO project, www.mainio.eu

169

also improves the dynamics of innovation (Edler & Georghiou, 2007). At best,
public procurement can have greater incentive effect on firms' innovation activities
than traditional public sector funded research and development activities. The
greatest obstacles for the implementation of innovative procurement do not lie so
much in the legal instruments guiding procurement, but in the procurement
departments ability to study and apply procedures that allow the development of
providing innovative solutions. In addition, innovative procurement often contain
higher risks, the result of which division of responsibilities of the risks caused by
procurement should be clearly specified in advance.
Public procurement potential as source of innovation is based, above all, on the
public sectors large but still widely unutilized economic volume. Although the
public procurement has already been used as a tool to promote innovativeness in
many countries, innovative public procurement can be viewed as a rather new
phenomenon. The most common cases for innovative public procurement have so
far been the major infrastructure projects and defence industry procurement. The
most advanced in the development of innovative procurement are generally seen
the United Kingdom and the United States. In Finland, the promotion of
innovation has not usually been the main goal of public procurement even in new
procurement, but rather as a minor aim of the projects. The choice of innovative
procurement method in Finnish public procurement projects has been justified
mostly with achievable savings instead of innovativeness.
Not only exclusive existing demand gives rise to innovation, thus essential for
innovative purchase models is a dialogue between end users and other key actors
considering the functionality of a procurement. Demand-driven public
procurement can say to reach for higher customer orientation in procurement.
Publicly acquired infrastructure and services should not be judged not so much
according to decision-makers interest, but based on what the end-user sees as
valuable. In practice, the customer and the service provider should pay attention to,
as well as the procurement of current and future functional requirements, and be
able to define the demand in more broader and clearer scale. The interaction
between the actors in innovative procurement is seen in present research, as a triad
of the customer, the supplier and the value co-producer (Havila et al., 2004;
Majamaa, 2008). The triadic nature of the focal procurement relationships means
the existence of an intermediary that has constant contact with both the selling
party (supplier) and the buying party (procurer), while the selling party and the
buying party have also their own simultaneous contact with each other (Havila et
al. 2004). Important for the triadic relationships are social bonds and trust
especially at the operational level (Holma 2012).
A starting point for many of the new procurement practices is considered to be the
rise of New Public Management movement gaining its popularity since early 1990s.
The main aim of NPM trend has been to modernize and enhance public sector
operations with a more market-based approach on organizing public services
(Hood, 1991). Market-based approach on public procurement opens opportunities
both for mobilizing innovation and at the same time better achieving public policy
goals and delivering better service to the citizens. Public procurement practitioners
tend to often lack a clear understanding of who the client of the public service is
and, therefore, do not know whose needs they are supposed to satisfy (Bovaird
2007). Even though no generally agreed upon definition of public procurement

170

partnerships is known to exist (e.g. Lawther & Martin, 2005; Yescombe, 2007), some
market-based models like public-private partnership and pre-commercial
procurement (PCP) have gained vast interest from both the researchers and
practitioners in the public procurement field.
In Public-Private Partnership (PPP) model aka life cycle model the supplier carries
a larger liability of the procured object or service for a longer period of time also
called the life cycle of the procurement. In other words, public sector sets end
targets for the outcomes of the procurement, but doesnt define in advance how to
reach these goals (Yescombe, 2007). PPP model has emerged as a worldwide trend
by public procurement practitioners especially within infrastructure acquisitions
(Ng et al., 2013). The most vital aspect on describing innovations within PPP model
is the added value the innovation creates for the end-users (Yliherva 2006). If PPP
method was chosen more regularly on the basis of value gained by the end-users,
then cooperation, commitment and networks would more often be considered as
the prime benefits of the model instead of the financial arguments (Lhdesmki &
Kilkki 2008).
PCP model can be applied to innovative procurement in which more research and
development is needed. The PCP model aims to achieve user-friendly solutions
developed together with businesses, avoiding monopolistic structures by involving
at least potential suppliers in field test phase. The benefit of the model for the
customer in procurement is sharing the risks related to product development
between public customers and potential suppliers (Mattila & Silander, 2015).
3 Living Labs role in innovative public procurement
As open innovation emerges in new service development, elaborate networks
where organizations co-create to generate new products and services have been
increasingly researched and established (Chesbrough & Appleyard, 2007). The
open innovation model (Chesbrough, 2006) shows that ideas are generated both
inside and outside organizations. External knowledge is seen to play an equal role
as that afforded to internal knowledge in the earlier conception (Chesbrough,
2006). Open innovation builds strongly on voluntary collaboration, which makes
Living Labs a fruitful environment for deploying open innovation (Chesbrough &
Crowther, 2006; West et al.2006). Though there are several definitions of Living
Lab, no coherent definition yet exists. According to most recent research Living Lab
can be defined an embodiment of both the open and user innovation paradigms
(Schuurman, 2015). According to Leminen et al. (2012) a Living Lab is a network that
integrates both user-centred research and open innovation. Living Labs are driven
by two ideas: involving users as co-creators on equal footing with the other
participants and experiments in real-world settings (Almirall, Lee & Wareham,
2012). Living Labs are thus seen as separate from other innovation approaches due
to two dimensions; a high degree of realism and a high degree of user involvement
(Schuurman & De Marez, 2012). Compared to, for instance, field trials or user
testing, a Living Lab involves users in all stages of R&D and all stages of the product
development lifecycle (Ballon, Pierson & Delaere, 2005). Following Almirall &
Wareham (2012) and Leminen et al. (2014), Living Labs can also be defined as an
organized approach (as opposed to an ad hoc approach) to innovation consisting of
real-life experimentation and active user involvement by means of different

171

methods involving multiple stakeholders, as is implied in the Public-Private-People

character of Living Labs. (Schuurman, 2015).
Living Labs role in innovative public procurement can be seen significant as
implementing innovative procurement requires cooperation among all actors;
customer, supplier and end users (Mattila & Silander, 2015). Even though the
cooperation with users and the surrounding community has been recognized
essential for public procurements success, the resources given to end-users
engagement to the public procurement processes are often slim (Bovaird &
Loeffler, 2012). The desires of end-users get neglected too often which leads to
solutions unsuitable for the actual user of the public service creating e.g. financial
losses caused by the additional fixing costs and dissatisfactions effecting on the
suppliers life cycle payments (Ng et al. 2013; Satish & Shah 2009). The early
detection of user requirements and needs guides the procurement project towards
better end results, efficiency and innovative solutions straight from the beginning
(Laine & Junnonen 2006; Majamaa et al. 2008; Satish & Shah 2009). In addition to
creative mind, users can also bring other resources to the process for instance by
positively influencing other users and lowering the public opposition (Bovaird &
Loeffler, 2012). Thus, Living Lab user communities can be useful when involving
end users in innovative public procurement cases. The supplier and end user share
a common need to develop a new product or service for the market, aiming to
create added value e.g. through better quality, more efficient production processes,
lower life cycle costs, environment friendliness or usability (Mattila & Silander,
2015). By developing innovative procurement, end users are able to participate in
the process from the early planning to implementation. Living Labs basic idea,
early involvement of end users makes possible corrections or changes cost-effective
also in procurement cases.
4 Methodology
The study applies case study as the primary research approach. According to Yin
(2003) a case study design should be applied when: the focus of the study is to
answer how and why questions, or behaviour of those involved in the study
cannot be manipulated, or contextual conditions are thought to be relevant to the
phenomenon under study, or the boundaries are not clear between the
phenomenon and context.
In this study, Living Lab approach and multiple Living Lab methods were used in
order to collect diverse data and form a comprehensive view of the Keyless home
care case. Interviews, online end user involvement tool PATIO (www.patiolla.fi),
quantitative measurement, and surveys were used in this study. Primary data of the
study is qualitative: project team was informally interviewed and identified key
persons in-depth interviews were conducted. All interviews were recorded and
transcribed into text documents. Approximate duration of an interview was 1 hour
each. The interviewees were:
Technology Specialist from the City of Oulu who was at the time of
implementation of the Keyless home care case in a role of Purchase Planner
in the City of Oulus strategic procurement department.
Project Manager of MAINIO project who was actively participating in
planning and in charge of product testing as a part of purchase, and

172

Usability Specialist in main role of product testing

implementation and reporting in Keyless home care case.

planning,

Furthermore, qualitative data was collected through online discussion where test
users were able to provide feedback of product testing they were involved in.
Quantitative data was collected through product testing which included objective
technical measurement such as time (duration of opening a door), and several
questionnaires for test users. Project related internal reports and documents as well
as public information available regarding the Keyless home care case were used as
secondary data.
5 Case Keyless home care
Involving users may make a traditional purchase innovative, which is growing
trend in public procurement, and also included in the strategy of City of Oulu: the
aim for 2016 is that 20% of the purchases have been made using innovative
procurement procedures. The Keyless home care project was implemented during
approximately eight months in 2013-2014 as a pilot project of an EU funded project
in cooperation with the University of Oulu and the City of Oulu.
The need for this kind of pilot project arose from initiation of the Social Welfare
and Health Services and specifically home care services of the City of Oulu, which
aims to search for cost-efficient services and cost-efficient tools for service
production. Home care workers visit over ten patients in their homes during a shift,
thus using ten different keys for opening patients doors. The keys are stored in the
office of home care workers from where they have to pick them up individually
between patient visits. The need for Keyless door opening system arose thus from
practical reasons need to save home care workers time, ease their work and
improve safety through minimizing the risk of losing keys. In few cities in Finland
the home care has been already made 100% keyless. Good general experiences of
Keyless home care based on pre-study and practical needs of home care in the City
of Oulu led to a project where product testing was for first time in significant role as
a part of public procurement in the City of Oulu. Living Lab resources and user
involvement as well as usability knowhow that were necessary for implementing
the Keyless home care pilot project, were provided by OULLabs. In addition,
expertise of the field (home care) and project management were needed in the
project.
The Keyless home care mobile door opening service was purchased partly (40%) on
the basis of a product testing of four door opening system (lock module, application
and software for administration of the door opening). Product testing was planned
and implemented by OULLabs specialists. Planning took two months and
implementation including official decision making process six months, overall
duration being thus eight months. Relatively long time was spent in planning as
product testing within public procurement of the City of Oulu was novel:
preliminary work including search for references, minimization of risks and
consideration of legacy aspects was time-consuming.
Selected group of users, seven home care workers and their four supervisors (n=11)
performed product testing of the four mobile door opening systems delivered from
different companies during their two week shifts in a real home care environment,

173

sheltered housing. The more precise task of the home care workers was to test the
opening of lock modules and use of a key: a mobile application installed in their
work mobile phones through which the lock modules can be opened. The
products were anonymized and coded with colors: red, green, yellow, and blue. The
supervisors tested PC software through which the system is administered, e.g.
access rights given to users. In addition, a usability specialist tested both the lock
modules and software of all companies. The test was based on usability standard
ISO9241-11 (ISO, 1998) and they were implemented in such way that all test cases
were measurable. Evaluation feedback and scores were collected from users
through a questionnaire. Additionally, a Living Lab user involvement online
platform PATIO (www.patiolla.fi) was used for collecting feedback in a private
online discussion area from home care workers about product testing process in
order to develop the procedure of innovative procurement within the City of Oulu.
Criteria
Price
Quality
Questionnaire, users
Questionnaire, administrator users
Timing/lock
Timing/software

Score %
60
40
12,5
12,5
7,5
7,5

Table 1. Evaluation Criteria

The evaluation criteria for purchase were price (with a 60% weighting) and quality
(with a 40% weighting). Scores given by users with scores given by the usability
specialist together formed quality: user feedback (questionnaire) and supervisors
(administrator user) feedback both with a 12,5 % weighting, product efficiency 7,5%
and time spent for software usage 7,5%, altogether with a 40% weighting (Table 1).
Figure 1 illustrates the overall process of the Keyless home care public procurement
process. The process consisted of several elements such as planning, bidding,
installation, product testing, analysis and decision. Duration of planning phase was
two months as the project was unique without existing references to be exploited in
planning. Living Lab was involved in the planning phase as expertise of conducting
product testing and user involvement was needed. In the bidding phase, official
bidding announcement in co-operation with project team was made. On the basis
of announcement, four offers from suppliers were received and four products (lock
module and software) were installed in a testing environment. Suppliers
introduced their products for usability specialists to enable planning and
implementing and to ensure efficient testing. Product testing was conducted in a
real use environment during two weeks. At the same period, usability specialists
conducted usability testing for products and software. Analysis of product testing
results was made in cooperation with public procurer after which official decision
making process was started.

174

Figure 1. Process of Keyless home care innovative public procurement

6 Results
On the basis of scores formed by quality features and price, mobile door opening
system was procured. The overall size of the purchase was 500 lock modules and
software, the total value of the purchase being 250 000 EUR. Compared to
traditional public procurement where product with lowest price usually becomes
selected, the result of the Keyless homecare procurement differ considerably: the
selected mobile door opening system was not the cheapest one but the one with
highest scores obtained from overall price and quality together. This means that
quality and user assessment were in significant role in the procurement. What was
also interesting result was the fact that the product with the lowest price did not
receive good scores on quality.
As there were several stakeholders involved, the results of the Keyless home care
project can be viewed from different angles: from the City point of view, first-hand
experience on conducting a successful procurement pilot was valuable considering
the further development of procurement. Costs were saved, effectiveness of home
care working hours was improved and home care workers satisfaction was
increased as they themselves were able to choose their tools. The effect of
increased work satisfaction was not studied here but there might be far reaching
positive impact. From users, home care workers point of view, usability of a daily
used product was ensured through comprehensive product testing: in addition to
basic testing, the users were able to express their opinions and feel obtaining a
significant role in the project. From home care patients point of view, service
provided for them was also improved as home care workers are able to enter
patients homes more quickly without additional office visits between patient visits.
To set up the service, patients lock modules had to be changed and a small service
fee was set by the City of Oulu but on the other hand, other costs such as ordering
new keys were saved by patients at the same time. From the Living Lab point of
view, valuable experience of successful use of a Living Lab in innovative public
procurement was obtained, and conditions for using Living Lab in future public
procurement cases were created. Furthermore, experience of the suitability of

175

Living Lab methodology for innovative public procurement where i.a. legal aspects
must be taken in account, was gathered.
Figure 2 illustrates the setting of the Keyless home care public procurement,
especially the role of Living Lab in the case. The importance of Living Lab services
and expertise was crucial in the planning phase where a product testing plan,
which was an important part of bidding announcement, was planned and
implemented by Living Lab usability specialist.

Figure 2. Living Lab as a part of Keyless home care innovative public procurement

7 Conclusion
The study presented a public procurement case where product testing service
provided by a Living Lab was for the first time included in a public procurement in
the City of Oulu. The case of Keyless home care was unique and raised a lot of
interest among stakeholders. Successful implementation of the procurement
process as well as exceptional results are extremely important for all stakeholders,
but especially the City of Oulu and Living Labs. New means of shifting from
traditional procurement towards innovative public procurement were identified
and tested in the process. Product testing within public procurement was carefully
planned and documented and thus serve as a reference and easily exploited in the
coming procurement cases where product testing is included.
In general, the promising results of the case Keyless home care indicate that Living
Labs expertise is useful in implementing innovative public procurement. For
instance, in a case where citizens would be involved in a public procurement, the
role of Living Lab could be even more significant as user engagement tools and
methods for facilitating users are among the strengths of Living Labs. First
experiences such as Keyless home care provide opportunity to practice cooperation and build trust between operators in multi-stakeholder projects. For
instance, experience of close co-operation between local Living Lab and the City of
Oulu in several different development activities has built strong confidence and
knowledge of each others expertise between the organizations, which cannot be
underestimated when operating in such strictly regulated field as public
procurement. However, there is need for further research in order to efficiently

176

benefit from user engagement and Living Lab approach in forming a wellestablished innovative public procurement process.
8 References
Aho, E., Cornu, J., Georghiou, L. & Subira, A. (2006). Creating an innovative
Europe. Report of the independent expert group on R&D and innovation
appointed following the Hampton Court Summit.
Almirall, E., Lee, M. & Wareham, J. (2012). Mapping Living Labs in the landscape of
innovation methodologies. Technology Innovation Management Review,
September, 1219.
Almirall, E. & Wareham, J. (2011). Living Labs: arbiters of mid-and ground-level
innovation. Technology Analysis & Strategic Management, 23(1), 87-102.
Ballon, P., Pierson, J. & Delaere, S. (2005, Month). Open innovation platforms for
broadband innovation: Examining European practice. Paper presented at 16th
European Regional Conference by the International Telecommunications Society
(ITS), Porto, Portugal.
Bovaird, T. & Loeffler, E. (2012). From engagement to co-production: The
contribution of users and communities to outcomes and public value. Voluntas
23(4): 1119-1138.
Chesbrough, H. & Appleyard, M. (2007). Open innovation and strategy. California
Management Review, 50(1), 5777.
Chesbrough, H. (2006). In: Chesbrough, Vanhaverbeke & West: Open Innovation:
Researching a new paradigm. Oxford University Press.
Chesbrough, H. & Crowther, A.K. (2006). Beyond high tech: early adopters of open
innovation in other industries. R&D Management, 36 (3), 229236.
Edler, J. & Georghiou, L. (2007). Public procurement and innovation-Resurrecting
the demand side. Research Policy 36(7): 949-963.
Hood, C. (1991). A public management for all season. Public Administration 69(1), 319.
Holma, A-M. (2012). Interpersonal interaction in business triad - Case studies in
corporate travel purchasing. Journal of Purchasing and Supply Management,
18(2):101-112.
ISO 9241-11 (1998). Ergonomic requirements for office work with visual display
terminals (VDTs) -- Part 11: Guidance on usability.
Jamali, D. (2007). A study of customer satisfaction in the context of a public private
partnership. International Journal Quality & Reliability Management 24(4): 370-385.
Laine, V. & Junnonen, J. (2006). Julkisen elinkaarihankkeen hankintaprosessi.
Vammala: Rakennusteollisuus RT Oy.
Lawther, W.C. & Martin, L.L. (2005). Innovative practices in public procurement
partnerships: the case of the United States. Journal of Purchasing and Supply
Management, 11(5), 212-220.
Lee, S.M., Olson, D.L. & Trimi, S. (2012). Co-innovation: convergenomics,
collaboration, and co-creation for organizational values. Management Decision, 50
(5), 817-831.
Leminen, S, Westerlund, M. & Nystrm, A-G. (2012). Living labs as openinnovation networks. Technology Innovation Management Review, 612.

177

Leminen, S., Westerlund, M. & Nystrm, A-G. (2014). On becoming creative

consumersuser roles in living labs networks. International Journal of Technology
Marketing, 9(1), 33-52.
Lhdesmki, K. & Kilkki, S. (2008). New public management principles and
practices in producing public utilities and services. Administratie si Management
Public (10): 114.
MAINIO project, Pilot Project Report, D15: Report pf Pilot project 2, Keyless home
care product testing. 16.06.2014.
Majamaa, W., Junnila, S., Doloid, H. & Niemist, E. (2008). End-user oriented
public-private partnerships in real estate industry. International Journal of
Strategic Property Management 12(1): 1-17.
Mattila, P. & Silander, P. (Eds.) (2015), How to create the School of the Future
Revolutionary thinking and design from Finland, Multprint, Oulu.
News on OULLabs web page. Retrieved from http://www.oullabs.fi/oullabsinkuulumisia/, 1.2.2015
News
on
OULLabs
web
page.
Retrieved
from
http://www.oullabs.fi/en/referenssit/oullabs-specialist-team-carried-out-producttesting-as-a-part-of-an-innovative-public-procurement/ , 1.4.2015
News on Oulu-lehti: Avaimeton kotihoito alkanut Oulussa. Retrieved from
http://www.oululehti.fi/etusivu/avaimeton_kotihoito_alkanut_oulussa_6972213.html
19.6.2014
Ng, S.T., Wong, J.M.W. & Wong, K.K.W. (2013). A public private people
partnerships (P4) process framework for infrastructure development in Hong Kong.
Cities, Vol. 31, pp. 370-381.
Pekkarinen, S., Hennala, L., Harmaakorpi, V. & Tura, T. (2011) Clashes as potential
for innovation in public service sector reform. International Journal of Public
Sector Management 24(6): 507-532.
Procurement Announcement. Keyless home care Mobile door opening service.
Retrieved from HILMA http://www.hankintailmoitukset.fi/ 1.3.2014
Satish, D. & Shah, P. (2009). A study of Public Private Partnership models. IUP
Journal of Infrastructure 7(1): 22-38.
Schuurman, D. & De Marez, L. (2012). Structuring user involvement in panel-based
Living Labs. Technology Innovation Management Review,2(9), 3139.
Schuurman, D. (2015). Bridging the gap between Open and User Innovation?:
exploring the value of Living Labs as a means to structure user contribution and
manage distributed innovation. Ghent University. Faculty of Political and Social
Sciences; Vrije Universiteit Brussel. Faculty of Economic and Social Sciences,
Ghent; Brussels, Belgium.
TEKES, (2008). Innovatiiviset julkiset hankinnat. Tekes reports 225/2008.
Uyarra, E. & Flanagan, K. (2009). Understanding the innovation impacts of public
procurement. Manchester Business School Working Paper, Number 574.
West, J., Vanhaverbeke, W. & Chesbrough, H. (2006). Open innovation: A research
agenda. H. Chesbrough, W. Vanhaverbeke and J. West (Eds.), Open innovation:
Researching a new paradigm, Oxford University Press, Oxford, 285307.
Yescombe, E.R. (2007). Public-private Partnerships: Principles of Policy and
Finance. Amsterdam: Elsevier.
Yin, R. K. (2003). Case study research: Design and methods (3rd ed.). Thousand
Oaks, CA: Sage.

178

Yliherva, J. (2006). Tuottavuus, innovaatiokyky ja innovatiiviset hankinnat. Sitras

reports 2006. Edita Prima, Helsinki.

179

Towards a sustainable panel-based

living lab approach
in older adult care innovation
Juul Lemey1, Charlotte Brys2, Koen Vervoort3, Patricia De Vriendt2,4 & An Jacobs3,5
on behalf of KIO6 and PO3
1

Department of Bachelor in Nursing, Artevelde University College Ghent, Ghent, Belgium

2
Frailty in Ageing (FRIA) Faculty of Medicine and Pharmacy, Vrije Universiteit Brussel,
Brussels, Belgium
3
iMinds, Flanders digital strategic research centre, Ghent, Belgium
4
Department of Bachelor in Occupational Therapy, Artevelde University College Ghent,
Ghent, Belgium
5
iMinds, SMIT, Vrije Universiteit Brussel, Brussels, Belgium
6
KIO, interuniversity consortium studying innovations in older adult care in Flanders.
Juul.Lemey@arteveldehs.be
This submission has been previously published elsewhere.

Abstract
To tackle the current challenges in the care for older adults innovative solutions
should be created. Herefore a panel-based living lab approach can be used. This
requires a sustainable involvement of older adults and their caregivers, which is
challenging. Based on the experiences in the Care Living Labs Flanders program
this paper will discuss how a panel based approach can be achieved. This case
study was a combination of a plan evaluation and action research. A comparison
between the initial plans and the steps towards a sustainable panel for the Care
Living labs is provided.
Keywords
Older adult care, user involvement, panel-based living lab
1 Introduction
Worldwide the care for older adults faces problems as budgetary restrictions and
staff shortages (Ellenbecker, 2010) while the demand for care is rising (Hussein &
Manthorpe, 2005). In addition most older persons indicate that they prefer to live at
home and close to their familiar social environment (Buffel et al., 2014) which
results in a growing pressure on informal caregivers. To tackle these challenges

180

policy makers funded several care living labs (CLL) to create social and technical
solutions that enable older adults to live longer and independently at their homes.
Essential in these CLL is the active participation of older adults, informal caregivers
and professional caregivers in the creation and testing processes. In a panel-based
living lab approach, they form the panel, which is the core of the lab. Nevertheless
the formation of a sustainable panel with these groups is associated with several
difficulties, such as getting access to the future users, striving for representativeness
and keeping participants attached to a panel. In this paper possible strategies to
tackle them will be discussed.
1.1 Care Living Labs Flanders
The case study used in this paper to reflect on the challenges for a panel-based
living lab approach is the program Care Living Labs Flanders. It is initiated by the
Flemish government to tackle the challenges in the care for older adults. Currently
six CLL (CareVille, InnovAGE, AIPA, AzoB, Online Buurten and LiCalab) and 23
projects are funded from 2013 to 2016. A project is a string of activities creating and
testing an innovation iteratively. In the first year (2013-2014) the CLLs started with
the creation of their panels. Between the CLL different strategies are observed to
recruit and commit their end-users, which will be discussed in this paper.
1.2 The panel-based living lab
Panel research is a term used in literature with many different connotations, most
often it refers to repeatedly asking questions or profiling people over time. In social
sciences and market research it is used for longitudinal survey studies on change in
behaviour and attitudes resulting in larger scale results with stronger validity (e.g.
Estrada, 2014). In medical practice one is experimenting with panel approaches to
peruse an enhanced profiling and thus better care (e.g. Neuwirth et al, 2007).
Schuurman & De Marez (2012) defined panel in panel-based living lab as a part of
the infrastructure, where users are recruited thematically and are profiled. They
also studied its added value versus the more traditional living lab approach by
analysing several Flemish cases. Ease of user recruitment based on specific
characteristics is key. It makes implementation of a test phase for example faster:
there is accurate and up to date data on potential participants to make the selection
more specific, they already opted in regarding privacy and other operational
aspects are also already covert, pre-activity information is available and it is easier
to collect ex-post measurements because of the sustained relationship with the
panel members over the different projects. But they also warn that to cherish these
advantages time and effort needs to go to the recruitment and management of your
panel.
Also in classic social sciences longitudinal survey studies retention of participants is
a challenge. Based on social influence and relationship research Estrada et al. (2014)
built a tailored management approach, which makes sure communal norms are
created that stimulate commitments to stay active as a participant during the
subsequent studies. Commitment is created by working on compensation (built
reciprocity by giving compensation before activities, focus on intrinsic motivation),
communication (bidirectional, choice to multiple channels, personalised

181

communication and tailored to the participants, creation of group identity),

consistency (messages predictable and recognisable over time, dont overburden,
be clear on the rules, roles and values of engagement) and credibility (the organiser
and the study are perceived as legitimate).
This advice is in line with the panel management experience within iMinds in
domains not related to older adult care. Vervoort (2012) presented seven steps to
create and maintain a panel based living lab on his teams experiences with panel
management in several Flemish living lab cases (table 1).
Steps
Step 1: Define the purpose of the panel

Step 2: recruit users

Step 3: support the panel

Step 4: live the lab

Step 5: handle privacy

Step 6: reward users

Step 7: maintain the eco-system

Explanation and instructions

What is the role of the panel? What are
the parameters of the panel? What are
possible
motivators
for
the
participants?
Define the profile of the panel
members, the recruitment channels,
the communication and the project
flow (who when why). Participating
should be fun.
Organise the helpdesk. Give the living
lab a face through a single point of
contact (SPOC). Training and inform
panel members. Set up a system to
share experiences. Feedback is a twoway street.
Capture central data. Measuring is
knowing. What is the mixture? Keep
track of linking devices and services to
members.
Protect members personal space.
Develop a procedure to handle
personal data towards stakeholders (on
need to know base).
Provide motivators to cooperate
(intrinsically, financially) (offer a
mixture).
Define an entry and exit strategy. Build
the community through member-getmember. Expectation management is
a continuous effort over the lifetime of
the panel.

Table 1 Steps to create and maintain a panel in living labs

This experience is used for the coaching of the build-up of the CLL in Flanders.
This experience was merged with specific knowledge on older adult care to
improve the coaching as it is challenging to involve the multiple actors in this
sector.

182

1.3 The challenge of multi actors and heterogeneity

The Flemish government required to include older adults as the central target
group. Today, 20% of the Flemish people are 65 years and over. This ratio will
increase to 25% in 2030 and 33% in 2040 (Federal Public Service Economy, 2015). A
similar demographic change occurs in Europe (European Commission, 2015).
However, 'the' older person does not exist. In fact they are a heterogeneous group
with diverse needs and capacities. Using solely a certain age range is misleading
(European Commission and Spanish Ministry of Health and Social Affairs, 2010).
Differences between older adults are characterized by their health status, degree of
dependency, social network, cultural background, socio-economic status, level of
citizen participation, ... These aspects change during the life course.
Including this heterogeneous group of older adults to create and test innovations
for elder care is necessary. Nevertheless there are other involved actors, like
informal and professional caregivers. They may experience major direct and
indirect effects from the innovations that are created. For instance social
innovations where older adults help each other out may lead indirect to a
decreasing burden of family caregivers (Lemey et al., 2015). There are also
innovations (e.g. communication technology) to support caregivers directly. In both
situations family caregivers, volunteers and professional caregivers should
participate in the CLL.
Informal caregivers are also a heterogeneous group. There are spouses, children or
other family members and volunteers. In Europe 21% - 43% of the noninstitutionalised population of 65 years of age or over receive help or support at
least sometimes on an informal basis (Riedel & Kraus, 2011). In Belgium the age of
informal caregivers varies between 20 and 89 years (Panel study on Belgian
households, 2001). About 6% of the European population aged 50 or over provides
personal care for an older relative or family member. Almost 60 % of them is
female (Riedel & Kraus, 2011). Other differences can be seen in terms of
employment, marital status and hours of provided care.
The group of professional caregivers is also very broad and consists of nurses,
nurse-assistants, doctors, dentists, pharmacists, physiotherapists and other
paramedical professionals like dieticians, occupational therapists, speech
therapists, hearing-aid specialists, audiologists, podiatrists, pharmacist assistants
and non-urgent medical transporters. Also social workers and other staff (e.g.
hygienic staff) from healthcare institutions and nursing homes should be included
as they are part of the network of older adults. These professional caregivers also
differs for variables like age, experience, job motivation,
This heterogeneity of the multiple actors is an important point of attention for CLL
when building a panel.
1.4 Recruiting older adults and their caregivers as future users
This heterogeneity and the dynamic characteristics of the population of older
persons and their caregivers makes their recruitment for a panel in a CLL a
challenging task. Several barriers can prevent them to participate (Mody et al.,
2008, McMurdo et al., 2011; Law, Russ & Connelly, 2013; Wilding et al., 2013). The
most common practical barriers are high travel costs and lack of time. Health

183

related barriers can be divided in physical and psychological problems (e.g.

cognitive impairment). Older adults with health related problems need assistance
to participate in a CLL. Social and cultural issues also occur frequent. These are
problems with language and literacy, financial problems or different cultures.
Immigrants are often more sceptical towards research.
To conquer these barriers a fine recruitment plan should be made. The
identification of the target groups needs to be the starting point. Hereby inclusion
and exclusion criteria should be set. These criteria should be in line with the goals
of the innovation. Moreover it is important to make an overview of the sources
were future users can be recruited. Community-based settings such as senior
centres are effective sources to find community-dwelling older adults (Wilding et
al., 2013). Care institutions and nursing homes can also be useful sources to find
older adults and their caregivers. Gatekeepers, such as family, community leaders,
institutional leaders and physicians, nurses, or other direct care workers can
provide access to those sources, identify potential subjects and can be a decisive
influence on participants decisions to enrol (Mody et al., 2008; Wilding et al., 2013).
Creation of an advisory board with these gatekeepers can be of value with
information about the needs and concerns of the future users (Mody et al., 2008).
During the identification of the future users it is also important to identify their
motivation to participate in a CLL. Intrinsic values (loyalty, civic duty and the wish
to improve the government), personal traits (education, family composition), being
responsible, trust in co-creation initiatives and the perceived abilities influence the
willingness of older adults and their caregivers to participate (Elad et al., 2000;
Law, Russ & Connelly, 2013; Voorberg et al., 2014).
In a next step the future users needs to be contacted and a trust relationship must
be built up (McHenry et al., 2012). Personalized outreach such as in-person contact
or personalized invitations is highly effective across all populations (McHenry et al.,
2012; Wilding et al., 2013). Face-to-face contact can be achieved through
gatekeepers, active participants of the CLL or representatives of the CLL, like a
panel manager. Both written and digital media can be used. Written folders with
information about the CLL should be placed in spaces where older adults and their
caregivers come along (e.g. waiting rooms for practitioners or churches). Other
possibilities are an article in the local newspaper or an announcement on the local
radio or television. Social media can be used to recruit younger caregivers.
Explaining the goal and expectations is important during this contact, as well as
convincing them of the possible benefits like access to helpful treatments, services,
or diagnostic tests; social interactions with staff or other participants; recognition of
ones contribution; or general altruism (Mody et al., 2008).
2 Method
In this case study insights from a plan evaluation and action research were used to
map how the CLLs and their projects plan and organise to recruit future users for
their panels.

184

2.1 Plan evaluation

A plan evaluation of the CLL was done through a document analysis of the
submitted proposal of the CLLs and semi-structured interviews with their
representatives. A content analysis on six themes (innovation goals, target
population, networks, labour organisation, geographical area and technology) was
performed in several steps. At first a close reading of the proposals of the CLL,
submitted to the funding agency IWT, was performed. Meanwhile all information
relevant for the six themes was selected. In the second phase this information was
used to explore the CLLs views on innovation within the different themes.
Thereafter these visions were compared with prevailing theories through deductive
analysis. Data-extraction and the first step of the analysis were performed
independently by two researchers. Differences were discussed with all researchers
(n=5) leading to research triangulation (Denzin 1989).
Additionally 29 semi-structured interviews were conducted with the CLLs
representatives to collect missing data and to verify the results (membercheck)
(Hannes, 2011). The coordinators of the CLLs were contacted by email. They
invited other representatives for the interview through mail, telephone or personal
contact. Mostly there were two researchers to perform the interview. One
researcher led the interview, while the other took notes. Interviews were recorded
digitally (45-150 minutes). The interview locations were chosen by the CLL
coordinators. The same method for analysis as in the documents was used for the
interviews. Additional insights from the interviews were integrated into the results.
2.2 Action research coaching session
The action research was presented to the CLL as a coaching trajectory. It started in
2013 with a half day workshop where all representatives of the CLL were invited. At
least the panel manager and the CLLs coordinator was expected. The four CLLs
which were active at that time were represented. During this workshop
presentations and Q&A was given on the background knowledge of iMinds on
good practices (see Panel-Based approach). When two additional CLLs joined the
CLL, an information meeting was kept for them communicating the information of
the workshop.
Based on the information of the workshop each CLL received the opportunity to
upgrade their plan to recruit participants for one of their first projects. A three hour
coaching session with the coordinator and panel manager took place with two
iMinds coaches, one with experience in panel management and one with
experience in user research on digital tools for older adult care. This two-way
process learned the starting CLLs how to make the advice operational for their
own situation, and the iMinds coaches gained insight in the specificities of panel
management in the domain of older adult care.
3 Results
These two methods showed different lenses on the CLL case. First, insights from
the plan evaluation will be discussed, subsequently those of the action research
approach.

185

3.1 The initial plans of the six platforms

LiCalab, AzoB and CareVille wanted to set up a stratified panel in line with the
demographical composition of their regions (using age, sex, nationality and social
economic status as variables). Only three CLLs (LiCalab, AIPA and CareVille)
described the plan to include informal caregivers in their panel. AzoB and Online
Buurten stated that they wanted to include informal caregivers during the
interview. InnovAGE did not plan to include informal caregivers, but uses an
organization of informal caregivers as representative for the actual informal
caregivers.
Only few CLLs set a goal for the size of their panel. AzoB wanted two samples from
the inhabitants of two different districts in two different urban regions. Each
sample had to consist of 200 older adults. AIPA aimed at a broad and
representative test population of at least 1000 older adults and professional
caregivers in homecare services. The participants will all be inhabitants from semiurban and rural regions. CareVille wanted to include 250 informal caregivers in
their panel. InnovAGE stated that they have access to 8800 65+ vulnerable older
adults from an urban region. However they did not write explicitly the intention to
include them all in their panel.
All CLLs described inclusion criteria to recruit older adults. Only three CLL
(LiCalab, AIPA and CareVille) described inclusion criteria for informal caregivers.
Five CLL will include both healthy and dependent older adults. One CLL
(InnovAGE) will only include frail older adults with a complex care situation.
CareVille will include older adults with acute and chronical care requirements.
AIPA, AZOB, LiCalab and Online Buurten described broader criteria which are not
related to pathologies or care requirements. Criteria for ages varied between al
CLL. All inclusion criteria are enlisted in table 2.
CLL
AIPA
LiCalab

CareVille

InnovAGE

Inclusion criteria
Every type of older adult, 50-55+, living
at home, 20% socially vulnerable
20% 50-60, 50% 60-70, 20% 70-80, 10%
80+; 40%-45% light care requirements,
30% in need for chronical care, 15-20%
high care requirements, 10-15%
immigrants living at home, healthy
older adults
65+ older adults, with acute or
chronical care requirements, older
adults from different cultures, living at
home and in residential settings,
attention for deprivation
Fragile older adults with complex care
requirements, 60+, living at home , a
service flat or in residential setting,
attention for socio-economical profile

186

AzoB

65+, living at home or in a service flat,

attention for socio economic status and
origin
Wide focus, 65+, healthy older adults,
living at home, social vulnerable older
adults

Online Buurten

Table 2 Initial inclusion criteria for older adults of the CLL based on a plan evaluation

The CLLs planned to recruit older adults through various channels such as senior
organisations, social housing companies, city councils or municipalities, several aid
and care actors (like hospitals, nursing homes, public centre for social welfare, ...)
media, personal contact, actions on events, fairs and campaigns. Table 3 provides
an overview of the recruiting channels from the CLL.
AIPA LiCalab CareVille InnovAGE AzoB Online
Buurten
Senior
organisation

Social housing X
company
City council or x
municipality

Aid and care x

actors

Media

Personal
contact

Actions
on
social events

Actions on fairs
and campaigns

x
x

Table 3 Initial recruiting channels from the CLL based on a plan evaluation

Methods to capture the wishes and needs of the older adults and informal
caregivers varied. Most CLL planned to use a combination of online surveys, focus
groups, interviews, written surveys, brainstorming, user testing. AIPA only
mentioned the use of focus groups.
Plans to recruit voluntary caregivers and professional caregivers were not found in
the initial documents of the CLL. However it is possible the CLL had such plans as
the researcher did not check this during the interviews.

187

3.2 The adapted approach of the six platforms

Due to the heterogeneous group of older adults and the need of involving multiple
actors a more systematic approach was needed without dropping the seven steps
(see table 1) within the CLL case. Because both living labs and projects were starting
up at the same time, the first two steps needed a lot of attention.
Step 1 Define the purpose of the panel: start with mapping multiple actors in the
projects
Within the first step Defining the purpose of the panel an extensive analysis of
the living lab panel was performed, based on the findings that all CLL defined their
panel too closed and focused on older adults, forgetting all other involved panel
members needed within a multiple actors panel.
To help them make this analysis we mapped together their users on a diagram with
concentric circles with the primary user in the middle (figure 1). For example in one
project the medication process of 50 residents in a nursing home will be studied
and adapted. This implies not only that 50 residents need to be included, but also at
least their family and professional caregivers. They are involved to give valuable
feedback and are gatekeepers for access to the residents. In addition, organisations
giving access to care professionals or family should not be forgotten, to provide
access and give feedback. In every coaching session each CLL was made aware of
the need to map the actors, to include and create a panel plan for every project.

Figure 1 Example of mapping multiple actors for a project to be supported by a CLL

Step 2 Recruit users: keeping a systematic overview with a panel matrix

188

The heterogeneity within and between projects made it necessary to develop

another tool to support the panel manager in knowing when to recruit who for
what. In figure 2 an example is provided of the basic format for the same project
example about the medication process and adaptation. The matrix has to be used
in two steps. First there must be identified which panel members are required
during which activity in each project phase by ticking to right boxes in the matrix.
Next, the number of people with that profile to be recruited need to be filled in
those ticked boxes. Doing this exercise for all projects helped the CLLs to
determinate their communication and recruitment strategy. If vulnerable
inhabitants in nursing homes are needed in the first project activity, there is less
need to do a large communication campaign to recruit active older adults. This
implies also that the organisational plan of the helpdesk becomes more clear and a
financial plan on the budget for all panel activities can be made. It also reveals
which type and how many of the needed panel members are not reached yet. For
example it is classic that people of different origin and weaker socioeconomic status
are less likely to be reached with classical city communication. Other activities are
then planned, like house calls for example. The matrix is of course a living
document, to be updated over time. The nearby activities need to be more detailed,
than the ones planed in the further future. It is essential to determine the success
parameters for these nearby activities up front. These parameters can also be used
during the evaluation. The matrix can also be used to check if a new project request
fits with the current profile of the CLL panel.

Figure 2 Support and interact with the panel: use of a SPOC and feedback is a two-way street

Step 3 Support and interact with the panel: use of a SPOC and feedback is a twoway street

189

In the CLLs plans the importance of the function of a panel manager as a fulltime
job offering a centralised approach, with a known and trusted person for the panel
(SPOC), was often underestimated. The workshop introduction, the coaching and
the day-to-day experience of panel management made that CLLs without this
function decided that they need it and had to find budget for it. The background of
people performing this function (e.g. knowledge institute, care organisation, city) is
divers, and not every CLL was able to create one full time position. AIPA for
example tackles this challenge by involving a call centre, which is a partner in the
CLL, to monitor and triage the questions and to provide first line information on
planned activities. Every platform was suggested to create a helpdesk structure.
Also the creation of material to make participation as easy as possible, as well as
ways to communicate feedback and results about the activities are essential
ingredients. In the start-up phase the material was created to inform people about
the projects on websites, leaflets, during information sessions on a broader topic,
These activities are closely related to the next step.
Step 4 Live the lab: keep your panel involved
This step targets long term involvement over different activities supporting
different projects. Crucial advice at the start was to change the plans of bulk
recruitment at the start, which some CLLs had. Making use of the panel matrix
helped to alter these plans. Creating a brand for the CLL, overarching the different
projects was another tip, to enable to group identification leading to commitment
and norms of reciprocity as described in the Tailored management approach (e.g.
figure 3). Platforms developed a logo and a website as a starting point, but differ in
communication to present the CLL to stakeholders as well as older adults. Herefore
folders, videos and events were used. At the start it was difficult to communicate
both the goal of the CLL and the project in one message. This resulted in
miscommunication on the expectations to the participants. Most opted for a double
strategy communicating in general about the CLL and specific about the projects to
the people involved, informing them on the fact that this is part of a larger
initiative. The heterogeneous group of older adults makes it not possible to take
the use of internet and social media for granted. Although some platforms like
Online Buurten, Licalab, AzoB and CareVille plan to use digital tools to
communicate with their panel members, which is a project endeavour in itself, but
opens up opportunities for the future tailored communication.

190

Figure 3 Example of visual and branding of a CLL (Careville)

Step 5 handle privacy: beyond privacy when discussing care related subjects
In every living lab handling the personal information of participants carefully is the
key to build a trust relationship. It is important to offer information about the
reason to collect information, the person who can see and use it and the purpose it
is required for. The panel manager was suggested to be the key person to both
inform and decide what the need to know basis of information is of every request
made. Understandable informed consents for all participants, information sharing
agreements with partners accessing this data are essential, as well as filling the
platform activities with the Belgian privacy commission. Some care institutions, as
gatekeepers, demanded extra precaution with their clients. The caregiver was then
the contact person for the panel, serving as a trusted person, collecting the data for
the project, without personal information. All subsequent contacts in the panel
with this client have to go through the trusted caregiver. This request implied
adaptations in the initial panel management approach of this CLL. In Before the
coaching sessions and workshop on panel management took place, a dedicated
workshop on privacy and the need for an ethical committee was organised. In every
project in this domain one needs to reflect on the need to file for an approval of a
medical ethics committee. Not every project innovating in the domain of care is
required to do that, but it is an additional point of attention.
Step 6 reward users: recognition and fun more important than payment
In all living lab projects, and also the CLL projects the experience is that feedback is
more important than presents or payment (e.g. Logghe et al., 2014). Today the first
results and practical examples of the activities that can help to motivate current
members as well as recruit new participants for future activities become visible (e.g.

191

co-creation movie https://youtu.be/88IhYOAeriY). Providing a museum visit with a

field test was an approach AIPA planned. Licalab organised two days to hear and
celebrate their panel members. Three hundred and fifty people took part and 106
questions and suggestions were made. The event was celebrated on different media
channels (twitter, their website and the local radio station).
Step 7 Maintain the eco-system: working with older adults, needs more follow up
and attention to exit strategies
To let the panel grow over time, the advice is to create an environment where it is
easy for current members to suggest new members, to pamper panel ambassadors,
give people a central place where information about several projects can be shared
(e.g. a website, a newsletter, ). It is important to realize that people can also leave
the panel. Understanding why they do not wish to participate further in the CLL is
crucial. Older adults can experience a lot of life changing events (e.g. illness,
increased dependencies, loss of loved ones). These life events may change their
commitment to the community, or may lead to the perception that activities are not
appropriate for their condition. CLL should look for adaptations in the format of
the activity to improve the participation of frailer older adults. Another confronting
lesson learned early in the creation of the panel is that panel members pass away.
Specific attention should be paid to communicate in the right way to relatives.
4 Conclusion
The plan evaluation revealed different plans to form a representative panel of older
adults. However few CLLs had a strategy to include informal and professional
caregivers. Nevertheless the innovations developed in the CLL's will also affect
them. Thus, it important to define the target population clearly before the start of
the recruitment and to set inclusion and exclusion criteria from the beginning.
Most CLL planned a personalized approach for the recruitment, which proved to
be successful in earlier research (McHenry et al., 2012; Wilding et al., 2013). Few
CLL's had thought about a way to engage the end-users to their panel.
The seven step approach used in other domains of panel-based living labs is a
valuable guideline for the creation of a panel-based living lab in the domain of
innovation in older adult care. Because of the multiple actors a more systematic
approach with the concentric circles and a panel matrix was designed, showing the
assets and needs while growing as a panel over different projects. The essential
function of a panel manager was reaffirmed. Extra attention within this domain
should be paid to the existing care relations between care organisations and their
clients, the need for filing of the project to the medical ethics committee and the
delicate follow up of exit of panel members.

192

5 References
Buffel, T., De Donder, L., Phillipson, C., De Witte, N., Dury, S. & Vert, D. (2014).
Place Attachment Among Older Adults Living in Four Communities in Flanders,
Belgium. Housing Studies, 29, 6, 800-822
Ellenbecker, C. (2010). Preparing the nursing workforce of the future. Policy,
Politics, & Nursing Practice, 11(2), 115-25.
Estrada, M., Woodcock, A., & Schultz, P. W. (2014). Tailored Panel Management: A
Theory-Based Approach to Building and Maintaining Participant Commitment to a
Longitudinal Study. Evaluation Review, 38(1), 3-28.
European Commision (2015). Public Health. Consulted at 20 April 2015 at
http://ec.europa.eu/health/ageing/policy/index_en.htm
European Commission and Spanish Ministry of Health and Social Affairs (2010).
Mental Health and Well - Being in Older People Making it Happen. Conference
28th June 2010, Madrid.
Denzin, N. (1989). The research act: a theoretical introduction to sociological
methods. Englewood Cliffs: Prentice Hall.
Federal Public Service Economy (2015). Population - Statistics Belgium. Consulted
at 20 April 2015 at http://statbel.fgov.be/nl/statistieken/cijfers/bevolking/
Hannes, K. (2011). Chapter 4: Critical appraisal of qualitative research. In
Supplementary Guidance for Inclusion of Qualitative Research in Cochrane
Systematic Reviews of Interventions. Consulted at 23 October 2014 at
http://cqrmg.cochrane.org/supplemental-handbook-guidance
Hussein, S., & Manthorpe, J. (2005). An international review of the long-term care
workforce. Journal of aging & social policy, 17, 4, 75 - 94.
Lemey, J., Gorus, E., Brys, C., & De Vriendt, P. (2015) Living labs Flanders
opportunities for informal care. Paper presented at 38th Winter Meeting, Ostend,
Belgium.
McHenry, J., Insel, K., Einstein, G., Vidrine, A., Koerner, K., & Morrow, D. (2012).
Recruitment of older adults: succes may be in the details. Consulted online at 4 may
2015
at
http://gerontologist.oxfordjournals.org/content/early/2012/08/09/geront.gns079.lon
g
Logghe, S., Baccarne, B., Veeckman, C., Lievens, B.& Schuurman, D. (2014). Uit
passie of voor de poen? Een exploratie van gebruikersmotivaties voor deelname
aan innovatie onderzoek in Living Labs. Paper presented at Etmaal van de
Communicatiewetenschappen 2014, Wageningen, The Netherlands.
Mody et al. (2008). Recruitment and retention of older adults in aging research.
Journal of the American Geriatrics Society, 56(12), 2340-2348.
Neuwirth, E. E. B., Schmittdiel, J. A., Tallman, K., & Bellows, J. (2007).
Understanding panel management: a comparative study of an emerging approach
to population care. The Permanente Journal,11(3), 12. Retrieved from
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3057714/
Panel Study on Belgian Households. (2001). University of Antwerp & Universit de
Lige.
Riedel, M., & Kraus, M. (2011). Informal care provision in Europe: regulation and
profile of providers. European Network of Economic Policy Research Institutes
(ENEPRI) Research report no. 96.

193

Schuurman, D., & De Marez, L. (2012). Structuring user involvement in panel-based

living labs. Consulted at 7 May at http://timreview.ca/article/606
Vervoort, K (2013) Working with users: Living labs bottom up, WS7, 4th ENoLL
Living Lab Summer School, Manchester. Consulted at 7 May at
https://vimeo.com/channels/4thenollsummerschool/80471087
Voorberg, W., Bekkers, V., & Tummers, L. (2014). A Systematic Review of CoCreation and Co-Production: Embarking on the social innovation journey.
Consulted
online
at
4
May
at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2444075
Wilding, M., Seegert, L., Griffin, M., Kachnowski, S., & Parasuraman, S. (2013).
Falling short: Recruiting elderly individuals for a fall study. Ageing Research
Reviews, 12, 552-560.

194