PRACTITIONERS TOOLKIT

BEING SMART WITH DATA,

USING INNOVATIVE
SOLUTIONS

Employment, MARCH 2017

Social Affairs
and Inclusion
 Europe Direct isaservice tohelp you find answers
to your questions about the European Union.

Freephone number (*):

00 800 6 7 8 9 10 11
(*) The information given isfree, asare most calls
(though some operators, phone boxes orhotels may charge you).

More information onthe European Union isavailable onthe internet (http://europa.eu).

Luxembourg: Publications Office ofthe European Union, 2017

ISBN 978-92-79-66962-0
doi:10.2767/797031

European Union, 2017

Reproduction isauthorised provided the source isacknowledged.

Cover picture: European Union

The European Network ofPublic Employment Services was created following aDecision ofthe European Parliament and Council inJune 2014
(DECISION No573/2014/EU). Its objective isto reinforce PES capacity, effectiveness and efficiency. This activity has been developed within
the work programme ofthe European PES Network. For further information: http://ec.europa.eu/social/PESNetwork.

This activity has received financial support from the European Union Programme for Employment and Social Innovation "EaSI" (2014-2020).
For further information please consult: http://ec.europa.eu/social/easi

LEGAL NOTICE
This document has been prepared for the European Commission however itreflects the views only ofthe authors, and the Commission cannot
beheld responsible for any use which may bemade ofthe information contained therein.
PRACTITIONERS TOOLKIT

BEING SMART WITH DATA,

USING INNOVATIVE
SOLUTIONS

Written byDr. Willem Pieterson,

the Center for e-Government Studies,
in collaboration with ICF

MARCH 2017
Contents

CHAPTER 1. INTRODUCTION 6
1.1 What isthis toolkit about? 6
1.2 Who this toolkit isfor 8
1.2.1 PES with little orno experience working with data 8
1.2.2 PES with some orintermediate levels of experience 8
1.2.3 PES with advanced levels ofexperience 8
1.2.4 Tactical oroperational managers 8
1.3 Strategic managers 9
1.4 Scope ofthis toolkit 9
1.5 Reading guide 9

CHAPTER 2. GETTING STARTED WITH DATA 10

2.1 Creating aplan 10
2.1.1 Deductive approaches 10
2.1.2 Inductive approaches 11
2.2 Creating your data team 12
2.2.1 Leadership 12
2.2.2 Data team members 13
2.3 Setting upthe data infrastructure 14
2.4 Creating adata-catalogue 15
2.5 Costs and budgeting 16

CHAPTER 3. ORGANISING DATA 19

3.1 Cleaning & Sanitising 19
3.2 Describing data & data characteristics 20
3.3 Quality control 21
3.4 Integrating data sources 21
3.5 Security and Data Protection 22

CHAPTER 4. ANALYSING DATA 25

4.1 Overview 25
4.2 Statistics 26
4.3 Data mining & KDD 29
 4.4 Advanced Analytics 31
4.4.1 Artificial Intelligence 31
4.4.2 Machine Learning 33
4.4.3 Deep Learning 34
4.5 Combinations & Derivations 35

CHAPTER 5. PRESENTING & REPORTING 36

5.1 Why move away from traditional reports? 36
5.2 (Interactive) Visualisations 37
5.3 Interactive Tools & Dashboards 38
5.4 Open data 40

CHAPTER 6. EVALUATION & CONTINUATION 41

6.1 Evaluation 41
6.2 Continuation and scale-up of pilots 43

APPENDICES 45
Appendix 1 | Safe Harbor De-identification types 45
6

Chapter 1. The role and importance ofdata inour society

isgrowing fast. Not only dowe collect more and

Introduction
more data, but advancements incomputing power
aswell asthe tools and algorithms toanalyse data
allow organisations touse data inentirely new ways.
This also applies toPublic Employment Services
(PES). Afew PES are exploring the use ofBig Data
toimprove efficiency and effectiveness ofprocesses,
improve customer satisfaction and/or innovate
inorder totransform how the PES functions. This
toolkit could help them onthis journey. Most PES,
however, are atthe start ofthis journey and this
toolkit can also help PES who want tostart using
their data inbetter and smarter ways.

1.1 What isthis toolkit about?

This toolkit isabout data and the use ofdata tocre-
ate abetter functioning PES. Among the chief rea-
sons tohave atoolkit about data isthat PES, like
most other organisations, are collecting more and
more data. And asdata isbeing stored inproduction
systems and/or data warehouses, the possibility
arises touse this data for all kinds ofpurposes. The
infographic below illustrates a) the tremendous
growth in(world-wide) data production, b) the poten-
tial (cost) benefits ofusing so-called big data and
c), the anticipated future growth ofdata production
(e.g. due tothe growth ofthe Internet ofThings).

Figure 1: Infographic about big data

0,5 %
100.000.000.000 of all data
Potential savings in operational are being
efficiency at governments in developed analysed
European countries due to big data* and that
percentage
is shrinking**

44x
Growth in data
production
26.000.000.000
2009-2020**** Number of connected devices
2010 2030
on the Internet of Things (IoT)
in 2020*****
The amount of data
produced doubles
every year***

* http://www.csc.com/insights/flxwd/78931-big_data_universe_beginning_to_explode
** http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/
big-data-the-next-frontier-for-innovation
*** https://www.technologyreview.com/business-report/big-data-gets-personal/
**** http://www.gartner.com/newsroom/id/2684616
***** http://www.emc.com/about/news/press/2011/20110628-01.htm
 PRACTITIONERS TOOLKIT

With the expected future growth ofdata production,
several challenges arise:
How tounlock the potential ofdata wealready
have toimprove efficiency, effectiveness,
customer satisfaction orany other goal?
How toset-up the infrastructure inthe organi-
sation now sothat wegain experience with data
analytics before wedrown inasea ofdata?
How tointegrate data-analytics into the DNA
ofthe organisation sothat organisational
decision making improves and organisational
agility increases?
This toolkit aims tohelp organisations, specifically
PES, toget started with finding answers tothese
questions.
Even though the topic ofthis toolkit isdata, the true
goal isto help PES transform their existing data into
Information, Knowledge, and subsequently Wisdom.
Ineach stage oftransformation ofdata, the data
isenriched, asthe schema below illustrates.

Data Information Knowledge Wisdom

Raw Meaning Context Application

Figure 2: Data transformation process

STEP TRANSFORMATION EXPLANATION EXAMPLE(S)

Data None This isplainly data, the 63 [just the plain number]
(Raw data) numbers how you would
extract those from any
system.

Information Adding Meaning When transforming data into
information, weadd basic
meaning tothe information.
Very often this happens
byadding units, variables
and definitions.
Knowledge Adding context When adding context, weare
able tomake sense ofthe
data, itstarts telling astory.
Wisdom Adding application Turning knowledge into
action isthe last step of
the process. Bycombining
data points, you can create
actionable results.
63% oflower educated
clients are unhappy with
PES service levels
63% oflower educated clients
are unhappy with PES service
levels, compared to48%
ofhigher educated clients.
63% oflower educated clients
are unhappy with PES service
levels, compared to48%
ofhigher educated clients.
This correlates with their
evaluation oflanguage
difficulty onthe PES website.
Changing language level
could solve this problem
8
The toolkit isbased onthe thematic review workshop
onmodernising PES through data and ITsystems1.
The workshop revealed aneed tounderstand the
topic ofdata inmore detail and explore how PES
can benefit from advancements inthis field. Core
concept inthis toolkit isthe concept ofsmart data.
Byusing the concept smart data, wewant toprevent
bias towards data being anormative goal initself.
Smart data isseen asthe sum of:
(Big) Data [the data itself]
Utility [the potential utility derived from
the data]
Semantics [the semantic understanding
ofthe data]
Data Quality [the quality ofthe data collected]
Security [the ways data are managed securely]
Data Protection [how privacy and confidenti-
ality are guarded]
These different topics are woven throughout the
body ofthis toolkit. Similarly, tostay focused,
westreamline this toolkit along the lines ofthe PDCA
(Plan, Do, Check, Act) Cycle. The first content chapter
[2] focuses onhow toget started and create aplan.
The following chapters [3-5] focus onthe actual
doing. While wefocus throughout the toolkit and
the proper checks and balances, chapter 6, isspecifi-
cally devoted tothe topic ofchecking and evaluating.
While this isapractitioners toolkit, most content
inthis toolkit this actionable, but once again, specific
action points after the analytical process are dis-
cussed inthe final chapter.
1.2 Who this toolkit isfor
The primary audience for this toolkit consists ofPES
who have little tono experience working with (big)
data. Asapractitioners toolkit, managers ontactical
and operational levels may benefit the most from
the content inthis toolkit. For example managers
who have been tasked with analytics ordata science.
However, there will beuses for other audiences
aswell. Below welay out how the different audiences
could benefit from this toolkit.
1 The Thematic Review Workshop took place
inCroatia in6-7 July 2016; itwas developed
under the PES Mutual Learning Programme.
1.2.1 PES with little orno experience
working with data
This toolkit can serve asastarting point for those
PES who want toget started with data. Itgives
anoverview ofrelevant actions and provides practical
tips onhow toget started and where.
PES with little orno experience are advised tostart
reading at: Chapter 2 Getting started with data
1.2.2 PES with some orintermediate levels
of experience
For PES with some experience with data, especially
the sections onadvanced analytics and the various
examples provided throughout this toolkit may beof
interest. Furthermore, the sections onreporting and
presenting data may provide new insights. Lastly,
the parts ondata security and protection could serve
asagood refresh onmatters related tosecurity.
PES with some experience, are advised tostart read-
ing at: Chapter 3 Organising data
1.2.3 PES with advanced levels ofexperience
PES that have much experience working with data
may still benefit from this toolkit. Onthe one hand,
the toolkit may provide some new insights, especially
regarding more novel developments inthe advanced
analytics section. Furthermore, the toolkit can serve
asan introduction for new employees who join data
teams. Lastly, even advanced PES may find interest-
ing examples from other PES throughout this toolkit.
The most relevant section for PES with advanced
levels ofexperience will besection 4.4.
1.2.4 Tactical oroperational managers
Tactical oroperational managers tasked with setting
updata functions and/or capabilities within PES may
benefit the most this toolkit. The entire toolkit should
berelevant for this audience. The toolkit will help
you get familiar with much ofthe jargon used inthe
world ofdata analytics and should give you enough
guidance toget started. While this toolkit ismeant
toprovide ageneric introduction and practical tips
and tricks, most sections will provide links toother
resources that could help you further.

1.3 Strategic managers
Biggest value for strategic managers ofthis toolkit
istwofold. The first isthat ithelps higher level execu-
tives infamiliarising themselves with the possibilities
ofdata analytics. The second isthat the toolkit can
support strategic managers intheir decision making
processes around data analytics, for example
interms ofhiring adata team, and the goals where
data analytics could make adifference.
Specific strategic points are added throughout this
toolkit that are relevant for strategic managers
(See pages 10, 11, 13, 15, 16, 20, 22, 23 and 31).
Strategic manager will benefit the more from read-
ing chapters 2 & 6.
1.4 Scope ofthis toolkit
This toolkit isabout the use ofdata that PES collect
intheir systems orthrough other methods. Data cur-
rently stored inproduction systems ordata ware-
houses are examples ofthis type ofdata. Wealso
include data that are available inaPES that donot
stem from primary processes, such asresearch data
(e.g. data-sets from survey research), data shared
from other organisations (e.g. educational data)
oreven reports, books, etc. Insum; wefocus onall
data the PES already has and less soon data the PES
would need tocollect toachieve certain goals. Con-
ducting research (e.g. how toconduct surveys, inter-
views, focus-groups, etc.) isnot part ofthis toolkit.
In terms ofanalytics; the focus ison the more novel
types ofanalytics and specifically those more com-
monly associated with big data. Where appropriate,
wewill discuss more traditional types ofanalytics
(such asmore common statistical methods), and
tools (such asExcel, SPSS, SAS, etc.), but given the
abundance of(online) resources, wewill link tothose
resources instead ofproviding more comprehensive
information inthis toolkit.
PRACTITIONERS TOOLKIT
The same applies tothe presentation ofinformation.
While traditional research reports (e.g. print, pdfs,
etc.) are still widely used and can begood ways
topresent data and while (static) graphics created
in for example spreadsheet software can provide
compelling insights, weagain choose tofocus onthe
more novel solutions topresent data. This includes
tools tocreate interactive graphs and online
dashboards.
1.5 Reading guide
The content ofthis toolkit isorganised asfollows:
1. Getting started with analytics. Creating aplan,
aswell asadata team. Setting upthe data infra-
structure and hecreation ofan inventory ordata
catalogue.
2. Organising data. Cleaning and describing data.
Ensuring the quality ofdata, integration ofdata
sources. How tosecurity data and protect privacy
and confidentiality.
3. The actual analysis ofdata. Different types ofdata
analysis including statistical methods, data mining
and advanced topics such asartificial intelligence
and machine learning.
4. Presenting and reporting ofdata. What are novel
ways topresent results and what are considera-
tions when reporting outcomes? Also discussion
ofopen data.
5. Evaluation, continuation and implementation. How
togo from small pilot orexperiment toabroader
implementation? What are the key technical and
organisational considerations?
10
Chapter 2. Does the organisation have quantifiable goals
Getting started
with data
This chapter deals with the question ofhow toget
started. What are the main things tothink about
when anorganisation wants tostart doing more
with the existing data? What kind ofskills and knowl-
edge would you need?
The following topics form the heart ofthis chapter:
Provide overview ofthings tothink ofwhen asthe composition ofthe data team involved and
starting towork with data
Help create ateam ofpeople who can work
with data
Setting upthe organisational infrastructure
The following strategic questions are answered inthis
chapter:
What isthe best way toget started and what
can Iexpect from working with data?
Whom should bein charge and where inthe
organisation should weposition the data
function?
What can Ido toensure success?
The following tactical questions are answered inthis
chapter:
What are the things weneed todo actually
start working with data?
2.1 Creating aplan
The most important question toanswer when starting
to work with data is: why? Depending onthe answer
to that question the approach taken when implementing
analytics and working with data will be different.
Is there aconcrete (organisational) problem
that needs tobe solved?
Figure 3: Deductive research approaches
Theory Hypothesis
and data available totrack progress towards
meeting those goals?
Does the organisation simply want tolearn
about working with data sothat experience
isgained that could beuseful for the future?
Does the organisation want toinnovate and see
what improvements can bemade from tinkering
with (data) resources available?
When the why question ofthe organisation looks
like the former two questions, the organisation isbet-
ter off starting with adeductive approach tothe use
ofdata. When itlooks like latter two questions,
aninductive approach may bebetter. The utility
derived from each approach will bedifferent, aswell
the place ofthis team inthe organisation.
Smart data = (Big) Data + Utility
+ Semantics + Data Quality + Security
+ Data Protection
Utility refers tothe usefulness ofsmart data.
The more itaids insolving business problems
and/or the increase ofknowledge inthe organi-
sation, the higher the utility is. However, depend-
ing onthe approach taken when starting with
analytics (deductive orinductive) expectations
regarding utility can bedifferent and should
bemanaged assuch.
2.1.1 Deductive approaches
In deductive approaches, the organisation has a(rela-
tively) clear understanding ofwhat itwants todo,
for example what the problem isthat needs tobe
solved.
For instance, aPES may find that anew automated
vacancy matching system yields lower rates ofsuc-
cessful matches that the previous manual matching
Observation Confirmation

Figure 4: Inductive research approaches
Observation Pattern
process. Inthis case, the PES could formulate the
expectation ortheory that the new matching system
isworking improperly. This theory could betranslated
into aseries ofconcrete expectations orhypotheses
that could betested using data (e.g. The matching
algorithms donot include certain important variables
orthe weight ofcertain variables incalculating
matches istoo high/low). Ifsuch hypotheses exist,
adata team could start working ontesting these
hypothesis and work closely with the business pro-
cess owners inthe PES tocollect and analyse data.
Concept: Data team
In the context ofthis toolkit, wedefine the data
team asthe group ofpeople within the PES
tasked with data analytics. This could overlap
(partially) with existing research teams (whose
focus typically ismore onstatistics (see section
4.2) and/or data mining (see section 4.3). Focus
ofthe data team ison the analysis of(big)
data that can beextracted from systems
orimported from other sources.
In such asituation, the analytics function would bemore
embedded within the relevant business processes
toensure smooth communication tosolve the problem
athand. Furthermore, higher levels ofvalidity ofand
confidence inthe results are required ifoutcomes are
being used inproduction processes.
Concept: Data analytics2
The whole process offormulating data goals,
collecting, organising, analysing and presenting
(big) data.
2 Some equate data analytics todata analysis,
wedecide todifferentiate for the sake ofclarity
(and inline with many publications dealing with big
data). Wesee analysis asthe process ofanalysing
data, whereas wesee analytics asamuch broader
term involving the setting ofgoals, collection,
organisation, analysis and presentation ofdata.
PRACTITIONERS TOOLKIT
11
Hypothesis Theory
2.1.2 Inductive approaches
The set-up ofinductive approaches isdifferent. There
isno clear understanding ofaproblem and the focus
ison:
1. Learning from the data and/or finding ways to
generate value from this data.
2. Learning about analytics and gaining knowledge
about the process and (smart) data applications.
Therefore, ininductive approaches there are fewer (if
any) explicit expectations orhypothesis regarding
outcomes. The focus ison letting the data speak and
discovering interesting patterns inthe data. For exam-
ple, suppose aPES gets high levels ofunstructured
emails from clients (for example with questions,
complains, comments), adata team could start ana-
lysing these emails tosee ifthere are any interesting
patterns inthis data, for example regarding:
The word choice ofpeople (that could beused
tochange the tone ofvoice incommunication
orthe use ofcertain synonyms inwritten
communication).
The questions people have (that could be
used tochange the way information isbeing
displayed onwebsites).
The language (skill) level ofcertain groups
ofjob seekers (that could beused tocreate
customer segments that could betargeted
using different communication styles).
Only after certain patterns have been discovered
inthe data after analysing high numbers ofobserva-
tions, certain hypothesis and eventually theories
could beformulated regarding the discovered pat-
terns (which insubsequent rounds ofdeductive
analysis could befurther tested).
In such asituation, the analytics function would take
more the shape ofalaboratory orexperimental unit.
This means itoperates more onastand-alone basis,
has more freedom toexperiment and isless tied
12

Deductive vsInductive

Primary focus
Management focus
Position inthe organisation
Team values
Team composition
DEDUCTIVE
Solving business problems
Integration with the business, bridging parts
of the organisation. Introducing the team
to the organisation (so they understand
the processes).
Close to/part ofbusiness processes
Validity, robustness, value driven
Focused, mostly ondata engineers
and scientists
INDUCTIVE
Learning and innovation
Shielding the data team from the business,
to make sure they can focus ontheir work.
Making sure data isaccessible and the team
works with the data.
Independent/removed from the business
processes
Creativity, making mistakes, trial and error
Broad, including social scientists & people
with creative profiles

tospecific business processes. The following table
compares differences between the different approaches.
PES wanting tostart with analytics are typically
better ofstarting with inductive approaches. Ittypi-
cally allows for smaller scale experiments that allow
both the data team, and the PES, togrow accus-
tomed toworking with data and slowly turn into
adata driven organisation. Creating asmall team
that operates relatively independently from the
organisation inorder toprove value inthe long term
isagood starting point. Once the team has experi-
ence and shown value, the team could bebrought
into the organisation more and start shifting tomore
deductive approaches.
2.2 Creating your data team
Crucial tothe success ofworking with data, isthe
composition ofthe data team. Relevant questions
inthis respect are; what isthe approach weare taking
(inductive, deductive, oracombination)? How many
resources can wemake available? What isthe time
pressure todeliver results? Inthis section wediscuss
team leadership (and variations therein) and several
tiers ofpotential positions within the team.
2.2.1 Leadership
In smaller scale settings, the team will besmaller
and team leadership will generally have aless senior
position. Inthis case asenior data-scientist orman-
ager ofdata analytics would beafitting role tolead
the team. Asfor the position inthe organisation,
arole under the following functional leadership role
ispossible:
CDO (Chief Data Officer) [or equivalent]
In practice, only large (data) mature organisa-
tion will have aleading data position. The CDO
isresponsible for governance and utilisation
ofdata across the entire organisation. This
means that deCDO oversees all data initiatives
and coordinates all analytics activities within
the organisation.
CIO (Chief Innovation Officer) [or equivalent]
The first interpretation ofCIO isthat of
Innovation officer. This role isconcerned with
innovation and change management within
the organisation. Ifthe data team ispositioned
under the Innovation Officer, the focus ofthe
data team will most likely beon more inductive
approaches, trying tocreate innovative data-
driven solutions.
CIO (Chief Information Officer) [or equivalent]
The second interpretation isthat ofInformation
officer. This role isreserved for the highest
ranking officer who isresponsible for informa-
tion technology and computer systems inside
the organisation. Ifpositioned under aCIO,
the data team will probably befocused more
onsupporting the technology role inthe
organisation and hence have amore deductive
orientation.
CTO (Chief Technology Officer) [or equivalent]
The CTO isin charge of(the broader) technology
used bythe organisation, but could also focus
oncore technologies iftechnology isimportant
inclient facing processes (with the increasing
levels ofautomation used inPES, that seems
toapply here). Ifpositioned under aCTO, the
data team will probably focus ondeductive,
client oriented and technology related issues.

Director ofR&D [or equivalent]
The last role isthat ofthe director ofR&D.
While there issome overlap with the Chief
Innovation Officers role, this CIO istypically
occupied with more short term change manage-
ment and the implementation ofinnovations.
The director ofR&D istypically charges with
longer term research and development.
Ifpositioned here, the data team will bemore
experimental and focused onthe development
oflonger term innovations.
Strategic insight
The purpose ofthis overview ofroles isnot
toprescribe where the data team should
bepositioned. Rather, itis meant toraise the
awareness that the position inthe organisation
will impact the expectations one should have
ofthe team and what the focus ofthe team
will have. This could impact hiring ofteam
members aswell.
2.2.2 Data team members
Depending onthe (desired) size, workload and focus
ofthe team, different types ofteam members are
needed tostart asuccessful data analytics practice.
Wedivide these types inthree tiers:
Key Roles
These are the must have members asyou start
building your data team.
Secondary roles
These are the good tohave roles and will
become more relevant asthe team grows
insize.
Tertiary roles
These are the nice tohave roles that add value
tothe team, but are less critical than the others.
They will most likely become relevant once the
team reaches high levels ofmaturity and has
alarge size.
We can define three key roles inworking with the
actual data:
Data engineers
The data engineer typically sets upand works
with the data infrastructure. That means that
they set updatabases and work onExtraction,
Transformation and Loading (ETL, see below)
tasks, they support data analysts and data
scientists intheir roles, and lastly they make
PRACTITIONERS TOOLKIT
13
sure the system works smoothly and performs
well. Very often they have abackground
insoftware engineering.
Data analysts
Data Analysts are the professionals intheir
organization who query and process data.
Furthermore, they typically create data reports
and summaries, visualizations. This role ismore
closely associated with Data Mining and
Statistical Analysis.
Data scientists
The data scientist isthe most important role
inthe context ofthis toolkit. Key role ofthe
data scientist isto generate valuable and
actionable insights from the data and help solve
problems inthe organization using data. Data
scientists apply (mostly) advanced analytics,
such asmachine learning, todata.
Secondary roles:
Social scientists
Team members with asocial science back-
ground (e.g. Sociology, psychology, communica-
tion, marketing, etc.) can perform two important
roles onthe team. The first isto help create
theories and hypotheses can beanswered/
tested using deductive approaches. The second
isto help make sense ofthe outcome ofanaly-
sis when more inductive approaches are taken.
Inthis way, social scientist hold akey role
intranslating organisational goals and/or
problems into the actual data work and
subsequently translate the outcome ofthe
data work into implications and actions for
the organisation.
Software engineers
Even though there isawealth oftools available,
toorganise, analyse and present data, very
often anorganisation will discover that the
tools donot fit their needs entirely. Software
engineers built custom service solutions (in
conjunction with the rest ofthe team) that
help maximize results across the organization.
Difference with data engineers isthat the
software engineer inthis context typically works
onmore front end (or customer) facing solu-
tions (for example dashboards ormobile
applications toaccess data and outcomes).
Some tertiary roles (that wewont discuss into detail):
Graphical and/or interface designers
To help design useful and usable applications
and visualisations.
14
Philosophers
To help interpret data and help create theories
and hypotheses.
Mathematicians & Statisticians
To aid inthe further development ofcomplicated
mathematical models and/or complicated
statistical analyses.
2.3 Setting upthe data
infrastructure
Once ateam isin place, itis necessary tocreate the
data infrastructure onwhich the data team can work.
The most important challenge here isto create (a)
database(s) from which the data team can pull the
data needed toperform their analyses. Incomputing,
Extract, Transform, Load (ETL) refers tothe process
indatabase usage and especially indata warehous-
ing ofperforming:
Extraction. The process ofextracting data from
different data sources
Transformation. Transformation the data for
storing itin the proper format orstructure for
the purposes ofquerying and analysis
Loading loads itinto the final target (data-
base, more specifically, operational data store,
data mart, ordata warehouse, or(in this case)
ananalytics database from which analytics
are performed).
Figure 5: Potential database architecture
Process Process Process
Servers Databases Warehouse
Many configurations ofETL are possible and exist
inpractice. Most ofthese approaches have pros
and cons:
Working from production systems/data bases
In this setup, data engineers would collect data
directly from production systems (e.g. live
profiling ormatching systems). While this
allows the team towork with the most current
data. The downside isthat itcreates security
risks and the risk ofmessing with live data.
Furthermore, ifdata from multiple processes
are needed, the team has toextract and
transform data multiple times.
Working from acentral data warehouse
If the organisation has (one ormore) central
(virtual3) warehouse(s), itis not uncommon for
data teams towork directly with the data from
the warehouse. Asthe data inawarehouse
typically isnot used inproduction settings,
production processes should not beimpacted.
However, this setup does create problems that
are (especially) relevant inthe government
context. The first isthat inmany cases (such
aslegal archive reasons) adata warehouse
serves asan important data store orarchive.
3 It isfairly common tohave avirtual database that
uses wrappers togather data from multiple different
data sources.
Analytics Distributed
Databases Analytics Computing
Server Server
Presentation
Server
Open data
Database
Web
Server
 PRACTITIONERS TOOLKIT

15

Getting started

PACKAGE/TOOL EXPLANATION LINK

Hadoop Apache Hadoop isasoftware library http://hadoop.apache.org

that enables for the distributed
processing oflarge data sets across
clusters ofcomputers using simple
programming models.
Spark Apache Spark isan engine for (distributed) http://spark.apache.org/
large scale data processing.
Scriptella ETL Scriptella ETL Scriptella isan open http://scriptella.org/
source ETL tool.

Working with data from the warehouse directly
could pose arisk regarding the integrity ofthis
archive. The second isthat the warehouse may
contain sensitive data orPII [see below] that
should not beused for analytics purposes.
Creating adedicated analytics database
The third and last scenario isthat of(a)
dedicated database(s) solely for the purpose
ofanalytics.
In this case, relevant data are being pulled from
the warehouse (and other sources) and after being
sanitised loaded into ananalytics database. This
database would then bethe primary source for the
data team towork from. This database could be
supported byseveral types ofservers toaid in the
analytics:
Analytics Servers
Servers whose sole purpose itis torun
(computational) analytics. These servers
typically have lots ofprocessing power torun
heavy and complicated analytics.
Distributed Computing Servers
When datasets become too large, itmay no
longer befeasible torun onthem onasingle
server. Inthis situation itis common toset
upadistributed computing environment. What
this entails is(conceptually) straightforward:
adataset isbeing broken down insmaller
pieces, send todifferent servers (computers),
analysed onthese servers, and results ofthe
analysis are bundled back into one single
outcome. Many commercial distributed
environments are available (such asAmazon
Web Services (AWS), Google Compute Engine,
and many others), but depending onthe
sensitivity ofthe data, aPES could consider
setting upadistributed environment.
Presentation and visualisation servers
These servers typically run applications
toshow results (such asdashboards) and/or
the data catalogue.
Concept: PII
PII stands for Personal Identifiable Information.
This isinformation that helps identify individual
people. Classic examples ofPII are names,
addresses and unique person identifiers such
associal security orcitizen numbers. One
problem with PII and analytics isthat, asmore
and more different types ofdata are being
combined, itbecomes increasingly possible
to indirectly pinpoint individuals. Itis impor-
tant, especially asdata are being opened (see
section 5.4) toensure that individuals cannot
beidentified.
2.4 Creating adata-catalogue
In order for any organisation tostart answering ques-
tion, itis important toknow what information the
organisation already has. This prevents the organisa-
tion from collecting the same information again and
ityields anoverview ofdata stored within the organi-
sation that can beused for other purposes.
A data catalogue isan instrument that provides
anoverview of(all) data present inthe organisation,
it(semantically) explains the data and variables,
and adds meta-data tothese data sets. This means
that the catalogue contains descriptions ofthe vari-
ables and nature ofthe data.
16
Concept: Data catalogue
A data catalogue isan overview ofthe existing
data indatabases and provides descriptions using
meta data ofthe nature and state ofthe data,
such asthe base tables, volume, definitions, prop-
erties, synonyms, annotations and tags. Assuch,
the data catalogue isan important part ofthe
semantics part ofthe Smart Data equation.
Smart data = (Big) Data + Utility
+ Semantics + Data Quality + Security
+ Data Protection
Semantics refers tothe meaning ofdata. Know-
ing exactly what data you have and what the
data can tell you about reality isan important
aspect ofsmart data. Semantics here does not
just refer todescriptions ofdata and variables,
but also totheir meaning inreal life. For exam-
ple, does ameasure orproxy ofcustomer
satisfaction really signify satisfaction ofclients
inreal life?
A data catalogue can becompared toan index
inalibrary. Alibrary isacollection ofbooks, written
bydifferent people ondifferent subjects atdifferent
points intime ofdifferent lengths for different
audiences. Anindex inthe library contains the
overview ofexactly what you can find inthe library.
Furthermore, the index allows you tofind the
resources you need.
The process ofcreating adata catalogue ispretty
straightforward. Members ofthe data team will
have todocument all available data inthe organisa-
tion and use meta-data todescribe this data. For
example, they have to:
Create anoverview ofall data sources,
databases and datasets available and the
nature ofthese data sets (e.g. what kind
ofdatabases dowe have?).
Document and describe all variables within
these datasets.
Document the number ofrecords and changes
inthese records (e.g. How often are databases
updated).
Concept: Meta-data
Meta data isdata that describes other data. For
example, meta-data about avacancy database
could include descriptions ofall variables stored
inthe database, the nature ofthese variables
(e.g. Are they text, numbers (and what kind
ofnumber format), etc.) and such things asthe
number ofrecords. Inessence, the data-catalogue
isan overview ofall the meta-data.
While the primary function ofadata catalogue
isto document the data stored inthe organisations
databases, more types ofdata could beincluded
inadata catalogue, such as:
Research data (such assurvey data sets).
Externally available (analytics) data (such
aswhat iscollected through Google analytics
orother trackers).
Other relevant data (such associal media use,
relevant (technical) documents.
While astatic data-catalogue ispossible (literally
acatalogue describing data), most organisations
choose tocreate adatabase based data-catalogue
with asearchable front-end (often using web tech-
nologies). While technically anopen data catalogue,
the catalogue athttp://catalog.data.gov/dataset
provides agood anexample ofwhat atypically
data catalogue looks like.
2.5 Costs and budgeting
We finish this chapter byfocusing onthe costs
associated with starting ananalytics practice.
Budgeting for analytics activities proves tobe
achallenging task. Astudy byGartner4 found that
more than half ofall analytics projects failed
because they were not completed within budget
oron schedule. Key reason for this isthat itis very
difficult toestimate the costs ofearly analytics
projects, for example because:
Early on, itis often unclear which information
the organisation has and how useful this
information is.
Organising and cleaning data take upconsider-
able amounts oftime (and resources) and
organisations tend tounderestimate the amount
oftime ittakes toget all data organised.
4 http://www.gartner.com/newsroom/id/2637615

Especially with inductive projects, the expected
outcomes are difficult toforesee, this creates
many uncertainties regarding timeline and costs.
Nevertheless, itis possible tocreate abudget and
allocate costs for execution ofanalytics. The cost
ofany analytics endeavour typically breaks down
inthe following:
The set-up cost for the relevant infrastructure
(databases, analytics servers, data extraction,
transformation & loading, other hardware and
software). The good news regarding these
costs isthat, while still significant, the cost
ofacquiring, storing and managing data keeps
ongoing down. Especially using scalable
solutions (see 2.3) itis possible toset up
arelatively affordable infrastructure.
The recurring/ongoing costs for the infrastruc-
ture (power, licensing fees, maintenance, etc.).
Total sum ofthese costs depends onmany
factors such asthe use offreely available
software, orcommercial software, the size
ofthe server park, etc.
Personnel costs. This entails the costs ofthe
data-team such ashiring and salary costs.
While analytics isdriven bytechnology, human
labour still makes upalarge portion ofthe
total cost ofany analytics initiatives (in most
cases the bulk). Reason for this isthat organ-
izing, sanitizing, cleaning and subsequently
analysing and interpreting ofdata isavery
labour intensive process (also see chapter 3).
The more ambitious the project, the higher
the labour costs will be.
However, the degree towhich all costs occur depend
onanumber offactors, most notably the following
choices will impact the cost:
In-house oroutsourcing?
The list ofcosts above assumes the organisa-
tion will want toown their own infrastructure
and have all personnel onstaff. There are,
however anumber ofalternative scenarios:
Use ofexternal orcloud infrastructure?
Many organisations host their analytics servers
elsewhere, often inthe cloud. Benefits ofusing
external providers for storage and/or analytics
solutions isthat a) you only pay for the
capacity needed, b) itis easy toscale the
capacity needed upor down and c) you dont
have toworry (as much) about administration,
maintenance, etc. However, there are anumber
ofdrawbacks, the first isalower level ofcon-
trol over the infrastructure (which could for
PRACTITIONERS TOOLKIT
17
example conflict with the needs ofthe organi-
sation when certain requirements cannot
bemet [for example when acertain tool does
not work onthe platform provided]). The
second isthat certain legal requirements could
prohibit the organisation from storing data
outside ofthe organisation, the government and/
or the country. Itis wise toconsider the specific
legal requirements (see also section 3.5) before
making this decision.
Internal data team orexternal service provider?
Organisations that want toengage inanalytics
activities face animportant (cost related)
decision when itcomes tothe personnel aspect;
hire adata-team oroutsource the personnel
aspect. Several (consulting) providers provide
analytics services that could take care ofthe
personnel requirement (and often the infrastruc-
ture aswell). Furthermore, the organisation could
consider using freelance personnel orvirtual (job)
marketplaces, but given the sensitivity ofthe
data often involved these options are often not
realistic. The benefits ofusing anexternal
provider are that a) itcould becost effective
ifanalytics are only needed onaproject basis
(and not continuously), b) you dont have toworry
about training orcapacity. Downsides are that
a) the organisation builds little experience and
knowledge about analytics internally, b) costs
can be(much) higher inthe long run. Soif the
organisation plans toseriously build analytics
capabilities, creating the capacity internally
seems the right course ofaction.
Hybrid ornot?
Lastly, the organisation could choose amix
ofdoing part ofthe infrastructure internally
(such asdata storage) and part externally
(such asrunning certain analysis onan external
server provided byathird party) and hire part
ofthe data team and use anexternal provider
for specific expertise areas. Inpractice, most
advanced organisations use some kind ofhybrid
version. Reasons for this are that a) analytics
needs can become very specific (or advanced)
and using and external provider ismore
cost-effective than building the capability
internally for just one specific type ofanalysis
b) depending onthe nature and scope ofthe
analytics activities, the organisation may only
need acertain amount ofcapacity torun the
normal business but during peak times orfor
specific activities, extra capacity could be
needed inwhich case partnering with athird
party isreasonable.
 18

Furthermore, the following are relevant considera-
tions when itcomes tocosts:
As mentioned above, many projects overrun
their budgets. This means that there isaten-
dency tounderestimate costs. For this reasons
itis advisable toeither lower project require-
ments ornot beoverly optimistic when starting
with analytics.
Costs ofanalytics donot scale linearly. Not only
isthere typically alarge start-up cost involved,
but asanalytics activities become more
advanced, sodoes the complexity ofthe work.
Especially when amultitude ofdata sources are
being used and models become more compli-
cated, costs can rise atahigher than linear rate.
The best way tokeep costs manageable isto
start small. Smaller datasets can beanalysed
onrelatively cheap hardware and smaller teams
are needed for smaller projects. Inthat sense, if
budgets are non-existent, orsmall, itis advisable
tostart asmaller analytics practice and grow this
over time asthe team proves its value.

PRACTICAL EXAMPLE

In May 2016, the Executive Office ofthe United States President released areport onthe opportunities ofBig Data. The report contains
acase study onthe potential ofBig Data for employment. Askey problem inemployment related todata, the report recognises that
traditional hiring practices may unnecessarily filter out applicants whose skills match the job opening. Tosolve this problem, big data
isseen asan opportunity: Big data can beused touncover orpossibly reduce employment discrimination. For example, big data
analytics can beused:
To prevent affinity bias orlike mebias inthe hiring process (for example where hiring managers tend toselect candidates like them
orwhom they like).
To find potential job-candidates who otherwise might have been overlooked based onthe more traditional educational orworkplace-
experience related job-requirements. For example, bylooking atthe skills and knowledge areas that have made other employees
successful, amatching system could use pattern matching torecognize the characteristics that made current employees successful
and thus need tobe looked for infuture employees.
Large data analytics systems could help prevent biases often seen intraditional hiring practices that could lead todiscrimination.
Analgorithm could bedesigned tonot look atfactors like age, gender, race orany factor whereas itis much more difficult toblock
such (implicit) factors asahuman.
Beyond supporting orrecommending matching/hiring decisions, advanced algorithms create the possibility ofsolving long-term
employment challenges related todiscrimination, such asthe wage gap oroccupational segregation, for example bygoing beyond
formal job qualifications, but finding the person for the job based oncultural orother factors.
Using data-analytics new kinds ofcandidate scores ormatching scores can becreated byusing diverse and new sources
ofinformation onjob candidates. The report mentions how one employment research firm found that distance employees commute
towork tobe one ofthe strongest predictors ofhow long customer service employees will stay with their jobs. Such variables
and data could beused toimprove matching algorithms for specific (customer service) job vacancies.
Finally, machine learning based algorithms could help decide what kinds ofemployees are likely tobe successful byreviewing
the past performance ofexisting employees ofcertain companies orjob seekers who worked for certain firms orby analysing
the preferences ofhiring managers asshown bytheir past decisions. This could also apply tosuch things asemployee turnover
and the likelihood that certain people will retain jobs incertain industries.
 PRACTITIONERS TOOLKIT

19

Chapter 3. In this chapter wefocus onthe organisation ofthe

data. Wediscuss how data can becleaned and

Organising data
sanitised and how existing data from within and
outside ofthe organisation can beextracted, trans-
formed and loaded into adatabase that can beused
for analytics.

The following topics form the main content ofthis

chapter:
Provide overview ofwhat organising data entails.
Focus onareas such assecurity, data protection
and privacy.

Strategic questions this chapter answers:

How can weguarantee security?

Tactical questions this chapter answers:

How toorganise and protect data?
What are reasonable expectations regarding
data organising.

3.1 Cleaning & Sanitising

Organising data isacomplicated process that
requires alot oftime and devotion. Data scientists
can spend upto 90% oftheir time organising data.
This has several reasons:

1. The data are inan undesired format. For example,

databases can store times and dates using many
different formats (e.g. think about the European
DD/MM/YY format versus the USMM/DD/YY format).
Importing different data sources inone database
can create inconsistencies incertain variables that
need tobe fixed. Furthermore, data might have
tobe reorganised sothat analytics can berun.

2. The data need tobe cleaned. Many databases

contain noise that could berelevant for one pro-
cess, but irrelevant for analytical processes. Clean-
ing the analytics database helps reduce noise (and
could aid indatabase performance). Inadata-
catalogue, certain variables that are less relevant
for analytics could bemarked and deleted when
creating adedicated analytics database.

3. The data need tobe sanitised. This refers tothe

process ofridding the database from unwanted
information. For example, personal identifiable
information (PII) (see section 2.3) needs tobe
removed from the datasets. When variables are
documented properly, sanitisation can bedone using
scripts. When that isnot the case, the data-team
will have tocheck the data manually and sanitise
when needed.
20

Getting started

The following might beuseful toget started onorganising and cleaning data:

PACKAGE/TOOL EXPLANATION LINK

Encog Machine Learning Framework Encog isan advanced machine learning http://www.heatonresearch.com/encog/
framework that supports avariety of
advanced algorithms, aswell assupport
classes tonormalize and process data.

4. The data need tobe transformed. Even though
transformation ispart ofthe ETL process (see
section 2.3). Additional transformation can be
needed while preparing for data analytics. For
example, decimal places could have tobe fixed
or floating numbers could have tobe converted
to integers. Another example isto transform
unstructured data into structured data.
Concept: Structured vs. Unstructured data
Structured data isdata with ahigh level
oforganisation (and formatting). For example,
atable with records, variables and labelled data
(such asatable with jobseekers demographic
information) isstructured information. Inits most
simple form, unstructured data isdata lacking
such organisation. Adatabase with PDFs orpho-
tocopies ofjobseekers resumes isan example
ofunstructured data.
5. Missing data might have tobe imputed. Imputation
isthe process ofsubstituting missing data with
calculated values. Indatabases where data
ismissing, scientists may choose torun algorithms
toimpute these missing data points. One way
todo this isby looking atpatterns inthe data. For
example, ifpeople with similar characteristics
consistently have the same value regarding acer-
tain variable, the likelihood increases that the
missing values are similar.
It isextremely important that data are cleaned and
sanitised properly. For this reason, PES starting to
work with data could create peer review processes
to make sure data are being reviewed after being
organised byother team members.
Smart data = (Big) Data + Utility
+ Semantics + Data Quality + Security
+ Data Protection
Data Quality refers tovarious aspects ofthe data
itself. These are a) completeness (do Ihave
enough data about everybody tomake claims),
b) cleanness (are the data well cleaned, sanitised
and maintained), c) have high levels ofvalidity
and d) impact (have they been analysed insuch
away that they retain their validity and create
relevant meaning for the organisation).
3.2 Describing data & data
characteristics
Descriptive analytics help usunderstand the data.
When describing data, welook athow the data
isdistributed (e.g. normal, power-law, linear), what
the key characteristics ofthe data are (e.g. the mode,
median and mean) and wecheck for such things
asthe outliers inthe data. Descriptives are important
because they:
Help usunderstand the nature ofthe data
(e.g. what isthe nature ofthe variables).
Help usdraw initial conclusions about the data
(e.g. Based onthe distribution ofdata wecan
observe that certain jobseekers have certain
characteristics).
Help usprepare for further analysis (e.g. by
removing outliers that distort the data).
Furthermore, descriptive statistics are another way
tocheck the quality ofthe data and the organisation
ofthe data. For example, itcould show inconsisten-
cies inthe data collected and areas where data have
been improperly transformed.

Getting started
The following methods, tools and/or applications can
beused tocalculate descriptive analytics:
Most spreadsheet software (e.g. Microsoft Excel,
LibreOffice Calc) include basic functions for
most descriptive analytics
Most statistical software (e.g. SPSS, SAS) have
dedicated functions for descriptive analysis
Modelling languages (e.g. R& Python) have
packages for descriptive analysis.
3.3 Quality control
High quality data isan important element ofsmart
data. Ensuring quality and having the proper checks
and balances inplace can help aPES being confident
inthe quality ofthe available data. The most impor-
tant way tocontrol quality isto have control mecha-
nisms during every single step ofthe data process.
This means that during collection, organisation,
analysis, presentation, and evaluation activities have
tobe deployed tocheck for the quality ofthe data.
These are ways tocontrol the quality ofthe data
during various stages ofthe process. Examples
include:
When preparing for data collection:
Checking the quality oftheories orhypotheses
using expert evaluations orliterature reviews.
When collecting data:
Taking multiple measurements, observations
orsamples
Using standardised methods, instruments and
protocols when collecting data (which could
beincluded inthe data catalogue).
When organising data:
Setting upvalidation rules when entering data
Using strict protocols when cleaning data
Documenting database structures properly
(and reviewing this)
Creation ofautomated scripts for organisation
tasks (and checking their performance using
reviews)
Having peer reviews oforganisation methods
and practices.
When analysing data and presenting results:
Model testing using multiple (independent)
samples
PRACTITIONERS TOOLKIT
21
Code and model reviews either byteam
members orexternal experts
Combining model fit measures with theoretical
evaluations
Automated testing ofcode before models are
being executed orany code isbeing published
orpushed toaproduction environment.
3.4 Integrating data sources
Focus ofthis section isnot onthe integration ofdata
inbusiness processes. Rather, the focus ison the
combination ofdifferent types ofdata from within
and between organisations.
Data integration isthe process ofcombining data
housed indifferent sources and giving users auni-
fied view ofthe data. While inmany organisations,
adata warehouse will already have integrated the
most important data sources, there are many sce-
narios inwhich data will have tobe integrated
orcombined tocreate aunified view necessary for
analytics, such as:
Missing data
Imagine job seekers registering online, but not
completing all information during the registra-
tion process (e.g. their educational background).
These data are missing inthe database and this
could beaproblem ifwe want todo education
related analysis. If, during the registration
process, the job seeker has also uploaded
aresume, wemay beable toextract the data
from the resume and integrate the two data
sources inorder tocreate acomplete record.
Triangulation
Triangulation refers tothe combination ofdiffer-
ent data sources tocheck for the quality ofone
(or both). For example, data scientists could use
predictive modelling topredict satisfaction
ofjobseekers with amatching tool and use
survey data tocheck the results ofthe model.
Increased quantity
The most important reason, however, tointe-
grate data sources isto have more data
available. For example, data oncoaching or
training outcomes could beused toenhance
the profile ofajob seeker.
While the actual implementation ofdata integration
isatechnical question, the main challenge ofdata
integration isorganisational innature. Itinvolves
such activities as:
 22
Convincing different stakeholders ofopening
uptheir data silos.
Getting cooperation from ITdepartments
toactually get the data.
Coordinate privacy and security risks with
relevant stakeholders.
Working with relevant stakeholders tounder-
stand the nature ofthe data (i.e. Add tothe
data-catalogue).
Making agreements around updates ofthe data
orSLAs (i.e. how often and towhat standards
data are being shared).
Concept: Service Level Agreement (SLA)
An agreement specifying the quality ofservices
delivered from one (part ofan) organisation
toanother. For example about the uptime
ofservers orthe refresh rate ofdata.
The stronger the mandate ofthe leader ofthe data
team and the higher the position inthe organisation,
the more (formal) organisational power the team
has ingetting the data itneeds. This isan important
consideration when starting with data analytics.
Amore experimental team focused oninductive
approaches, may have more difficulty inintegrating
data sources from across the organisation versus
ateam with aposition more closely tied tobusiness
processes.
PRACTICAL EXAMPLE
The X-Road isEstonias infrastructure that connects databases
from amultitude ofgovernmental agencies. Itis best described
asadistributed service bus which allows databases tointeract, making
integrated e-services possible. This, however, also creates opportunities
for data integration that can beused for data analytics.
Currently, 219 databases are connected toX-Road and these result
inover 1700 services being offered. Byintegrating data sources,
citizens only have tosupply many pieces ofinformation once and
italready allows fraud prevention through analytics.
Please, find further information inthe following link.
This applies even more strongly todata integration
across organisations. Ifthe success ofthe data team
depends ondata from other organisations, itwill
need astronger position inthe organisation, prefer-
ably with support from the highest levels ofleader-
ship inthe organisation. Logical data sharing partners
for PES include:
Other governments:
Ministries ofeducation (or similar)
For example regarding data about the future
workforce, which could behelpful inpredictive
models for future matching applications
orunemployment forecasting.
Tax agencies (or similar)
Regarding financial data about job seekers
(e.g. For benefit fraud detection).
Social security institutions (or similar)
Regarding social security orbenefit information.
This isespecially relevant when there isalegal
obligation for data collection and/or sharing.
Statistics bureaus
For various types ofinformation such aspopu-
lation mobility (which could beused tofine-tune
job recommendations) orhousehold develop-
ments (which could impact the labour force).
Regional orlocal governments
For data regarding specific local orregional
circumstances (e.g. Local employment
initiatives).
Businesses:
For example regarding job developments
(are business going toadd orremove positions?),
their future needs.
Other organisations:
Such asfoundations working inthe labour
market (for example, overviews ofactivities
could help ininterpreting specific labour
market fluctuations).
3.5 Security and Data Protection
Security and Data Protection are two other key ele-
ments from the Smart Data equation. Protecting (user)
data and having good security should beamong the
highest priorities ofboth the data team, aswell asthe
leadership ofthe team and the parts ofthe organisa-
tion involved inthat data used bythe team. Several
types ofsecurity are important:

Physical security
Where are the data stored? How easily can these
machines beaccessed? Storing data insafe locations
where only authorised personnel has access isone
ofthe key steps toensure. The following can help
improving physical security:
Have strict access controls torooms containing
data servers
Have protocols for the use of(sensitive) data
ondesktops, laptops and other devices
Have guidelines for the use ofmobile devices
(e.g. Laptops, tables, phones) outside ofsecure
areas
Have protocols for the use ofremovable media
(e.g. USB drives, removable hard drives, etc.).
Virtual access security
Once being close toamachine (or accessing
itremotely), how easy isit togain access? Are (safe)
passwords inplace? Isdata encrypted? The following
can help inimproving virtual access security:
Develop guidelines for the encryption ofdata
(especially onthose devices with easier physical
access)
Develop policies for the use offirewalls and
ant-virus software
Have strict protocols regarding passwords,
password sharing and password changes
Limit the use ofAPIs (and other ways toaccess
data) tonon-sensitive data and/or open data.
Smart data = (Big) Data + Utility
+ Semantics + Data Quality + Security
+ Data Protection
Security refers tothe ways the data are being
securely stored and managed. This applies not
only tothe physical security (who can access
servers and related systems?) but also virtual
security (who has access todata secured
onsystems?). Security isimportant tomake
sure systems are being hacked and/or data
does not leak oris being stolen.
In addition, the following can help with security
ingeneral:
Makes sure software isalways upto date
Having aregular security meeting todiscuss
and refresh members ofthe data-teams
memories onsecurity related matters
Make security training astandard part of(new)
data-team members, sothat aculture ofsecu-
rity awareness isinstilled from the start.
PRACTITIONERS TOOLKIT
23
The second topic weare discussing inthis toolkit
isthat ofdata protection.
Smart data = (Big) Data + Utility
+ Semantics + Data Quality + Security
+ Data Protection
Data protection refers tothe ways privacy and
confidentiality are safeguarded. Are PII replaced
byother identifiers? Can data betraced (in
combination ornot) toindividuals? Are users
aware ofthe use oftheir data? Protecting data
properly will protect (vulnerable) individuals
and minimise organisational risks.
For purposes ofthis toolkit, webreak down data
protection intwo topics; privacy and confidentiality.
While privacy applies tothe person that needs tobe
protected, confidentiality applies tothe persons data.
When data can beused toidentify aperson, privacy
issues may arise. When data about aperson can
beused maliciously (for example byjudging aperson
using data that was supposed tobe collected anony-
mously), confidentiality issues can arise.
Guarding the privacy ofindividuals and the confi-
dentiality oftheir data isimportant toensure no
harm isdone toany individual ororganisation. Main
consideration regarding privacy consists ofthe
applicable laws and regulations. Onan EUlevel, the
following are important:
Directive 95/46/EC | Protection ofpersonal data.
Directive 95/46/EC sets uparegulatory frame-
work which seeks tostrike abalance between
ahigh level ofprotection for the privacy
ofindividuals and the free movement ofper-
sonal data within the European Union (EU).
Todo so, the Directive sets strict limits onthe
collection and use ofpersonal data and
demands that each Member State set upan
independent national body responsible for the
supervision ofany activity linked tothe process-
ing ofpersonal data [quoted from URL below]
URL: http://eur-lex.europa.eu/legal-content/EN/
TXT/?uri=URISERV%3Al14012
Regulation (EU) 2016/679 | General Data
Protection Regulation
This Regulation isset toreplace directive 95/46/EC.
Itwas adopted on27 April 2016 and enters into
application on25 May 2018.
More info: http://ec.europa.eu/justice/data-
protection/index_en.htm
24
Besides applicable European regulations, every
single member states will have their own applicable
laws and regulations. These should beconsulted
before starting analytics projects. When data across
multiple countries are being collected, laws from
multiple countries may apply. Special care should
also begiven tothe (cloud) storage ofdata incoun-
tries other than the home country and/or outside
ofthe EU.
Next toabiding bythe law, the following good prac-
tices can help toestablish good data practices and
ensure privacy protection:
Implement solid de-identification protocols
and capabilities.
This consists ofthe removal (or replacement)
ofPII (see section 2.3), aswell ashaving checks
and balances inplace that ensure that noPII
enters the process atany time and/or individu-
als can beidentified using analytics (e.g.
Bycombining amultitude ofvariables, itcould
bepossible tonarrow data sets down toindi-
viduals). The U.S. Department ofHealth and
Human Services (HHS) describes two (commonly
accepted) methods tode-identify information5:
Expert Determination method:
In this scenario qualified experts apply statisti-
cal orscientific principles torender information
tobe not individually identifiable.
Safe Harbor method:
The removal of18 types ofdata related
toindividuals from the dataset completely
(see Appendix 1) for anoverview.
Always use privacy impact assessments (PIA). procedure).
PIAs are tools used toidentify and mitigate
privacy risks. Section 3, article 33 ofthe new
EUGeneral Data Protection Regulation [Data
protection impact assessment and prior
authorisation] already stipulates controllers and
processors tocarry out adata protection impact
assessment prior torisky processing operations.
However, agood practice could beto assess the
privacy impact for any project related toindi-
vidual people and/or cases. Atthe very least,
such aPIA should:
5 See http://www.hhs.gov/hipaa/for-professionals/
privacy/special-topics/de-identification/index.html
Assess whether the information used complies
with all (privacy-related) legal and regulatory
requirements.
Make aninventory ofpotential risks ofworking
with PII.
Assess processes for handling information
toreduce ormitigate potential privacy risks.
Investigate the consent methods (see below)
used toask individuals permission for the use
oftheir data.
Record the outcomes ofthe assessment and
make them available.
Implement solutions for any risks orproblems
discovered.
Implement privacy bydesign principles (also
required according toarticle 23 ofthe new
EU General Data Protection Regulation).
This means that any project, process orservice
needs tobe designed from the start toadhere
tothe strictest possible privacy considerations.
For example, when creating ananalytics
database, PII should never beincluded insuch
adatabase inthe first place. This prevents the
data team towork with PII inthe first place.
To safeguard confidentiality, two actions are important:
Use consent procedures when collecting
information. For example ask people for consent
touse their data when they register asunem-
ployed, orwhen they fill out surveys.
Actively inform individuals ofthe purposes
for which their data isbeing used (very often
this happens inconjunction with the consent
 PRACTITIONERS TOOLKIT

25

Chapter 4. In the fourth chapter wediscuss the analyses ofthe

data. Wediscuss more traditional types ofanalytics

Analysing data
(such asstatistical methods and data mining), but
also discuss novel and innovative types ofanalytics
such asmachine learning and artificial intelligence.
The focus ofthese innovative types isnot necessarily
onhow PES are using these types, but onpotential
use cases for the future.

The following topics form the main content ofthis

chapter:
Provide overview ofanalytical techniques and
tools
Provide guidance onwhat goals can beachieved
using what methods.

These strategic question are being answered inthis

chapter:
How can analytics help achieving specific
strategic organisational goals?

The following tactical questions are central inthis

chapter:
What are types ofanalytics that are available
and how can they help PES?
Which analytics tochoose and what are their
pros/cons?

4.1 Overview
Before westart our overview oftechniques towork
with data, wefirst give anoverview ofsome common
analytical approaches and their differences/overlap6.
Asthe graph makes clear, there isan abundance
ofmethods, tools and approaches available totrans-
form data into value. The specific use case ofeach
approach depends onthe goal (see Chapter 2) the
PES wants toachieve.

6 Goal isnot tobe complete inthis overview,

but toprovide anoverview ofrelevant approaches
and toshow their relations.
26

Key Data Analytics Concepts

Figure 6: Key data analytics concepts

(Semi)-automatic analysis of (big) data
to extract new information and transform
data into an understandable structure
for further use
[Knowledge Discovery in Databases]
KDD the process of discovering useful
knowlegdge from a collection of data

Data
Mining

Statistics
Collection, organization,
Quantitative analysis, interpretation,
and presentation
Subset of machine Deep of data
learning that attempts Learning
to model high level Qualitative
abstractions in data Quantitative data
Machine collection and analysis
Learning
Constructing Qualitative data
algorithms that collection and analysis
learn from and Artificial
make predictions Intelligence Creation of intelligence
based on data exhibited by machines

4.2 Statistics ofqualitative data (for example, if many interviews

Statistics refers to(traditional types) ofresearch
tocollect data from people orother units ofresearch.
Typically, statistics isseen asmerely traditional
social science research orjust the statistical analysis
ofdata. While both are not entirely correct, wewill
try and explain the concept inmore detail inthis
section. For the purpose ofthis toolkit, our focus
ison the data collection and analysis part and
welimit ourselves tostatistics inthe more traditional
social science sense. Statistical analysis isbeing used
inpretty much every other method ofanalyses,
hence our focus.
We focus ontwo key approaches within statistics:
1. Quantitative methods
These are methods typically used tocollect and ana-
lyse data that has tobe asked from entities and
cannot begathered using other methods. Put simply,
quantitative methods are used totest hypotheses.
Most commonly, quantitative methods are associated
with surveys and questionnaires, however there are
many more quantitative methods of collecting data,
such aslarge scale observations or quantifications
are being held, certain responses could bequantified
and used for quantitative analysis).
Quantitative methods are, even inthe age ofbig data,
still very useful. While most data pulled from data-
bases are objective, factual data, quantitative methods
can beused tomeasure more subjective elements.
The most typical example iscustomer satisfaction
surveys. Inthe context ofinnovation, surveys are
agood way totest assumptions derived from theories
orget structured (quantified) feedback onprototypes
orideas. Other methods, apart from surveys, are:
A/B tests7
Where different groups ofrespondents are
being exposed todifferent versions ofaproduct
(or treatment). Measuring behaviour after
exposure can help infer effects about the
treatment. This could beused byPES toexperi-
ment with different versions ofwebsites
orother online tools.
7 Depending on the design of the study, most of the
methods explained can take a more qualitative or
quantitative nature. We mention them there where
they in our view are used more commonly.
 PRACTITIONERS TOOLKIT

27

The following table gives more detail ofeach approach and potential use cases for PES.

APPROACH DESCRIPTION POTENTIAL USE CASE

Statistics Collection, organisation, analysis, Traditional types ofresearch tocollect
interpretation and presentation ofdata data from people orother units
ofresearch.
Example(s) Surveys, observations, interviews and the subsequent analysis ofthis data.
Quantitative statistics 8
The use ofquantitative methods Quantitative methods ofdata collection
tocollect and analyse data. Surveys are typically used tocollect data that has
are acommon method tocollect tobe asked from entities and cannot
quantitative data. begathered using other methods.
Example(s) Surveys tomeasure customer satisfaction.
Qualitative statistics Use ofqualitative methods togather Qualitative methods ofdata collection
data that focuses onquality, rather are typically used togain adeeper
than quantity. Common methods are descriptive understanding ofatopic.
interviews, focus groups, etc.
Example(s) Interviews tounderstand why people contact aPES.
Data mining Data mining isthe application ofspecific Data mining isused tocondense large
algorithms inorder toextract patterns amounts ofdata and/or transform them.
from data.
Example(s) Mining usage statistics ofservice channels and showing trends/developments.
KDD (Knowledge Discovery KDD isthe process ofdiscovering KDD isused tobuild upon data mining
inDatabases) knowledge and patterns inlarge and transform data into knowledge.
amounts ofdata.
Example(s) Fraud detection ofbenefits.
Artificial intelligence Goal ofAI isto create intelligence Create smarter technologies that can
exhibited bymachines. make decisions orsupport decision
making.
Example(s) Smart job search systems that work based oncustomer profiles.
Machine learning Algorithms that learn from data Machine learning isused tocreate better
processed tomake predictions and functioning algorithms and models
improve outcomes. bylearning from ongoing analysis.
Example(s) Improvement ofmatching systems byanalysing reasons for previous matches
and/or mismatches.
Deep learning Algorithms that learns and models based Deep learning isused toexplore data
onhigh level abstract and layers that ishighly unstructured and abstracted
ormanifestations ofunstructured data and tries tocreate abstractions from this
(such aswritten text, pictures, videos and data.
many combinations ofdata sources).
Example(s) Drawing inferences from writing styles and formatting ofresumes toimprove
vacancy matching ortraining.

Eye tracking
Where respondents use adevise that tracks
their eye movements. This helps understand
how respondents navigate products, which parts
draw attention, etc. This could beused indevel-
opment stages ofnew (online) tools tohelp
understand how people navigate pages.
8 One could argue that most approaches are
quantitative and therefore fall in this bucket.
However, we use quantitative statistics in a social
science context, i.e. Referring to the analysis is
quantitative data collected through social science
research methods, such as surveys.
 28

PRACTICAL EXAMPLE
The USState ofNew Mexico Department ofWorkforce Solutions (DWS)
noticed that many benefits applications made mistakes (purposefully
ornot) while applying for (unemployment) benefits, resulting inimproper
benefits payments.
DWS partnered with Deloitte toconduct atwo stage project. The first was
touse quantitative statistical analysis tomodel suspect behaviours. The
next step was togently nudge individuals into more desirable behaviours.
The key tothis was todesign and test communications and notifications
for claimants atthree moments: 1) during the vetting process for eligibility,
2) when individuals report work and earnings, and 3) while determining
anaction plan toseek new employment.
By field testing different types ofcommunications, DWS was able toanalyse
the best working solution and subsequently implement this. DWS was
able tosubstantially influence claimants behaviour. The State successfully
increased accurate reporting while reducing improper payments.
(see https://www2.deloitte.com/us/en/pages/deloitte-analytics/articles/
business-analytics-case-studies.html)
Plus-minus methods
Where respondents are asked tomark part
of aproduct (e.g. anonline tool, orphysical
product) they like ordislike. Subsequently
respondents are asked toelaborate ontheir
choices. Often these are used totest brochures
and other physical products, but could beused
inonline, product environments aswell.
Like quantitative methods, qualitative research
remains valuable inthe age ofbig and/or smart
data. The key function ofqualitative methods isto
help make sense ofthe world and/or get adeeper
understanding ofphenomenon that simply cannot
begenerated through other means ofanalysis.
Ininnovation settings, qualitative methods are most
commonly used inconjunction with other methods,
throughout the innovation process.
Getting started

The following methods, tools and/or applications can

2. Qualitative methods
These are methods are typically used togain adeeper
descriptive understanding ofatopic. Put simply,
qualitative methods are commonly used tocreate
hypotheses. Most often, people associate qualitative
methods with (group) interviews, but aswell aswith
quantitative methods, there are many more
approaches. Relevant examples inthis context are:
Think aloud methods
Where people are asked toperform atask and
explain their thought process while performing
these tasks. This could beused togain deeper
insights into the choices people make when using
(online) tools orapplications (such asmobile apps).
beused tocollect and analyse (statistical) data:
There are many online (free) survey tools
available, such asSurveyMonkey [surveymon-
key.com]. Non-free products (such asQualtrics
[qualitrics.com]) often have more functionality
and/or better support
Most spreadsheet software (e.g. Microsoft Excel,
LibreOffice Calc) include basic statistical
functions
Dedicated statistical software (e.g. SPSS, SAS,
PSPP) can beused toperform (complex) statisti-
cal analysis on(mostly) quantitative data
Various tools exist for the transcription and
analysis ofqualitative data. Examples are:

PACKAGE/TOOL EXPLANATION LINK

QDA Miner Lite Free computer assisted qualitative https://provalisresearch.com/products/
analysis software. Can beused for the qualitative-data-analysis-software/
analysis oftextual data such asinterview freeware/
and news transcripts, open-ended
responses, etc.
Weft QDA Free tool toanalyse qualitative data http://www.pressure.to/qda/
such asinterview transcripts, fieldnotes
and other documents.
Transana Software toa) various types ofdata https://www.transana.com/
inasingle analysis, b) categorize and
code segments ofthe data, c) explore
coded data through text reports,
graphical reports, and searches.
 PRACTITIONERS TOOLKIT

29

Open Source and Free dedicated tools and

packages can befound inthe able below:

PACKAGE/TOOL EXPLANATION LINK

R R isafree software environment for www.r-project.org
analytics and visualisations. Itis one
of the most popular programming
language for analytics. Itis highly
modular and alarge support/
documentation community online.
Python Python isaprogramming language https://www.python.org
that isbeing used heavily for analytics
(sometimes inconjunction with R).
Like R, itis very modular and various
packages exist for specific types
ofanalytics (or visualisations).
DataMelt DataMelt isafree mathematics software. http://jwork.org/dmelt/
Itcan beused for numeric computation,
statistics, symbolic calculations, data
analysis and data visualization.

4.3 Data mining & KDD
Data Mining and Knowledge Discovery inDatabases
(KDD) are closely related (yet different) types ofana-
lytics. Their relationship can beseen infigure 7.
Wecould argue that KDD isaway ofturning informa-
tion (see figure 7) gathered through data mining
into valuable knowledge.
Figure 7: KDD process
Data mining refers the application ofspecific algo-
rithms inorder toextract patterns from data. Data
mining was invented when datasets became too
large tobe analysed byhumans soresearchers
invented ways tocondense large datasets and
extract useful types ofinformation. Therefore the
key difference (in this context) between the statistical
methods discussed inthe previous section isthat
data mining isaimed atthe automation ofthe
analysis and presentation ofresults.

Data Mining KDD

ETL Data Patterns Knowledge

30

Two common applications ofdata mining are:
Automated prediction oftrends and behaviours.
For example, based onprevious purchases,
marketers can estimate the likelihood ofcus-
tomers buying other products orwhen they
are likely tobuy the same product again.
Nowadays, data mining isoften used inconjunction
with more advanced types ofanalytics (as wewill
discuss below). For example, certain probabilities
ofoccurrences discovered using data mining can
beused asinputs for machine learning models.
Likewise, machine learning could beused toimproved
algorithms used for data mining.

Within PES, this could for example beused to:
Estimate the likelihood that (certain) job seekers
find (certain) jobs inacertain period oftime.
Estimate the probability ofbenefit fraud
occurring among certain groups ofpeople with
benefits.
Unemployment forecasting based onhistorical
data.
KDD inmany ways isafollow-up step todata mining.
Itis important tomention here asit illustrates two
points:
While data mined can have value initself, the
true value lies inthe interpretation ofdata and
its transformation into knowledge.
It requires extra effort toturn data into interpre-
tative knowledge (and actionable wisdom.

Automated discovery ofpreviously unknown
patterns.
By combing different variables and many types
of data, data mining can beused todiscover
patterns in data that were previously unknown.
For example, this type ofdata mining isused
inmarketing todiscover ifcustomers who
purchase certain products are also buying
other products (for Example, Amazon uses this
togive recommendations tocustomers: people
who bought this product, also bought that
product).
Within PES, this could for example beused to:
If job seekers with certain similar aspects on
their resumes are more likely tofind jobs quicker.
If certain combinations ofjob seeker character-
istics would also make them agood fit for
vacancies not directly fitting with their past
experiences.
To move from data mining toKDD, PES can dotwo
things:
Enrich the data mined (e.g. bycombining
variables) sothat more value iscreated
orpatterns become more obvious. For example,
bycombining unemployment trends with labour
market seasonality, itbecomes possible
tocorrect for seasonal variations inunemploy-
ment and assess the true trend (if any).
Use experts tointerpret results. Making sense
ofresults can beextremely difficult without the
proper subject matter expertise and knowledge
ofthe context inwhich the analysis takes place.
Itis for these reasons that having social
scientists (and other experts) onthe data team
can enhance its value multifold.
Getting started
The following methods, tools and/or applications can
beused for data mining and/or KDD:

PACKAGE/TOOL EXPLANATION LINK

RapidMiner Open Source platform tomine data https://rapidminer.com/
(and run data science applications).
Data Applied Cloud based, free data mining http://www.data-applied.com/
and visualisation platform.
Orange Open source data mining and http://orange.biolab.si/
visualisation platform (with machine
learning capabilities).

4.4 Advanced Analytics
4.4.1 Artificial Intelligence
Artificial intelligence (AI) isused tocreate smarter
technologies that can make decisions orsupport deci-
sion making. The main goal ofAI isto create technolo-
gies that are sosmart that they can think and act like
humans. The Turing test isthe benchmark for AIafter
which itcan beconsidered human smart.
Artificial Intelligence isabroad concept that encom-
passes machine learning, deep learning and inter-
sects with other types ofanalytics, such asdata
mining and statistics. Inthis toolkit werestrict
ourselves tothe intelligent applications ofAI where
the application exhibits certain levels ofsmartness
based onlearning and creativity.
The Turing test
The Turing test was developed bymathematician
Alan Turing in1950 inorder totest the ability
ofamachine toexhibit intelligent behaviour.
The common version ofthe Turing test requires
acomputer topretend tobe ahuman. Ahuman
interrogates the machine (without knowing itis
one) and subsequently decides whether he/she
was talking toahuman ormachine. Ifthe
machine wins, ithas passed the Turing test.
Understanding AIworks best bylooking atreal world
applications. Perhaps the earliest application ofAI
that received world-wide attention was when IBMs
chess computer Deep Blue beat world champion
chess player Gary Kasparov in1996. This computer
was able toprocess and analyse enormous amounts
ofdata (previous chess games) and based onKasp-
arovs playing style reason which move was best
out ofall available moves. This ability ofreasoning
isakey characteristic ofAI.
More recently (March 2016), AlphaGo (by Google
DeepMind) was the first artificial intelligence applica-
tion tobeat ahuman atthe game Go. This isan
important feat for anumber ofreasons. The first
isthat, compared tochess, Gois amuch more com-
plicated game with many more moves. Analysing
PRACTITIONERS TOOLKIT
31
all the potential moves ofagame ofGo requires
tremendous processing power and the ability
ofAlphaGo tocrunch the numbers isatestament
tothe progress made incomputational power. The
second isthat AlphaGo beat ahuman against the
expectations of(many) AIexperts who did not expect
the application tobe sophisticated enough interms
ofreasoning and decision making. Ifyou want tosee
more current examples ofhow AIis developing,
https://aiexperiments.withgoogle.com/ isagood
website toget inspiration.
Figure 8: Components ofArtificial Intelligence
(Big) Data
Artificial
Intelligence
Learning Computational
algorithms Power
These two examples, and the progress they show-
case, help usunderstand the more foundational
properties ofAI (see figure 8). Furthermore, ithelps
usunderstand what aPES needs inorder tostart
using AI:
Large amounts ofdata that are available for
AIto process.
The availability ofsufficient amounts ofpro-
cessing power.
Algorithms that can learn from the data and
become increasingly smart.
32

EXAMPLES OFAI POTENTIAL FOR PES

Chat bots are more and more commonplace inthe private Similarly, bots could beused inservice settings. They could
sector. These bots are able toprovide basic customer service ease the burden ofcall centre agents byresponding tosimple
and help clients solve basic problems. inquiries leaving tothe complex and ambiguous tasks for
human agents.
Virtual Assistants such asSiri, OkGoogle, Cortana, that Virtual Assistants that guide the job seeker through the entire
assist people throughout their days bygiven (localised process ofregistering tofinding ajob and exiting the system.
and personalised) recommendations, pro-actively serve Itreminds job seekers ofwhen todo things, gives them advice
information and answer questions. onhow todo things and answers questions about the process
(e.g. Give writing tips when ajobseeker creates aresume
orapplication letter).
Video Games where artificial intelligence controls the behaviour Serious gaming where jobseekers can practice job interviews
ofopponents inshooter games. These opponents can learn and based onAI the interviewer asks relevant questions (and
from the players behaviour and subsequently change their subsequently gives feedback).
routines.

Examples and potential for PES Getting started

In the table above, wedescribe several existing
applications orAI and how PES could develop similar
applications ofAI.
With the novelty toAI and the lack ofexperiences
ofPES (or other governments) that could bereadily
implemented within PES, itseems advisable tostart
small. Smaller scale experiments with AIallow PES
toexplore the possibilities, reduce the amount
ofdata needed and the complexity ofthe algorithms.
The following tools could beof help.

PACKAGE/TOOL EXPLANATION LINK

Open Cog OpenCog isan open-source software http://opencog.org/
project aimed creating general Artificial
Intelligence applications.
Watson AI Watson isbeing marketed byIBM http://www.ibm.com/watson/
asrelatively generic application
tobusiness and governments and isused
invarious areas such as(structured and
unstructured) analytics, virtual assistant,
data integration and search.
 PRACTITIONERS TOOLKIT

33

4.4.2 Machine Learning
Machine learning isused tocreate better functioning
algorithms and models bylearning from ongoing
analysis. Machine learning isasubset ofartificial
intelligence and there isdisagreement about the
exact difference between the two concepts. Wesee
the difference asmachine learning being mostly
used toanalyse large volumes ofdata, discover
patterns inthese data and subsequently learn from
the data. Artificial intelligence goes one step further
and includes systems that can make decisions,
combine elements, reason and thus show behaviour
comparable tohuman thinking. Herein also lies akey
difference between machine learning and data-
mining/KDD. Inmachine learning there isaclear
emphasis inlearning from the data and the applied
analysis for future iterations.
Several types ofmachine learning exist; inthe table
below welist anumber ofcommon types ofmachine
learning:

TYPE EXPLANATION PES POTENTIAL APPLICATION

Supervised Learning The team feeds clearly labelled data into
the computer with desired outcomes and
the machine learns how tomanipulate
inputs into outputs.
Unsupervised Learning In this type oflearning, the machine tries
tocreate inferences byitself based
onunlabelled orunstructured data.
Reinforced Learning The machine must achieve agoal and
has create its own feedback mechanisms
toassess whether itis getting closer
toits goal (such asself-driving cars).
Learn how jobseekers best match certain
jobs based onpre-defined inputs.
Analyse job-seekers resumes, create
certain skill categories based onthese
resumes and match these skills
tovacancies.
Running simulations tominimise
unemployment over time.

Machine learning isbeing widely applied incom-
mercial settings and healthcare. Virus scanners on
computers are based onmachine learning and soare
smart thermostats that learn about your heating
preferences and adjust heating cycles accordingly.
In the following table welist some more applications
from machine learning from other domains and how
these could beapplied within PES.

EXAMPLES OFMACHINE LEARNING POTENTIAL FOR PES

Recommender systems from companies such asNetflix
and Amazon that become smarter bylearning from users
behaviours and people similar tothe user.
Fraud detection systems such asused bybanks, insurance,
and credit card companies rely onmachine learning todetect
potentially fraudulent transactions.
Chat application Skype9 uses machine learning totranslate
voice orinstant messaging conversations inamultitude
of languages inreal time. Its machine learning algorithms
make itbetter asit gets used more frequently.
Improvement ofjob matching systems based ontechnology
similar to(commercial) recommender systems. For example,
based onsuccessful matches of(previous) jobseekers, with
similar characteristics, more tailored recommendations for
vacancies could bemade.
Detection ofpotentially fraudulent benefit applications
byanalysing patterns inpast known applications and learning
the difference between fraudulent and non-fraudulent
applications.
Facilitate international job interviews orcross-regional
interviews incountries with multiple languages. Itcould also
beused for job (language) training and/or counselling.

9 See https://www.skype.com/en/features/
skype-translator/
 34

Examples and potential for PES Getting started

Within PES, machine learning has not been widely Several ofthe tools already mentioned can beused
applied. One notable exception isthe application for machine learning (such asR& Python). The table
oflearning algorithms atthe Flemish PES (VDAB) below lists more relevant tools.
(see below).

PACKAGE/TOOL EXPLANATION LINK

Apache Mahout An environment for quickly creating https://mahout.apache.org/
scalable machine learning applications
aswell asafree library ofmachine
learning algorithms.
OpenNN A C++ library implementing neural http://www.opennn.net/
networks.
TensorFlow A software library for machine learning. https://www.tensorflow.org/
Encog Machine Learning Framework Encog isan advanced machine learning http://www.heatonresearch.com/encog/
framework that supports avariety
ofadvanced algorithms, aswell
assupport classes tonormalize and
process data.

PRACTICAL EXAMPLE
The Flemish PES (VDAB) isworking toimprove job matching byusing
Big Data torecommend vacancies tojobseekers when they access
the VDABs vacancy system. In2016 aRecommender system was
developed based onatwofold objective: finding out which users are
interested inwhich vacancies (by looking atwhat they click on, what
they read, open and look at); and predicting ajobseekers interest
inother vacancies (by looking atwhat similar users have looked
at, analysing behaviours). VDAB seeks tomake both accurate and
extended recommendations tojobseekers through this system, looking
toopen the pool ofjobs that ajobseeker could find interesting based
onarange ofpreferences that are expressed inthe vacancy.
The Recommender system iscurrently being tested onaset ofsub-
users. This system has been developed byVDABs Innovation Lab.
More information can befound about the Lab onaspecific fiche
accessible inthe PES Practices website.
4.4.3 Deep Learning
is asubset ofartificial intelligence. Inour view10,
the key difference between machine learning and
deep learning isthat deep learning focuses heavily
onunstructured and abstract data aswell asthe
combination ofmany layers ofdata. Machine learn-
ing tends tofocus onstructured data and discovering
patterns indata that are well organised.
Deep learning isfor example used toautomatically
organise and tag photos. Companies like Google and
Facebook can recognise people and locations inpho-
tos and can use this information totag and catego-
rise the photos.
Thinking along these lines, apotential application
for deep learning inPES isto have algorithms learn
from job seekers CVs orresumes and discover useful
information. For example, formatting styles, fonts
used, colours and pictures could tell ussomething
about the job seeker that could beuseful when
recommending jobs orto help them optimise their
resumes. Similarly, recordings (with consent and for

Deep learning isused toexplore data that ishighly 10 Once again, many different interpretations exist,
unstructured and abstracted and tries tocreate so the reader may have come across different
abstractions from this data. Deep learning isasub- definitions. We have tried to create an easy to
set of machine learning (which, asexplained above) understand common definition.

PACKAGE/TOOL EXPLANATION
Deeplearning4j Open-source, distributed deep learning
framework.
Keras Python library for development and
evaluation ofdeep learning models.
research purposes) could beused toanalyse tone
ofvoice and emotions and this could beused toper-
sonalise service delivery processes.
Getting started
As with machine learning, many ofthe tools already
mentioned can beused for deep learning (such
asR& Python). The table above lists more relevant
tools and/or specific packages.
4.5 Combinations & Derivations
In the previous section wehave described what cur-
rently (in our view) the most important and promising
types ofanalytics are for PES. However, many sub-
types, combinations and derivations ofthese main
classes exist. Inthis section webriefly mention
several ofthese types ofanalytics. For each type,
wedefine and explain the concept, describe how
itrelates toother types ofanalytics, and outline how
itmay beof value for PES.
1. Predictive/Prescriptive analytics
Based onmodels, trying toextrapolate from previous
data points tofuture data points. Many recommender
systems are based onpredictive analytics and soare
well known examples such asweather forecasts.
Within PES, this could, for example beused in:
Unemployment orlabour market forecasting
Job seeker profiling applications (e.g. Byesti-
mating job seekers developments and training
needs)
Matching applications (e.g. Bypredicting the
ease with which vacancies can befilled).
2. Natural language processing (NLP)
NLP refers toabroad class ofmethods tointerpret
normal people ornatural language (and translate
that inother types oflanguage). This isused
byspeech recognition and translation software.
PRACTITIONERS TOOLKIT
35
LINK
https://deeplearning4j.org/
https://keras.io/
Within PES, this could beused to:
Better understand customer service communi-
cation (and for example create content that
better aligns with clients language)
Interpret jobseekers resumes, and better
match the languages used byjobseekers
and employers
Create better classification schemes (e.g. ESCO)
bymapping jargon and technical terms
tohuman language.
3. Image recognition
This isaspecial class ofmachine/deep learning focused
onunderstanding orlearning from (digital) images.
Asmentioned above, the underlying deep learning
algorithms are used totag orcategorize content).
Within PES, this could beused to:
Analyse profile pictures used for resumes and
make recommendations for job seekers
pictures tobetter match certain jobs.
4. Speech recognition
Closely related tonatural language processing,
speech recognition isused tounderstand spoken
language. Combined with natural language process-
ing and other types ofmachine learning and AI, this
could beused tocreated social robots and/or chat
bots. Currently, speech recognition isused totran-
scribe spoken communication. This helps create
archives ofcommunication and allows organisations
tounderstand content (such asquestions customers
have and their accompanying emotions).
Within PES, this has the following potential applications:
Understanding tone ofvoice and emotions
incustomer service interactions tobetter
understand perceived problems and obstacles
Allow communication tobe continued and
stored onother channels
Better understand word choices and use these
toupdate web and written content.
36

Chapter 5. In this chapter wefocus onpresenting results and

creating reports. Wedo not focus ontraditional (writ-

Presenting
ten) reports (given the abundance ofresources
onreporting). Wedo focus on(interactive) visualisa-
tions and dashboards aswell assharing data with

& Reporting the public using open data. Throughout, wediscuss

pros and cons and considerations.

The following topics form the main content ofthis

chapter:
Understand why traditional reports may not
bethe best way topresent results from smart
data analytics
Provide overview of(novel) ways topresent
and report data
Considerations when reporting data and who
toopen results upto.

Strategic questions this chapter answers:

How can Iget the best possible insights tohelp
my(strategic) work?

Tactical questions this chapter answers:

What are available ways topresent and report
data?
What are the use cases for each ofthose and
how will they help PES?

5.1 Why move away from

traditional reports?
While the traditional report isstill widely used and
inmany cases agood way toconvey information
and report research findings, many organisations
are moving away from written reports when reporting
about smart data. Several reasons for this exist:
Keep results upto date
One problem with traditional reports istheir
static nature. These creates problems with
dynamic data and especially high velocity data
that creates acontinuous stream ofinsights
with nonatural stopping points. Tobenefit from
the possibility torefresh results, more dynamic
ways toreport outcomes are desired.
Say more with less
Text isoften not the best way todescribe
results and the static nature ofmany graphs
inreports donot allow todig deeper into the
data. This isaproblem solved byinteractive
visualisations that allow tofilter, sort and
search through the data and show those
insights that matter.

Be more appealing and actionable
One ofthe bigger problems with traditional
reports istheir linear nature and the often dry
nature ofwriting that does not compel readers
toread the entire report, let alone follow upon
its recommendations. This isaproblem that
many interactive tools try tosolve byoffering
personalised insights and offer more dynamic
routes toexplore outcomes.
5.2 (Interactive) Visualisations
We start this overview with ashort discussion ofthe
role ofvisualisations. Visualisations are apowerful
way toshow findings and interesting patterns indata.
Key benefit ofvisualisations isthat they allow to:
More easily show relationships and develop-
ments (over time) than using text.
Provide abetter way ofconveying and remem-
bering information than using text for many
people (depending ontheir learning styles).
However, visualisations also have drawbacks, most
notably:
Only visual representations may lead tofalse
conclusions, for example when agraph suggests
arelationship between variables while inreal
life there isno significant relationship.
Complex graphs can beoverwhelming and
distract from the message that the sender
wants toconvey. Especially the volume aspect
ofbig data can cause problems when trying
tovisualise too many data points atonce.
Very often itstill requires skills and (domain)
expertise tointerpret results. When just visuali-
sations are presented itcan bedifficult tojudge
the value ofthe results presented.
While great for presenting results and data
visualisations are not the best vehicle toraise
concerns, points for discussion and describe
context.
In general, when information isvery complex and
much contextual information isneeded simple visu-
alisations asastand-alone way toreport information
are not the best way toconvey amessage. For these
reasons, the following seem good use cases touse
(interactive) visualisations:
In production environments when the information
issimple enough tobe understood without too
much contextual information. For example, visual
PRACTITIONERS TOOLKIT
37
elements could beused toshow the extent
towhich job seekers match tocertain jobs.
In situations when summaries ofinformation
are presented (for example inevaluations
ofpilots). Condensed versions ofinformation
(like summaries) reduce the complexities
ofinformation and make iteasier touse
visualisations).
When room for contextual information isavail-
able (e.g. indashboards orinteractive web
pages) more complex information can be
transmitted using visualisations, provided
the contextual information allows tointerpret
information correctly.
When experts (e.g. from the data team) are
available tohelp interpret information, more
complex visualisations can beused and the
experts can help resolve any ambiguities.
Getting started
There are many ways tocreate following methods,
tools and/or applications can beused tocalculate
descriptive analytics:
Most spreadsheet software (e.g. Microsoft Excel,
LibreOffice Calc) include basic graph functional-
ity aswell asrudimentary capabilities for
interactivity.
Most statistical software (e.g. SPSS, SAS) have
rudimentary visualisation capabilities that allow
for basic manipulations.
Many packages orprogramming languages
can beused for data visualisations. Some of
the most common ones are the jQuery, Chart.js
orlibraries for JavaScript. For some (interactive)
examples see http://bl.ocks.org/mbostock.
Many online services are built ontop ofthese
packages (and other code) and allow tocreate
visualizations online without the need for coding
(e.g. Datawrapper.de).
Certain dedicated software tools exist for data
visualisations. Tableau (.com) isawell-known
example, but others exist, such asSisense
(.com). Very often these tools blur the line
between simple visualisation tools and (online)
dashboards that can beused toanalyse and
manipulate data.
 38

PRACTICAL EXAMPLE

WollyBI (wollybi.com) isaspin-off from the Department ofStatistics and Quantitative Methods (CRISP), University ofMilan-Bicocca.
Together with regional public employment service, CRISP created the WollyBI platform asameans tovisually explore the labour
market invarious regions inItaly based onvacancies, location and skills. Itcurrently allows to inauser friendly manner visualise
various analyses ofover 1.5 million job vacancies.

5.3 Interactive Tools & Dashboards
In this section wediscuss (novel) ways topresent
data using interactive (web) tools and online dash-
boards. The benefit ofthese tools isthat they allow
the user tointeract with the data and thus allow the
user toa) personalise the data tofit his/her needs
(e.g. Through searching, sorting, and filtering), b)
explore patterns more easily while going from one
section toanother (related) section and c) get help
and contextual information more easily (e.g. Through
embedded help functions).
We define aDashboard inthis context asan interac-
tive tool, most often based onweb technologies,
working directly ontop ofdata sources that allow
tomanipulate and visualise information and provide
additional textual and contextual information.
Interactive tools and dashboard aim toresolve many
ofthe issues existing with stand-alone visualisations.
For example:
Dashboards typically allow for more sophisti-
cated types ofdata manipulations such
assorting, searching and filtering.
Dashboards can include additional information
explaining data points that help interpret
information.
Dashboards can include contextual information
that help understand the setting inwhich the
data was collected and analysed. This can
increase understanding ofthe data.
Dashboards can include recommendations or
conclusions that build upon the data presented.
Dashboards allow tolink between different
sections and thus allow for more dynamic
routing ofinformation.
 PRACTITIONERS TOOLKIT

39

Dashboards can link toinformation outside
ofthe dashboard, allowing tolink toother
relevant subjects.
Dashboards allow tointegrate communication
tools (e.g. Chat applications) orlink tocommuni-
cation tools, sothat the user ofthe dashboard
can contact the data team orother support
staff toaid with the use ofthe dashboard and/
or interpretation ofinformation.
In many cases dashboards are cloud based or
through other means accessible onthe internet
or intranet. This allows toshare information easily
and allow flexible access policies. Furthermore,
because dashboards typically work ontop ofanalyt-
ics data sources they allow for easy discrete orcon-
tinuous data updates.
risk exists oftrying toadd everything toadash-
board leading tosomething called feature creep,
namely adding every single possible data point
and type ofinformation. This could severely
distract from the goal which should beachieved.
Maintaining access rights requires resources
and solid policies for this have tobe created.
When dashboards are open (i.e. without user
authorisation oraccessible without credentials),
extra care needs tobe taken toprevent misin-
terpretation and abuse ofdata.
Even with the possibilities toexplain and add
context, interpreting data remains complicated
and even with the best dashboard, experts may
still beneeded. The risk exists that dashboards
are used toreplace instead ofcomplement
experts.

However, dashboards could suffer from the following Getting started

drawbacks:
Because ofthe flexible nature ofthe dashboards The following tools can help getting started with
and the many features than can beadded, the more interactive visualisations ordashboards.

PACKAGE/TOOL EXPLANATION LINK

Shiny Web application toturn (R Based) http://shiny.rstudio.com/
analytics into interactive applications.
Google Charts Web application tocreate (interactive) https://developers.google.com/chart/
graphs from various types ofdata.
Tableau One ofthe most popular applications http://tableau.com
to create interactive graphs
ordashboards. Available desktop,
server and cloud solution. [has afree
trial, afterwards paid for]
40

5.4 Open data The following examples ofopen data can be

A special class ofreporting isthat ofopen data. Open
data isslightly outside ofthe scope ofthis toolkit and
fir this reason, weonly discuss the concept briefly.
Inopen data, data sources (such asraw data orana-
lysed data) are opened upfor broader audiences
touse. Four main reasons for open data exist:
Create transparency
By opening updata about governments and
processes, public organisations allow the public
tosee what governments do, how they function
and how money isbeing spend.
Accountability
A second reason, and tied tothe first, isthat
of accountability. Using open data, citizens
can check upon the promises and actions
ofgovernments.
Improve services orprocesses
Using open data, external parties can help
governments improve processes (e.g. byfinding
data patterns orresults not previously discov-
ered bygovernments themselves). Furthermore,
itcreates the opportunity for external parties
tocompete with PES and docertain things more
effectively and/or efficiently.
Innovate
Using open data, third parties could develop
new, innovative applications that could provide
new oradditional services for job seekers
oremployers. This could indirectly benefit PES.
inspirational:
LMI for All (see http://www.lmiforall.org.uk)
Online data portal, containing sources ofhigh
quality, reliable labour market information (LMI)
from the UK.
Data.gov (http://data.gov)
Open data portal from the United States
Government. Contains awide variety ofopen
data sets and agood searchable interface.
European Union Open Data Portal (https://data.
europa.eu/euodp/en/data)
The EUOpen Data Portal isan access point
toagrowing range ofdata produced bythe
institutions and other bodies ofthe European
Union.
Canadian Open Data Portal (http://open.canada.
ca/en/open-data)
Is interesting because itshowcased applications
developed based onthe open data available
inthe portal.

While open data have these potential benefits, the

following points, directly tied tothe topic ofthis
toolkit need tobe kept inmind:
Develop good policies for privacy and confiden-
tiality when opening updata tothe public.
Also bear inmind that, through combinations
ofvariables and smart inferences, itsometimes
ispossible toidentify (possible) individuals even
when PII isremoved from datasets. For this
reason, open data should bescreened extra
carefully.
Security isalways important and when creating
anopen data infrastructure, the PES basically
creates a(vulnerable) entrance toits data
systems. Tominimise security risks, itis
advisable tocreate aseparate infrastructure
for open data that isnot connected tocritical
systems orinfrastructures.
To make sure open data isused and interpreted
properly, itis important that open data are
documented and described properly.
 PRACTITIONERS TOOLKIT

41

Chapter 6. In the sixth and final chapter wefocus onaspects

pertaining toevaluation, and continuation. Evaluat-

Evaluation &
ing both the outcomes and the process are often
neglected parts ofany analytical process. However,
they are extremely important injudging the quality

Continuation ofboth process and outcomes and therefore are

acrucial step indeciding what todo with the out-
comes ofan analytics process. For example; should
results ofapilot beimplemented organisation wide?
Only athorough evaluation can help answer this
question properly.

The following topics form the main content ofthis

chapter:
Help understand the importance ofevaluation
Create anongoing data analytics infrastructure
and culture.

In this chapter, weanswer the following strategic

questions:
How dowe ensure our analytics are correct?
What doIneed todo tomake this part
of(ongoing work in) the PES?

The following tactical questions are answered:

How doIevaluate the process and the
outcomes?
What can Ido toensure continuation?
How doIscale upfrom pilots/small scale
projects tosomething larger?

6.1 Evaluation
Note: this section does not focus onevaluation
asaresearch method. Itfocuses onthe evaluation
ofany analytics process.

Evaluation isthe last step inany (stand-alone) ana-

lytics process and should beapart ofany ongoing
orcontinuous activity. There are many reasons why
evaluations are very important, such as:
Learning from the experience inorder
toimprove future activities orprevent mistakes
from happening again.
Assessing the quality ofthe results sothat they
can bejudged bytheir true value.
Assessing the quality ofthe process toverify
whether nomistakes have kept inand the
results are reflective ofthe desired situation.
Documenting the experiences sothat other
teams can replicate the process and/or draw
the same learnings.
42
EVALUATION FOCUS Continuous
Ad-hoc
Understanding whether the results ofthe
analytics activities are worth the (financial)
investments.
Determining whether one time projects orpilots
can orneed tobe implemented throughout the
organisation orbe scaled up.
Given the many reasons toconduct evaluations itis
important toinclude evaluations inany data related
activity and this entails:
Incorporation ofevaluation asastep oractivity
inany data analytics process
Dedication ofresources toconduct the
evaluation.
Not every evaluation isthe same though and wecan
distinguish between four different types ofevalua-
tions11 based onthe combination ofthe evaluation
focus (what isbeing evaluated) and evaluation
moment (when isthe evaluation taking place). The
following figure gives anoverview. Different types
ofevaluations can also take place simultaneously-
they are not mutually exclusive.
Each ofthese evaluations has the following
characteristics:
Process flow evaluations
These are ongoing evaluations ofanalytical
processes and aim toanswer such questions as:
Isthe analytical system working asexpected?
Are there any errors? Are the results within
anticipated margins and are their large varia-
tions inthe time ittakes torun models?
Process flow evaluations are typically used
todetermine ifthe process itself isfunctioning
well. For example, when arecommender system
takes much longer togenerate arecommenda-
tion incertain situations, there could beabug
orerror inthe model ordata input.
System outcome evaluations
These evaluated the outcomes ofan analytical
model orsystem continuously. They aim
toanswer questions as: are the outcomes ofthe
11 Also see the Analytical Paper on his topic,
Pieterson (2016).
EVALUATION FOCUS
Process oriented Outcome oriented
Process flow evaluations System outcome evaluations
Evaluations Assessments
analytics asexpected? Do(predictive) analytics
match actual outcomes orcan wetriangulate
outcomes toother data-sources and assess
their validity?
For example, when aPES implements apredic-
tive analytics tool topredict the likelihood of
job seekers finding anew job within acertain
amount oftime, system outcome evaluations
will focus onmeasuring whether the predicted
result (eventually) will lead tojob seekers
finding jobs.
Process performance evaluations
In these evaluations, usually when aproject
orpilot isconcluded oraprocess isevaluated
more thorough, the process itself isbeing
evaluated. Usually, these evaluations are
broader than process flow evaluations and
process performance evaluations, inthe context
ofanalytics, focus onthe entire analytical
process and could include such questions as:
was the team working well together, was the
process effective and efficient, are wehappy
with the tools and methods used? Are outcomes
being used properly?
For example, when aPES conducts apilot
toimplement anew profiling system, aprocess
performance evaluation could consist ofinter-
views with members ofthe data team and PES
employees using the new profiling system
toevaluate the process.
Outcome assessments
This fourth type ofevaluation focuses oneither
the actual outcomes ofapilot orone-time
project orthe (overall orcumulative) outcomes
of an ongoing process atclearly defined points
in time. This helps answer questions such as:
re analytics creating correct outcomes over
longer periods oftime (e.g. When adjusting
for seasonal fluctuations), are the outcomes of
apilot satisfactory? Are the outcomes ofanew
analytical tool/method more robust, valid or
reliable compared tothe old method (and should
wetherefore implement the new method?).

For example, one could compare the quality of
human generated vacancy matches with those
of anew analytics platform after acertain amount
of time and certain number ofmatches made. The
comparison ofthe two ways ofmatching could tell
the organisation which one performs better (and
should therefore beimplemented).
Getting started
The following can help ingetting started
with evaluations:
Include evaluations inany analytics plan
orproposal. This ensures commitment toevalu-
ation from the start. This should also include
the exact evaluation moment. For continuous
evaluations, many moments toassess these
evaluations could bescheduled (for example
recurring evaluation meetings).
Try toconnect evaluations toKPIs orpractical
relevant outcomes inthe organisation. For
example, certain performance goals for anew
tool can bespecified and evaluations can
beused tomeasure progress towards these
tools. This way, evaluations become amore
practical and useful tool.
Similarly to(for example) security training,
(new) data team members could betrained
regularly onthe importance ofevaluations.
Especially concerning more important initiatives,
itcould behelpful toget the help ofexternal
auditors orevaluation agencies. Furthermore,
external auditors can beuseful ingeneral touse
(randomly) onprojects toguarantee integrity
and guard bias within the data team.
6.2 Continuation and scale-up
of pilots
While ongoing evaluation isincredibly important and
anorganisation should keep apulse ofany ongoing
analytical activities, one types ofevaluation isvery
important interms ofthe decision the organisation
has tomake:
For any data analytics project that starts small, at
some point intime the organisation has todecide
whether this project (whether itis aresearch activity,
pilot orexperiment) will move from being asmall scale
pilot tobeing rolled out inthe entire organisation.
This decision regarding continuation orfull scale-up
orimplementation inthe organisation isimportant
PRACTITIONERS TOOLKIT
43
for the organisation. Itwill likely impact how (parts)
ofthe organisation are working, could impact service
delivery for jobseekers and/or employers and could
consume valuable resources inthe organisation.
Besides these organisational aspects, there are
several technical considerations when scaling upini-
tiatives. Here are some examples ofboth:
Technical considerations:
From adata standpoint, one has tobe abso-
lutely certain that the results are valid, reliable
and can begeneralised (if that isthe aim). This
typically requires alot oftesting and running
models toassure model fit isright.
When scaling up, the assumption isthat the
new tool orproduct has obvious (and measured)
benefits outweighing the previous tool orprod-
uct. Acost-benefit analysis can help inthis
decision making process. Similarly, this can help
indeciding whether tocontinue certain analytics
projects.
Broader implementation requires that the tool
iscapable ofbeing scaled-up. This isnot always
the case. For example, code may not always
bevery efficient which isnot necessarily
aproblem working with small datasets
insmaller settings, but could lead toperfor-
mance issues inlarge production environments.
Atechnical assessment and requirements
analysis can help indetermining what needs
tohappen interms ofproduct (performance)
requirements and stability before atool isready
for aproduction environment.
The interface ofany product ortool could work
well inalaboratory setting but not necessarily
inaproduction environment. Highly trained
members ofadata-team used toworking with
data and tools have different needs than PES
staff members working with production tools.
For example interms of: a) the ease ofUI/UX,
b) explanations/context surrounding datapoints,
c) help/support functionality. User interface
experts can help readying anexperimental tool
for aproduction environment.
Organisational considerations:
Very often anexpansion orscale-up ofaproject
orimplementation ofaproduct requires training
and support for new staff members. This
requires resources and time that may not
bedirectly available and need tobe planned.
When the purpose ofadata-team issolely
tofocus onexperimenting and innovation,
 44
PRACTICAL EXAMPLE
The Finish PES developed anew statistical profiling tool that was
implemented inthe organisation in2007. The profiling tool was part
ofan integrated ITsystem that calculated arisk estimate for the
jobseeker atregistration using administrative data. The new model
was found tobe 90 per cent effective atestimating the likelihood
ofajobseeker being unemployed for over 12 months. However, case
workers did not think the tool was useful and did not trust the results
from the tool. Asaresult the tool was withdrawn from the production
environment (see Kurekov, 2014).
the data-team may not always bethe most
logical place toown aproduction tool orfacil-
ity. Ifthat isthe case, aconsideration becomes
who the functional owner ofthe tool should
be(and who isresponsible for support, further
development and maintenance).
Perhaps most importantly, and often over-
looked, isdealing with resistance inthe organi-
sation, aswell ascreating adata-driven or
innovation oriented culture inthe organisation.
Without the support from the employees inthe
organisation, any innovation isdoomed tofail.
Proper communication plans and cultural
initiatives are essential inthe scale-up orimple-
mentation ofany new tool. Current attitudes
and cultures should betaken into account when
making implementation decisions.
In sum, while ananalytics activity may result in
auseful result, oreven application that could beused
within the organisation, there usually isafairly long
way togo before results can beused inthe everyday
work ofthe PES. Planning the implementation pro-
cess carefully and taking the considerations above
into account can help greatly increating asmooth
process.
 PRACTITIONERS TOOLKIT

45

Appendices
Appendix 1 | Safe Harbor
De-identification types
This list gives anoverview ofthe HHS types ofPII
that need tobe removed from data sets following
the Safe Harbor method. For the full text, consult
http://www.hhs.gov/hipaa/for-professionals/privacy/
special-topics/de-identification/index.html

a) Names

b) Address/geographic information (such asstreet

address, city, county, precinct, (full) ZIP code)

c)All elements ofdates (except year) directly related

toan individual (such asbirth date, admission
date, discharge date, death date)

d) Telephone numbers

e) Vehicle identifiers and serial numbers, including

license plate numbers

f) Fax numbers

g) Device identifiers and serial numbers

h) Email addresses

i) Web Universal Resource Locators (URLs)

j) Social security numbers [or any other national

equivalent]

k) Internet Protocol (IP) addresses

l) Medical record numbers

m) Biometric identifiers, including finger and voice

prints

n) Health plan beneficiary numbers

o) Full-face photographs and any comparable images

p) Account numbers

q) Any other unique identifying number, character-

istic, orcode

r) Certificate/license numbers
 HOW TOOBTAIN EUPUBLICATIONS

Free publications:
one copy:
via EUBookshop (http://bookshop.europa.eu)
more than one copy orposters/maps:
from the European Unions representations (http://ec.europa.eu/represent_en.htm);
from the delegations innon-EU countries (http://eeas.europa.eu/delegations/index_en.htm);
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere inthe EU) (*).

(*) The information given isfree, asare most calls (though some operators, phone boxes orhotels may charge you).

Priced publications:
via EUBookshop (http://bookshop.europa.eu).
KE-02-17-300-EN-N